Commit Graph
17 Commits
Author SHA1 Message Date
Ryan HuberandClaude Opus 5.5 21075ba32e udp/darwin: batch receives and connected-socket sends with recvmsg_x/sendmsg_x
internal/msgx is the one copy of xnu's private batched socket calls: the
msghdr_x layout, Send/Recv through the raw syscall (EINTR retried) and
Unsupported for the errnos a kernel or sandbox refuses them with.

Every socket the listener owns reads with recvmsg_x: the listener and each
connected socket run their own reader goroutine, and hand whole batches to
ListenOut over a channel. ListenOut runs the EncReader over a batch and
flushes once, so the reader stays single-threaded without a lock that
readers contend on, and the next recvmsg_x overlaps the decrypt of the last
batch. Batch buffers (64 x MTU) come from a per-listener free list that
keeps up to 11; a reader takes one only while its socket is readable, so
idle connected sockets hold none. A batch with a header intact msghdr_x
entries can't produce, or ENOSYS/EPERM/EOPNOTSUPP, switches every reader to
one datagram per read for good.

Connected sockets send runs with sendmsg_x, which xnu batches through the
stack on a connected socket. EAGAIN/ENOBUFS keep #1901's single-datagram
stall handling; a refused sendmsg_x retries the datagram with write and
turns sendmsg_x off once that goes through.

The connected-socket knobs lose their NEBULA_CONNSOCK* environment
overrides and keep #1901's defaults.

Based on Caleb Jasik's #1895 (recvmsg_x) and #1897 (sendmsg_x), and Jack
Doan's internal/msgx shape from #1898.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 11:11:48 +00:00
Caleb JasikandClaude Opus 5.5 37032a22bf udp/darwin: send to busy established peers on connected sockets
On darwin an unconnected sendto runs the address and MAC policy checks
and the EndpointSecurity send check for every datagram, and while
xprotectd throttles it each send takes about 185us, which caps a
tunnel's upload and, through its ACKs, its download. A connected socket
runs those checks once, at connect, and it is the only UDP send that
reports the interface queue's flow control, as ENOBUFS.

Only the current direct remote of an established tunnel is eligible.
udp.EstablishedPeerConn is an optional Conn interface, implemented on
darwin; establishedRemotes refcounts remotes over the main hostmap and
tells each writer on the first and last reference: handshake completion
adds one, deleting or retiring a hostinfo removes it, SetRemote (roaming,
SetRemoteIfPreferred, control, ssh) moves it. Pending and relayed
hostinfos hold none, and a rehandshake's two hostinfos share one.

An eligible peer that Nebula sends NEBULA_CONNSOCK_RUN (64) datagrams to
within NEBULA_CONNSOCK_WINDOW (1s) gets a connected socket: SO_REUSEPORT,
bound to the source address the listener's route to the peer picks on
the listener's port, carrying the listener's IP_BOUND_IF. The peer sees
the same address and port as from the listener, and the socket's reader
delivers through the listener's reader. The listener itself never sets
SO_REUSEPORT (listen routines still do), so no other uid can bind the
port; with a specific listen.host and no routines connected sockets stay
off, since sharing an exact address needs SO_REUSEPORT on both.

All routines listening on one address share one set of connected
sockets: at most NEBULA_CONNSOCKS (8), one per peer, with opens from a
bucket that refills one per NEBULA_CONNSOCK_OPEN_EVERY (1s). A socket
closes when its peer stops being eligible, after NEBULA_CONNSOCK_IDLE
(30s) without traffic, on Rebind (of every routine) and Close, and on an
error that means the connected path is gone. Closing never runs under
the hostmap's or the eligibility lock; the peer can't reopen until the
close finishes. A send that gets ENOBUFS or EAGAIN waits within one
10ms budget shared by every writer since the last send that went
through, then drops; nothing waits in the netpoller without a deadline.
When nothing is eligible or open, a send costs one atomic load before
the listener path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 09:59:09 +00:00
Wade Simmons 0dc8a28d1f apply go fix for Go 1.26 (#1867)
* apply `go fix` for Go 1.26

This applies the `go fix` recommendations for go1.26

* for all of ./...
2026-09-14 09:25:35 -04:00
Jack Doan edc3c5e018 the definitive tun offloads branch (#1704) 2026-08-21 11:57:38 -05:00
Nate BrownandJack Doan 7902ce674e Rebind for MacOS (#1816)
Co-authored-by: Jack Doan <me@jackdoan.com>
2026-07-23 09:26:24 -05:00
Jack Doan eaad4896c1 udp_darwin: don't call the EncReader on a UDP error (#1755) 2026-06-10 09:26:31 -05:00
Nate Brown d0f02ba873 Switch to slog, remove logrus (#1672) 2026-04-27 09:41:47 -05:00
Jack Doan e80b9830a3 Remove more os.Exit calls and give a more reliable wait for stop function (attempt 3) (#1661) 2026-04-20 16:08:26 -05:00
Nate Brown 7aff313a17 Relax the restriction on routines from the config (#1531) 2025-11-19 13:10:11 -06:00
Nate Brown 99faab505c Fix a potential bug with udp ipv4 only on darwin (#1532) 2025-11-19 09:56:58 -06:00
Nate Brown 52623820c2 Drop inactive tunnels (#1427) 2025-07-03 09:58:37 -05:00
Nate Brown c2420642a0 Darwin udp fix (#1428) 2025-07-02 15:50:22 -05:00
Nate Brown e264a0ff88 Switch most everything to netip in prep for ipv6 in the overlay (#1173) 2024-07-31 10:18:56 -05:00
Nate Brown f7e392995a Fix rebind to not put the socket in blocking mode (#972) 2023-09-07 11:56:09 -05:00
Nate Brown a3e59a38ef Use registered io on Windows when possible (#905) 2023-07-10 12:43:48 -05:00
Nate Brown 3bbf5f4e67 Use an interface for udp conns (#901) 2023-06-14 10:48:52 -05:00
Nate Brown bcabcfdaca Rework some things into packages (#489) 2021-11-03 20:54:04 -05:00