internal/msgx is the one copy of xnu's private batched socket calls: the
msghdr_x layout, Send/Recv through the raw syscall (EINTR retried) and
Unsupported for the errnos a kernel or sandbox refuses them with.
Every socket the listener owns reads with recvmsg_x: the listener and each
connected socket run their own reader goroutine, and hand whole batches to
ListenOut over a channel. ListenOut runs the EncReader over a batch and
flushes once, so the reader stays single-threaded without a lock that
readers contend on, and the next recvmsg_x overlaps the decrypt of the last
batch. Batch buffers (64 x MTU) come from a per-listener free list that
keeps up to 11; a reader takes one only while its socket is readable, so
idle connected sockets hold none. A batch with a header intact msghdr_x
entries can't produce, or ENOSYS/EPERM/EOPNOTSUPP, switches every reader to
one datagram per read for good.
Connected sockets send runs with sendmsg_x, which xnu batches through the
stack on a connected socket. EAGAIN/ENOBUFS keep #1901's single-datagram
stall handling; a refused sendmsg_x retries the datagram with write and
turns sendmsg_x off once that goes through.
The connected-socket knobs lose their NEBULA_CONNSOCK* environment
overrides and keep #1901's defaults.
Based on Caleb Jasik's #1895 (recvmsg_x) and #1897 (sendmsg_x), and Jack
Doan's internal/msgx shape from #1898.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On darwin an unconnected sendto runs the address and MAC policy checks
and the EndpointSecurity send check for every datagram, and while
xprotectd throttles it each send takes about 185us, which caps a
tunnel's upload and, through its ACKs, its download. A connected socket
runs those checks once, at connect, and it is the only UDP send that
reports the interface queue's flow control, as ENOBUFS.
Only the current direct remote of an established tunnel is eligible.
udp.EstablishedPeerConn is an optional Conn interface, implemented on
darwin; establishedRemotes refcounts remotes over the main hostmap and
tells each writer on the first and last reference: handshake completion
adds one, deleting or retiring a hostinfo removes it, SetRemote (roaming,
SetRemoteIfPreferred, control, ssh) moves it. Pending and relayed
hostinfos hold none, and a rehandshake's two hostinfos share one.
An eligible peer that Nebula sends NEBULA_CONNSOCK_RUN (64) datagrams to
within NEBULA_CONNSOCK_WINDOW (1s) gets a connected socket: SO_REUSEPORT,
bound to the source address the listener's route to the peer picks on
the listener's port, carrying the listener's IP_BOUND_IF. The peer sees
the same address and port as from the listener, and the socket's reader
delivers through the listener's reader. The listener itself never sets
SO_REUSEPORT (listen routines still do), so no other uid can bind the
port; with a specific listen.host and no routines connected sockets stay
off, since sharing an exact address needs SO_REUSEPORT on both.
All routines listening on one address share one set of connected
sockets: at most NEBULA_CONNSOCKS (8), one per peer, with opens from a
bucket that refills one per NEBULA_CONNSOCK_OPEN_EVERY (1s). A socket
closes when its peer stops being eligible, after NEBULA_CONNSOCK_IDLE
(30s) without traffic, on Rebind (of every routine) and Close, and on an
error that means the connected path is gone. Closing never runs under
the hostmap's or the eligibility lock; the peer can't reopen until the
close finishes. A send that gets ENOBUFS or EAGAIN waits within one
10ms budget shared by every writer since the last send that went
through, then drops; nothing waits in the netpoller without a deadline.
When nothing is eligible or open, a send costs one atomic load before
the listener path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>