WriteBatch hands up to 64 datagrams to one sendmsg_x call instead of
one sendto each. The socket is unconnected, so xnu still runs every
message through sendit, per-send MAC checks included; the saving is the
syscall crossings.
xnu's per-message loop stops at the first message that fails. For
EWOULDBLOCK, ENOBUFS and EMSGSIZE it reports how many went out, or the
errno alone when none did; for any other error it returns only the
errno, with the messages before it already sent. So each call carries
one run of datagrams to a single destination, and nothing is resent
after the kernel may have taken it. A datagram that fails on its own
account (a full interface queue, an oversize packet) goes to sendto,
which delivers or reports it, so it costs only itself, as without
batching; a refused destination costs its run. ENOSYS/EOPNOTSUPP, or
EPERM/EINVAL on a datagram sendto then delivers, switch the socket
back to sendto for good.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Passthrough.Flush hands its whole lane to a new tio.BatchWriter in one
WriteBatch call when the queue implements it, and NewSingleQueue
exposes WriteBatch when its source does. On darwin, tun.WriteBatch
writes up to 64 packets per sendmsg_x on the utun control socket,
which is a connected datagram socket, so xnu takes the batched
sendit_x path rather than one sendit per message. utun registers no
send_list handler, so the kernel still passes each packet to utun on
its own: this saves syscalls and fd locking, not per-packet kernel work.
Error handling follows what sendmsg_x reports:
- A short count with no error (a packet past the send buffer
mid-call, or a later send failing with ENOBUFS): the remainder goes
out in the next sendmsg_x.
- EMSGSIZE: only returned when the first packet is too large, before
the kernel takes any, so that packet alone goes through Write, which
reports it, and the rest are batched again.
- ENOSYS, EPERM, EOPNOTSUPP: sendmsg_x is private API, so WriteBatch
logs once and falls back to writev per packet for good.
- Anything else: sendmsg_x returns the error without the count it
took, so the call is reported rather than retried, which could
deliver packets twice.
One loss mode has no userspace fix. sendit_x counts each packet as
sent once it is copied into an mbuf; if a later allocation fails, it
frees the copied prefix unsent, returns its length, and sendmsg_x
drops the ENOBUFS because the count is nonzero. That looks exactly
like a real short count, so under mbuf exhaustion WriteBatch silently
drops packets that writev would have delivered.
The iOS tun path (tun_ios.go) is unchanged.
Inside a macOS Network Extension, download goes from 640 to 832 Mbps
against the recvmsg_x change alone in three paired rounds, at 15% less
CPU per Mbps; the likely cause is that its utun charges more per write
than a raw one. Raw dnclient already runs download near line rate,
and there the CPU difference is within noise.
TestTunWriteBatch* round-trip through a connected AF_UNIX datagram
socketpair, which takes sendmsg_x the same way, over both the batched
and the fallback path: order, AF prefixes, skipped invalid packets,
and an oversize packet mid-call and first-in-call. Dropping Namelen
from msghdrX fails them. Mutation testing with gremlins leaves only
equivalent survivors in WriteBatch, plus which error is reported first
when several fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ListenOut drains the socket with recvmsg_x, xnu's private batched
recvmsg (SYS_RECVMSG_X, bsd/sys/socket_private.h), up to 64 datagrams
per syscall, and calls flush once per batch instead of once per packet.
Each msghdr_x carries its own sockaddr, so this works on the shared
unconnected socket. The send side stays on sendto: xnu only batches
sendmsg_x through the stack on connected sockets, and on an unconnected
socket like this one it falls back to one sendit per message, with
msg_name documented as unsupported.
recvmsg_x is private, so ListenOut falls back to one
ReadFromUDPAddrPort per datagram, logging once, if the kernel or a
sandbox refuses it (ENOSYS, EPERM, EOPNOTSUPP) or if a batch comes back
in a shape intact msghdr_x entries can't produce: more datagrams than
headers, a length past MTU, or a sender that isn't a matching
AF_INET/AF_INET6 sockaddr. That batch is dropped rather than delivered. IPv6 sources now arrive without their
zone, matching the Linux recvmmsg path; WriteTo already drops it.
macOS raw dnclient to a Windows peer on the same gigabit LAN, iperf3
10 s, median of 3 alternating rounds against 7cfa47d:
download, 1/4/8 streams 606/600/604 -> 869/873/863 Mbps
download CPU ~159% -> ~123%
upload 340 -> 323 Mbps
UDP 500M loss 0.32% -> 0.31%
Download is now close to the 902 Mbps the LAN gives without a tunnel.
TestMsghdrXKernelRoundTrip and FuzzMsghdrXKernelRoundTrip pin msghdrX
against the kernel's struct msghdr_x, which the SDK doesn't ship: they
round-trip bursts through recvmsg_x on loopback and check every field
the kernel writes. recvmsg_x never reports MSG_TRUNC (soreceive_m_list
doesn't set it), so a datagram past MTU arrives cut to MTU with Flags
0, the same as ReadFromUDP. TestListenOutSingleFallback covers the
fallback loop, TestCheckMsghdrX the shape check, and the round trip
asserts the check accepts every batch the kernel returns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
remove runtime.LockOSThread() because it makes things worse now
remove the "custom" Write() method from tun_linux.go, the stdlib path via os.File performs better
We should change our guidance around number of routines, ~2 per thread (that you wish to use for Nebula) seems to be about right now
If the host OS is already big endian, we were swapping bytes when we
shouldn't have. Use the Go helper to make sure we do the endianness
correctly
Fixes: #1189
On some systems, IPv6 is disabled (for example, CIS benchmark recommends to disable it when not used), but currently all UDP connections are using AF_INET6 sockets.
When we are binding AF_INET6 socket to an address like ::ffff:1.2.3.4 (IPv4 addresses are parsed by net.ParseIP this way), we can't send or receive IPv6 packets anyway, so this will not break any scenarios.
---------
Co-authored-by: Wade Simmons <wsimmons@slack-corp.com>
* add calculated_remotes
This setting allows us to "guess" what the remote might be for a host
while we wait for the lighthouse response. For networks that hard
designed with in mind, it can help speed up handshake performance, as well as
improve resiliency in the case that all lighthouses are down.
Example:
lighthouse:
# ...
calculated_remotes:
# For any Nebula IPs in 10.0.10.0/24, this will apply the mask and add
# the calculated IP as an initial remote (while we wait for the response
# from the lighthouse). Both CIDRs must have the same mask size.
# For example, Nebula IP 10.0.10.123 will have a calculated remote of
# 192.168.1.123
10.0.10.0/24:
- mask: 192.168.1.0/24
port: 4242
* figure out what is up with this test
* add test
* better logic for sending handshakes
Keep track of the last light of hosts we sent handshakes to. Only log
handshake sent messages if the list has changed.
Remove the test Test_NewHandshakeManagerTrigger because it is faulty and
makes no sense. It relys on the fact that no handshake packets actually
get sent, but with these changes we would send packets now (which it
should!)
* use atomic.Pointer
* cleanup to make it clearer
* fix typo in example
JackDoan