Compare commits

..
Author SHA1 Message Date
Caleb JasikandClaude Opus 5.5 10e1924489 udp/darwin: batch sends with sendmsg_x
WriteBatch hands up to 64 datagrams to one sendmsg_x call instead of
one sendto each. The socket is unconnected, so xnu still runs every
message through sendit, per-send MAC checks included; the saving is the
syscall crossings.

xnu's per-message loop stops at the first message that fails. For
EWOULDBLOCK, ENOBUFS and EMSGSIZE it reports how many went out, or the
errno alone when none did; for any other error it returns only the
errno, with the messages before it already sent. So each call carries
one run of datagrams to a single destination, and nothing is resent
after the kernel may have taken it. A datagram that fails on its own
account (a full interface queue, an oversize packet) goes to sendto,
which delivers or reports it, so it costs only itself, as without
batching; a refused destination costs its run. ENOSYS/EOPNOTSUPP, or
EPERM/EINVAL on a datagram sendto then delivers, switch the socket
back to sendto for good.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 16:10:26 -05:00
Caleb JasikandClaude Opus 5.5 4d191f805f overlay/darwin: drain several tun packets per read wakeup
The darwin tun returned one packet per Read, and listenIn flushes its
UDP send batch after every Read, so each outbound packet went to the
underlay as its own batch. tunQueue.Read now keeps calling readv until
the utun would block, up to tunReadBatch packets, so a burst reaches the
send batch together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 12:44:00 -05:00
Caleb JasikandClaude Opus 5.5 61947ff1c2 overlay/darwin: batch tun writes with sendmsg_x
Passthrough.Flush hands its whole lane to a new tio.BatchWriter in one
WriteBatch call when the queue implements it, and NewSingleQueue
exposes WriteBatch when its source does. On darwin, tun.WriteBatch
writes up to 64 packets per sendmsg_x on the utun control socket,
which is a connected datagram socket, so xnu takes the batched
sendit_x path rather than one sendit per message. utun registers no
send_list handler, so the kernel still passes each packet to utun on
its own: this saves syscalls and fd locking, not per-packet kernel work.

Error handling follows what sendmsg_x reports:

- A short count with no error (a packet past the send buffer
  mid-call, or a later send failing with ENOBUFS): the remainder goes
  out in the next sendmsg_x.
- EMSGSIZE: only returned when the first packet is too large, before
  the kernel takes any, so that packet alone goes through Write, which
  reports it, and the rest are batched again.
- ENOSYS, EPERM, EOPNOTSUPP: sendmsg_x is private API, so WriteBatch
  logs once and falls back to writev per packet for good.
- Anything else: sendmsg_x returns the error without the count it
  took, so the call is reported rather than retried, which could
  deliver packets twice.

One loss mode has no userspace fix. sendit_x counts each packet as
sent once it is copied into an mbuf; if a later allocation fails, it
frees the copied prefix unsent, returns its length, and sendmsg_x
drops the ENOBUFS because the count is nonzero. That looks exactly
like a real short count, so under mbuf exhaustion WriteBatch silently
drops packets that writev would have delivered.

The iOS tun path (tun_ios.go) is unchanged.

Inside a macOS Network Extension, download goes from 640 to 832 Mbps
against the recvmsg_x change alone in three paired rounds, at 15% less
CPU per Mbps; the likely cause is that its utun charges more per write
than a raw one. Raw dnclient already runs download near line rate,
and there the CPU difference is within noise.

TestTunWriteBatch* round-trip through a connected AF_UNIX datagram
socketpair, which takes sendmsg_x the same way, over both the batched
and the fallback path: order, AF prefixes, skipped invalid packets,
and an oversize packet mid-call and first-in-call. Dropping Namelen
from msghdrX fails them. Mutation testing with gremlins leaves only
equivalent survivors in WriteBatch, plus which error is reported first
when several fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 10:04:03 -05:00
Caleb JasikandClaude Opus 5.5 a33a6356c6 udp/darwin: batch receives with recvmsg_x
ListenOut drains the socket with recvmsg_x, xnu's private batched
recvmsg (SYS_RECVMSG_X, bsd/sys/socket_private.h), up to 64 datagrams
per syscall, and calls flush once per batch instead of once per packet.
Each msghdr_x carries its own sockaddr, so this works on the shared
unconnected socket. The send side stays on sendto: xnu only batches
sendmsg_x through the stack on connected sockets, and on an unconnected
socket like this one it falls back to one sendit per message, with
msg_name documented as unsupported.

recvmsg_x is private, so ListenOut falls back to one
ReadFromUDPAddrPort per datagram, logging once, if the kernel or a
sandbox refuses it (ENOSYS, EPERM, EOPNOTSUPP) or if a batch comes back
in a shape intact msghdr_x entries can't produce: more datagrams than
headers, a length past MTU, or a sender that isn't a matching
AF_INET/AF_INET6 sockaddr. That batch is dropped rather than delivered. IPv6 sources now arrive without their
zone, matching the Linux recvmmsg path; WriteTo already drops it.

macOS raw dnclient to a Windows peer on the same gigabit LAN, iperf3
10 s, median of 3 alternating rounds against 7cfa47d:

  download, 1/4/8 streams  606/600/604 -> 869/873/863 Mbps
  download CPU             ~159% -> ~123%
  upload                   340 -> 323 Mbps
  UDP 500M loss            0.32% -> 0.31%

Download is now close to the 902 Mbps the LAN gives without a tunnel.

TestMsghdrXKernelRoundTrip and FuzzMsghdrXKernelRoundTrip pin msghdrX
against the kernel's struct msghdr_x, which the SDK doesn't ship: they
round-trip bursts through recvmsg_x on loopback and check every field
the kernel writes. recvmsg_x never reports MSG_TRUNC (soreceive_m_list
doesn't set it), so a datagram past MTU arrives cut to MTU with Flags
0, the same as ReadFromUDP. TestListenOutSingleFallback covers the
fallback loop, TestCheckMsghdrX the shape check, and the round trip
asserts the check accepts every batch the kernel returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 16:30:55 -05:00
21 changed files with 1359 additions and 2353 deletions

No files matched your search

+1 -2
View File
@@ -203,8 +203,7 @@ func (c *Control) RebindUDPServer() {
} }
// A failure here means we are likely still pinned to the interface we came up on, so the rest of this is // A failure here means we are likely still pinned to the interface we came up on, so the rest of this is
// unlikely to help. Say so instead of silently carrying on as if we rebound. Only the first writer is rebound; // unlikely to help. Say so instead of silently carrying on as if we rebound.
// darwin's Rebind clears every listen routine's listener.
if err := c.f.outside.Rebind(); err != nil { if err := c.f.outside.Rebind(); err != nil {
c.l.Error("Failed to rebind udp socket", "error", err) c.l.Error("Failed to rebind udp socket", "error", err)
} }
-90
View File
@@ -1,90 +0,0 @@
package nebula
import (
"net/netip"
"sync"
"github.com/slackhq/nebula/udp"
)
// establishedRemotes tells the udp conns that implement udp.EstablishedPeerConn which addresses are the current
// direct remote of at least one established tunnel. A hostinfo holds a reference on its remote while it is in the
// main hostmap, and SetRemote moves the reference; a relayed tunnel has no direct remote and holds none.
type establishedRemotes struct {
mu sync.Mutex
conns []udp.EstablishedPeerConn
refs map[netip.AddrPort]int
}
// newEstablishedRemotes returns nil when no conn wants to know, which every method accepts.
func newEstablishedRemotes(conns []udp.Conn) *establishedRemotes {
var pc []udp.EstablishedPeerConn
for _, c := range conns {
if p, ok := c.(udp.EstablishedPeerConn); ok {
pc = append(pc, p)
}
}
if len(pc) == 0 {
return nil
}
return &establishedRemotes{conns: pc, refs: map[netip.AddrPort]int{}}
}
// add makes hostinfo hold a reference on its remote, as it joins the main hostmap. A hostinfo holds references in
// one establishedRemotes at a time, so an add while it is in another is ignored.
func (e *establishedRemotes) add(hostinfo *HostInfo) {
if e == nil {
return
}
e.mu.Lock()
defer e.mu.Unlock()
if !hostinfo.establishedIn.CompareAndSwap(nil, e) && hostinfo.establishedIn.Load() != e {
return
}
e.syncLocked(hostinfo)
}
// remove drops hostinfo's reference, as it leaves the main hostmap.
func (e *establishedRemotes) remove(hostinfo *HostInfo) {
if e == nil {
return
}
e.mu.Lock()
defer e.mu.Unlock()
if hostinfo.establishedIn.CompareAndSwap(e, nil) {
e.syncLocked(hostinfo)
}
}
// syncLocked moves hostinfo's reference to the remote it should hold now: its current remote while it is
// established, none otherwise. Reading both under mu makes the last call win however add, remove and SetRemote
// interleave. Only the establishedRemotes that hostinfo is in, or has just left, may call it, since
// hostinfo.establishedRemote is a reference in that one's refs.
func (e *establishedRemotes) syncLocked(hostinfo *HostInfo) {
var want netip.AddrPort
if hostinfo.establishedIn.Load() == e {
want = hostinfo.GetRemote()
}
held := hostinfo.establishedRemote
if want == held {
return
}
if held.IsValid() {
e.refs[held]--
if e.refs[held] == 0 {
delete(e.refs, held)
for _, c := range e.conns {
c.SetEstablishedPeer(held, false)
}
}
}
if want.IsValid() {
e.refs[want]++
if e.refs[want] == 1 {
for _, c := range e.conns {
c.SetEstablishedPeer(want, true)
}
}
}
hostinfo.establishedRemote = want
}
-173
View File
@@ -1,173 +0,0 @@
package nebula
import (
"net/netip"
"sync"
"testing"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// establishedPeerConn records which destinations the hostmap has told it are established peers, and fails the test
// on a call that doesn't change anything, which would mean a reference count went wrong.
type establishedPeerConn struct {
udp.NoopConn
t *testing.T
mu sync.Mutex
peers map[netip.AddrPort]bool
}
func (c *establishedPeerConn) SetEstablishedPeer(dst netip.AddrPort, established bool) {
c.mu.Lock()
defer c.mu.Unlock()
assert.NotEqual(c.t, established, c.peers[dst], "SetEstablishedPeer(%v, %v) repeated", dst, established)
if established {
c.peers[dst] = true
} else {
delete(c.peers, dst)
}
}
func (c *establishedPeerConn) established() []netip.AddrPort {
c.mu.Lock()
defer c.mu.Unlock()
var out []netip.AddrPort
for dst := range c.peers {
out = append(out, dst)
}
return out
}
// newEstablishedTestHostMap returns a hostmap whose conns are a recording conn and one that doesn't track peers.
func newEstablishedTestHostMap(t *testing.T) (*HostMap, *establishedPeerConn) {
hm := newHostMap(test.NewLogger())
hm.preferredRanges.Store(&[]netip.Prefix{netip.MustParsePrefix("192.168.0.0/16")})
c := &establishedPeerConn{t: t, peers: map[netip.AddrPort]bool{}}
hm.established = newEstablishedRemotes([]udp.Conn{c, udp.NoopConn{}})
require.NotNil(t, hm.established)
return hm, c
}
func newEstablishedTestHostInfo(vpnAddr netip.Addr, index uint32) *HostInfo {
return &HostInfo{
vpnAddrs: []netip.Addr{vpnAddr},
localIndexId: index,
remotes: NewRemoteList([]netip.Addr{vpnAddr}, nil),
}
}
func TestNewEstablishedRemotesWithoutTrackingConns(t *testing.T) {
assert.Nil(t, newEstablishedRemotes([]udp.Conn{udp.NoopConn{}}))
}
// A conn hears about a remote while at least one hostinfo in the main hostmap has it as its direct remote: from
// the add, through roaming and SetRemoteIfPreferred, until the last such hostinfo is deleted or retired by
// MaxHostInfosPerVpnIp. Pending and relayed hostinfos never count.
func TestHostMap_EstablishedRemotes(t *testing.T) {
hm, c := newEstablishedTestHostMap(t)
f := &Interface{}
a := netip.MustParseAddr("10.0.0.1")
r1 := netip.MustParseAddrPort("203.0.113.1:4242")
r2 := netip.MustParseAddrPort("203.0.113.2:4242")
r3 := netip.MustParseAddrPort("192.168.1.3:4242")
// A pending hostinfo, as a handshake sets its remote, isn't established.
h1 := newEstablishedTestHostInfo(a, 1)
h1.SetRemote(r1)
assert.Empty(t, c.established())
// Handshake completion adds it to the main hostmap.
hm.unlockedAddHostInfo(h1, f)
assert.ElementsMatch(t, []netip.AddrPort{r1}, c.established())
// A relayed tunnel has no direct remote.
relayed := newEstablishedTestHostInfo(netip.MustParseAddr("10.0.0.2"), 2)
hm.unlockedAddHostInfo(relayed, f)
assert.ElementsMatch(t, []netip.AddrPort{r1}, c.established())
// A rehandshake's new hostinfo shares the remote, which stays established until both are gone.
h3 := newEstablishedTestHostInfo(a, 3)
h3.SetRemote(r1)
hm.unlockedAddHostInfo(h3, f)
hm.DeleteHostInfo(h1)
assert.ElementsMatch(t, []netip.AddrPort{r1}, c.established())
// Roaming moves the reference.
h3.SetRemote(r2)
assert.ElementsMatch(t, []netip.AddrPort{r2}, c.established())
require.True(t, h3.SetRemoteIfPreferred(hm, ViaSender{UdpAddr: r3}))
assert.ElementsMatch(t, []netip.AddrPort{r3}, c.established())
// A relayed tunnel that hears from its peer directly roams onto a direct remote.
relayed.SetRemote(r1)
assert.ElementsMatch(t, []netip.AddrPort{r1, r3}, c.established())
hm.DeleteHostInfo(relayed)
// A deleted hostinfo lets go, and a SetRemote after that changes nothing.
hm.DeleteHostInfo(h3)
assert.Empty(t, c.established())
h3.SetRemote(r2)
assert.Empty(t, c.established())
hm.DeleteHostInfo(h3)
// Adding past MaxHostInfosPerVpnIp retires the oldest hostinfo and its remote.
var hs []*HostInfo
for i := range MaxHostInfosPerVpnIp + 1 {
h := newEstablishedTestHostInfo(a, uint32(10+i))
h.SetRemote(netip.AddrPortFrom(netip.MustParseAddr("198.51.100.1"), uint16(1000+i)))
hm.unlockedAddHostInfo(h, f)
hs = append(hs, h)
}
assert.Len(t, c.established(), MaxHostInfosPerVpnIp)
assert.NotContains(t, c.established(), hs[0].GetRemote())
}
// Roaming on the receive path races tunnel teardown on the connection manager; whatever the order, a deleted
// hostinfo holds no reference.
func TestHostMap_EstablishedRemotesRoamRacesDelete(t *testing.T) {
hm, c := newEstablishedTestHostMap(t)
f := &Interface{}
a := netip.MustParseAddr("10.0.0.1")
for i := range 200 {
h := newEstablishedTestHostInfo(a, uint32(i+1))
h.SetRemote(netip.MustParseAddrPort("203.0.113.1:4242"))
hm.Lock()
hm.unlockedAddHostInfo(h, f)
hm.Unlock()
var wg sync.WaitGroup
wg.Go(func() { h.SetRemote(netip.AddrPortFrom(netip.MustParseAddr("203.0.113.2"), uint16(i+1))) })
wg.Go(func() { hm.DeleteHostInfo(h) })
wg.Wait()
require.Empty(t, c.established(), "round %d", i)
}
}
// A hostinfo holds its reference in the first tracking hostmap it joins; another's add and delete of it change
// nothing, so neither hostmap's counts go wrong.
func TestHostMap_EstablishedRemotesSingleOwner(t *testing.T) {
hm1, c1 := newEstablishedTestHostMap(t)
hm2, c2 := newEstablishedTestHostMap(t)
f := &Interface{}
r := netip.MustParseAddrPort("203.0.113.1:4242")
h := newEstablishedTestHostInfo(netip.MustParseAddr("10.0.0.1"), 1)
h.SetRemote(r)
hm1.unlockedAddHostInfo(h, f)
hm2.unlockedAddHostInfo(h, f)
assert.ElementsMatch(t, []netip.AddrPort{r}, c1.established())
assert.Empty(t, c2.established())
hm2.DeleteHostInfo(h)
assert.Equal(t, map[netip.AddrPort]int{r: 1}, hm1.established.refs)
assert.Empty(t, hm2.established.refs)
assert.ElementsMatch(t, []netip.AddrPort{r}, c1.established())
hm1.DeleteHostInfo(h)
assert.Empty(t, hm1.established.refs)
assert.Empty(t, hm2.established.refs)
assert.Empty(t, c1.established())
assert.Empty(t, c2.established())
}
-16
View File
@@ -69,8 +69,6 @@ type HostMap struct {
Hosts map[netip.Addr]*HostInfo Hosts map[netip.Addr]*HostInfo
moreHosts map[netip.Addr][]*HostInfo moreHosts map[netip.Addr][]*HostInfo
preferredRanges atomic.Pointer[[]netip.Prefix] preferredRanges atomic.Pointer[[]netip.Prefix]
// established hears about each hostinfo that joins or leaves the hostmap; nil when no conn wants to know.
established *establishedRemotes
l *slog.Logger l *slog.Logger
} }
@@ -278,11 +276,6 @@ type HostInfo struct {
// This value will be behind against actual tunnel utilization in the hot path. // This value will be behind against actual tunnel utilization in the hot path.
// This should only be used by the ConnectionManagers ticker routine. // This should only be used by the ConnectionManagers ticker routine.
lastUsed time.Time lastUsed time.Time
// establishedIn is set while the hostinfo is in a main hostmap whose conns track established remotes, and
// establishedRemote is the remote it holds a reference on there, guarded by establishedIn's mu.
establishedIn atomic.Pointer[establishedRemotes]
establishedRemote netip.AddrPort
} }
type ViaSender struct { type ViaSender struct {
@@ -523,7 +516,6 @@ func (hm *HostMap) unlockedDeleteHostInfo(hostinfo *HostInfo) bool {
if len(hm.Indexes) == 0 { if len(hm.Indexes) == 0 {
hm.Indexes = map[uint32]*HostInfo{} hm.Indexes = map[uint32]*HostInfo{}
} }
hm.established.remove(hostinfo)
if hm.l.Enabled(context.Background(), slog.LevelDebug) { if hm.l.Enabled(context.Background(), slog.LevelDebug) {
hm.l.Debug("Hostmap hostInfo deleted", hm.l.Debug("Hostmap hostInfo deleted",
@@ -670,7 +662,6 @@ func (hm *HostMap) unlockedAddHostInfo(hostinfo *HostInfo, f *Interface) {
hm.Indexes[hostinfo.localIndexId] = hostinfo hm.Indexes[hostinfo.localIndexId] = hostinfo
hm.RemoteIndexes[hostinfo.remoteIndexId] = hostinfo hm.RemoteIndexes[hostinfo.remoteIndexId] = hostinfo
hm.established.add(hostinfo)
hostinfo.markOut(f.rebindEpoch.Load()) hostinfo.markOut(f.rebindEpoch.Load())
if f.connectionManager != nil { // f.connectionManager is only nil in some unit tests if f.connectionManager != nil { // f.connectionManager is only nil in some unit tests
@@ -851,13 +842,6 @@ func (i *HostInfo) SetRemote(remote netip.AddrPort) {
if i.GetRemote() != remote { if i.GetRemote() != remote {
i.remote.Store(&remote) i.remote.Store(&remote)
i.remotes.LearnRemote(i.vpnAddrs[0], remote) i.remotes.LearnRemote(i.vpnAddrs[0], remote)
if e := i.establishedIn.Load(); e != nil {
e.mu.Lock()
if i.establishedIn.Load() == e {
e.syncLocked(i)
}
e.mu.Unlock()
}
} }
} }
-1
View File
@@ -213,7 +213,6 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
} }
hostMap := NewHostMapFromConfig(l, c) hostMap := NewHostMapFromConfig(l, c)
hostMap.established = newEstablishedRemotes(udpConns)
punchy := NewPunchyFromConfig(l, c, udpConns[0]) punchy := NewPunchyFromConfig(l, c, udpConns[0])
connManager := newConnectionManagerFromConfig(l, c, hostMap, punchy) connManager := newConnectionManagerFromConfig(l, c, hostMap, punchy)
lightHouse, err := NewLightHouseFromConfig(ctx, l, c, pki.getCertState(), udpConns[0], punchy) lightHouse, err := NewLightHouseFromConfig(ctx, l, c, pki.getCertState(), udpConns[0], punchy)
+62
View File
@@ -435,3 +435,65 @@ func testPP(pkt []byte) *firewall.ParsedPacket {
} }
return pp return pp
} }
// fakeTunBatchWriter records each WriteBatch call and each plain Write separately.
type fakeTunBatchWriter struct {
writes [][]byte
batches [][][]byte
}
func (w *fakeTunBatchWriter) Write(p []byte) (int, error) {
w.writes = append(w.writes, bytes.Clone(p))
return len(p), nil
}
func (w *fakeTunBatchWriter) WriteBatch(pkts [][]byte) error {
var b [][]byte
for _, p := range pkts {
b = append(b, bytes.Clone(p))
}
w.batches = append(w.batches, b)
return nil
}
// TestPassthroughUsesWriteBatch pins that a writer implementing tio.BatchWriter gets the whole
// passthrough lane in one WriteBatch per Flush, in order, and no plain Writes; an empty Flush
// makes no call.
func TestPassthroughUsesWriteBatch(t *testing.T) {
w := &fakeTunBatchWriter{}
m := newTestMultiCoalescer(t, w)
k := &keySeq{epoch: 1}
var want [][]byte
for i := range 5 {
icmp := make([]byte, 28)
icmp[0] = 0x45
icmp[3] = 28
icmp[9] = 1
icmp[20] = byte(i)
want = append(want, icmp)
if err := m.Commit(icmp, k.next(), testPP(icmp)); err != nil {
t.Fatal(err)
}
}
if err := m.Flush(); err != nil {
t.Fatal(err)
}
if err := m.Flush(); err != nil {
t.Fatal(err)
}
if len(w.writes) != 0 {
t.Fatalf("got %d plain Writes, want 0", len(w.writes))
}
if len(w.batches) != 1 {
t.Fatalf("got %d WriteBatch calls, want 1", len(w.batches))
}
for i, p := range w.batches[0] {
if !bytes.Equal(p, want[i]) {
t.Fatalf("packet %d: got % x, want % x", i, p, want[i])
}
}
if len(w.batches[0]) != len(want) {
t.Fatalf("got %d packets, want %d", len(w.batches[0]), len(want))
}
}
+13 -2
View File
@@ -2,18 +2,23 @@ package batch
import ( import (
"io" "io"
"github.com/slackhq/nebula/overlay/tio"
) )
// Passthrough is MultiCoalescer's verbatim lane: no batching, packets are written at Flush in the // Passthrough is MultiCoalescer's verbatim lane: no coalescing, packets are written at Flush in the
// order enqueued. // order enqueued, in one WriteBatch when out implements tio.BatchWriter.
type Passthrough struct { type Passthrough struct {
out io.Writer out io.Writer
bw tio.BatchWriter // nil: Flush writes one packet per Write
slots [][]byte slots [][]byte
} }
func NewPassthrough(w io.Writer) *Passthrough { func NewPassthrough(w io.Writer) *Passthrough {
bw, _ := w.(tio.BatchWriter)
return &Passthrough{ return &Passthrough{
out: w, out: w,
bw: bw,
slots: make([][]byte, 0, 128), slots: make([][]byte, 0, 128),
} }
} }
@@ -26,12 +31,18 @@ func (p *Passthrough) enqueue(pkt []byte) error {
func (p *Passthrough) Flush() error { func (p *Passthrough) Flush() error {
var firstErr error var firstErr error
if p.bw != nil {
if len(p.slots) > 0 {
firstErr = p.bw.WriteBatch(p.slots)
}
} else {
for _, s := range p.slots { for _, s := range p.slots {
_, err := p.out.Write(s) _, err := p.out.Write(s)
if err != nil && firstErr == nil { if err != nil && firstErr == nil {
firstErr = err firstErr = err
} }
} }
}
clear(p.slots) clear(p.slots)
p.slots = p.slots[:0] p.slots = p.slots[:0]
return firstErr return firstErr
+17 -2
View File
@@ -15,9 +15,13 @@ type singleQueue struct {
// NewSingleQueue wraps a one-datagram-per-Read ReadWriteCloser (a legacy tun device) into a Queue. // NewSingleQueue wraps a one-datagram-per-Read ReadWriteCloser (a legacy tun device) into a Queue.
// bufSize is the per-queue read scratch size and must be at least the largest datagram the source can return. // bufSize is the per-queue read scratch size and must be at least the largest datagram the source can return.
// Close closes rwc. // Close closes rwc. The Queue implements BatchWriter when rwc does.
func NewSingleQueue(rwc io.ReadWriteCloser, bufSize int) Queue { func NewSingleQueue(rwc io.ReadWriteCloser, bufSize int) Queue {
return &singleQueue{rw: rwc, closer: rwc, buf: make([]byte, bufSize)} q := &singleQueue{rw: rwc, closer: rwc, buf: make([]byte, bufSize)}
if bw, ok := rwc.(BatchWriter); ok {
return &batchSingleQueue{singleQueue: q, bw: bw}
}
return q
} }
// NewSingleQueueNoClose is NewSingleQueue for a source owned by someone else, // NewSingleQueueNoClose is NewSingleQueue for a source owned by someone else,
@@ -47,3 +51,14 @@ func (q *singleQueue) Close() error {
} }
return q.closer.Close() return q.closer.Close()
} }
// batchSingleQueue is a singleQueue whose source also implements BatchWriter.
// It is a separate type so a plain singleQueue never advertises WriteBatch.
type batchSingleQueue struct {
*singleQueue
bw BatchWriter
}
func (q *batchSingleQueue) WriteBatch(pkts [][]byte) error {
return q.bw.WriteBatch(pkts)
}
+11
View File
@@ -145,3 +145,14 @@ func SupportsGSO(w io.Writer, want GSOProto) (GSOWriter, bool) {
return gw, false return gw, false
} }
} }
// BatchWriter is implemented by Queues that can emit several already-segmented packets in one call.
// Backends without a batched write do not implement it and callers loop over Write.
//
// pkts are borrowed: WriteBatch must not retain them past its return.
// A returned error doesn't say which packets were delivered, so callers must not retry the batch.
// Safe for concurrent use, like Write.
type BatchWriter interface {
io.Writer
WriteBatch(pkts [][]byte) error
}
+244 -9
View File
@@ -8,6 +8,7 @@ import (
"log/slog" "log/slog"
"net/netip" "net/netip"
"os" "os"
"sync"
"sync/atomic" "sync/atomic"
"syscall" "syscall"
"unsafe" "unsafe"
@@ -33,6 +34,35 @@ type tun struct {
// and routes for it. NEPacketTunnelProvider on darwin does this. // and routes for it. NEPacketTunnelProvider on darwin does this.
hostOwned bool hostOwned bool
l *slog.Logger l *slog.Logger
// noSendmsgX is set once sendmsg_x is refused; WriteBatch then writes one packet at a time.
noSendmsgX atomic.Bool
batchMu sync.Mutex
batch tunBatch
}
// tunWriteBatch is how many packets one sendmsg_x call may carry.
const tunWriteBatch = 64
// tunBatch is WriteBatch's scratch, guarded by tun.batchMu. The kernel reads every entry in place.
type tunBatch struct {
heads [tunWriteBatch][4]byte
iovs [tunWriteBatch][2]unix.Iovec
hdrs [tunWriteBatch]msghdrX
pkts [tunWriteBatch][]byte
}
// msghdrX mirrors xnu's struct msghdr_x (bsd/sys/socket_private.h).
// Keep in sync with msghdrX in udp/udp_darwin.go.
type msghdrX struct {
Name *byte
Namelen uint32
Iov *unix.Iovec
Iovlen int32
Control *byte
Controllen uint32
Flags int32
Datalen uint64
} }
type ifReq struct { type ifReq struct {
@@ -607,6 +637,74 @@ func (t *tun) Read(to []byte) (int, error) {
return n - 4, nil return n - 4, nil
} }
// tunReadBatch is the most packets one tunQueue.Read returns.
const tunReadBatch = 64
// tunReadArena is tunQueue's receive buffer. Draining stops once less than defaultBatchBufSize of it
// is left, so every readv has room for the largest packet utun can return at any device MTU.
const tunReadArena = 4 * defaultBatchBufSize
// tunQueue is the darwin tun's Queue. Read drains every packet already queued on the utun, up to
// tunReadBatch, so the caller encrypts and sends them as one batch rather than one per wakeup.
type tunQueue struct {
t *tun
buf []byte
pkts [tunReadBatch]tio.Packet
}
// Read waits for the utun to become readable, then reads packets until it would block. An error after
// at least one packet is dropped in favor of returning those packets; a persistent one recurs on the
// next Read.
func (q *tunQueue) Read() ([]tio.Packet, error) {
rc, err := q.t.f.SyscallConn()
if err != nil {
return nil, err
}
var head [4]byte
n, off := 0, 0
var callErr error
err = rc.Read(func(fd uintptr) bool {
for n < tunReadBatch && len(q.buf)-off >= defaultBatchBufSize {
iovecs := [2]unix.Iovec{
{Base: &head[0], Len: 4},
{Base: &q.buf[off], Len: uint64(len(q.buf) - off)},
}
l, e := tunReadv(int(fd), iovecs[:])
if e != nil {
if errno, ok := e.(syscall.Errno); ok && errno.Temporary() {
// Park on the poller only while there is nothing to hand back.
return n > 0
}
callErr = e
return true
}
if l < 4 {
// A datagram too short to carry the AF prefix, or end of file; stop rather than spin on it.
return true
}
end := off + l - 4
q.pkts[n] = tio.Packet{Bytes: q.buf[off:end:end]}
n++
off = end
}
return true
})
if n > 0 {
return q.pkts[:n], nil
}
if err != nil {
return nil, err
}
return nil, callErr
}
func (q *tunQueue) Write(p []byte) (int, error) { return q.t.Write(p) }
func (q *tunQueue) WriteBatch(pkts [][]byte) error { return q.t.WriteBatch(pkts) }
func (q *tunQueue) Close() error { return q.t.Close() }
// Write pushes one IP packet onto the utun device. Safe for concurrent use: // Write pushes one IP packet onto the utun device. Safe for concurrent use:
// the AF prefix and iovecs are per-call stack state, and the fd write itself // the AF prefix and iovecs are per-call stack state, and the fd write itself
// serializes on the runtime's fd mutex (see the Queue contract in tio.go). // serializes on the runtime's fd mutex (see the Queue contract in tio.go).
@@ -615,16 +713,12 @@ func (t *tun) Write(from []byte) (int, error) {
return 0, syscall.EIO return 0, syscall.EIO
} }
ipVer := from[0] >> 4
var head [4]byte var head [4]byte
switch ipVer { af, err := tunAF(from)
case 4: if err != nil {
head[3] = syscall.AF_INET return 0, err
case 6:
head[3] = syscall.AF_INET6
default:
return 0, fmt.Errorf("unable to determine IP version from packet")
} }
head[3] = af
// Grab rc as a local so the compiler can devirtualize the call and keep the closure on the stack. // Grab rc as a local so the compiler can devirtualize the call and keep the closure on the stack.
rc, err := t.f.SyscallConn() rc, err := t.f.SyscallConn()
@@ -657,6 +751,147 @@ func (t *tun) Write(from []byte) (int, error) {
return n - 4, nil return n - 4, nil
} }
// tunAF returns the utun address-family prefix byte for an IP packet.
func tunAF(pkt []byte) (byte, error) {
switch pkt[0] >> 4 {
case 4:
return syscall.AF_INET, nil
case 6:
return syscall.AF_INET6, nil
default:
return 0, fmt.Errorf("unable to determine IP version from packet")
}
}
// WriteBatch writes pkts to the utun device with sendmsg_x, xnu's private batched sendmsg, up to
// tunWriteBatch packets per syscall. The kernel still hands each packet to utun on its own, so this
// saves syscalls, not per-packet kernel work. Safe for concurrent use.
//
// An empty packet or one with no IP version is skipped and reported, as Write would. Any other
// error from sendmsg_x means the kernel took an unknown prefix of that call and dropped the rest, so
// it is reported rather than retried, which could deliver packets twice.
//
// Under mbuf exhaustion, sendmsg_x silently drops packets that writev would have delivered: the
// kernel counts each packet as sent once it is copied in, and when a later allocation fails it
// frees that prefix unsent and returns its length as a short count, which is indistinguishable
// from a real one.
func (t *tun) WriteBatch(pkts [][]byte) error {
if t.noSendmsgX.Load() {
return t.writeEach(pkts)
}
t.batchMu.Lock()
defer t.batchMu.Unlock()
b := &t.batch
var firstErr error
for len(pkts) > 0 {
n := 0
for len(pkts) > 0 && n < tunWriteBatch {
p := pkts[0]
pkts = pkts[1:]
var af byte
err := error(syscall.EIO)
if len(p) > 0 {
af, err = tunAF(p)
}
if err != nil {
if firstErr == nil {
firstErr = err
}
continue
}
b.heads[n] = [4]byte{3: af}
b.iovs[n] = [2]unix.Iovec{
{Base: &b.heads[n][0], Len: 4},
{Base: &p[0], Len: uint64(len(p))},
}
b.hdrs[n] = msghdrX{Iov: &b.iovs[n][0], Iovlen: 2}
b.pkts[n] = p
n++
}
if n == 0 {
continue
}
for off := 0; off < n; {
sent, err := t.sendmsgX(off, n)
switch {
case err == nil && sent > 0:
// The kernel stops early without an error on a packet larger than the socket's send
// buffer, or when sending a later packet fails with ENOBUFS; resend the remainder.
off += sent
case err == nil:
if werr := t.writeEach(b.pkts[off:n]); werr != nil && firstErr == nil {
firstErr = werr
}
off = n
case err == unix.EMSGSIZE:
// Only the first packet being larger than the socket's send buffer fails the whole call,
// before the kernel takes any; Write reports that one and the rest go out in the next call.
if werr := t.writeEach(b.pkts[off : off+1]); werr != nil && firstErr == nil {
firstErr = werr
}
off++
case err == unix.ENOSYS || err == unix.EPERM || err == unix.EOPNOTSUPP:
// sendmsg_x is private API: fall back to writev if a kernel or sandbox refuses it.
if t.noSendmsgX.CompareAndSwap(false, true) {
t.l.Warn("sendmsg_x unavailable on the tun device, writing one packet per syscall", "error", err)
}
if werr := t.writeEach(b.pkts[off:n]); werr != nil && firstErr == nil {
firstErr = werr
}
clear(b.iovs[:n])
clear(b.pkts[:n])
if werr := t.writeEach(pkts); werr != nil && firstErr == nil {
firstErr = werr
}
return firstErr
default:
if firstErr == nil {
firstErr = err
}
off = n
}
}
clear(b.iovs[:n])
clear(b.pkts[:n])
}
return firstErr
}
// sendmsgX hands entries off through n-1 of t.batch to sendmsg_x and returns how many the kernel took.
func (t *tun) sendmsgX(off, n int) (int, error) {
rc, err := t.f.SyscallConn()
if err != nil {
return 0, err
}
var sent uintptr
var errno syscall.Errno
err = rc.Write(func(fd uintptr) bool {
sent, _, errno = unix.Syscall6(unix.SYS_SENDMSG_X, fd, uintptr(unsafe.Pointer(&t.batch.hdrs[off])), uintptr(n-off), 0, 0, 0)
// sendmsg_x reports EAGAIN only when it took nothing; a partial batch returns its count.
return !errno.Temporary()
})
if err != nil {
return 0, err
}
if errno != 0 {
return 0, errno
}
return int(sent), nil
}
// writeEach writes pkts one at a time with Write.
func (t *tun) writeEach(pkts [][]byte) error {
var firstErr error
for _, p := range pkts {
if _, err := t.Write(p); err != nil && firstErr == nil {
firstErr = err
}
}
return firstErr
}
func (t *tun) Networks() []netip.Prefix { func (t *tun) Networks() []netip.Prefix {
return t.vpnNetworks return t.vpnNetworks
} }
@@ -666,5 +901,5 @@ func (t *tun) Name() string {
} }
func (t *tun) Queues(int) ([]tio.Queue, error) { func (t *tun) Queues(int) ([]tio.Queue, error) {
return []tio.Queue{tio.NewSingleQueue(t, defaultBatchBufSize)}, nil return []tio.Queue{&tunQueue{t: t, buf: make([]byte, tunReadArena)}}, nil
} }
+201
View File
@@ -0,0 +1,201 @@
//go:build !ios && !e2e_testing
package overlay
import (
"bytes"
"os"
"syscall"
"testing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/sys/unix"
)
// newSocketpairTun returns a tun writing to one end of a connected AF_UNIX datagram socketpair, which
// takes sendmsg_x the way a utun control socket does, and the other end's fd for reading back.
// sndbuf is the writer's send buffer, which caps the largest datagram it accepts.
func newSocketpairTun(t *testing.T, sndbuf int) (*tun, int) {
t.Helper()
fds, err := unix.Socketpair(unix.AF_UNIX, unix.SOCK_DGRAM, 0)
require.NoError(t, err)
require.NoError(t, unix.SetsockoptInt(fds[0], unix.SOL_SOCKET, unix.SO_SNDBUF, sndbuf))
require.NoError(t, unix.SetsockoptInt(fds[1], unix.SOL_SOCKET, unix.SO_RCVBUF, 1<<20))
// A dropped datagram fails the read-back instead of hanging it.
require.NoError(t, unix.SetsockoptTimeval(fds[1], unix.SOL_SOCKET, unix.SO_RCVTIMEO, &unix.Timeval{Sec: 2}))
require.NoError(t, unix.SetNonblock(fds[0], true))
f := os.NewFile(uintptr(fds[0]), "socketpair")
t.Cleanup(func() {
_ = f.Close()
_ = unix.Close(fds[1])
})
return &tun{f: f, l: test.NewLogger()}, fds[1]
}
// testTunPkt returns packet i of size bytes, alternating IPv4 and IPv6, and the datagram the utun
// device should receive for it.
func testTunPkt(i, size int) (pkt, wire []byte) {
pkt = make([]byte, size)
pkt[0] = 0x45
af := byte(syscall.AF_INET)
if i%2 == 1 {
pkt[0] = 0x60
af = syscall.AF_INET6
}
pkt[1] = byte(i)
pkt[2] = byte(i >> 8)
return pkt, append([]byte{0, 0, 0, af}, pkt...)
}
// readTunBack asserts r receives exactly want, in order.
func readTunBack(t *testing.T, r int, want [][]byte) {
t.Helper()
buf := make([]byte, 4096)
for i, w := range want {
n, _, err := unix.Recvfrom(r, buf, 0)
require.NoError(t, err, "datagram %d", i)
if !bytes.Equal(w, buf[:n]) {
t.Fatalf("datagram %d: got % x, want % x", i, buf[:n], w)
}
}
_, _, err := unix.Recvfrom(r, buf, unix.MSG_DONTWAIT)
assert.ErrorIs(t, err, unix.EAGAIN, "extra datagrams delivered")
}
// forEachTunWritePath runs fn once over sendmsg_x and once over the per-packet writev fallback.
func forEachTunWritePath(t *testing.T, fn func(t *testing.T, fallback bool)) {
for _, fallback := range []bool{false, true} {
name := "sendmsg_x"
if fallback {
name = "writev"
}
t.Run(name, func(t *testing.T) { fn(t, fallback) })
}
}
// TestTunWriteBatch pins that WriteBatch delivers every packet whole, in order and behind the right
// utun AF prefix, across several sendmsg_x calls, and returns nil.
func TestTunWriteBatch(t *testing.T) {
forEachTunWritePath(t, func(t *testing.T, fallback bool) {
tn, r := newSocketpairTun(t, 1<<20)
tn.noSendmsgX.Store(fallback)
var pkts, want [][]byte
for i := range 3*tunWriteBatch + 5 {
p, w := testTunPkt(i, 60+i)
pkts = append(pkts, p)
want = append(want, w)
}
require.NoError(t, tn.WriteBatch(pkts))
assert.Equal(t, fallback, tn.noSendmsgX.Load())
readTunBack(t, r, want)
})
}
// TestTunWriteBatchSkipsInvalid pins that an empty packet or one with no IP version is skipped
// without shifting the packets after it, and that the first such error is the one reported.
func TestTunWriteBatchSkipsInvalid(t *testing.T) {
forEachTunWritePath(t, func(t *testing.T, fallback bool) {
tn, r := newSocketpairTun(t, 1<<20)
tn.noSendmsgX.Store(fallback)
var pkts, want [][]byte
for i := range 20 {
p, w := testTunPkt(i, 60)
pkts = append(pkts, p)
want = append(want, w)
}
pkts = append(pkts[:10], append([][]byte{{0x10, 1, 2}, {}}, pkts[10:]...)...)
err := tn.WriteBatch(pkts)
assert.ErrorContains(t, err, "IP version")
readTunBack(t, r, want)
})
}
// TestTunWriteBatchOversize pins that a packet larger than the socket's send buffer drops only
// itself and is reported, whether sendmsg_x reaches it mid-call (a short count) or first (EMSGSIZE
// for the whole call). The send buffer is 1024 bytes and the oversize packet 2000.
func TestTunWriteBatchOversize(t *testing.T) {
for _, tc := range []struct {
name string
at int
}{
{"mid-call", tunWriteBatch + 6},
{"first-in-call", tunWriteBatch},
} {
t.Run(tc.name, func(t *testing.T) {
forEachTunWritePath(t, func(t *testing.T, fallback bool) {
tn, r := newSocketpairTun(t, 1024)
tn.noSendmsgX.Store(fallback)
var pkts, want [][]byte
for i := range 2*tunWriteBatch + 10 {
size := 100
if i == tc.at {
size = 2000
}
p, w := testTunPkt(i, size)
pkts = append(pkts, p)
if i != tc.at {
want = append(want, w)
}
}
err := tn.WriteBatch(pkts)
assert.ErrorIs(t, err, unix.EMSGSIZE)
assert.Equal(t, fallback, tn.noSendmsgX.Load())
readTunBack(t, r, want)
})
})
}
}
// TestTunQueueReadDrains pins that Read hands back every queued packet, AF prefix stripped and in
// order, capped at tunReadBatch per call, and doesn't wait for more once the queue is empty.
func TestTunQueueReadDrains(t *testing.T) {
tn, w := newSocketpairTun(t, 1<<20)
// An AF_UNIX datagram send is bounded by the receiver's buffer, here the tun's end.
rc, err := tn.f.SyscallConn()
require.NoError(t, err)
require.NoError(t, rc.Control(func(fd uintptr) {
require.NoError(t, unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_RCVBUF, 1<<20))
}))
q := &tunQueue{t: tn, buf: make([]byte, tunReadArena)}
total := tunReadBatch + 3
var want [][]byte
for i := range total {
pkt, wire := testTunPkt(i, 60+i)
_, err := unix.Write(w, wire)
require.NoError(t, err)
want = append(want, pkt)
}
var got [][]byte
for _, size := range []int{tunReadBatch, 3} {
pkts, err := q.Read()
require.NoError(t, err)
require.Len(t, pkts, size)
for _, p := range pkts {
got = append(got, p.Clone().Bytes)
}
}
assert.Equal(t, want, got)
}
// TestTunQueueReadClipsPackets pins that each returned packet's capacity ends at its own bytes, so
// appending to one can't overwrite the next.
func TestTunQueueReadClipsPackets(t *testing.T) {
tn, w := newSocketpairTun(t, 1<<20)
q := &tunQueue{t: tn, buf: make([]byte, tunReadArena)}
for i := range 2 {
_, wire := testTunPkt(i, 100)
_, err := unix.Write(w, wire)
require.NoError(t, err)
}
pkts, err := q.Read()
require.NoError(t, err)
require.Len(t, pkts, 2)
assert.Equal(t, len(pkts[0].Bytes), cap(pkts[0].Bytes))
}
-8
View File
@@ -48,14 +48,6 @@ type Conn interface {
Close() error Close() error
} }
// EstablishedPeerConn is a Conn that treats the remotes of established tunnels differently from other
// destinations, as darwin's gives busy ones connected sockets.
type EstablishedPeerConn interface {
// SetEstablishedPeer reports whether dst is the current remote of at least one established tunnel. Calls for
// one dst must not race each other.
SetEstablishedPeer(dst netip.AddrPort, established bool)
}
type NoopConn struct{} type NoopConn struct{}
func (NoopConn) Rebind() error { func (NoopConn) Rebind() error {
-115
View File
@@ -1,115 +0,0 @@
//go:build !e2e_testing
package udp
import (
"log/slog"
"net"
"net/netip"
"testing"
"time"
)
// benchConnSocks opens a wildcard listener with ListenOut running, with the connected socket knobs pinned to
// their defaults and then cfg applied, and a loopback peer that never reads, so its sends cost only the sender.
func benchConnSocks(b *testing.B, s Settings, cfg func(*connSockConfig)) (*StdConn, netip.AddrPort) {
b.Helper()
for k, v := range map[string]string{
"NEBULA_CONNSOCKS": "8",
"NEBULA_CONNSOCK_RUN": "64",
"NEBULA_CONNSOCK_WINDOW": "1s",
"NEBULA_CONNSOCK_IDLE": "30s",
"NEBULA_CONNSOCK_OPEN_EVERY": "1s",
} {
b.Setenv(k, v)
}
c, err := NewListener(slog.New(slog.DiscardHandler), s)
if err != nil {
b.Fatal(err)
}
b.Cleanup(func() { _ = c.Close() })
u := c.(*StdConn)
if cfg != nil {
cfg(&u.socks.cfg)
}
go func() { _ = u.ListenOut(func(netip.AddrPort, []byte) {}, func() {}) }()
for u.reader.Load() == nil {
time.Sleep(time.Millisecond)
}
pc, err := net.ListenPacket("udp4", "127.0.0.1:0")
if err != nil {
b.Fatal(err)
}
b.Cleanup(func() { _ = pc.Close() })
return u, pc.LocalAddr().(*net.UDPAddr).AddrPort()
}
// BenchmarkConnSockWriteTo measures one WriteTo of a full-size datagram down each path: the listener with
// connected sockets off, to a destination that is no tunnel's remote while another is, to an established peer
// that never gets busy, and on a connected socket.
func BenchmarkConnSockWriteTo(b *testing.B) {
pkt := make([]byte, 1400)
cases := []struct {
name string
cfg func(*connSockConfig)
setup func(b *testing.B, u *StdConn, dst netip.AddrPort)
}{
{name: "listener/off", cfg: func(c *connSockConfig) { c.max = 0 }},
{name: "listener/ineligible", setup: func(b *testing.B, u *StdConn, dst netip.AddrPort) {
u.SetEstablishedPeer(netip.AddrPortFrom(dst.Addr(), dst.Port()+1), true)
}},
{name: "listener/eligible-not-busy", cfg: func(c *connSockConfig) { c.run = 1 << 30 },
setup: func(b *testing.B, u *StdConn, dst netip.AddrPort) { u.SetEstablishedPeer(dst, true) }},
{name: "connected", setup: func(b *testing.B, u *StdConn, dst netip.AddrPort) {
u.SetEstablishedPeer(dst, true)
for range u.socks.cfg.run {
_ = u.WriteTo(pkt, dst)
}
if u.connSockFor(dst) == nil {
b.Fatal("no connected socket")
}
}},
}
for _, c := range cases {
b.Run(c.name, func(b *testing.B) {
u, dst := benchConnSocks(b, Settings{Listen: netip.MustParseAddrPort("0.0.0.0:0")}, c.cfg)
if c.setup != nil {
c.setup(b, u, dst)
}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
_ = u.WriteTo(pkt, dst)
}
})
}
}
// BenchmarkConnSockDispatch measures what WriteTo adds before the listener's sendto, without the syscall: a v4
// listener refuses a v6 destination before sending, with connected sockets off or with another peer established.
func BenchmarkConnSockDispatch(b *testing.B) {
pkt := make([]byte, 1400)
dst := netip.MustParseAddrPort("[::1]:4242")
for _, c := range []struct {
name string
cfg func(*connSockConfig)
establish bool
}{
{name: "off", cfg: func(c *connSockConfig) { c.max = 0 }},
{name: "ineligible", establish: true},
} {
b.Run(c.name, func(b *testing.B) {
u, peer := benchConnSocks(b, Settings{Listen: netip.MustParseAddrPort("127.0.0.1:0"), Multi: true}, c.cfg)
if c.establish {
u.SetEstablishedPeer(peer, true)
}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if u.WriteTo(pkt, dst) == nil {
b.Fatal("sent a v6 datagram on a v4 listener")
}
}
})
}
}
-803
View File
@@ -1,803 +0,0 @@
//go:build !e2e_testing
package udp
import (
"context"
"errors"
"fmt"
"log/slog"
"maps"
"net"
"net/netip"
"os"
"slices"
"strconv"
"sync"
"sync/atomic"
"syscall"
"time"
"golang.org/x/sys/unix"
)
// A connected socket is a UDP socket connected to one busy peer, on the listener's port and bound to the source
// address the listener's route to that peer picks, so the peer sees the same address and port from either. A send
// on a connected socket skips the per-datagram address and MAC policy checks, and the EndpointSecurity send check
// that xprotectd can hold every unconnected sendto on, and it is the only UDP send that reports the interface
// queue's flow control, as ENOBUFS. A connected socket wins xnu's lookup for its peer's datagrams; it never sees
// anyone else's.
//
// A connected socket sets SO_REUSEPORT and binds a specific address beside a wildcard listener, which sets
// SO_REUSEPORT only for listen routines. xnu refuses a wildcard SO_REUSEPORT bind beside a holder without the
// option, and refuses another uid's bind to an address a socket holds specifically, so no other user can join the
// port while connected sockets open and close: uid nobody got EADDRINUSE on every bind against a root listener
// with connected sockets in the 09-25 probes. A connected socket is closed, never disconnected, because xnu widens
// a disconnected socket's local address to the wildcard.
//
// Only the current remote of an established tunnel may get one: Nebula names those with SetEstablishedPeer, so an
// unauthenticated sender can't earn a connected socket by drawing replies such as recv_error or handshake
// responses.
type connSock struct {
dst netip.AddrPort
peer *connSockPeer
conn *net.UDPConn
rc syscall.RawConn
// used is the monotonic time in nanoseconds of the connected socket's last send or receive.
used atomic.Int64
// stallAt is the monotonic time of the first ENOBUFS or EAGAIN since a send last went through, or 0, and
// dropping is set once that stall has dropped a datagram, so each stall logs its drops once.
stallAt atomic.Int64
dropping atomic.Bool
// parkMu serializes the writes that wait for a full send buffer, since the write deadline bounding each wait
// is the descriptor's, not the writer's.
parkMu sync.Mutex
// sys writes one datagram on the descriptor: writeFd, or a stand-in for the interface in tests.
sys func(fd int, b []byte) error
}
func writeFd(fd int, b []byte) error {
_, err := syscall.Write(fd, b)
return err
}
// errStallSpent is a write that found the connected socket's stall budget spent, so its datagram is dropped.
var errStallSpent = errors.New("connected socket stall budget spent")
// connSockWriter carries one write through RawConn.Write. They are pooled with fn bound once, since a closure
// passed through the RawConn interface escapes and would allocate on every send.
type connSockWriter struct {
s *connSock
b []byte
park bool
err error
fn func(fd uintptr) bool
}
var connSockWriters = sync.Pool{New: func() any {
w := &connSockWriter{}
w.fn = w.write
return w
}}
func (w *connSockWriter) write(fd uintptr) bool {
for {
w.err = w.s.sys(int(fd), w.b)
switch {
case w.err == syscall.EINTR:
case w.err == syscall.EAGAIN && w.park:
return false
default:
return true
}
}
}
// try writes b on s once and returns nil, the write's errno, os.ErrDeadlineExceeded, or net.ErrClosed. With park
// set, EAGAIN waits in the netpoller until the socket is writable or the write deadline passes; without it, EAGAIN
// is returned.
func (s *connSock) try(b []byte, park bool) error {
w := connSockWriters.Get().(*connSockWriter)
w.s, w.b, w.park = s, b, park
rerr := s.rc.Write(w.fn)
err := w.err
*w = connSockWriter{fn: w.fn}
connSockWriters.Put(w)
switch {
case rerr == nil:
return err
case errors.Is(rerr, os.ErrDeadlineExceeded):
return os.ErrDeadlineExceeded
default:
return net.ErrClosed
}
}
// connSockConfig holds the knobs. connSockConfigFromEnv reads all but backoff and enobufsMax from the environment,
// so one binary can be tuned without a rebuild.
type connSockConfig struct {
// max is how many connected sockets the listen routines may have open at once; 0 turns connected sockets off.
max int
// run is how many datagrams to one destination within one window earn it a connected socket. darwin's tun
// hands over one packet per read, so a peer's sends arrive one WriteTo or one-datagram batch at a time and
// busyness has to be counted across calls.
run int
window time.Duration
// idle closes a connected socket after it has neither sent nor received for this long.
idle time.Duration
// backoff keeps a destination whose connected socket failed on the listener for this long.
backoff time.Duration
// openEvery is how often the open bucket earns back an open once max are spent, which bounds connected socket
// churn however fast senders earn connected sockets and errors close them.
openEvery time.Duration
// enobufsMax is how long a connected socket's sends may wait out ENOBUFS without one going through before they
// drop instead, so a suspended interface costs one write per datagram rather than stalling the tun reader.
enobufsMax time.Duration
}
func connSockConfigFromEnv() connSockConfig {
c := connSockConfig{
max: 8,
run: 64,
window: time.Second,
idle: 30 * time.Second,
backoff: 30 * time.Second,
openEvery: time.Second,
enobufsMax: 10 * time.Millisecond,
}
if v, err := strconv.Atoi(os.Getenv("NEBULA_CONNSOCKS")); err == nil {
c.max = v
}
if v, err := strconv.Atoi(os.Getenv("NEBULA_CONNSOCK_RUN")); err == nil && v > 0 {
c.run = v
}
if v, err := time.ParseDuration(os.Getenv("NEBULA_CONNSOCK_WINDOW")); err == nil && v > 0 {
c.window = v
}
if v, err := time.ParseDuration(os.Getenv("NEBULA_CONNSOCK_IDLE")); err == nil && v > 0 {
c.idle = v
}
if v, err := time.ParseDuration(os.Getenv("NEBULA_CONNSOCK_OPEN_EVERY")); err == nil && v > 0 {
c.openEvery = v
}
return c
}
// connSockGroups holds the connSocks that the listen routines bound to one address share, so the limits, the
// connected sockets and Rebind cover every routine, and a peer has at most one connected socket.
var connSockGroups = struct {
sync.Mutex
m map[netip.AddrPort]*connSocks
}{m: map[netip.AddrPort]*connSocks{}}
// joinConnSocks returns the connSocks u shares with the other listen routines bound to la, and makes it for the
// first.
func joinConnSocks(u *StdConn, la netip.AddrPort, s Settings) *connSocks {
g := &connSockGroups
g.Lock()
defer g.Unlock()
if cs := g.m[la]; s.Multi && cs != nil {
cs.mu.Lock()
cs.members = append(cs.members, u)
cs.mu.Unlock()
return cs
}
cs := &connSocks{cfg: connSockConfigFromEnv(), key: la, members: []*StdConn{u}, clock: monoNow, sleep: time.Sleep}
if cs.cfg.max > 0 && !u.listenHost.IsUnspecified() && !s.Multi {
// A connected socket on a specific listen.host binds the listener's own address, which xnu allows only when
// both sockets set SO_REUSEPORT, and a listener with it lets another uid bind a dual-stack wildcard on the
// port.
cs.off.Store(true)
u.l.Debug("connected sockets: off for a specific listen.host", "addr", s.Listen)
}
if s.Multi {
g.m[la] = cs
}
return cs
}
type connSocks struct {
cfg connSockConfig
// key is the listeners' bound address, which connSockGroups files this under.
key netip.AddrPort
// off keeps connected sockets from opening, when the listener's address can't be shared with them.
off atomic.Bool
// table maps each destination that is the remote of an established tunnel, and that a connected socket could
// reach, to its peer; it is nil when there are none, so a send with nothing to consider costs one load. Every
// send reads it without a lock, and mu guards copying it. An open that holds mu and still finds its peer here
// can't outlive the tunnel.
table atomic.Pointer[map[netip.AddrPort]*connSockPeer]
// reserved counts the connected sockets that are open, opening or closing, against cfg.max; mu guards changing it.
reserved atomic.Int32
mu sync.Mutex
closed bool
// members are the listen routines' listeners that share these connected sockets.
members []*StdConn
// pending holds each destination whose connected socket is opening or closing, and not on its peer. xnu
// refuses to connect a 4-tuple that is still bound, so a destination in pending can't open another.
pending map[netip.AddrPort]struct{}
// rebinds counts Rebinds, so an open that copied the listener's interface before one is discarded.
rebinds uint64
// closers tracks the goroutines that close detached connected sockets.
closers sync.WaitGroup
// done stops the reaper; it is made when ListenOut starts the connected sockets.
done chan struct{}
// opens bounds connected socket opens to cfg.max at once and one per cfg.openEvery after.
opens gcra
// logs bounds connected socket log lines at info or warn; see connSockLogBurst.
logs gcra
// clock returns monotonic nanoseconds and sleep waits out ENOBUFS: monoNow and time.Sleep, or stand-ins in
// tests.
clock func() int64
sleep func(time.Duration)
// dialHook and closeHook, when set, run just before an open's dials and a close's Close, outside mu; tests
// use them to hold an open or close in flight.
dialHook, closeHook func(dst netip.AddrPort)
}
// connSockPeer is what connSocks keeps about an established tunnel's remote: its connected socket, how busy it is,
// and how long a failed connected socket keeps it on the listener.
type connSockPeer struct {
sock atomic.Pointer[connSock]
// windowMu guards start, the monotonic time the current window began, and n, the datagrams sent in it, so a
// window's reset and its counts can't interleave with another sender's.
windowMu sync.Mutex
start int64
n int
// retryAt is the monotonic time before which a peer whose connected socket failed stays on the listener;
// connSocks.mu guards it.
retryAt int64
}
// busy counts n more datagrams to p and reports whether it has now sent cfg.run within one window. A busy peer's
// count starts over, so a peer that can't get a connected socket retries at most once per cfg.run sends.
func (p *connSockPeer) busy(now int64, n int, cfg *connSockConfig) bool {
p.windowMu.Lock()
defer p.windowMu.Unlock()
if time.Duration(now-p.start) > cfg.window {
p.start, p.n = now, 0
}
p.n += n
if p.n < cfg.run {
return false
}
p.n = 0
return true
}
// connSockLogBurst and connSockLogEvery size the bucket that lets connected socket opens, closes and failures log
// at info or warn; past it they log at debug, so churn can't flood the log.
const (
connSockLogBurst = 32
connSockLogEvery = 10 * time.Second
)
var monoStart = time.Now()
func monoNow() int64 { return int64(time.Since(monoStart)) }
// gcra is a token bucket kept as the monotonic time it is next full (the generic cell rate algorithm), so it needs
// no refill tick.
type gcra struct{ tat int64 }
// take spends a token from a bucket of burst that earns one back per every, and reports whether there was one to
// spend.
func (g *gcra) take(now int64, burst int, every time.Duration) bool {
e := int64(every)
tat := max(g.tat, now)
if tat-now > int64(burst-1)*e {
return false
}
g.tat = tat + e
return true
}
// logLevelLocked returns level while the log bucket has a token, and debug once it is spent.
func (cs *connSocks) logLevelLocked(level slog.Level) slog.Level {
if cs.logs.take(cs.clock(), connSockLogBurst, connSockLogEvery) {
return level
}
return slog.LevelDebug
}
// startConnSocks starts the idle reaper.
func (u *StdConn) startConnSocks() {
cs := u.socks
cs.mu.Lock()
defer cs.mu.Unlock()
if cs.closed || cs.done != nil || cs.cfg.max <= 0 || cs.off.Load() {
return
}
cs.done = make(chan struct{})
go u.reapConnSocks(cs.done)
}
// stopConnSocks closes every connected socket and keeps new ones from opening, once any listen routine's listener
// closes, as they do together at shutdown. It returns once their descriptors are closed.
func (u *StdConn) stopConnSocks() {
cs := u.socks
cs.mu.Lock()
cs.members = slices.DeleteFunc(cs.members, func(m *StdConn) bool { return m == u })
if !cs.closed {
cs.closed = true
if cs.done != nil {
close(cs.done)
}
u.detachAllLocked("close")
}
cs.mu.Unlock()
connSockGroups.Lock()
if connSockGroups.m[cs.key] == cs {
delete(connSockGroups.m, cs.key)
}
connSockGroups.Unlock()
cs.closers.Wait()
}
// siblings returns the other listen routines' listeners.
func (cs *connSocks) siblings(u *StdConn) []*StdConn {
cs.mu.Lock()
defer cs.mu.Unlock()
return slices.DeleteFunc(slices.Clone(cs.members), func(m *StdConn) bool { return m == u })
}
// connSockFor returns dst's connected socket, or nil.
func (u *StdConn) connSockFor(dst netip.AddrPort) *connSock {
if p := u.socks.peer(dst); p != nil {
return p.sock.Load()
}
return nil
}
func (cs *connSocks) peer(dst netip.AddrPort) *connSockPeer {
if t := cs.table.Load(); t != nil {
return (*t)[dst]
}
return nil
}
// SetEstablishedPeer makes dst eligible for a connected socket while it is the remote of an established tunnel,
// and closes the one it has once it no longer is. Nebula calls it for every listen routine under the hostmap's
// write lock, so repeats are no-ops and it makes no syscalls: the descriptor is closed by another goroutine.
func (u *StdConn) SetEstablishedPeer(dst netip.AddrPort, established bool) {
cs := u.socks
if cs.cfg.max <= 0 || cs.off.Load() || !u.connSockable(dst) {
return
}
cs.mu.Lock()
defer cs.mu.Unlock()
t := cs.table.Load()
p := cs.peer(dst)
if cs.closed || (p != nil) == established {
return
}
next := map[netip.AddrPort]*connSockPeer{}
if t != nil {
next = maps.Clone(*t)
}
if established {
p = &connSockPeer{start: cs.clock()}
next[dst] = p
} else {
delete(next, dst)
if s := p.sock.Load(); s != nil {
u.detachLocked(s, "tunnel gone", false)
}
}
cs.storeTableLocked(next)
}
// storeTableLocked publishes t, or nil for an empty one.
func (cs *connSocks) storeTableLocked(t map[netip.AddrPort]*connSockPeer) {
if len(t) == 0 {
cs.table.Store(nil)
return
}
cs.table.Store(&t)
}
// connSockable reports whether a connected socket to dst could send what the listener would. A v4 listener can't
// reach a v6 destination at all, and a connected socket bound to a specific listen.host must share its family.
func (u *StdConn) connSockable(dst netip.AddrPort) bool {
if !dst.IsValid() || dst.Port() == 0 {
return false
}
if u.isV4 {
return dst.Addr().Is4()
}
return u.listenHost.IsUnspecified() || u.listenHost.Unmap().Is4() == dst.Addr().Unmap().Is4()
}
// connSockRun sends bufs, all to p's destination, on p's connected socket, opening one if they make p busy. It
// returns how many went out; when that is short of len(bufs), err is the failure of bufs[sent], or nil to leave
// bufs[sent:] to the listener.
func (u *StdConn) connSockRun(p *connSockPeer, dst netip.AddrPort, bufs [][]byte) (sent int, err error) {
s := p.sock.Load()
if s == nil {
if s = u.maybeOpenConnSock(p, dst, len(bufs)); s == nil {
return 0, nil
}
}
sent, errno, closed := u.connSockWrite(s, bufs)
switch {
case sent == len(bufs):
case connSockDead(errno):
u.closeConnSock(s, errno.Error(), true)
case !closed:
return sent, &net.OpError{Op: "write", Err: errno}
}
return sent, nil
}
// maybeOpenConnSock counts n datagrams to p, and opens a connected socket to it once it is busy and nothing rules
// one out.
func (u *StdConn) maybeOpenConnSock(p *connSockPeer, dst netip.AddrPort, n int) *connSock {
cs := u.socks
if n <= 0 || u.reader.Load() == nil || int(cs.reserved.Load()) >= cs.cfg.max || !p.busy(cs.clock(), n, &cs.cfg) {
return nil
}
cs.mu.Lock()
if cs.closed || cs.peer(dst) != p {
cs.mu.Unlock()
return nil
}
if s := p.sock.Load(); s != nil {
cs.mu.Unlock()
return s
}
now := cs.clock()
if _, ok := cs.pending[dst]; ok || int(cs.reserved.Load()) >= cs.cfg.max || now < p.retryAt ||
!cs.opens.take(now, cs.cfg.max, cs.cfg.openEvery) {
cs.mu.Unlock()
return nil
}
if cs.pending == nil {
cs.pending = map[netip.AddrPort]struct{}{}
}
cs.pending[dst] = struct{}{}
cs.reserved.Add(1)
rebinds := cs.rebinds
cs.mu.Unlock()
if cs.dialHook != nil {
cs.dialHook(dst)
}
s, err := u.dialConnSock(p, dst)
cs.mu.Lock()
if err != nil {
delete(cs.pending, dst)
cs.reserved.Add(-1)
p.retryAt = cs.clock() + int64(cs.cfg.backoff)
level := cs.logLevelLocked(slog.LevelWarn)
cs.mu.Unlock()
u.l.Log(context.Background(), level, "connected socket: open failed, staying on the listener", "udpAddr", dst, "error", err)
return nil
}
if cs.closed || cs.rebinds != rebinds || cs.peer(dst) != p {
// The tunnel went away, or a Rebind or shutdown ran, while the socket was opening; it stays in pending until
// it is closed.
cs.mu.Unlock()
u.finishClose(s)
u.l.Debug("connected socket: discarded, its tunnel or route changed while it opened", "udpAddr", dst)
return nil
}
delete(cs.pending, dst)
p.sock.Store(s)
go u.readConnSock(s)
level := cs.logLevelLocked(slog.LevelInfo)
cs.mu.Unlock()
u.l.Log(context.Background(), level, "connected socket: opened", "udpAddr", dst, "local", s.conn.LocalAddr(), "open", cs.reserved.Load())
return s
}
// dialConnSock opens a socket on the listener's port with SO_REUSEPORT, bound to the configured listen.host or,
// for a wildcard listener, to the source address the route to dst picks, and connected to dst. It carries the
// listener's IP_BOUND_IF, so both route the same way.
func (u *StdConn) dialConnSock(p *connSockPeer, dst netip.AddrPort) (*connSock, error) {
raddr := net.UDPAddrFromAddrPort(netip.AddrPortFrom(dst.Addr().Unmap(), dst.Port()))
network := "udp6"
if raddr.IP.To4() != nil {
network = "udp4"
}
ifindex, err := u.boundIf()
if err != nil {
return nil, err
}
local := &net.UDPAddr{IP: u.listenHost.AsSlice(), Zone: u.listenHost.Zone()}
if u.listenHost.IsUnspecified() {
// A connected socket gets the source address xnu would pick for an unconnected send to dst.
d := net.Dialer{Control: connSockControl(false, ifindex)}
probe, err := d.Dial(network, raddr.String())
if err != nil {
return nil, err
}
local = probe.LocalAddr().(*net.UDPAddr)
_ = probe.Close()
}
local.Port = int(u.port)
d := net.Dialer{LocalAddr: local, Control: connSockControl(true, ifindex)}
c, err := d.Dial(network, raddr.String())
if err != nil {
return nil, err
}
uc := c.(*net.UDPConn)
rc, err := uc.SyscallConn()
if err != nil {
_ = uc.Close()
return nil, err
}
s := &connSock{dst: dst, peer: p, conn: uc, rc: rc, sys: writeFd}
s.used.Store(u.socks.clock())
return s, nil
}
// connSockControl sets SO_REUSEPORT when reuse is set, and IP_BOUND_IF or IPV6_BOUND_IF when ifindex isn't 0,
// before the socket binds.
func connSockControl(reuse bool, ifindex int) func(network, address string, c syscall.RawConn) error {
return func(network, _ string, c syscall.RawConn) error {
var serr error
err := c.Control(func(fd uintptr) {
if reuse {
serr = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
}
if serr == nil && ifindex != 0 {
if network == "udp4" {
serr = unix.SetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_BOUND_IF, ifindex)
} else {
serr = unix.SetsockoptInt(int(fd), unix.IPPROTO_IPV6, unix.IPV6_BOUND_IF, ifindex)
}
}
})
if err != nil {
return err
}
return serr
}
}
// boundIf returns the interface index the listener is scoped to, or 0. An error keeps the peer on the listener,
// since a connected socket opened without the listener's scope could route out an interface the listener can't.
func (u *StdConn) boundIf() (int, error) {
level, opt := unix.IPPROTO_IPV6, unix.IPV6_BOUND_IF
if u.isV4 {
level, opt = unix.IPPROTO_IP, unix.IP_BOUND_IF
}
v, err := unix.GetsockoptInt(int(u.sysFd), level, opt)
if err != nil {
return 0, fmt.Errorf("reading the listener's bound interface: %w", err)
}
return v, nil
}
// closeConnSock closes s if it is still open. With backoff set, the peer stays on the listener until the backoff
// runs out.
func (u *StdConn) closeConnSock(s *connSock, why string, backoff bool) {
u.socks.mu.Lock()
u.detachLocked(s, why, backoff)
u.socks.mu.Unlock()
}
// detachLocked takes s off its peer, if it is still there, and closes it on another goroutine, since callers such
// as SetEstablishedPeer hold locks that no socket syscall should run under. s.dst stays in pending until the close
// completes.
func (u *StdConn) detachLocked(s *connSock, why string, backoff bool) {
cs := u.socks
if !s.peer.sock.CompareAndSwap(s, nil) {
return
}
if backoff {
s.peer.retryAt = cs.clock() + int64(cs.cfg.backoff)
}
if cs.pending == nil {
cs.pending = map[netip.AddrPort]struct{}{}
}
cs.pending[s.dst] = struct{}{}
level := cs.logLevelLocked(slog.LevelInfo)
cs.closers.Add(1)
go func() {
defer cs.closers.Done()
open := u.finishClose(s)
u.l.Log(context.Background(), level, "connected socket: closed", "udpAddr", s.dst, "reason", why, "open", open)
}()
}
// finishClose closes s, whose destination is in pending, then lets the destination open again, and returns how
// many connected sockets are left. Closing waits only for a write already in a syscall: a write parked on a full
// buffer is woken, and one waiting out ENOBUFS holds no reference on the descriptor.
func (u *StdConn) finishClose(s *connSock) int32 {
cs := u.socks
if cs.closeHook != nil {
cs.closeHook(s.dst)
}
_ = s.conn.Close()
cs.mu.Lock()
defer cs.mu.Unlock()
delete(cs.pending, s.dst)
return cs.reserved.Add(-1)
}
// detachAllLocked detaches every connected socket, for a network change or shutdown.
func (u *StdConn) detachAllLocked(why string) {
if t := u.socks.table.Load(); t != nil {
for _, p := range *t {
if s := p.sock.Load(); s != nil {
u.detachLocked(s, why, false)
}
}
}
}
// closeConnSocks detaches every connected socket and discards the ones still opening, for a network change.
func (u *StdConn) closeConnSocks(why string) {
u.socks.mu.Lock()
u.socks.rebinds++
u.detachAllLocked(why)
u.socks.mu.Unlock()
}
func (u *StdConn) reapConnSocks(done <-chan struct{}) {
cs := u.socks
idle := cs.cfg.idle
tick := time.NewTicker(max(min(idle/4, 5*time.Second), time.Millisecond))
defer tick.Stop()
for {
select {
case <-done:
return
case <-tick.C:
}
t := cs.table.Load()
if t == nil {
continue
}
now := cs.clock()
for _, p := range *t {
if s := p.sock.Load(); s != nil && time.Duration(now-s.used.Load()) > idle {
u.closeConnSock(s, "idle", false)
}
}
}
}
// connSockDead reports whether a connected socket send or receive error means the connected path to the peer is
// gone, so its traffic belongs back on the listener. A connected UDP socket reports a peer's ICMP unreachable as
// ECONNREFUSED on its next call, which the unconnected listener never sees.
func connSockDead(errno syscall.Errno) bool {
switch errno {
case unix.ECONNREFUSED, unix.EHOSTUNREACH, unix.EHOSTDOWN, unix.ENETUNREACH, unix.ENETDOWN,
unix.EADDRNOTAVAIL, unix.EPIPE, unix.ENOTCONN, unix.EDESTADDRREQ:
return true
}
return false
}
// enobufsWait is how long a connected socket send sleeps when the interface queue is flow controlled. The kernel
// posts no wakeup when the advisory lifts, so this polls; 50us kept a gigabit link full without spinning a core in
// the 09-25 probes.
const enobufsWait = 50 * time.Microsecond
// connSockWrite writes bufs on s, one datagram per write. A full socket buffer parks the caller in the netpoller
// until the socket is writable, and a flow-controlled interface makes it wait out ENOBUFS, so a busy link slows
// the tun reader instead of dropping. Once the connected socket has waited cfg.enobufsMax without a send going
// through, either one drops the datagram instead, and a dropped datagram counts as sent, as the listener's
// unreported interface drops do; it never falls back to the listener, which would reorder it. closed reports that
// the connected socket was closed before bufs[sent] went out; any other error stops the run and is returned with
// how many went out.
func (u *StdConn) connSockWrite(s *connSock, bufs [][]byte) (sent int, errno syscall.Errno, closed bool) {
cs := u.socks
defer func() {
if sent > 0 {
s.used.Store(cs.clock())
}
}()
for sent < len(bufs) {
err := s.try(bufs[sent], false)
if err == syscall.EAGAIN || err == os.ErrDeadlineExceeded {
// os.ErrDeadlineExceeded here is another writer's parked deadline, passed but not yet cleared.
err = u.parkedWrite(s, bufs[sent])
}
switch err {
case nil:
sent++
if s.stallAt.Load() != 0 {
s.stallAt.Store(0)
s.dropping.Store(false)
}
continue
case errStallSpent:
u.dropStalled(s)
sent++
continue
}
errno, ok := err.(syscall.Errno)
if !ok {
return sent, 0, true
}
if errno != syscall.ENOBUFS {
return sent, errno, false
}
// The wait holds no reference on the descriptor, so a close doesn't wait for it.
if s.stalledFor(cs.clock()) < cs.cfg.enobufsMax {
cs.sleep(enobufsWait)
} else {
u.dropStalled(s)
sent++
}
}
return sent, 0, false
}
// parkedWrite writes b after EAGAIN, parked in the netpoller for what is left of s's stall budget, and returns
// errStallSpent once none is. golang/go#73919: a darwin UDP write parked on EAGAIN can wait forever for a
// writability event that never comes, so the wait always has a deadline.
func (u *StdConn) parkedWrite(s *connSock, b []byte) error {
cs := u.socks
s.parkMu.Lock()
defer s.parkMu.Unlock()
left := cs.cfg.enobufsMax - s.stalledFor(cs.clock())
if left <= 0 {
return errStallSpent
}
if err := s.conn.SetWriteDeadline(time.Now().Add(left)); err != nil {
return net.ErrClosed
}
err := s.try(b, true)
_ = s.conn.SetWriteDeadline(time.Time{})
if err == os.ErrDeadlineExceeded {
return errStallSpent
}
return err
}
// dropStalled logs the first datagram a stall drops.
func (u *StdConn) dropStalled(s *connSock) {
if !s.dropping.Swap(true) {
u.l.Debug("connected socket: sends stalled, dropping datagrams until one goes through", "udpAddr", s.dst)
}
}
// stalledFor records an ENOBUFS or EAGAIN on s at now and returns how long its sends have stalled: since the first
// ENOBUFS or EAGAIN after a send last went through. Every writer measures from that one time, so concurrent writers
// spend the budget no faster than one, and a send going through ends the stall for every writer, including one
// asleep in it.
func (s *connSock) stalledFor(now int64) time.Duration {
if s.stallAt.CompareAndSwap(0, now) {
return 0
}
at := s.stallAt.Load()
if at == 0 {
return 0
}
return time.Duration(now - at)
}
// readConnSock delivers a connected socket's datagrams through the listener's reader, serialized with ListenOut,
// until the connected socket is closed. Any receive error closes it: a connected socket reports its peer's
// unreachables here, and retrying an unexpected error could spin.
func (u *StdConn) readConnSock(s *connSock) {
rd := u.reader.Load()
buf := make([]byte, MTU)
for {
n, from, err := s.conn.ReadFromUDPAddrPort(buf)
if err != nil {
if errors.Is(err, net.ErrClosed) {
return
}
why := err.Error()
var errno syscall.Errno
if errors.As(err, &errno) && connSockDead(errno) {
why = errno.Error()
} else {
u.l.Warn("connected socket: unexpected receive error", "udpAddr", s.dst, "error", err)
}
u.closeConnSock(s, why, true)
return
}
s.used.Store(u.socks.clock())
u.readMu.Lock()
rd.r(netip.AddrPortFrom(from.Addr().Unmap(), from.Port()), buf[:n:n])
rd.flush()
u.readMu.Unlock()
}
}
File diff suppressed because it is too large. Load diff
-6
View File
@@ -1,6 +0,0 @@
//go:build !race && !e2e_testing
package udp
// raceEnabled reports a -race build, whose sync.Pool drops a random share of what is put back.
const raceEnabled = false
-6
View File
@@ -1,6 +0,0 @@
//go:build race && !e2e_testing
package udp
// raceEnabled reports a -race build, whose sync.Pool drops a random share of what is put back.
const raceEnabled = true
+273 -100
View File
@@ -10,7 +10,6 @@ import (
"log/slog" "log/slog"
"net" "net"
"net/netip" "net/netip"
"sync"
"sync/atomic" "sync/atomic"
"syscall" "syscall"
"unsafe" "unsafe"
@@ -25,25 +24,11 @@ type StdConn struct {
sysFd uintptr sysFd uintptr
l *slog.Logger l *slog.Logger
// listenHost and port are the configured address and the bound port, which a connected socket shares. noSendmsgX atomic.Bool
listenHost netip.Addr sb *sendScratch
port uint16
// rc reaches the listener's descriptor for another listen routine's Rebind.
rc syscall.RawConn
socks *connSocks
// reader is ListenOut's reader, which connected socket readers share; readMu serializes them, since an
// EncReader isn't safe for concurrent use.
reader atomic.Pointer[readerPair]
readMu sync.Mutex
}
type readerPair struct {
r EncReader
flush func()
} }
var _ Conn = &StdConn{} var _ Conn = &StdConn{}
var _ EstablishedPeerConn = &StdConn{}
func NewListener(l *slog.Logger, s Settings) (Conn, error) { func NewListener(l *slog.Logger, s Settings) (Conn, error) {
lc := NewListenConfig(s.Multi) lc := NewListenConfig(s.Multi)
@@ -53,13 +38,12 @@ func NewListener(l *slog.Logger, s Settings) (Conn, error) {
} }
if uc, ok := pc.(*net.UDPConn); ok { if uc, ok := pc.(*net.UDPConn); ok {
c := &StdConn{UDPConn: uc, l: l, listenHost: s.Listen.Addr()} c := &StdConn{UDPConn: uc, l: l}
rc, err := uc.SyscallConn() rc, err := uc.SyscallConn()
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to open udp socket: %w", err) return nil, fmt.Errorf("failed to open udp socket: %w", err)
} }
c.rc = rc
err = rc.Control(func(fd uintptr) { err = rc.Control(func(fd uintptr) {
c.sysFd = fd c.sysFd = fd
@@ -73,8 +57,6 @@ func NewListener(l *slog.Logger, s Settings) (Conn, error) {
return nil, err return nil, err
} }
c.isV4 = la.Addr().Is4() c.isV4 = la.Addr().Is4()
c.port = la.Port()
c.socks = joinConnSocks(c, la, s)
return c, nil return c, nil
} }
@@ -111,19 +93,6 @@ func NewListenConfig(multi bool) net.ListenConfig {
func sendto(s int, buf []byte, flags int, to unsafe.Pointer, addrlen int32) (err error) func sendto(s int, buf []byte, flags int, to unsafe.Pointer, addrlen int32) (err error)
func (u *StdConn) WriteTo(b []byte, ap netip.AddrPort) error { func (u *StdConn) WriteTo(b []byte, ap netip.AddrPort) error {
if t := u.socks.table.Load(); t != nil {
if p := (*t)[ap]; p != nil {
bufs := [1][]byte{b}
if sent, err := u.connSockRun(p, ap, bufs[:]); sent == 1 || err != nil {
return err
}
}
}
return u.writeTo(b, ap)
}
// writeTo sends b through the listener.
func (u *StdConn) writeTo(b []byte, ap netip.AddrPort) error {
var sa unsafe.Pointer var sa unsafe.Pointer
var addrLen int32 var addrLen int32
@@ -174,52 +143,163 @@ func (u *StdConn) writeTo(b []byte, ap netip.AddrPort) error {
} }
} }
// sendBatch is how many datagrams one sendmsg_x call may carry.
const sendBatch = 64
// sendScratch is WriteBatch's scratch. The kernel reads every entry in place.
type sendScratch struct {
names [sendBatch]unix.RawSockaddrInet6
iovs [sendBatch]unix.Iovec
hdrs [sendBatch]msghdrX
// idx maps each entry back to its index in WriteBatch's bufs.
idx [sendBatch]int
}
// WriteBatch sends bufs with sendmsg_x, darwin's private batched sendmsg, so one syscall carries up to
// sendBatch datagrams. On this unconnected socket xnu still runs each message through sendit, so the
// saving is the syscall crossings, not the per-send policy checks.
//
// xnu's per-message loop stops at the first message that fails. For the errors in sentNothing it reports
// how many went out; for any other it returns only the errno, with the messages before it already sent.
// Each call therefore carries one run of datagrams to the same destination: a destination the kernel
// refuses fails on its first datagram, so a failed call costs that run and nothing else, and no datagram
// is ever resent after the kernel may have taken it.
//
// TODO: WriteTo maps EWOULDBLOCK to an error, so a full send buffer
// silently drops the rest of a burst (linux blocks instead). Poll for
// writability on EAGAIN before giving up on the remainder.
func (u *StdConn) WriteBatch(bufs [][]byte, addrs []netip.AddrPort) (int, error) { func (u *StdConn) WriteBatch(bufs [][]byte, addrs []netip.AddrPort) (int, error) {
// An un-sendable destination costs its own packet, never the ones behind it in the batch. if u.noSendmsgX.Load() {
// TODO: writeTo maps EWOULDBLOCK to an error, so a full listener send buffer return u.writeEach(bufs, addrs), nil
// silently drops those packets (linux blocks instead). Poll for }
// writability on EAGAIN before giving up on them. if u.sb == nil {
u.sb = new(sendScratch)
}
b := u.sb
written := 0
for i := 0; i < len(bufs); {
n := 0
for ; i < len(bufs) && n < sendBatch; i++ {
namelen, err := u.putSockaddr(&b.names[n], addrs[i])
if err != nil {
u.l.Debug("failed to write packet in batch", "udpAddr", addrs[i], "error", err)
continue
}
b.iovs[n] = unix.Iovec{}
if len(bufs[i]) > 0 {
b.iovs[n].Base = &bufs[i][0]
b.iovs[n].SetLen(len(bufs[i]))
}
b.hdrs[n] = msghdrX{Name: (*byte)(unsafe.Pointer(&b.names[n])), Namelen: namelen, Iov: &b.iovs[n], Iovlen: 1}
b.idx[n] = i
n++
}
for off := 0; off < n; {
dst := addrs[b.idx[off]]
end := off + 1
for end < n && addrs[b.idx[end]] == dst {
end++
}
sent, errno := u.sendmsgX(off, end)
switch {
case errno == 0 && sent > 0:
written += sent
off += sent
continue
case sentNothing(errno):
// The datagram at off failed on its own account. sendto delivers or reports it, as without
// batching, and the rest of the run goes back to sendmsg_x.
j := b.idx[off]
written += u.writeEach(bufs[j:j+1], addrs[j:j+1])
off++
continue
case errno == 0:
case errno == unix.ENOSYS || errno == unix.EOPNOTSUPP:
u.fallBack(errno)
case errno == unix.EPERM || errno == unix.EINVAL:
// Either can refuse one destination, but a datagram that sendto delivers was refused by
// sendmsg_x itself, as a sandbox or an older kernel would.
j := b.idx[off]
if u.writeEach(bufs[j:j+1], addrs[j:j+1]) == 0 {
off = end
continue
}
written++
off++
u.fallBack(errno)
default:
u.l.Debug("failed to write packets in batch", "udpAddr", dst, "packets", end-off, "error", errno)
off = end
continue
}
// The kernel took nothing from this call, so the rest can go out one datagram at a time.
for _, j := range b.idx[off:n] {
written += u.writeEach(bufs[j:j+1], addrs[j:j+1])
}
off = n
if u.noSendmsgX.Load() {
clear(b.iovs[:n])
return written + u.writeEach(bufs[i:], addrs[i:]), nil
}
}
clear(b.iovs[:n])
}
return written, nil
}
// sendmsgX hands entries off through end-1 of u.sb to sendmsg_x and returns how many the kernel took.
// The socket is non-blocking, so a full send buffer returns EAGAIN rather than waiting.
func (u *StdConn) sendmsgX(off, end int) (int, syscall.Errno) {
for {
r0, _, errno := unix.Syscall6(unix.SYS_SENDMSG_X, u.sysFd, uintptr(unsafe.Pointer(&u.sb.hdrs[off])), uintptr(end-off), 0, 0, 0)
if errno == unix.EINTR {
continue
}
return int(r0), errno
}
}
// sentNothing reports whether sendmsg_x failing with errno means no datagram went out: once any has, xnu
// turns these errors into a short count instead.
func sentNothing(errno syscall.Errno) bool {
return errno == unix.EWOULDBLOCK || errno == unix.ENOBUFS || errno == unix.EMSGSIZE
}
// fallBack switches WriteBatch to one sendto per datagram for the life of the socket.
func (u *StdConn) fallBack(errno syscall.Errno) {
if u.noSendmsgX.CompareAndSwap(false, true) {
u.l.Warn("sendmsg_x unavailable, sending one datagram per syscall", "error", errno)
}
}
// writeEach sends bufs one at a time with WriteTo and returns how many were written.
func (u *StdConn) writeEach(bufs [][]byte, addrs []netip.AddrPort) int {
written := 0 written := 0
t := u.socks.table.Load()
if t == nil {
for i, b := range bufs { for i, b := range bufs {
if err := u.writeTo(b, addrs[i]); err == nil { if err := u.WriteTo(b, addrs[i]); err == nil {
written++ written++
} else { } else {
u.l.Debug("failed to write packet in batch", "udpAddr", addrs[i], "error", err) u.l.Debug("failed to write packet in batch", "udpAddr", addrs[i], "error", err)
} }
} }
return written, nil return written
}
// putSockaddr writes ap into sa in the socket's address family, as WriteTo would, and returns its length.
func (u *StdConn) putSockaddr(sa *unix.RawSockaddrInet6, ap netip.AddrPort) (uint32, error) {
if u.isV4 {
if ap.Addr().Is6() {
return 0, ErrInvalidIPv6RemoteForSocket
} }
for off := 0; off < len(bufs); { sa4 := (*unix.RawSockaddrInet4)(unsafe.Pointer(sa))
// A run of datagrams to one peer goes out together, on its connected socket if it has or earns one. *sa4 = unix.RawSockaddrInet4{Family: unix.AF_INET, Addr: ap.Addr().As4()}
dst := addrs[off] binary.BigEndian.PutUint16((*[2]byte)(unsafe.Pointer(&sa4.Port))[:], ap.Port())
end := off + 1 return unix.SizeofSockaddrInet4, nil
for end < len(bufs) && addrs[end] == dst {
end++
} }
if p := (*t)[dst]; p != nil { *sa = unix.RawSockaddrInet6{Family: unix.AF_INET6, Addr: ap.Addr().As16()}
for off < end { binary.BigEndian.PutUint16((*[2]byte)(unsafe.Pointer(&sa.Port))[:], ap.Port())
sent, err := u.connSockRun(p, dst, bufs[off:end]) return unix.SizeofSockaddrInet6, nil
written += sent
off += sent
if err == nil {
break
}
u.l.Debug("failed to write packet in batch", "udpAddr", dst, "error", err)
off++
}
}
// The listener sends what no connected socket took.
for ; off < end; off++ {
if err := u.writeTo(bufs[off], dst); err == nil {
written++
} else {
u.l.Debug("failed to write packet in batch", "udpAddr", dst, "error", err)
}
}
}
return written, nil
} }
func (u *StdConn) LocalAddr() (netip.AddrPort, error) { func (u *StdConn) LocalAddr() (netip.AddrPort, error) {
@@ -247,13 +327,121 @@ func NewUDPStatsEmitter(udpConns []Conn) func() {
return func() {} return func() {}
} }
// recvBatch is how many datagrams one recvmsg_x call may return.
const recvBatch = 64
// msghdrX mirrors xnu's struct msghdr_x (bsd/sys/socket_private.h). recvmsg_x reports each
// datagram's length in Datalen rather than in its return value.
// Keep in sync with msghdrX in overlay/tun_darwin.go.
type msghdrX struct {
Name *byte
Namelen uint32
Iov *unix.Iovec
Iovlen int32
Control *byte
Controllen uint32
Flags int32
Datalen uint64
}
// ListenOut drains the socket with recvmsg_x, darwin's private batched recvmsg, so one syscall reads
// up to recvBatch datagrams instead of one.
func (u *StdConn) ListenOut(r EncReader, flush func()) error { func (u *StdConn) ListenOut(r EncReader, flush func()) error {
rc, err := u.UDPConn.SyscallConn()
if err != nil {
return err
}
bufs := make([][]byte, recvBatch)
names := make([]unix.RawSockaddrInet6, recvBatch)
iovs := make([]unix.Iovec, recvBatch)
hdrs := make([]msghdrX, recvBatch)
for i := range hdrs {
bufs[i] = make([]byte, MTU)
iovs[i].Base = &bufs[i][0]
iovs[i].SetLen(MTU)
hdrs[i].Iov = &iovs[i]
hdrs[i].Iovlen = 1
hdrs[i].Name = (*byte)(unsafe.Pointer(&names[i]))
}
for {
var n int
var errno syscall.Errno
err := rc.Read(func(fd uintptr) bool {
for i := range hdrs {
hdrs[i].Namelen = unix.SizeofSockaddrInet6
hdrs[i].Flags = 0
hdrs[i].Datalen = 0
}
r0, _, e := unix.Syscall6(unix.SYS_RECVMSG_X, fd, uintptr(unsafe.Pointer(&hdrs[0])), recvBatch, 0, 0, 0)
if e == unix.EAGAIN {
return false
}
n, errno = int(r0), e
return true
})
if err != nil {
if errors.Is(err, net.ErrClosed) {
return err
}
u.l.Error("unexpected udp socket receive error", "error", err)
continue
}
if errno == unix.ENOSYS || errno == unix.EPERM || errno == unix.EOPNOTSUPP {
// recvmsg_x is private, so a kernel or sandbox (such as an iOS extension's) may refuse it.
u.l.Warn("recvmsg_x unavailable, reading one datagram per syscall", "error", errno)
return u.listenOutSingle(r, flush)
}
if errno != 0 {
if errno != unix.EINTR {
u.l.Error("unexpected udp socket receive error", "error", errno)
}
continue
}
if err := checkMsghdrX(hdrs, names, n); err != nil {
// The batch was read through a layout the kernel no longer writes, so none of it is trustworthy.
u.l.Warn("recvmsg_x returned an unexpected header, reading one datagram per syscall", "error", err)
return u.listenOutSingle(r, flush)
}
for i := 0; i < n; i++ {
addr, ok := sockaddrToAddrPort(&names[i])
if !ok {
continue
}
l := int(hdrs[i].Datalen)
r(addr, bufs[i][:l:l])
}
flush()
}
}
// checkMsghdrX rejects a recvmsg_x result that intact msghdr_x entries can't produce, which is how a change
// to xnu's private struct layout would show up.
func checkMsghdrX(hdrs []msghdrX, names []unix.RawSockaddrInet6, n int) error {
if n > len(hdrs) {
return fmt.Errorf("returned %d datagrams for %d headers", n, len(hdrs))
}
for i := range n {
if hdrs[i].Datalen > MTU {
return fmt.Errorf("datagram %d is %d bytes, past the %d byte buffer", i, hdrs[i].Datalen, MTU)
}
switch {
case names[i].Family == unix.AF_INET && hdrs[i].Namelen == unix.SizeofSockaddrInet4:
case names[i].Family == unix.AF_INET6 && hdrs[i].Namelen == unix.SizeofSockaddrInet6:
default:
return fmt.Errorf("datagram %d has address family %d with length %d", i, names[i].Family, hdrs[i].Namelen)
}
}
return nil
}
// listenOutSingle reads one datagram per recvfrom and flushes after each, for when recvmsg_x is refused or misbehaves.
func (u *StdConn) listenOutSingle(r EncReader, flush func()) error {
buffer := make([]byte, MTU) buffer := make([]byte, MTU)
u.reader.Store(&readerPair{r: r, flush: flush})
u.startConnSocks()
for { for {
// Just read one packet at a time
n, rua, err := u.ReadFromUDPAddrPort(buffer) n, rua, err := u.ReadFromUDPAddrPort(buffer)
if err != nil { if err != nil {
if errors.Is(err, net.ErrClosed) { if errors.Is(err, net.ErrClosed) {
@@ -263,17 +451,22 @@ func (u *StdConn) ListenOut(r EncReader, flush func()) error {
continue continue
} }
u.readMu.Lock()
r(netip.AddrPortFrom(rua.Addr().Unmap(), rua.Port()), buffer[:n:n]) r(netip.AddrPortFrom(rua.Addr().Unmap(), rua.Port()), buffer[:n:n])
flush() flush()
u.readMu.Unlock()
} }
} }
// Close closes the connected sockets of every listen routine, then the listener. func sockaddrToAddrPort(sa *unix.RawSockaddrInet6) (netip.AddrPort, bool) {
func (u *StdConn) Close() error { switch sa.Family {
u.stopConnSocks() case unix.AF_INET:
return u.UDPConn.Close() sa4 := (*unix.RawSockaddrInet4)(unsafe.Pointer(sa))
port := binary.BigEndian.Uint16((*[2]byte)(unsafe.Pointer(&sa4.Port))[:])
return netip.AddrPortFrom(netip.AddrFrom4(sa4.Addr), port), true
case unix.AF_INET6:
port := binary.BigEndian.Uint16((*[2]byte)(unsafe.Pointer(&sa.Port))[:])
return netip.AddrPortFrom(netip.AddrFrom16(sa.Addr).Unmap(), port), true
}
return netip.AddrPort{}, false
} }
func (u *StdConn) SupportsMultipleReaders() bool { func (u *StdConn) SupportsMultipleReaders() bool {
@@ -282,10 +475,7 @@ func (u *StdConn) SupportsMultipleReaders() bool {
// Rebind clears the interface the kernel scoped this socket to, so that sends are routed against the current // Rebind clears the interface the kernel scoped this socket to, so that sends are routed against the current
// routing table instead of the interface we happened to be on when the socket was created. Darwin pins sockets // routing table instead of the interface we happened to be on when the socket was created. Darwin pins sockets
// this way on its own, which is what strands us after the underlying network changes. Control.RebindUDPServer // this way on its own, which is what strands us after the underlying network changes.
// rebinds only the first writer, so this clears the other listen routines' listeners too. Connected sockets are
// closed after the listeners are cleared, and later ones copy the cleared scope and pick their source addresses
// afresh.
func (u *StdConn) Rebind() error { func (u *StdConn) Rebind() error {
var err error var err error
if u.isV4 { if u.isV4 {
@@ -293,23 +483,6 @@ func (u *StdConn) Rebind() error {
} else { } else {
err = syscall.SetsockoptInt(int(u.sysFd), syscall.IPPROTO_IPV6, syscall.IPV6_BOUND_IF, 0) err = syscall.SetsockoptInt(int(u.sysFd), syscall.IPPROTO_IPV6, syscall.IPV6_BOUND_IF, 0)
} }
for _, m := range u.socks.siblings(u) {
err = errors.Join(err, m.clearBoundIf())
}
u.closeConnSocks("rebind")
return err
}
// clearBoundIf clears the interface the listener is scoped to through its RawConn, which fails on a listener that
// closed concurrently instead of reaching whatever reused its descriptor number.
func (u *StdConn) clearBoundIf() error {
level, opt := syscall.IPPROTO_IPV6, syscall.IPV6_BOUND_IF
if u.isV4 {
level, opt = syscall.IPPROTO_IP, syscall.IP_BOUND_IF
}
var serr error
if err := u.rc.Control(func(fd uintptr) { serr = syscall.SetsockoptInt(int(fd), level, opt, 0) }); err != nil {
return err return err
}
return serr
} }
+85
View File
@@ -0,0 +1,85 @@
//go:build !e2e_testing
package udp
import (
"log/slog"
"net"
"net/netip"
"sync"
"sync/atomic"
"testing"
)
// BenchmarkListenOut measures the receive path on loopback: senders flood datagrams sized like a full tunnel
// packet, and each op is one datagram delivered to the reader. There are enough senders that the reader, not the
// senders, sets the pace. flushes/pkt is 1 when every receive is its own syscall and falls as receives batch. The
// senders share the machine, so ns/op is only comparable between runs on the same host.
func BenchmarkListenOut(b *testing.B) {
for _, tc := range []struct{ name, listen string }{
{"v4", "127.0.0.1"},
{"v6", "::1"},
} {
b.Run(tc.name, func(b *testing.B) {
c, err := NewListener(slog.New(slog.DiscardHandler), Settings{Listen: netip.AddrPortFrom(netip.MustParseAddr(tc.listen), 0)})
if err != nil {
b.Fatal(err)
}
la, err := c.LocalAddr()
if err != nil {
b.Fatal(err)
}
const size, senders = 1300, 4
txs := make([]net.Conn, senders)
for i := range txs {
if txs[i], err = net.Dial("udp", la.String()); err != nil {
b.Fatal(err)
}
defer txs[i].Close()
}
target := int64(b.N)
var got, flushes atomic.Int64
reached := make(chan struct{})
done := make(chan struct{})
go func() {
_ = c.ListenOut(func(netip.AddrPort, []byte) {
if got.Add(1) == target {
close(reached)
}
}, func() { flushes.Add(1) })
close(done)
}()
// Writes fail with ENOBUFS while the send queue is full; the flood only has to keep the receiver busy.
stop := make(chan struct{})
var sent sync.WaitGroup
for _, tx := range txs {
sent.Go(func() {
payload := make([]byte, size)
for {
select {
case <-stop:
return
default:
_, _ = tx.Write(payload)
}
}
})
}
b.SetBytes(size)
b.ResetTimer()
<-reached
b.StopTimer()
close(stop)
sent.Wait()
if err := c.Close(); err != nil {
b.Fatal(err)
}
<-done
b.ReportMetric(float64(flushes.Load())/float64(got.Load()), "flushes/pkt")
})
}
}
+302
View File
@@ -0,0 +1,302 @@
//go:build !e2e_testing
package udp
import (
"bytes"
"fmt"
"log/slog"
"net"
"net/netip"
"sync"
"syscall"
"testing"
"time"
"unsafe"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/sys/unix"
)
// A burst larger than recvBatch arrives whole, with each datagram's own bytes and source address, and
// flush runs once per recvmsg_x batch rather than once per datagram.
func TestListenOutBatchesBurst(t *testing.T) {
for _, tc := range []struct{ name, listen string }{
{"v4", "127.0.0.1"},
{"v6", "::1"},
} {
t.Run(tc.name, func(t *testing.T) {
c, err := NewListener(slog.New(slog.DiscardHandler), Settings{Listen: netip.AddrPortFrom(netip.MustParseAddr(tc.listen), 0)})
require.NoError(t, err)
la, err := c.LocalAddr()
require.NoError(t, err)
tx, err := net.Dial("udp", la.String())
require.NoError(t, err)
defer tx.Close()
txAddr := netip.MustParseAddrPort(tx.LocalAddr().String())
const total = recvBatch*3 + 7
var mu sync.Mutex
var got []string
var froms []netip.AddrPort
flushes := 0
// Queued before the reader starts, so recvmsg_x sees full batches.
for i := 0; i < total; i++ {
_, err := tx.Write([]byte(fmt.Sprintf("pkt-%03d", i)))
require.NoError(t, err)
}
done := make(chan struct{})
go func() {
_ = c.ListenOut(func(addr netip.AddrPort, payload []byte) {
mu.Lock()
got = append(got, string(payload))
froms = append(froms, addr)
mu.Unlock()
}, func() {
mu.Lock()
flushes++
mu.Unlock()
})
close(done)
}()
require.Eventually(t, func() bool {
mu.Lock()
defer mu.Unlock()
return len(got) == total
}, 5*time.Second, 10*time.Millisecond)
require.NoError(t, c.Close())
<-done
for i, p := range got {
assert.Equal(t, fmt.Sprintf("pkt-%03d", i), p)
assert.Equal(t, txAddr, froms[i])
}
assert.Equal(t, (total+recvBatch-1)/recvBatch, flushes)
})
}
}
// msghdrX round-trips through xnu's recvmsg_x on a loopback socket, v4 and v6. The kernel reads Name, Namelen,
// Iov and Iovlen from each entry and writes Namelen, Flags and Datalen back, so a field at the wrong offset or a
// wrong stride between entries shows up as a wrong length, sender, or payload.
// TestListenOutSingleFallback pins the path ListenOut takes when the kernel refuses recvmsg_x.
func TestListenOutSingleFallback(t *testing.T) {
c, err := NewListener(slog.New(slog.DiscardHandler), Settings{Listen: netip.MustParseAddrPort("127.0.0.1:0")})
require.NoError(t, err)
la, err := c.LocalAddr()
require.NoError(t, err)
tx, err := net.Dial("udp", la.String())
require.NoError(t, err)
defer tx.Close()
txAddr := netip.MustParseAddrPort(tx.LocalAddr().String())
const total = 5
for i := 0; i < total; i++ {
_, err := tx.Write([]byte(fmt.Sprintf("pkt-%03d", i)))
require.NoError(t, err)
}
var mu sync.Mutex
var got []string
flushes := 0
done := make(chan struct{})
go func() {
_ = c.(*StdConn).listenOutSingle(func(addr netip.AddrPort, payload []byte) {
assert.Equal(t, txAddr, addr)
mu.Lock()
got = append(got, string(payload))
mu.Unlock()
}, func() {
mu.Lock()
flushes++
mu.Unlock()
})
close(done)
}()
require.Eventually(t, func() bool {
mu.Lock()
defer mu.Unlock()
return len(got) == total
}, 5*time.Second, 10*time.Millisecond)
require.NoError(t, c.Close())
<-done
for i, p := range got {
assert.Equal(t, fmt.Sprintf("pkt-%03d", i), p)
}
assert.Equal(t, total, flushes)
}
func TestCheckMsghdrX(t *testing.T) {
valid := func() ([]msghdrX, []unix.RawSockaddrInet6) {
hdrs := make([]msghdrX, 2)
names := make([]unix.RawSockaddrInet6, 2)
hdrs[0].Namelen, hdrs[0].Datalen = unix.SizeofSockaddrInet4, 1
names[0].Family = unix.AF_INET
hdrs[1].Namelen, hdrs[1].Datalen = unix.SizeofSockaddrInet6, MTU
names[1].Family = unix.AF_INET6
return hdrs, names
}
hdrs, names := valid()
require.NoError(t, checkMsghdrX(hdrs, names, 2))
require.NoError(t, checkMsghdrX(hdrs, names, 0))
for _, tc := range []struct {
name string
n int
mutate func([]msghdrX, []unix.RawSockaddrInet6)
}{
{"count past headers", 3, func([]msghdrX, []unix.RawSockaddrInet6) {}},
{"datalen past buffer", 2, func(h []msghdrX, _ []unix.RawSockaddrInet6) { h[1].Datalen = MTU + 1 }},
{"unknown family", 2, func(_ []msghdrX, a []unix.RawSockaddrInet6) { a[0].Family = unix.AF_UNIX }},
{"family and length disagree", 2, func(h []msghdrX, _ []unix.RawSockaddrInet6) { h[0].Namelen = unix.SizeofSockaddrInet6 }},
} {
t.Run(tc.name, func(t *testing.T) {
hdrs, names := valid()
tc.mutate(hdrs, names)
assert.Error(t, checkMsghdrX(hdrs, names, tc.n))
})
}
}
func TestMsghdrXKernelRoundTrip(t *testing.T) {
for _, v6 := range []bool{false, true} {
t.Run(fmt.Sprintf("v6=%v", v6), func(t *testing.T) {
// Loopback delivery goes through the dlil input thread, so the settle gives all three time to
// queue and come back from one call, which is what exercises the stride.
got := recvmsgXRoundTrip(t, v6, []int{1, 300, 1400}, 4, 50*time.Millisecond)
assert.Equal(t, 3, got, "largest batch from one recvmsg_x call")
})
}
}
// Any burst, of any datagram lengths up to the kernel's maxdgram, into any number of headers, comes back in order
// with every entry's fields intact, and headers past the returned count untouched.
func FuzzMsghdrXKernelRoundTrip(f *testing.F) {
f.Add(false, uint8(3), []byte{0, 1, 1, 44, 5, 120})
f.Add(true, uint8(0), []byte{0, 0, 0x23, 0x29}) // one empty datagram, then one at MTU
f.Add(false, uint8(1), []byte{0x23, 0x2a, 0x24, 0x00, 0}) // past MTU: cut to MTU, no MSG_TRUNC
f.Add(true, uint8(recvBatch-1), bytes.Repeat([]byte{0x05, 0xdc}, recvBatch))
f.Fuzz(func(t *testing.T, v6 bool, spare uint8, sizes []byte) {
// Two bytes per datagram, big-endian, folded into [0, maxdgram].
var lens []int
for i := 0; i+1 < len(sizes) && len(lens) < recvBatch; i += 2 {
lens = append(lens, (int(sizes[i])<<8|int(sizes[i+1]))%(maxDgram+1))
}
if len(lens) == 0 {
return
}
recvmsgXRoundTrip(t, v6, lens, 1+int(spare)%recvBatch, time.Millisecond)
})
}
// maxDgram is darwin's default net.inet.udp.maxdgram, the largest datagram a loopback sender may write.
const maxDgram = 9216
// recvmsgXRoundTrip sends one datagram per entry of lens to a fresh loopback socket, drains them with recvmsg_x
// into nh headers, and checks every field the kernel writes. Neighboring datagrams differ in sender, length and
// fill byte, so no field can pass by reading a neighbor's value. It returns the largest batch one call returned.
func recvmsgXRoundTrip(t *testing.T, v6 bool, lens []int, nh int, settle time.Duration) int {
t.Helper()
network, lo := "udp4", netip.MustParseAddr("127.0.0.1")
if v6 {
network, lo = "udp6", netip.IPv6Loopback()
}
rx, err := net.ListenUDP(network, net.UDPAddrFromAddrPort(netip.AddrPortFrom(lo, 0)))
require.NoError(t, err)
defer rx.Close()
txs := make([]*net.UDPConn, 3)
for i := range txs {
txs[i], err = net.DialUDP(network, nil, rx.LocalAddr().(*net.UDPAddr))
require.NoError(t, err)
defer txs[i].Close()
}
senders := make([]netip.AddrPort, len(lens))
for i, l := range lens {
tx := txs[i%len(txs)]
senders[i] = tx.LocalAddr().(*net.UDPAddr).AddrPort()
_, err := tx.Write(bytes.Repeat([]byte{byte(i + 1)}, l))
require.NoError(t, err)
}
time.Sleep(settle)
// Namelen goes in as the v6 size on both families, so the v4 size coming back proves the kernel wrote it.
const sentinel = 0xa5a5a5a5a5a5a5a5
wantNamelen := uint32(unix.SizeofSockaddrInet4)
if v6 {
wantNamelen = unix.SizeofSockaddrInet6
}
bufs := make([][]byte, nh)
names := make([]unix.RawSockaddrInet6, nh)
iovs := make([]unix.Iovec, nh)
hdrs := make([]msghdrX, nh)
for i := range hdrs {
bufs[i] = make([]byte, MTU)
iovs[i].Base = &bufs[i][0]
iovs[i].SetLen(MTU)
}
// A datagram dropped on loopback would otherwise block rc.Read forever.
require.NoError(t, rx.SetReadDeadline(time.Now().Add(5*time.Second)))
rc, err := rx.SyscallConn()
require.NoError(t, err)
largest := 0
for next := 0; next < len(lens); {
for i := range hdrs {
clear(bufs[i])
hdrs[i] = msghdrX{
Name: (*byte)(unsafe.Pointer(&names[i])),
Namelen: unix.SizeofSockaddrInet6,
Iov: &iovs[i],
Iovlen: 1,
Flags: -1,
Datalen: sentinel,
}
}
var n int
var errno syscall.Errno
require.NoError(t, rc.Read(func(fd uintptr) bool {
r0, _, e := unix.Syscall6(unix.SYS_RECVMSG_X, fd, uintptr(unsafe.Pointer(&hdrs[0])), uintptr(nh), 0, 0, 0)
n, errno = int(r0), e
return e != unix.EAGAIN
}), "received %d of %d datagrams", next, len(lens))
require.Zero(t, errno)
require.Positive(t, n)
require.LessOrEqual(t, n, min(nh, len(lens)-next))
largest = max(largest, n)
require.NoError(t, checkMsghdrX(hdrs[:nh], names[:nh], n), "ListenOut would fall back on this batch")
for i := range n {
d := next + i
h := &hdrs[i]
want := min(lens[d], MTU)
require.EqualValues(t, want, h.Datalen, "datagram %d Datalen", d)
require.Equal(t, wantNamelen, h.Namelen, "datagram %d Namelen", d)
from, ok := sockaddrToAddrPort(&names[i])
require.True(t, ok, "datagram %d family %d", d, names[i].Family)
require.Equal(t, senders[d], from, "datagram %d sender", d)
// Flags goes in as -1 and must come back 0. xnu's list receive (soreceive_m_list) never sets
// MSG_TRUNC, so a datagram past MTU arrives silently cut to MTU, as recvfrom through net.UDPConn does.
require.Zero(t, h.Flags, "datagram %d Flags", d)
require.Equal(t, bytes.Repeat([]byte{byte(d + 1)}, want), bufs[i][:want], "datagram %d payload", d)
if want < MTU {
require.Zero(t, bufs[i][want], "datagram %d wrote past Datalen", d)
}
}
for i := n; i < nh; i++ {
require.EqualValues(t, uint64(sentinel), hdrs[i].Datalen, "spare header %d Datalen", i)
require.EqualValues(t, unix.SizeofSockaddrInet6, hdrs[i].Namelen, "spare header %d Namelen", i)
}
next += n
}
return largest
}
+150
View File
@@ -0,0 +1,150 @@
//go:build !e2e_testing
package udp
import (
"fmt"
"log/slog"
"net"
"net/netip"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// recvAll reads n datagrams from pc, or fails the test after a second without one.
func recvAll(t *testing.T, pc net.PacketConn, n int) []string {
t.Helper()
var got []string
buf := make([]byte, 64)
for len(got) < n {
require.NoError(t, pc.SetReadDeadline(time.Now().Add(time.Second)))
l, _, err := pc.ReadFrom(buf)
require.NoError(t, err, "after %d of %d datagrams", len(got), n)
got = append(got, string(buf[:l]))
}
return got
}
// A batch longer than sendBatch reaches each peer whole and in order through sendmsg_x, without falling
// back to sendto: a run to one peer that spans two calls, then datagrams alternating between peers.
func TestWriteBatchSendsEachToItsPeer(t *testing.T) {
for _, tc := range []struct{ name, listen string }{
{"v4", "127.0.0.1"},
{"v6", "::1"},
} {
t.Run(tc.name, func(t *testing.T) {
c, err := NewListener(slog.New(slog.DiscardHandler), Settings{Listen: netip.AddrPortFrom(netip.MustParseAddr(tc.listen), 0)})
require.NoError(t, err)
defer c.Close()
var peers [2]net.PacketConn
var dsts [2]netip.AddrPort
for i := range peers {
peers[i], err = net.ListenPacket("udp", net.JoinHostPort(tc.listen, "0"))
require.NoError(t, err)
defer peers[i].Close()
dsts[i] = netip.MustParseAddrPort(peers[i].LocalAddr().String())
}
const total = sendBatch*2 + 5
bufs := make([][]byte, total)
addrs := make([]netip.AddrPort, total)
var want [2][]string
for i := range bufs {
bufs[i] = fmt.Appendf(nil, "pkt-%03d", i)
p := i % 2
if i < sendBatch+6 {
p = 0
}
addrs[i] = dsts[p]
want[p] = append(want[p], string(bufs[i]))
}
n, err := c.WriteBatch(bufs, addrs)
require.NoError(t, err)
assert.Equal(t, total, n)
assert.False(t, c.(*StdConn).noSendmsgX.Load(), "fell back to sendto")
for i := range peers {
assert.Equal(t, want[i], recvAll(t, peers[i], len(want[i])))
}
})
}
}
// Destinations that can't be sent to cost only their own packets: an IPv6 peer on an IPv4 socket is
// skipped before the call, and a broadcast address without SO_BROADCAST is refused by the kernel
// mid-batch. Neither stops the packets around it, delivers any twice, or turns sendmsg_x off.
func TestWriteBatchSkipsUnsendableDestinations(t *testing.T) {
c, err := NewListener(slog.New(slog.DiscardHandler), Settings{Listen: netip.MustParseAddrPort("127.0.0.1:0")})
require.NoError(t, err)
defer c.Close()
peer, err := net.ListenPacket("udp", "127.0.0.1:0")
require.NoError(t, err)
defer peer.Close()
good := netip.MustParseAddrPort(peer.LocalAddr().String())
bcast := netip.AddrPortFrom(netip.MustParseAddr("255.255.255.255"), good.Port())
v6 := netip.MustParseAddrPort("[::1]:4242")
addrs := []netip.AddrPort{good, bcast, bcast, good, v6, good, bcast}
bufs := make([][]byte, len(addrs))
for i := range bufs {
bufs[i] = fmt.Appendf(nil, "pkt-%d", i)
}
n, err := c.WriteBatch(bufs, addrs)
require.NoError(t, err)
assert.Equal(t, 3, n)
assert.False(t, c.(*StdConn).noSendmsgX.Load(), "fell back to sendto")
assert.Equal(t, []string{"pkt-0", "pkt-3", "pkt-5"}, recvAll(t, peer, 3))
require.NoError(t, peer.SetReadDeadline(time.Now().Add(100*time.Millisecond)))
_, _, err = peer.ReadFrom(make([]byte, 64))
assert.Error(t, err, "a datagram arrived twice")
}
// A datagram the kernel refuses as its own fault (EMSGSIZE here, as ENOBUFS would be under a full
// interface queue) costs only itself, whether it fails after others in the same call (a short count) or
// first (the errno alone): the rest of its run still arrives, in order, and sendmsg_x stays on.
func TestWriteBatchLosesOnlyTheFailedDatagram(t *testing.T) {
for _, tc := range []struct {
name string
big int
}{
{"mid-call", 2},
{"first", 0},
} {
t.Run(tc.name, func(t *testing.T) {
c, err := NewListener(slog.New(slog.DiscardHandler), Settings{Listen: netip.MustParseAddrPort("127.0.0.1:0")})
require.NoError(t, err)
defer c.Close()
peer, err := net.ListenPacket("udp", "127.0.0.1:0")
require.NoError(t, err)
defer peer.Close()
dst := netip.MustParseAddrPort(peer.LocalAddr().String())
bufs := make([][]byte, 5)
addrs := make([]netip.AddrPort, len(bufs))
var want []string
for i := range bufs {
addrs[i] = dst
if i == tc.big {
// Past the 65535-byte IP datagram limit, so the kernel can only refuse it.
bufs[i] = make([]byte, 70000)
continue
}
bufs[i] = fmt.Appendf(nil, "pkt-%d", i)
want = append(want, string(bufs[i]))
}
n, err := c.WriteBatch(bufs, addrs)
require.NoError(t, err)
assert.Equal(t, len(want), n)
assert.False(t, c.(*StdConn).noSendmsgX.Load(), "fell back to sendto")
assert.Equal(t, want, recvAll(t, peer, len(want)))
})
}
}