multiport: SO_REUSEPORT groups per port, routines is per port

Under multiport each port had exactly one socket, so each port had exactly
one core. That hurt worst on the base port, which carries every handshake,
every lighthouse and punch packet, every peer without multiport, and every
tunnel whose lanes are down or firewalled.

Bind multiport.ports consecutive ports and give each one a full group of
`routines` sockets sharing it through SO_REUSEPORT, so the kernel hashes
each arriving 4-tuple onto one of the group. `routines` is now per port and
the total worker count is routines * multiport.ports, computed for you: with
`routines: 8` and `multiport.ports: 4` you get 8 routines per port, 32
total. A routine still owns exactly one socket, which is what lets the read
path own its state without locking.

Sockets are laid out port-major, writers[s*routinesPerPort+r] being the r'th
socket on port listen.port+s, so socket selection is a pure function of
(queue, lane) with no borrowing: laneSock(q,s) = s*R + q%R, and base traffic
takes laneSock(q,0). Sibling routines land on different sockets of the same
port, so a lane's port is served by its whole group.

multiport.ports is a new key and must be > 1; there is no default, since
under these semantics defaulting it would silently multiply the worker
count. The multiport decision moves above tun creation because tun queues
are sized by the total.

Also drop the base-port fallback for unanswered lane probes. A lane now
aims only at its own paired peer port: sharing the base port's destination
buys the lane a source port of its own but costs the peer the receive
spread lanes exist to create, so a lane that cannot reach its port stays
down and its flows ride the base tunnel.
This commit is contained in:
Wade Simmons committed 2026-09-03 14:24:57 -04:00
1 parent 53c565eb29
commit af439dadbf
6 files changed
+423 -165

No files matched your search

+26 -8
View File
@@ -546,17 +546,35 @@ func (f *Interface) SendVia(via *HostInfo, relay *Relay, ad, nb, out []byte, noc
// egressSock picks the socket a tunnel packet leaves from.
//
// With multiport, everything that is not lane data plane egresses socket 0: handshakes, keepalives, close packets,
// rejects and relay carriers all belong to the base tunnel's 4-tuple, which is the only one a peer's spoof/roam checks
// and a vanilla peer's expectations know about. Lane data is sent directly through writers[lane] and never comes here.
// Everything that is not lane data plane leaves from the base port: handshakes, keepalives, close packets, rejects and
// relay carriers all belong to the base tunnel's 4-tuple, which is the only one a peer's spoof/roam checks and a
// vanilla peer's expectations know about. Lane data goes through laneSock instead and never comes here.
//
// Without multiport every socket shares one port under SO_REUSEPORT, so the source address is identical either way and
// we keep q, the socket the triggering packet arrived on, to avoid contending on socket 0's fd.
// Which socket on the base port doesn't matter — they share an address, so they produce identical packets — so keep to
// this routine's own share of the group and leave the rest of it uncontended. Without multiport that is q itself, since
// every socket is on the base port.
func (f *Interface) egressSock(q int) int {
if f.multiport {
return 0
return f.laneSock(q, 0)
}
// laneSock returns the index in writers of a socket bound to lane s's port, for a
// routine that reads queue q.
//
// Under multiport the sockets are laid out port-major — writers[s*routinesPerPort
// + r] is the r'th socket on port listen.port+s — so every routine has a sibling
// socket on every port and the arithmetic is a lane index away. Routines pick the
// sibling matching their own position in their group, which spreads the writers
// for one port over that port's whole group rather than funnelling them onto its
// first socket. It is a pure function of (q, s), so a flow always leaves from the
// same socket and cannot reorder itself across two of them.
//
// Without multiport there is one port and every socket is on it, so any lane
// resolves to q's own socket.
func (f *Interface) laneSock(q, s int) int {
if !f.multiport {
return q
}
return q
return s*f.routinesPerPort + q%f.routinesPerPort
}
func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType, ci *ConnectionState, hostinfo *HostInfo, remote netip.AddrPort, p, nb, out []byte, q int) {