mirror of
https://github.com/slackhq/nebula.git
synced 2026-11-15 15:16:52 +01:00
multiport: SO_REUSEPORT groups per port, routines is per port
Under multiport each port had exactly one socket, so each port had exactly one core. That hurt worst on the base port, which carries every handshake, every lighthouse and punch packet, every peer without multiport, and every tunnel whose lanes are down or firewalled. Bind multiport.ports consecutive ports and give each one a full group of `routines` sockets sharing it through SO_REUSEPORT, so the kernel hashes each arriving 4-tuple onto one of the group. `routines` is now per port and the total worker count is routines * multiport.ports, computed for you: with `routines: 8` and `multiport.ports: 4` you get 8 routines per port, 32 total. A routine still owns exactly one socket, which is what lets the read path own its state without locking. Sockets are laid out port-major, writers[s*routinesPerPort+r] being the r'th socket on port listen.port+s, so socket selection is a pure function of (queue, lane) with no borrowing: laneSock(q,s) = s*R + q%R, and base traffic takes laneSock(q,0). Sibling routines land on different sockets of the same port, so a lane's port is served by its whole group. multiport.ports is a new key and must be > 1; there is no default, since under these semantics defaulting it would silently multiply the worker count. The multiport decision moves above tun creation because tun queues are sized by the total. Also drop the base-port fallback for unanswered lane probes. A lane now aims only at its own paired peer port: sharing the base port's destination buys the lane a source port of its own but costs the peer the receive spread lanes exist to create, so a lane that cannot reach its port stays down and its flows ride the base tunnel.
This commit is contained in:
1 parent
53c565eb29
commit
af439dadbf
6 files changed
+423
-165
No files matched your search
@@ -546,17 +546,35 @@ func (f *Interface) SendVia(via *HostInfo, relay *Relay, ad, nb, out []byte, noc
|
||||
|
||||
// egressSock picks the socket a tunnel packet leaves from.
|
||||
//
|
||||
// With multiport, everything that is not lane data plane egresses socket 0: handshakes, keepalives, close packets,
|
||||
// rejects and relay carriers all belong to the base tunnel's 4-tuple, which is the only one a peer's spoof/roam checks
|
||||
// and a vanilla peer's expectations know about. Lane data is sent directly through writers[lane] and never comes here.
|
||||
// Everything that is not lane data plane leaves from the base port: handshakes, keepalives, close packets, rejects and
|
||||
// relay carriers all belong to the base tunnel's 4-tuple, which is the only one a peer's spoof/roam checks and a
|
||||
// vanilla peer's expectations know about. Lane data goes through laneSock instead and never comes here.
|
||||
//
|
||||
// Without multiport every socket shares one port under SO_REUSEPORT, so the source address is identical either way and
|
||||
// we keep q, the socket the triggering packet arrived on, to avoid contending on socket 0's fd.
|
||||
// Which socket on the base port doesn't matter — they share an address, so they produce identical packets — so keep to
|
||||
// this routine's own share of the group and leave the rest of it uncontended. Without multiport that is q itself, since
|
||||
// every socket is on the base port.
|
||||
func (f *Interface) egressSock(q int) int {
|
||||
if f.multiport {
|
||||
return 0
|
||||
return f.laneSock(q, 0)
|
||||
}
|
||||
|
||||
// laneSock returns the index in writers of a socket bound to lane s's port, for a
|
||||
// routine that reads queue q.
|
||||
//
|
||||
// Under multiport the sockets are laid out port-major — writers[s*routinesPerPort
|
||||
// + r] is the r'th socket on port listen.port+s — so every routine has a sibling
|
||||
// socket on every port and the arithmetic is a lane index away. Routines pick the
|
||||
// sibling matching their own position in their group, which spreads the writers
|
||||
// for one port over that port's whole group rather than funnelling them onto its
|
||||
// first socket. It is a pure function of (q, s), so a flow always leaves from the
|
||||
// same socket and cannot reorder itself across two of them.
|
||||
//
|
||||
// Without multiport there is one port and every socket is on it, so any lane
|
||||
// resolves to q's own socket.
|
||||
func (f *Interface) laneSock(q, s int) int {
|
||||
if !f.multiport {
|
||||
return q
|
||||
}
|
||||
return q
|
||||
return s*f.routinesPerPort + q%f.routinesPerPort
|
||||
}
|
||||
|
||||
func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType, ci *ConnectionState, hostinfo *HostInfo, remote netip.AddrPort, p, nb, out []byte, q int) {
|
||||
|
||||
Reference in new issue
Block a user