mirror of
https://github.com/slackhq/nebula.git
synced 2026-11-15 16:26:52 +01:00
Build multiport lanes lazily, on data-plane demand
Lanes were established for every slot as soon as the base tunnel came up, so a host with `routines: N` paid N-1 extra Noise handshakes, sessions and keepalives for every peer it talked to, including peers it exchanges a trickle with. Only routines that actually carry traffic to a peer need a lane. Add a per-slot demand flag to laneState. sendInsideMessage already loads txLanes[laneSlot] on every direct-path packet; that load missing is now the signal, and EnsureLanes only starts slots the data plane asked for. The miss path is unchanged otherwise: the packet rides the base tunnel, the same fallback used while a lane is down. noteLaneDemand load-guards its store so repeated misses on a lane-less slot are plain reads rather than a cache line ping-ponging between the routines sharing it. The demand check is EnsureLanes' last condition, so a slot that is pending or inside its backoff keeps the flag for the tick that can act on it. Consuming the flag also stops a lane whose routine went quiet from being rebuilt forever after it dies. The wire format, lane negotiation and port-offset scheme are untouched, so a lazy node interoperates with an eager one in both directions, and inbound lane handshakes from a busy peer are still accepted regardless of our own demand.
This commit is contained in:
1 parent
192d994273
commit
11528c6e02
6 files changed
+154
-11
No files matched your search
+32
@@ -325,6 +325,14 @@ type laneState struct {
|
||||
// data-plane routine that Loads non-nil always sees a usable tunnel.
|
||||
txLanes []atomic.Pointer[HostInfo]
|
||||
|
||||
// txDemand[i] is set by the data plane when a routine riding slot i has
|
||||
// traffic for this peer and no established lane, and consumed when
|
||||
// EnsureLanes claims the slot. Lanes exist only where traffic asked for
|
||||
// one: a peer we exchange a trickle with never costs more than the base
|
||||
// tunnel, no matter how many routines are configured. Written without
|
||||
// the Mutex.
|
||||
txDemand []atomic.Bool
|
||||
|
||||
// Under Mutex: per-slot handshake-in-flight flag, consecutive failure
|
||||
// count, and earliest next attempt, driving ensureLanes' backoff.
|
||||
txPending []bool
|
||||
@@ -342,12 +350,36 @@ func newLaneState(laneCount int, peerPortCount, peerBasePort, portOffset uint16)
|
||||
peerBasePort: peerBasePort,
|
||||
portOffset: portOffset,
|
||||
txLanes: make([]atomic.Pointer[HostInfo], laneCount),
|
||||
txDemand: make([]atomic.Bool, laneCount),
|
||||
txPending: make([]bool, laneCount),
|
||||
txFails: make([]uint8, laneCount),
|
||||
txRetryAt: make([]time.Time, laneCount),
|
||||
}
|
||||
}
|
||||
|
||||
// noteLaneDemand records that a data-plane routine has traffic for lane slot
|
||||
// i but found the slot empty. Called from the TX hot path on every packet
|
||||
// that misses, so the store is load-guarded: once the flag is up, further
|
||||
// misses are plain reads and cannot ping-pong a cache line shared with the
|
||||
// neighbouring slots' flags. Slot 0 is the base tunnel and never a lane.
|
||||
func (ls *laneState) noteLaneDemand(i int) {
|
||||
if i <= 0 || i >= len(ls.txDemand) {
|
||||
return
|
||||
}
|
||||
if !ls.txDemand[i].Load() {
|
||||
ls.txDemand[i].Store(true)
|
||||
}
|
||||
}
|
||||
|
||||
// takeLaneDemand consumes slot i's demand flag, reporting whether the data
|
||||
// plane had asked for the lane since the last time we looked.
|
||||
func (ls *laneState) takeLaneDemand(i int) bool {
|
||||
if i <= 0 || i >= len(ls.txDemand) {
|
||||
return false
|
||||
}
|
||||
return ls.txDemand[i].Swap(false)
|
||||
}
|
||||
|
||||
// laneTargetPort returns the peer port that owned lane i handshakes to and
|
||||
// egresses toward. The caller must ensure peerPortCount != 0.
|
||||
func (ls *laneState) laneTargetPort(i int) uint16 {
|
||||
|
||||
Reference in new issue
Block a user