mirror of
https://github.com/slackhq/nebula.git
synced 2026-11-15 18:46:52 +01:00
tun: Share replay-window reservation credit across readers
Charge staged reservations to each tunnel instead of dividing a static share of the window by routines. Several readers sending on one tunnel then share its credit: the limit is half the peer replay window, assumed from ReplayWindow, which must be a power of two of at least 8192 (checked at compile time). The other half allows finite unreserved traffic (inline and control sends, relay forwarding through SendVia) and network reordering; unlimited bypass traffic or delay can still exceed it. An outstanding-count cap alone would let a fast reader repeatedly send and reacquire credit, overtaking a paused reader without bound. So once readers overlap, spent credit is only recycled when every reservation has drained. A sole ordered reader gets each completion refunded, since its later batches cannot overtake its earlier ones. A reader that fails to acquire ends sole ownership, so a busy owner cannot keep a larger waiter blocked forever. This restores progress, not FIFO fairness. All transitions are a single CAS. Waiting readers flush their own oldest batch before retrying and never hold a partial acquisition: a failed multi-tunnel (relay) acquisition rolls back. Credit is released only after the flush or an explicit drop. A read that alone exceeds a tunnel's credit, such as a tiny-MSS superpacket, is sent segment by segment, synchronously. Acquisition is per read, so pressure on one tunnel also delays the other tunnels in the same read. The combined staged transmit, receive and shared credit measured 15 to 25 Gbit/s over one tunnel with eight TCP streams between two 100 GbE hosts. Tests model overtaking, ordered reuse, a blocked reader behind a busy owner, and relay rollback with both window sizes, and run eight competing readers plus unreserved control traffic on one tunnel, checking every packet is accepted by the receiver's replay window in wire order.
This commit is contained in:
1 parent
db8f9cd2e9
commit
e4f659a207
8 files changed
+427
-28
No files matched your search
@@ -19,11 +19,11 @@ import (
|
||||
// the crypto and then drops packets whose destination has closed. TX uses eight slots and RX six; at listen.batch=64
|
||||
// with GRO their buffers take about 9 MiB and 24 MiB per routine. Small-MSS TX arenas can grow.
|
||||
//
|
||||
// Metrics are registered only for active pipelines. flow.{tx,rx}.batches counts reads handed to the ordered stage.
|
||||
// flow.stage.<stage>.q<queue>.busy_ns locates the limiting stage: tx_read, tx_encrypt, tx_send, rx_read, rx_decrypt
|
||||
// or rx_write. Take rates; 1e9 ns/s fills a serial stage. These are elapsed times that exclude read syscalls and
|
||||
// downstream waits; descheduling inflates them and worker times add up across the pool. Small RX batches sent to the
|
||||
// writer include decryption in writer time.
|
||||
// Metrics are registered only for active pipelines. flow.{tx,rx}.batches counts reads handed to the ordered stage and
|
||||
// flow.tx.inline oversized reads the reader sent itself. flow.stage.<stage>.q<queue>.busy_ns locates the limiting
|
||||
// stage: tx_read, tx_encrypt, tx_send, rx_read, rx_decrypt or rx_write. Take rates; 1e9 ns/s fills a serial stage.
|
||||
// These are elapsed times that exclude read syscalls and downstream waits; descheduling inflates them and worker
|
||||
// times add up across the pool. Small RX batches sent to the writer include decryption in writer time.
|
||||
|
||||
// stagedConfig is tun.staged. workers is per routine and direction; zero keeps processing inline.
|
||||
type stagedConfig struct {
|
||||
|
||||
Reference in new issue
Block a user