tun: Share replay-window reservation credit across readers

Charge staged reservations to each tunnel instead of dividing a static share
of the window by routines. Several readers sending on one tunnel then share
its credit: the limit is half the peer replay window, assumed from
ReplayWindow, which must be a power of two of at least 8192 (checked at
compile time). The other half allows finite unreserved traffic (inline and
control sends, relay forwarding through SendVia) and network reordering;
unlimited bypass traffic or delay can still exceed it.

An outstanding-count cap alone would let a fast reader repeatedly send and
reacquire credit, overtaking a paused reader without bound. So once readers
overlap, spent credit is only recycled when every reservation has drained. A
sole ordered reader gets each completion refunded, since its later batches
cannot overtake its earlier ones. A reader that fails to acquire ends sole
ownership, so a busy owner cannot keep a larger waiter blocked forever. This
restores progress, not FIFO fairness. All transitions are a single CAS.

Waiting readers flush their own oldest batch before retrying and never hold a
partial acquisition: a failed multi-tunnel (relay) acquisition rolls back.
Credit is released only after the flush or an explicit drop. A read that alone
exceeds a tunnel's credit, such as a tiny-MSS superpacket, is sent segment by
segment, synchronously. Acquisition is per read, so pressure on one tunnel
also delays the other tunnels in the same read.

The combined staged transmit, receive and shared credit measured 15 to 25
Gbit/s over one tunnel with eight TCP streams between two 100 GbE hosts.

Tests model overtaking, ordered reuse, a blocked reader behind a busy owner,
and relay rollback with both window sizes, and run eight competing readers
plus unreserved control traffic on one tunnel, checking every packet is
accepted by the receiver's replay window in wire order.
This commit is contained in:
Ryan Huber committed 2026-10-16 21:22:11 +00:00
1 parent db8f9cd2e9
commit e4f659a207
8 files changed
+427 -28

No files matched your search

+5 -5
View File
@@ -19,11 +19,11 @@ import (
// the crypto and then drops packets whose destination has closed. TX uses eight slots and RX six; at listen.batch=64
// with GRO their buffers take about 9 MiB and 24 MiB per routine. Small-MSS TX arenas can grow.
//
// Metrics are registered only for active pipelines. flow.{tx,rx}.batches counts reads handed to the ordered stage.
// flow.stage.<stage>.q<queue>.busy_ns locates the limiting stage: tx_read, tx_encrypt, tx_send, rx_read, rx_decrypt
// or rx_write. Take rates; 1e9 ns/s fills a serial stage. These are elapsed times that exclude read syscalls and
// downstream waits; descheduling inflates them and worker times add up across the pool. Small RX batches sent to the
// writer include decryption in writer time.
// Metrics are registered only for active pipelines. flow.{tx,rx}.batches counts reads handed to the ordered stage and
// flow.tx.inline oversized reads the reader sent itself. flow.stage.<stage>.q<queue>.busy_ns locates the limiting
// stage: tx_read, tx_encrypt, tx_send, rx_read, rx_decrypt or rx_write. Take rates; 1e9 ns/s fills a serial stage.
// These are elapsed times that exclude read syscalls and downstream waits; descheduling inflates them and worker
// times add up across the pool. Small RX batches sent to the writer include decryption in writer time.
// stagedConfig is tun.staged. workers is per routine and direction; zero keeps processing inline.
type stagedConfig struct {