With staged transmit, an idle reader encrypted and sent a small read
itself, while larger reads went to the workers and left from the
ordered sender. Each read chose on its own, so under load a routine
alternated between the two paths. Their packets left from different
CPUs, and the receiver's TCP saw them out of order: on 100 GbE, summed
over four hosts, dsack_dups rose from 0-1 on master to about 25k and
reord_seen from 0 to 13k-19k.
Each routine now holds one mode, inline on the reader or fanout to the
workers and the sender. It switches only after tun.staged.gate_hold
(default 64) reads in a row want the other mode, and before going
inline it waits until every batch in flight is on the wire, so a
switch never lets a later packet overtake an earlier one. Going to
fanout nothing is in flight. flow.tx.gate_switches counts the switches.
On the same hosts dsack_dups fell to 20-52 and reord_seen to 0-2,
while throughput matched or beat per-read mixing (many-to-one 34.9
instead of 30.7 Gbit/s).
The affinity test waited only for the previous batch's packets to be
captured before the next read, but the sender returns the slot after
its write, so the reader could still see it in flight and stage the
small read. Switching to inline now drains, which makes the inline
send deterministic there.
The TX reader timed its slot wait with a second pair of clock reads only to
subtract it again. It now waits for a free slot before it starts the busy
clock, so a read costs one time.Now and one time.Since. The RX listener
likewise ends its busy time when it has to wait for a batch instead of
timing the wait, and pool workers time a run of queued jobs with one pair
of reads. A test checks the read stages still leave backpressure out.
The TX and RX slot counts were fixed at eight and six. TX now uses one slot
per worker plus the one being read and the one being sent, at least eight.
RX keeps up to 384 recvmmsg entries in flight in four to six slots, so a
larger listen.batch takes fewer slots and RX memory never exceeds today's.
tun.staged.tx_slots and rx_slots override either, down to two. The defaults
at listen.batch 64 with up to six workers are unchanged: eight TX slots of
about 1.1 MiB and six RX slots of 4 MiB with GRO.
With staging enabled, a reader whose pipeline is empty handles a batch below
256 TX segments or RX packets itself, without a worker handoff or buffer swap.
The threshold is the measured crossover, not another configuration knob.
Inline work only starts once that reader's ordered work has completed, so
ownership and order hold across switches.
With several readers, transmit still takes shared tunnel credit and the same
reservations as staged batches, releasing it after the synchronous flush; a
sole reader with an empty pipeline has nothing to overtake and uses the
original inline path. Small batches a reader sends itself run on the reader's
CPU, so they can use a different XPS TX ring than its sender.
Metrics: flow.tx.inline and flow.rx.inline count reads handled on the reader
(TX inline also counts oversized reads).
With receive IRQ CPUs kept free of Nebula threads, this improved one TCP
stream over 100 GbE from 16.7 to 21.1 Gbit/s; that measurement does not isolate
the threshold. Tests cover the switch in both directions, ordering behind an
in-flight batch, a paused reader against repeated small sends on another,
RecvError handling on the listener, and send affinity across mode switches.
Charge staged reservations to each tunnel instead of dividing a static share
of the window by routines. Several readers sending on one tunnel then share
its credit: the limit is half the peer replay window, assumed from
ReplayWindow, which must be a power of two of at least 8192 (checked at
compile time). The other half allows finite unreserved traffic (inline and
control sends, relay forwarding through SendVia) and network reordering;
unlimited bypass traffic or delay can still exceed it.
An outstanding-count cap alone would let a fast reader repeatedly send and
reacquire credit, overtaking a paused reader without bound. So once readers
overlap, spent credit is only recycled when every reservation has drained. A
sole ordered reader gets each completion refunded, since its later batches
cannot overtake its earlier ones. A reader that fails to acquire ends sole
ownership, so a busy owner cannot keep a larger waiter blocked forever. This
restores progress, not FIFO fairness. All transitions are a single CAS.
Waiting readers flush their own oldest batch before retrying and never hold a
partial acquisition: a failed multi-tunnel (relay) acquisition rolls back.
Credit is released only after the flush or an explicit drop. A read that alone
exceeds a tunnel's credit, such as a tiny-MSS superpacket, is sent segment by
segment, synchronously. Acquisition is per read, so pressure on one tunnel
also delays the other tunnels in the same read.
The combined staged transmit, receive and shared credit measured 15 to 25
Gbit/s over one tunnel with eight TCP streams between two 100 GbE hosts.
Tests model overtaking, ordered reuse, a blocked reader behind a busy owner,
and relay rollback with both window sizes, and run eight competing readers
plus unreserved control traffic on one tunnel, checking every packet is
accepted by the receiver's replay window in wire order.
Set tun.staged.workers to a positive count on Linux, with tun.use_offloads
enabled, to move encryption off each tun reader. The reader filters and
reserves counters for a whole read, hands the batch to that many encrypt
workers, and an ordered sender flushes the batches in read order. Eight slots
per routine bound the batches in flight; a full pipeline blocks the reader.
Zero workers, the default, keeps today's inline path, as do tun queues that
cannot hand off their read buffer and builds whose AES-GCM needs nonce-ordered
encryption (BoringCrypto, FIPS).
IMPORTANT: staged senders need peers with the 8192-entry replay window of
master. Released v1.11.x peers use 1024 and will drop reordered batches. The
window is not negotiated.
Readers keep today's pinning. The sender is only pinned with an explicit
tun.cpu_affinity: sender i uses entry i+1, unless that is its reader's CPU,
so a one-entry list leaves the sender unpinned rather than sharing it: two
busy transmit threads on one CPU serialise (eight TCP streams over 100 GbE
fell from 25.9 to 22.8 Gbit/s). Without a list the sender is not pinned. Placing it by queue index on a
hybrid CPU put it on an efficiency core, and one TCP stream over 100 GbE
reached 18.6-19.6 Gbit/s pinned against 20.2-20.6 unpinned.
New API, each needed across a package boundary:
- tio.Offload.SwapRxBuf hands the buffer of the last Read to the reader and
reads into the one given. It is a method on the concrete queue, found through
an unexported interface in nebula, so tio.Queue and its other
implementations are unchanged.
- batch.SendBatch.Reset drops queued packets without sending, for a sender
whose sockets are already closed; Flush now uses it.
Metrics, registered only for active pipelines: flow.tx.batches counts reads
handed to the sender, flow.stage.{tx_read,tx_encrypt,tx_send}.q<N>.busy_ns the
elapsed busy time of each stage, to find the limiting one.
Tests cover ordering and counters under backpressure with both ciphers (AES
skipped where nonce order is required), read errors with batches in flight,
shutdown, exhaustion, relays, fallback, firewall parity, and the actual reader
and send thread affinity for every placement, with pinning on and off: the
sender is pinned only to an explicit entry other than its reader's CPU.
Build on the inside-packet filtering split with shared preparation and encryption helpers for inline and reserved sends. Reserve each batch’s per-tunnel counter range before encryption, including relay counters, without enabling staged processing yet. Test reserved encryption against a real receiver and check segmentation counts.
Ryan Huber