The TX and RX slot counts were fixed at eight and six. TX now uses one slot
per worker plus the one being read and the one being sent, at least eight.
RX keeps up to 384 recvmmsg entries in flight in four to six slots, so a
larger listen.batch takes fewer slots and RX memory never exceeds today's.
tun.staged.tx_slots and rx_slots override either, down to two. The defaults
at listen.batch 64 with up to six workers are unchanged: eight TX slots of
about 1.1 MiB and six RX slots of 4 MiB with GRO.
With staging enabled, a reader whose pipeline is empty handles a batch below
256 TX segments or RX packets itself, without a worker handoff or buffer swap.
The threshold is the measured crossover, not another configuration knob.
Inline work only starts once that reader's ordered work has completed, so
ownership and order hold across switches.
With several readers, transmit still takes shared tunnel credit and the same
reservations as staged batches, releasing it after the synchronous flush; a
sole reader with an empty pipeline has nothing to overtake and uses the
original inline path. Small batches a reader sends itself run on the reader's
CPU, so they can use a different XPS TX ring than its sender.
Metrics: flow.tx.inline and flow.rx.inline count reads handled on the reader
(TX inline also counts oversized reads).
With receive IRQ CPUs kept free of Nebula threads, this improved one TCP
stream over 100 GbE from 16.7 to 21.1 Gbit/s; that measurement does not isolate
the threshold. Tests cover the switch in both directions, ordering behind an
in-flight batch, a paused reader against repeated small sends on another,
RecvError handling on the listener, and send affinity across mode switches.
With tun.staged.workers set, each Linux UDP listener classifies a recvmmsg
batch, hands its data packets to that many decrypt workers, and an ordered
writer commits the batch in arrival order: replay window, roaming, firewall,
control packets and tun writes go through the same helpers inline receive
uses. Six batches per routine bound memory and backpressure. Conns that cannot
hand off their buffers stay inline.
The listener's classification is speculative. Before touching replay state the
writer resolves each data packet through its own batch-local hostmap cache,
exactly where inline receive would, so data after a close in the same batch is
still delivered while data after a removal or replacement between batches is
dropped, with the same RecvError reply inline receive sends. Known replays are
marked by a precheck and skip decryption, but keep their place so that reply
stays identical when an earlier RecvError deleted their tunnel.
New API: udp.StdConn.ListenOutHandoff, and the same on TesterConn for e2e,
runs the receive loop while handing each batch's buffers to the caller in
exchange for replacements. It is found through an unexported interface in
nebula, so udp.Conn and its other implementations are unchanged.
Metrics: flow.rx.batches, and flow.stage.{rx_read,rx_decrypt,rx_write}.q<N>.busy_ns.
Tests cover both ciphers, ordering, duplicate and corrupted packets from other
addresses, window jumps, control packets mid-batch, RecvError then replay,
tunnel replacement, firewall parity, fallback, real UDP sockets, partial
batches at close, and a handshake and traffic through e2e that checks pipeline
use on Linux and the inline fallback elsewhere. No isolated receive throughput
claim is made.
Set tun.staged.workers to a positive count on Linux, with tun.use_offloads
enabled, to move encryption off each tun reader. The reader filters and
reserves counters for a whole read, hands the batch to that many encrypt
workers, and an ordered sender flushes the batches in read order. Eight slots
per routine bound the batches in flight; a full pipeline blocks the reader.
Zero workers, the default, keeps today's inline path, as do tun queues that
cannot hand off their read buffer and builds whose AES-GCM needs nonce-ordered
encryption (BoringCrypto, FIPS).
IMPORTANT: staged senders need peers with the 8192-entry replay window of
master. Released v1.11.x peers use 1024 and will drop reordered batches. The
window is not negotiated.
Readers keep today's pinning. The sender is only pinned with an explicit
tun.cpu_affinity: sender i uses entry i+1, unless that is its reader's CPU,
so a one-entry list leaves the sender unpinned rather than sharing it: two
busy transmit threads on one CPU serialise (eight TCP streams over 100 GbE
fell from 25.9 to 22.8 Gbit/s). Without a list the sender is not pinned. Placing it by queue index on a
hybrid CPU put it on an efficiency core, and one TCP stream over 100 GbE
reached 18.6-19.6 Gbit/s pinned against 20.2-20.6 unpinned.
New API, each needed across a package boundary:
- tio.Offload.SwapRxBuf hands the buffer of the last Read to the reader and
reads into the one given. It is a method on the concrete queue, found through
an unexported interface in nebula, so tio.Queue and its other
implementations are unchanged.
- batch.SendBatch.Reset drops queued packets without sending, for a sender
whose sockets are already closed; Flush now uses it.
Metrics, registered only for active pipelines: flow.tx.batches counts reads
handed to the sender, flow.stage.{tx_read,tx_encrypt,tx_send}.q<N>.busy_ns the
elapsed busy time of each stage, to find the limiting one.
Tests cover ordering and counters under backpressure with both ciphers (AES
skipped where nonce order is required), read errors with batches in flight,
shutdown, exhaustion, relays, fallback, firewall parity, and the actual reader
and send thread affinity for every placement, with pinning on and off: the
sender is pinned only to an explicit entry other than its reader's CPU.
The goroutines Main starts stop only when its context does, and only a caller
holding the Control can arrange that. Cancelling solely on error left a config
test, which returns no Control and no error, with the lighthouse query worker
and a hostname resolver per dns named static host running for the life of the
process. Every embedder that validates a config in process rather than by
exec'ing pays for that on each config load, dnclient and the apple clients
included.
Co-authored-by: Nate Brown <nbrown.us@gmail.com>
Ryan Huber