tun: Keep small idle batches on the reader

With staging enabled, a reader whose pipeline is empty handles a batch below
256 TX segments or RX packets itself, without a worker handoff or buffer swap.
The threshold is the measured crossover, not another configuration knob.
Inline work only starts once that reader's ordered work has completed, so
ownership and order hold across switches.

With several readers, transmit still takes shared tunnel credit and the same
reservations as staged batches, releasing it after the synchronous flush; a
sole reader with an empty pipeline has nothing to overtake and uses the
original inline path. Small batches a reader sends itself run on the reader's
CPU, so they can use a different XPS TX ring than its sender.

Metrics: flow.tx.inline and flow.rx.inline count reads handled on the reader
(TX inline also counts oversized reads).

With receive IRQ CPUs kept free of Nebula threads, this improved one TCP
stream over 100 GbE from 16.7 to 21.1 Gbit/s; that measurement does not isolate
the threshold. Tests cover the switch in both directions, ordering behind an
in-flight batch, a paused reader against repeated small sends on another,
RecvError handling on the listener, and send affinity across mode switches.
This commit is contained in:
Ryan Huber committed 2026-10-16 21:22:30 +00:00
1 parent e4f659a207
commit eb073d6c23
9 files changed
+199 -9

No files matched your search

+3
View File
@@ -284,6 +284,8 @@ tun:
# Staged reservations may use half the window, the other half absorbs inline and control traffic, relay forwarding
# and network reordering; unlimited forwarding or delay can still exceed it. Waiting for credit on a busy tunnel
# also delays the other tunnels of the same read, and readers are not guaranteed fairness.
# Idle batches below 256 TX segments or RX packets stay on the reader. With multiple routines, those sends still
# take tunnel reservation credit.
#staged:
# workers: 0
@@ -291,6 +293,7 @@ tun:
# batched sends flowing through one XPS-selected NIC TX ring, so packets within a flow stay ordered on the wire
# instead of being sprayed across multiple TX rings and reordered. Not reloadable. Coerced to false if routines <= 1.
# Staged senders are only pinned with an explicit cpu_affinity, see below; otherwise the scheduler places them.
# Small batches a reader sends itself can use a different TX ring than its sender.
#pin_threads: true
# pin_threads_key helps the CPU-auto-selector shuffle which CPUs are chosen for pinning.