tun: Keep small idle batches on the reader

With staging enabled, a reader whose pipeline is empty handles a batch below
256 TX segments or RX packets itself, without a worker handoff or buffer swap.
The threshold is the measured crossover, not another configuration knob.
Inline work only starts once that reader's ordered work has completed, so
ownership and order hold across switches.

With several readers, transmit still takes shared tunnel credit and the same
reservations as staged batches, releasing it after the synchronous flush; a
sole reader with an empty pipeline has nothing to overtake and uses the
original inline path. Small batches a reader sends itself run on the reader's
CPU, so they can use a different XPS TX ring than its sender.

Metrics: flow.tx.inline and flow.rx.inline count reads handled on the reader
(TX inline also counts oversized reads).

With receive IRQ CPUs kept free of Nebula threads, this improved one TCP
stream over 100 GbE from 16.7 to 21.1 Gbit/s; that measurement does not isolate
the threshold. Tests cover the switch in both directions, ordering behind an
in-flight batch, a paused reader against repeated small sends on another,
RecvError handling on the listener, and send affinity across mode switches.
This commit is contained in:
Ryan Huber committed 2026-10-16 21:22:30 +00:00
1 parent e4f659a207
commit eb073d6c23
9 files changed
+199 -9

No files matched your search

+1 -1
View File
@@ -142,6 +142,6 @@ func TestParseStagedConfig(t *testing.T) {
require.Equal(t, stagedConfig{}, staged(2))
return
}
want := stagedConfig{workers: 2, tx: !noiseutil.EncryptLockNeeded, rx: true}
want := stagedConfig{workers: 2, tx: !noiseutil.EncryptLockNeeded, rx: true, inlineBelow: 256}
require.Equal(t, want, staged(2))
}