mirror of
https://github.com/slackhq/nebula.git
synced 2026-11-15 10:36:52 +01:00
tun: Keep small idle batches on the reader
With staging enabled, a reader whose pipeline is empty handles a batch below 256 TX segments or RX packets itself, without a worker handoff or buffer swap. The threshold is the measured crossover, not another configuration knob. Inline work only starts once that reader's ordered work has completed, so ownership and order hold across switches. With several readers, transmit still takes shared tunnel credit and the same reservations as staged batches, releasing it after the synchronous flush; a sole reader with an empty pipeline has nothing to overtake and uses the original inline path. Small batches a reader sends itself run on the reader's CPU, so they can use a different XPS TX ring than its sender. Metrics: flow.tx.inline and flow.rx.inline count reads handled on the reader (TX inline also counts oversized reads). With receive IRQ CPUs kept free of Nebula threads, this improved one TCP stream over 100 GbE from 16.7 to 21.1 Gbit/s; that measurement does not isolate the threshold. Tests cover the switch in both directions, ordering behind an in-flight batch, a paused reader against repeated small sends on another, RecvError handling on the listener, and send affinity across mode switches.
This commit is contained in:
1 parent
e4f659a207
commit
eb073d6c23
9 files changed
+199
-9
No files matched your search
@@ -284,6 +284,8 @@ tun:
|
||||
# Staged reservations may use half the window, the other half absorbs inline and control traffic, relay forwarding
|
||||
# and network reordering; unlimited forwarding or delay can still exceed it. Waiting for credit on a busy tunnel
|
||||
# also delays the other tunnels of the same read, and readers are not guaranteed fairness.
|
||||
# Idle batches below 256 TX segments or RX packets stay on the reader. With multiple routines, those sends still
|
||||
# take tunnel reservation credit.
|
||||
#staged:
|
||||
# workers: 0
|
||||
|
||||
@@ -291,6 +293,7 @@ tun:
|
||||
# batched sends flowing through one XPS-selected NIC TX ring, so packets within a flow stay ordered on the wire
|
||||
# instead of being sprayed across multiple TX rings and reordered. Not reloadable. Coerced to false if routines <= 1.
|
||||
# Staged senders are only pinned with an explicit cpu_affinity, see below; otherwise the scheduler places them.
|
||||
# Small batches a reader sends itself can use a different TX ring than its sender.
|
||||
#pin_threads: true
|
||||
|
||||
# pin_threads_key helps the CPU-auto-selector shuffle which CPUs are chosen for pinning.
|
||||
|
||||
Reference in new issue
Block a user