tun: Hold one transmit mode per routine

With staged transmit, an idle reader encrypted and sent a small read
itself, while larger reads went to the workers and left from the
ordered sender. Each read chose on its own, so under load a routine
alternated between the two paths. Their packets left from different
CPUs, and the receiver's TCP saw them out of order: on 100 GbE, summed
over four hosts, dsack_dups rose from 0-1 on master to about 25k and
reord_seen from 0 to 13k-19k.

Each routine now holds one mode, inline on the reader or fanout to the
workers and the sender. It switches only after tun.staged.gate_hold
(default 64) reads in a row want the other mode, and before going
inline it waits until every batch in flight is on the wire, so a
switch never lets a later packet overtake an earlier one. Going to
fanout nothing is in flight. flow.tx.gate_switches counts the switches.
On the same hosts dsack_dups fell to 20-52 and reord_seen to 0-2,
while throughput matched or beat per-read mixing (many-to-one 34.9
instead of 30.7 Gbit/s).

The affinity test waited only for the previous batch's packets to be
captured before the next read, but the sender returns the slot after
its write, so the reader could still see it in flight and stage the
small read. Switching to inline now drains, which makes the inline
send deterministic there.

flow.go

@@ -23,10 +23,11 @@ import (
 // tun.staged.tx_slots and rx_slots override the counts. Small-MSS TX arenas can grow.
 //
 // Metrics are registered only for active pipelines. flow.{tx,rx}.batches counts reads handed to the ordered stage and
-// flow.{tx,rx}.inline those an idle reader handled itself (TX inline also counts oversized reads); their rates show
-// whether staging is used. flow.stage.<stage>.q<queue>.busy_ns locates the limiting stage: tx_read, tx_encrypt,
-// tx_send, rx_read, rx_decrypt or rx_write. Take rates; 1e9 ns/s fills a serial stage. These are elapsed times that
-// exclude read syscalls and downstream waits; descheduling inflates them and worker times add up across the pool.
+// flow.{tx,rx}.inline those the reader handled itself (TX inline also counts oversized reads); their rates show
+// whether staging is used. flow.tx.gate_switches counts transmit switches between inline and fanout mode.
+// flow.stage.<stage>.q<queue>.busy_ns locates the limiting stage: tx_read, tx_encrypt, tx_send, rx_read, rx_decrypt or
+// rx_write. Take rates; 1e9 ns/s fills a serial stage. These are elapsed times that exclude read syscalls and
+// downstream waits; descheduling inflates them and worker times add up across the pool.
 // Inline work counts at the reader; small RX batches sent to the writer include decryption in writer time.
 
 // stagedConfig is tun.staged. workers is per routine and direction; zero keeps processing inline.
@@ -34,12 +35,19 @@ type stagedConfig struct {
 	workers int
 	tx      bool
 	rx      bool
-	// inlineBelow is the TX segment or RX packet count below which an idle reader skips the worker handoff, 0 never
+	// inlineBelow is the TX segment or RX packet count below which a read may skip the worker handoff, 0 never: a TX
+	// read in inline mode, an RX read with nothing in flight
 	inlineBelow int
 	// txSlots and rxSlots are how many batches each routine can have in flight per direction
 	txSlots, rxSlots int
+	// gateHold is how many TX reads in a row must want the other mode before a routine switches between inline and
+	// fanout transmit
+	gateHold int
 }
 
+// stagedGateHold is the default tun.staged.gate_hold
+const stagedGateHold = 64
+
 const (
 	// stagedTxSlots is the default TX slot count, unless the workers need more
 	stagedTxSlots = 8
@@ -85,10 +93,12 @@ func parseStagedConfig(c *config.C, l *slog.Logger) stagedConfig {
 		l.Warn("tun.staged cannot encrypt in parallel with boringcrypto or fips140; transmit stays inline")
 		sc.tx = false
 	}
+	sc.gateHold = max(c.GetInt("tun.staged.gate_hold", stagedGateHold), 1)
 	sc.txSlots, sc.rxSlots = stagedSlots(sc.workers, c.GetInt("listen.batch", 64),
 		c.GetInt("tun.staged.tx_slots", 0), c.GetInt("tun.staged.rx_slots", 0))
 	l.Info("Staged packet processing enabled", "workers", sc.workers, "tx", sc.tx, "rx", sc.rx,
-		"inlineBelow", sc.inlineBelow, "txSlots", sc.txSlots, "rxSlots", sc.rxSlots)
+		"inlineBelow", sc.inlineBelow, "gateHold", sc.gateHold,
+		"txSlots", sc.txSlots, "rxSlots", sc.rxSlots)
 	return sc
 }