Nothing previously asserted the offload path works end to end -- a
silent fallback to per-packet sends would pass every test and only show
up as a throughput regression. Send a batch through a real StdConn over
loopback and assert (a) the run left as a single sendmmsg entry (GSO
engaged, via a spy around the real syscall) and (b) the kernel carved
the superpacket back into the exact original datagrams at the receiver.
Fails rather than skips when the probe reports no GSO on a UDP_SEGMENT-
capable kernel, so CI (make test, ubuntu-latest) guards engagement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rewind (resume after the kernel accepts fewer entries than
submitted) was the hairiest untested logic in the write path; a bug
there silently duplicates or loses packets under backpressure. Give
batchWriter an injectable sendFn and drive WriteBatch through scripted
partial-acceptance sequences over a mixed GSO-run/plain batch, decoding
what "reached the wire" straight from the prepared iovecs rather than
the entryEnd bookkeeping under test. Also pins the zero-progress abort
and the EIO runtime GSO-disable replay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The correctness sweeps only exercised the public Checksum dispatcher, so
wherever it resolved to the gvisor fallback (non-AVX2 amd64, fallback
architectures) the suite compared gvisor against itself and the AVX2
assembly went untested -- silently green. Per-arch export_test.go files
now enumerate the hand-written implementations and every sweep runs
against the dispatcher plus each of them, skipping with an explicit
message when the running CPU can't execute one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pseudoSumIPv4/IPv6, foldOnceNoInvert, ipv4HdrChecksum (batch) and
foldComplement (tio/virtio) feed the virtio NEEDS_CSUM contract; a wrong
seed means every coalesced packet is silently dropped by the receiver
with nothing failing on our side. Check them against an independent
reference built from explicit RFC pseudo-header bytes -- deliberately
not the production checksum code -- including the carry/fold edge cases
and an end-to-end seed -> kernel-completion -> receiver-accepts
property.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TX arena (128 x 9033B per routine) and GRO receive scratch
(listen.batch x 64KiB per socket) stay at their worst-case bounds by
design: the arena never grows past real demand and the GRO slots cannot
be smaller without truncating coalesced superpackets. Document the
listen.batch knob's memory implication so constrained hosts know what
to tune.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The length checks were fishy on four counts: an empty hdr/transportHdr
with real payload returned nil (silent drop with a success signal); the
HdrLen/GSOSize/CsumStart uint16 conversions could wrap unchecked;
nothing verified transportHdr covers csum_start+csum_offset, so the
kernel's NEEDS_CSUM write could land in payload bytes; and there was no
total-size bound even though every length field involved is 16-bit.
Malformed geometry is now a real error, and a single 65535 total-length
guard makes all the u16 conversions exact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PinThreadToCPU left the goroutine locked to its OS thread even when
sched_setaffinity failed. The lock only exists to make the affinity
stick; without it the kernel migrates the thread anyway, so a failed pin
kept a dedicated thread for zero benefit. Unwind on the error path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thread pinning trades scheduler freedom for TX-ring ordering; that's the
right trade on dedicated forwarders but not as a surprise default on
hosts sharing cores with other workloads. Make it opt-in and document
the default in the example config.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GSOSize came from len(pays[0]) while the iovec build skips empty
fragments, so a leading empty fragment emitted a TSO/USO header with
gso_size == 0 -- virtio_net_hdr_to_skb rejects that with EINVAL and the
whole superpacket is lost. Compute gso_size from the first non-empty
fragment and use the non-empty count to decide superpacket vs plain.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Write took &buf[0] before calling writeWithScratch, so the len==0 guard
in the helper could never run -- a zero-length buffer panicked on the
index instead of returning. Hoist the guard above the indexing and fold
writeWithScratch into Write since it was the only caller and duplicated
the iovec setup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Warn("==== end of batch ====") delimiter (and the `logged` flag that
fed it) was left over from debugging the cross-slot gap logging.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TUN_F_TSO_ECN is negotiated, so once ECN feedback flows the kernel hands
us TSO superpackets typed TCPV4|GSO_ECN (CWR set). protoFromGSOType
treated the qualifier bit as an unknown type and the read path dropped
every such superpacket - a latent bug that only fires when a congested
hop CE-marks the flow, exactly when drops hurt most. The segmenter
already handles CWR (first segment only); just mask the bit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SendBatch.Reserve duplicated Arena's grow-on-demand logic byte for byte.
Use an Arena for the slot backing so the borrow/grow/recycle semantics
live in one place.
Multi-disciplinary correctness review of the batched tun / GSO-GRO / sendmmsg
rework. Each fix has a regression test; the merged tree builds on
linux/darwin/openbsd/windows/freebsd/netbsd, vets clean, passes the unit and
e2e suites, and is -race clean.
Critical:
- C1 zero-length inner UDP datagram no longer panics the process (remote DoS):
the UDP coalescer routes payLen==0 to passthrough instead of seeding a GSO
slot, and WriteGSO skips empty payload iovecs as defense in depth.
- C2 segmenter no longer corrupts inner headers when gsoSize < headerLen: the
L3+L4 header is snapshotted once and each segment stamped from the copy,
replacing the destructive overlapping in-place slide (SegmentTCP + SegmentUDP).
High:
- H1 applyOuterECN updates the IPv4 header checksum (RFC 1624 incremental) when
folding outer CE into the inner ToS, so passthrough packets are no longer
dropped by the peer stack.
- H2 the GRO reject path caps the borrowed RX segment ([:n:n]) so a reject can
no longer overrun into the next coalesced segment's Nebula header. Note:
oversized ICMPv6 rejects that need >16B beyond the segment are now refused
rather than sent under GRO (safe; see TOFIX.md for the scratch-buffer follow-up).
- H3 WriteBatch falls back to per-packet WriteTo for a chunk when writeSockaddr
fails, so one bad-family destination costs only its own packet, not the batch.
- H4 UserDevice.Readers returns N distinct queue wrappers with private buffers
(sharing the pipes) so concurrent readers no longer race/overwrite borrowed
packet bytes.
- H5 Poll.Close / Offload.Close no longer null t.fd (matching master's
tunFile.Close), removing the data race with a concurrent readOne load.
Medium/Low:
- M1 the UDP GSO 127-segment gate moved from kernel >=5.5 to >=6.9 (the real
UDP_MAX_SEGMENTS 64->128 threshold), avoiding EINVAL + per-packet fallback on
5.5-6.8 kernels.
- M2 NewMultiQueueReader replays the offload mask newTun actually negotiated
instead of the TSO-only mask, so adding a queue no longer disables USO
device-wide; the advertised USO capability derives from the same mask.
- M3 the shutdown eventfd is closed in pollQueueSet.Close / offloadQueueSet.Close
(double-close guarded), fixing the per-lifecycle fd leak.
- M4 dual-stack ECN selects the cmsg by address family, not socket family: RX
parseRecvCmsg reads both IP_TOS and IPV6_TCLASS; TX writeEntryCmsg stamps
IP_TOS for v4/v4-mapped dests and IPV6_TCLASS for v6 (on-host verified).
- L1 newPoll no longer closes the fd on failure (matching newOffload), removing
the double-close on QueueSet.Add error.
JackDoan