The tunnel's real bottleneck queue - the UDP receive buffer feeding the
decrypt loop - is invisible to every kernel AQM, so under overload it
regulates ECN-capable flows with tail-drop loss like it's 1993. Sample
SK_MEMINFO once per recvmmsg batch (tunnels.ecn_mark_threshold, fraction
of rcvbuf, 0=off) and treat depth beyond the threshold as an outer CE:
the existing RFC 6040 fold then CE-marks ECT inner packets and senders
back off without loss.
A partial success left the remaining entries' iovecs, sockaddrs, and
cmsgs fully intact, then threw them away and replanned the remainder
from bufs -- doubling the packing work exactly when the socket is
congested. Give sendFn a start offset so the drain resumes the same
prepared array at the first unsent entry, and skip a kernel-rejected
entry in place the same way. Only the GSO-disable path still replans,
since its entries change shape; it now rewinds precisely to the failed
run instead of the whole chunk, so entries already sent are never
duplicated.
New scripted tests pin the two paths that didn't exist before: a mid-
chunk rejected entry (drop it, resume the rest, start offsets advance)
and a mid-chunk EIO (GSO off, replay only the failed run, no dup of
already-sent packets).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
Behavior untouched; each marker records a known gap and the intended
fix so the next visit doesn't rediscover it: transient zero-sent
sendmmsg errors drop a whole run; the non-vnet Poll queue lacks the
post-wake drain loop; recvmmsg controllen resets touch every entry;
cached handshake packets flush one syscall each; the routines clamp in
activate() would blackhole surplus REUSEPORT sockets if it ever became
reachable; darwin WriteBatch burst-drops on EWOULDBLOCK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
- multi_coalesce/batch: the ordering contract now states what Flush
actually guarantees -- per-flow DATA order -- and names the two
shapes later data may legally overtake (pure ACKs by design, and
unparseable in-flow shapes as an accepted tradeoff).
- validVnetHdr claimed DATA_VALID makes the stack skip L4 checksum
verification; the tun write path ignores that bit entirely. What the
header buys is the absence of NEEDS_CSUM.
- tun_darwin Write said "only valid for single threaded use"; it is
concurrency-safe and concurrent callers exist.
- udp_coalesce eviction comment said "Seal it" but never sets sealed.
- recordCapability: note the gauges are process-global while the state
is per-socket (last writer wins).
- drop a stale tunReadBufSize reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
Every non-coalesceable in-flow packet evicted the flow's open slot, so
a bidirectional connection's inbound data run was broken by each peer
ACK interleaved into it, largely defeating coalescing on concurrent
upload+download. A bare acknowledgment (zero payload, nothing beyond
ACK|PSH|ECE) carries no ordering obligation toward the flow's data --
delivered late it is just a stale ACK the receiver ignores -- so it
can ride the lane as a passthrough without the evict, same as kernel
GRO, which doesn't flush held data on pure ACKs. SYN/FIN/RST/CWR keep
sealing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
A slot that stays single-segment is byte-identical to the packet it
was seeded from, but flushSlot re-emitted it via WriteGSO with a
seeded pseudo-sum, forcing the kernel to software-checksum up to
~1400B that arrived with a perfectly valid checksum. Keep the borrowed
seed packet on the slot (valid until Flush per the Commit contract)
and emit it through the plain DATA_VALID path when numSeg is still 1
at flush time. appendPayload and mergeSlots only touch hdrBuf once
numSeg >= 2, so the raw bytes are pristine whenever the fast path
fires. This is every non-coalesced TCP/UDP packet: request/response
flows, many-flow fan-in, and each run's leftover tail.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
sendInsideEncrypt ran connectionManager.Out for every segment -- up to
~45 extra atomic stores per TSO superpacket, all inside writeLock when
boring crypto serializes encryption. One mark in sendInsideMessage
covers the whole superpacket on both the direct and relay paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
A failed IPV6_RECVTCLASS probe disabled ECN RX entirely, taking down
working IPv4 (v4-mapped) delivery with it, while the opposite failure
already only degraded. Treat both directions the same: each family
degrades to Not-ECT independently, and only a full-family failure
turns the cmsg parsing off.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
decodeRead failures were silently swallowed; a kernel emitting an
unnegotiated GSO type would blackhole all tun traffic with nothing in
the logs. Debug-gated per the usual idiom so the happy path pays
nothing, which means plumbing the logger down through the offload
queueset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
gsoTypeFromProto returns GSO_NONE when the IP version nibble is
neither 4 nor 6, so a multi-fragment superpacket went out as one
silent jumbo GSO_NONE packet -- exactly the silent mis-emission the
geometry checks promise not to allow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
The return lived inside a log-level-gated else-if, so with debug
logging off, control fell out of both switches. Nothing follows the
switch today, making it a silent drop by luck; any future code added
after the switch would have run for oversized Test requests only when
debug logging was disabled. Make the drop a guard clause.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ugV2edVqoz3tBvq9J6yWp
35596c7 added the udp-l4-ecn-rejected test but only the ECN mask, so
UDP_L4|ECN validated as plain UDP_L4. Mirror virtio_net_hdr_to_skb and
refuse ECN on anything but TCPV4/TCPV6.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nothing previously asserted the offload path works end to end -- a
silent fallback to per-packet sends would pass every test and only show
up as a throughput regression. Send a batch through a real StdConn over
loopback and assert (a) the run left as a single sendmmsg entry (GSO
engaged, via a spy around the real syscall) and (b) the kernel carved
the superpacket back into the exact original datagrams at the receiver.
Fails rather than skips when the probe reports no GSO on a UDP_SEGMENT-
capable kernel, so CI (make test, ubuntu-latest) guards engagement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rewind (resume after the kernel accepts fewer entries than
submitted) was the hairiest untested logic in the write path; a bug
there silently duplicates or loses packets under backpressure. Give
batchWriter an injectable sendFn and drive WriteBatch through scripted
partial-acceptance sequences over a mixed GSO-run/plain batch, decoding
what "reached the wire" straight from the prepared iovecs rather than
the entryEnd bookkeeping under test. Also pins the zero-progress abort
and the EIO runtime GSO-disable replay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The correctness sweeps only exercised the public Checksum dispatcher, so
wherever it resolved to the gvisor fallback (non-AVX2 amd64, fallback
architectures) the suite compared gvisor against itself and the AVX2
assembly went untested -- silently green. Per-arch export_test.go files
now enumerate the hand-written implementations and every sweep runs
against the dispatcher plus each of them, skipping with an explicit
message when the running CPU can't execute one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pseudoSumIPv4/IPv6, foldOnceNoInvert, ipv4HdrChecksum (batch) and
foldComplement (tio/virtio) feed the virtio NEEDS_CSUM contract; a wrong
seed means every coalesced packet is silently dropped by the receiver
with nothing failing on our side. Check them against an independent
reference built from explicit RFC pseudo-header bytes -- deliberately
not the production checksum code -- including the carry/fold edge cases
and an end-to-end seed -> kernel-completion -> receiver-accepts
property.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The TX arena (128 x 9033B per routine) and GRO receive scratch
(listen.batch x 64KiB per socket) stay at their worst-case bounds by
design: the arena never grows past real demand and the GRO slots cannot
be smaller without truncating coalesced superpackets. Document the
listen.batch knob's memory implication so constrained hosts know what
to tune.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The length checks were fishy on four counts: an empty hdr/transportHdr
with real payload returned nil (silent drop with a success signal); the
HdrLen/GSOSize/CsumStart uint16 conversions could wrap unchecked;
nothing verified transportHdr covers csum_start+csum_offset, so the
kernel's NEEDS_CSUM write could land in payload bytes; and there was no
total-size bound even though every length field involved is 16-bit.
Malformed geometry is now a real error, and a single 65535 total-length
guard makes all the u16 conversions exact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PinThreadToCPU left the goroutine locked to its OS thread even when
sched_setaffinity failed. The lock only exists to make the affinity
stick; without it the kernel migrates the thread anyway, so a failed pin
kept a dedicated thread for zero benefit. Unwind on the error path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thread pinning trades scheduler freedom for TX-ring ordering; that's the
right trade on dedicated forwarders but not as a surprise default on
hosts sharing cores with other workloads. Make it opt-in and document
the default in the example config.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GSOSize came from len(pays[0]) while the iovec build skips empty
fragments, so a leading empty fragment emitted a TSO/USO header with
gso_size == 0 -- virtio_net_hdr_to_skb rejects that with EINVAL and the
whole superpacket is lost. Compute gso_size from the first non-empty
fragment and use the non-empty count to decide superpacket vs plain.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JackDoan