JackDoan and Claude Fable 5
7fe4fab167
tun: default pin CPUs avoid NIC IRQ cores
...
When tun.pin_threads is on and tun.cpu_affinity is unset, pick pin CPUs
from the allowed set that do not service any up physical NIC's MSI vectors
(read-only walk of /sys/class/net/*/device/msi_irqs and
/proc/irq/*/effective_affinity_list). The old allowed[i] default pinned
encrypt threads onto exactly the cores drivers affine their first RX queue
IRQs to; a flow whose RSS queue fired there had NAPI fighting encrypt for
the core (measured 8.4 vs 10.2 Gbps REV bimodality). Falls back to the old
spread, with a log, when there aren't enough IRQ-free CPUs - e.g. drivers
that allocate one queue per core until the admin narrows them (ethtool -L).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-07-14 15:20:04 -05:00
JackDoan
37b924945d
unslop some comments
2026-07-14 13:38:39 -05:00
JackDoan
5631346b07
batch: back SendBatch with Arena instead of a hand-rolled slab
...
SendBatch.Reserve duplicated Arena's grow-on-demand logic byte for byte.
Use an Arena for the slot backing so the borrow/grow/recycle semantics
live in one place.
2026-07-14 12:19:50 -05:00
JackDoan
97eb3c635a
batch: move shared-arena Reset ownership from lanes to their owner
2026-07-14 12:19:50 -05:00
JackDoan
05f7923860
re-align to master
2026-07-14 11:38:43 -05:00
JackDoan
49028cb755
some tests
2026-07-14 11:00:55 -05:00
JackDoan
adf71d1458
simplify making new Queues
2026-07-14 11:00:55 -05:00
JackDoan
cefda6524c
make service test less annoying
2026-07-14 11:00:55 -05:00
JackDoan
d0f14de739
checkpt
2026-07-14 11:00:55 -05:00
JackDoan
9e7646ee62
checkpt
2026-07-14 11:00:55 -05:00
JackDoan
6e6cfc89db
more ram -> more speed
2026-07-14 11:00:55 -05:00
JackDoan
3719f135e3
more fixes!
2026-07-14 11:00:55 -05:00
JackDoan
2724b4a96c
more fixes!
2026-07-14 11:00:55 -05:00
JackDoan
e386e290ab
lint
2026-07-14 11:00:55 -05:00
JackDoan
0a44376403
datapath: fix 12 correctness findings from tun/UDP offload review
...
Multi-disciplinary correctness review of the batched tun / GSO-GRO / sendmmsg
rework. Each fix has a regression test; the merged tree builds on
linux/darwin/openbsd/windows/freebsd/netbsd, vets clean, passes the unit and
e2e suites, and is -race clean.
Critical:
- C1 zero-length inner UDP datagram no longer panics the process (remote DoS):
the UDP coalescer routes payLen==0 to passthrough instead of seeding a GSO
slot, and WriteGSO skips empty payload iovecs as defense in depth.
- C2 segmenter no longer corrupts inner headers when gsoSize < headerLen: the
L3+L4 header is snapshotted once and each segment stamped from the copy,
replacing the destructive overlapping in-place slide (SegmentTCP + SegmentUDP).
High:
- H1 applyOuterECN updates the IPv4 header checksum (RFC 1624 incremental) when
folding outer CE into the inner ToS, so passthrough packets are no longer
dropped by the peer stack.
- H2 the GRO reject path caps the borrowed RX segment ([:n:n]) so a reject can
no longer overrun into the next coalesced segment's Nebula header. Note:
oversized ICMPv6 rejects that need >16B beyond the segment are now refused
rather than sent under GRO (safe; see TOFIX.md for the scratch-buffer follow-up).
- H3 WriteBatch falls back to per-packet WriteTo for a chunk when writeSockaddr
fails, so one bad-family destination costs only its own packet, not the batch.
- H4 UserDevice.Readers returns N distinct queue wrappers with private buffers
(sharing the pipes) so concurrent readers no longer race/overwrite borrowed
packet bytes.
- H5 Poll.Close / Offload.Close no longer null t.fd (matching master's
tunFile.Close), removing the data race with a concurrent readOne load.
Medium/Low:
- M1 the UDP GSO 127-segment gate moved from kernel >=5.5 to >=6.9 (the real
UDP_MAX_SEGMENTS 64->128 threshold), avoiding EINVAL + per-packet fallback on
5.5-6.8 kernels.
- M2 NewMultiQueueReader replays the offload mask newTun actually negotiated
instead of the TSO-only mask, so adding a queue no longer disables USO
device-wide; the advertised USO capability derives from the same mask.
- M3 the shutdown eventfd is closed in pollQueueSet.Close / offloadQueueSet.Close
(double-close guarded), fixing the per-lifecycle fd leak.
- M4 dual-stack ECN selects the cmsg by address family, not socket family: RX
parseRecvCmsg reads both IP_TOS and IPV6_TCLASS; TX writeEntryCmsg stamps
IP_TOS for v4/v4-mapped dests and IPV6_TCLASS for v6 (on-host verified).
- L1 newPoll no longer closes the fd on failure (matching newOffload), removing
the double-close on QueueSet.Add error.
2026-07-14 11:00:55 -05:00
JackDoan
9e61269935
make mobile happy
2026-07-14 11:00:55 -05:00
JackDoan
a0836aa819
correctly shutdown the pprofserver
2026-07-14 11:00:55 -05:00
JackDoan
cf2800f7bd
SendVia: don't emit a zero-length packet when prepareSendVia fails
2026-07-14 11:00:55 -05:00
JackDoan
fc89b9d14c
adapt Control lifecycle tests to the batched tio.Queue Device interface
2026-07-14 11:00:55 -05:00
JackDoan
3e5326e60a
udp setsockopt correctness fixes
2026-07-14 11:00:55 -05:00
JackDoan
840841a53c
use less ram pls
2026-07-14 11:00:55 -05:00
JackDoan
cfb4ab24c3
clean up a comment a bit
2026-07-14 11:00:55 -05:00
JackDoan
67f9cfad91
drop in a logger
2026-07-14 11:00:55 -05:00
JackDoan
1c78f5e500
go mod tidy
2026-07-14 11:00:55 -05:00
JackDoan
ad8ff2e45e
lint
2026-07-14 11:00:55 -05:00
JackDoan
c13a1ff4ec
fix
2026-07-14 11:00:55 -05:00
JackDoan
8b66ebaf72
faster
...
grr heap usage!
2026-07-14 11:00:55 -05:00
JackDoan
bbdaf5c3f3
no
2026-07-14 11:00:55 -05:00
JackDoan
a515aeaf39
use clear()
2026-07-14 11:00:55 -05:00
JackDoan
0f6f14eaf6
remove udp-level RX reorder buf
2026-07-14 11:00:55 -05:00
JackDoan
dafdd34af9
make relays take the fast path maybe
2026-07-14 11:00:55 -05:00
JackDoan
f6791130df
scoot pinning around
2026-07-14 11:00:55 -05:00
JackDoan
145a6267fa
scoot stuff around for e2e
2026-07-14 11:00:55 -05:00
JackDoan
7716f1da23
disable sort-on-RX, CPU pinning seems to work for now
2026-07-14 11:00:55 -05:00
JackDoan
beb1d7d89f
switch to ASM vector checksum
2026-07-14 11:00:55 -05:00
JackDoan
fa0d593f28
GSO/GRO offloads, with TCP+ECN and UDP support
2026-07-14 11:00:55 -05:00
JackDoan
ff48040c78
better and batched tun interface
2026-07-14 11:00:55 -05:00
Jack Doan
861d3aabd7
correct directionality of firewall.inbound_action and firewall.outbound_action ( #1798 )
2026-07-13 08:40:42 -05:00
Jack Doan
86733864fe
don't make new relay state on a just-discarded tunnel ( #1796 )
2026-07-10 12:02:17 -05:00
Jack Doan
19ad3bb904
correctly discard nil proto addresses ( #1785 )
2026-07-07 14:41:01 -05:00
Jack Doan
95d98b1f4b
firewall: move conntrack check after cert+IP verification ( #1779 )
2026-07-07 13:10:11 -05:00
Jack Doan
6afca0f461
correctly handle a test packet with a payload longer than the header ( #1778 )
smoke-extra / freebsd-amd64 (push) Failing after 16s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 14s
smoke-extra / netbsd-amd64 (push) Failing after 15s
smoke-extra / openbsd-amd64 (push) Failing after 14s
smoke-extra / linux-386 (push) Failing after 15s
smoke / Run multi node smoke test (push) Failing after 1m26s
Build and test / Static checks (push) Successful in 1m44s
Build and test / Test linux (push) Failing after 1m39s
Build and test / Test linux-boringcrypto (push) Failing after 3m3s
Build and test / Test linux-pkcs11 (push) Failing after 3m11s
Build and test / Cross-build linux-arm (push) Successful in 3m3s
Build and test / Cross-build linux-mips (push) Successful in 3m44s
Build and test / Cross-build linux-other (push) Successful in 3m8s
Build and test / Cross-build windows (push) Successful in 1m1s
Build and test / Cross-build freebsd (push) Successful in 1m38s
Build and test / Cross-build netbsd (push) Successful in 1m35s
Build and test / Cross-build openbsd (push) Successful in 1m34s
Build and test / Cross-build mobile (push) Successful in 3m17s
smoke-extra / Run windows smoke test (push) Has been cancelled
Build and test / Test macos (push) Has been cancelled
Build and test / Test windows (push) Has been cancelled
Build and test / CI status (push) Has been cancelled
2026-07-03 11:06:22 -05:00
Jack Doan
16b302c11d
Relay log fix ( #1765 )
...
* ensure CreateRelayRequest gets logged correctly
* ensure CreateRelayResponse gets logged correctly
2026-06-16 11:38:34 -05:00
Jack Doan
2e9117da5b
fix tunnels that could permanently escape connection-manager monitoring ( #1752 )
smoke-extra / freebsd-amd64 (push) Failing after 16s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 15s
smoke-extra / netbsd-amd64 (push) Failing after 14s
smoke-extra / openbsd-amd64 (push) Failing after 16s
smoke-extra / linux-386 (push) Failing after 17s
smoke / Run multi node smoke test (push) Failing after 1m25s
Build and test / Static checks (push) Successful in 1m42s
Build and test / Test linux (push) Failing after 2m17s
Build and test / Test linux-boringcrypto (push) Failing after 3m9s
Build and test / Test linux-pkcs11 (push) Failing after 2m54s
Build and test / Cross-build linux-arm (push) Successful in 3m3s
Build and test / Cross-build linux-mips (push) Successful in 3m44s
Build and test / Cross-build linux-other (push) Successful in 3m7s
Build and test / Cross-build windows (push) Successful in 59s
Build and test / Cross-build freebsd (push) Successful in 1m33s
Build and test / Cross-build netbsd (push) Successful in 1m34s
Build and test / Cross-build openbsd (push) Successful in 1m33s
Build and test / Cross-build mobile (push) Successful in 3m15s
smoke-extra / Run windows smoke test (push) Has been cancelled
Build and test / Test macos (push) Has been cancelled
Build and test / Test windows (push) Has been cancelled
Build and test / CI status (push) Has been cancelled
2026-06-10 11:03:23 -05:00
Jack Doan
a690c904ba
improve rejection of malformed handshakes ( #1756 )
smoke-extra / freebsd-amd64 (push) Failing after 18s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 17s
smoke-extra / netbsd-amd64 (push) Failing after 15s
smoke-extra / openbsd-amd64 (push) Failing after 16s
smoke-extra / linux-386 (push) Failing after 16s
smoke / Run multi node smoke test (push) Failing after 1m28s
Build and test / Static checks (push) Successful in 1m42s
Build and test / Test linux (push) Failing after 1m31s
Build and test / Test linux-boringcrypto (push) Failing after 2m54s
Build and test / Test linux-pkcs11 (push) Failing after 3m7s
Build and test / Cross-build linux-arm (push) Successful in 3m5s
Build and test / Cross-build linux-mips (push) Successful in 3m48s
Build and test / Cross-build linux-other (push) Successful in 3m12s
Build and test / Cross-build windows (push) Successful in 1m1s
Build and test / Cross-build freebsd (push) Successful in 1m34s
Build and test / Cross-build netbsd (push) Successful in 1m35s
Build and test / Cross-build openbsd (push) Successful in 1m35s
Build and test / Cross-build mobile (push) Successful in 3m22s
smoke-extra / Run windows smoke test (push) Has been cancelled
Build and test / Test macos (push) Has been cancelled
Build and test / Test windows (push) Has been cancelled
Build and test / CI status (push) Has been cancelled
2026-06-10 09:28:07 -05:00
Jack Doan
e028e6bf1a
disallow negative stats intervals ( #1754 )
2026-06-10 09:27:40 -05:00
Jack Doan
3db406b8ac
fix a race in RelayState.CopyRelayIps ( #1753 )
2026-06-10 09:27:15 -05:00
Jack Doan
eaad4896c1
udp_darwin: don't call the EncReader on a UDP error ( #1755 )
2026-06-10 09:26:31 -05:00
Jack Doan
e6032f81aa
correctly record window counters for relayed packets in a tunnel ( #1751 )
smoke-extra / freebsd-amd64 (push) Failing after 18s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 16s
smoke-extra / netbsd-amd64 (push) Failing after 15s
smoke-extra / openbsd-amd64 (push) Failing after 16s
smoke-extra / linux-386 (push) Failing after 16s
smoke / Run multi node smoke test (push) Failing after 1m27s
Build and test / Static checks (push) Successful in 1m43s
Build and test / Test linux (push) Failing after 1m42s
Build and test / Test linux-boringcrypto (push) Failing after 2m57s
Build and test / Test linux-pkcs11 (push) Failing after 3m3s
Build and test / Cross-build linux-arm (push) Successful in 3m2s
Build and test / Cross-build linux-mips (push) Successful in 3m46s
Build and test / Cross-build linux-other (push) Successful in 3m7s
Build and test / Cross-build windows (push) Successful in 1m2s
Build and test / Cross-build freebsd (push) Successful in 1m36s
Build and test / Cross-build netbsd (push) Successful in 1m34s
Build and test / Cross-build openbsd (push) Successful in 1m33s
Build and test / Cross-build mobile (push) Successful in 3m16s
smoke-extra / Run windows smoke test (push) Has been cancelled
Build and test / Test macos (push) Has been cancelled
Build and test / Test windows (push) Has been cancelled
Build and test / CI status (push) Has been cancelled
2026-06-09 16:18:59 -05:00
Jack Doan
a82a8dc547
don't panic on bad ed25519 key lengths ( #1601 )
...
gofmt / Run gofmt (push) Failing after 4s
smoke-extra / Run extra smoke tests (push) Failing after 3s
smoke / Run multi node smoke test (push) Failing after 3s
Build and test / Build all and test on ubuntu-linux (push) Failing after 3s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
* don't panic on bad ed25519 key lengths
* don't allow mismatched curves
* add test
2026-05-06 17:00:07 -05:00
Jack Doan
ff91c37529
switch Bits to a packed u64 ( #1705 )
2026-05-06 10:22:26 -05:00
Jack Doan
5f890dbc34
noise: only type-assert once ( #1691 )
gofmt / Run gofmt (push) Failing after 2s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 3s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 3s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
2026-04-24 13:12:42 -05:00
Jack Doan
e80b9830a3
Remove more os.Exit calls and give a more reliable wait for stop function (attempt 3) ( #1661 )
2026-04-20 16:08:26 -05:00
Jack Doan
b3194236aa
udp_linux: wrap socket operations with syscall.RawConn for clean teardown ( #1654 )
...
gofmt / Run gofmt (push) Failing after 3s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 3s
Build and test / Build all and test on ubuntu-linux (push) Failing after 3s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
remove runtime.LockOSThread() because it makes things worse now
remove the "custom" Write() method from tun_linux.go, the stdlib path via os.File performs better
We should change our guidance around number of routines, ~2 per thread (that you wish to use for Nebula) seems to be about right now
2026-04-14 18:25:24 -05:00
Jack Doan
91d1f4675a
properly handle closetunnel packets ( #1638 )
2026-03-25 11:59:37 -05:00
Jack Doan
51308b845b
connection-track ICMP traffic ( #1602 )
...
gofmt / Run gofmt (push) Failing after 2s
smoke-extra / Run extra smoke tests (push) Failing after 3s
smoke / Run multi node smoke test (push) Failing after 2s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
* connection-track ICMP and ICMPv6 traffic
* icmpv6 only has identifier on echo
2026-02-18 23:19:37 -06:00
Jack Doan
353ad1f271
firewall: icmp no longer requires a port spec ( #1609 )
2026-02-13 11:10:40 -06:00
Jack Doan
f573e8a266
Merge commit from fork
...
gofmt / Run gofmt (push) Failing after 3s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 3s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 3s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
Newly signed P256 based certificates will have their signature clamped to the low-s form.
Update CHANGELOG.md
2026-02-06 14:26:51 -05:00
Jack Doan
42bee7cf17
Report if Nebula start fails because of tun device name ( #1588 )
...
gofmt / Run gofmt (push) Failing after 2s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 2s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
* specifically report if nebula start fails because of tun device name
* close all routines when closing the tun
2026-01-28 10:03:36 -06:00
Jack Doan
3ec527e42c
cert.MarshalSigningPublicKeyToPEM should emit the 'ECDSA' variant of the banner ( #1552 )
...
gofmt / Run gofmt (push) Failing after 2s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 2s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
* cert.MarshalSigningPublicKeyToPEM should emit the 'ECDSA' variant of the banner
* oof owie ouch my tests
2025-12-10 10:39:36 -06:00
Jack Doan
6d7cf611c9
improve nebula-cert sign version auto-select ( #1535 )
2025-11-20 13:27:27 -06:00
Jack Doan
297767b2e3
warn user if they configure a firewall rule that will allow way more traffic than you might expect ( #1513 )
...
* warn user if they accidentally configure a firewall rule that will allow way more traffic than you might expect
* add groups-contains-any warning
2025-11-19 11:06:34 -06:00
Jack Doan
a89f95182c
Firewall types and cross-stack subnet stuff ( #1509 )
...
* firewall can distinguish if the host connecting has an overlapping network, is a VPN peer without an overlapping network, or is a unsafe network
* Cross stack subnet stuff (#1512 )
* experiment with not filtering out non-common addresses in hostinfo.networks
* allow handshakes without overlaps
* unsafe network test
* change HostInfo.buildNetworks argument to reference the cert
2025-11-12 13:40:20 -06:00
Jack Doan
0f305d5397
don't block startup on failure to configure SSH ( #1520 )
2025-11-05 10:41:56 -06:00
Jack Doan
01909f4715
try to make certificate addition/removal reloadable in some cases ( #1468 )
...
* try to make certificate addition/removal reloadable in some cases
* very spicy change to respond to handshakes with cert versions we cannot match with a cert that we can indeed match
* even spicier change to rehandshake if we detect our cert is lower-version than our peer, and we have a newer-version cert available
* make tryRehandshake easier to understand
2025-11-03 19:38:44 -06:00
Jack Doan
770147264d
fix make bench ( #1510 )
2025-10-21 11:32:34 -05:00
Jack Doan
8824eeaea2
helper functions to more correctly marshal curve 25519 public keys ( #1481 )
2025-10-02 13:56:41 -05:00
Jack Doan
f1e992f6dd
don't require a detailsVpnAddr in a HostUpdateNotification ( #1472 )
...
* don't require a detailsVpnAddr in a HostUpdateNotification
* don't send our own addr on HostUpdateNotification for v2
2025-09-29 13:43:12 -05:00
Jack Doan
1ea5f776d7
update to go 1.25, use the cool new ECDSA key marshalling functions ( #1483 )
...
* update to go 1.25, use the cool new ECDSA key marshalling functions
* bonk the runners
* actually bump go.mod
* bump golangci-lint
2025-09-29 13:02:25 -05:00
Jack Doan
5cccd39465
update RemoteList.vpnAddrs when we complete a handshake ( #1467 )
2025-09-10 09:44:25 -05:00
Jack Doan
8196c22b5a
store lighthouses as a slice ( #1473 )
...
* store lighthouses as a slice. If you have fewer than 16 lighthouses (and fewer than 16 vpnaddrs on a host, I guess), it's faster
2025-09-10 09:43:25 -05:00
Jack Doan
65cc253c19
prevent linux from assigning ipv6 link-local addresses ( #1476 )
2025-09-09 13:25:23 -05:00
Jack Doan
768325c9b4
cert-v2 chores ( #1466 )
2025-09-05 15:08:22 -05:00
Jack Doan
932e329164
Don't delete static host mappings for non-primary IPs ( #1464 )
...
* Don't delete a vpnaddr if it's part of a certificate that contains a vpnaddr that's in the static host map
* remove unused arg from ConnectionManager.shouldSwapPrimary()
2025-09-04 14:49:40 -05:00
Jack Doan
4bea299265
don't send recv errors for packets outside the connection window anymore ( #1463 )
...
* don't send recv errors for packets outside the connection window anymore
* Pull in fix from #1459 , add my opinion on maxRecvError
* remove recv_error counter entirely
2025-09-03 11:52:52 -05:00
Jack Doan
fd1906b16f
minor text fixes ( #1135 )
2024-05-03 20:43:40 -05:00
