Commit Graph
22 Commits
Author SHA1 Message Date
Caleb JasikandClaude Opus 5.5 10e1924489 udp/darwin: batch sends with sendmsg_x
WriteBatch hands up to 64 datagrams to one sendmsg_x call instead of
one sendto each. The socket is unconnected, so xnu still runs every
message through sendit, per-send MAC checks included; the saving is the
syscall crossings.

xnu's per-message loop stops at the first message that fails. For
EWOULDBLOCK, ENOBUFS and EMSGSIZE it reports how many went out, or the
errno alone when none did; for any other error it returns only the
errno, with the messages before it already sent. So each call carries
one run of datagrams to a single destination, and nothing is resent
after the kernel may have taken it. A datagram that fails on its own
account (a full interface queue, an oversize packet) goes to sendto,
which delivers or reports it, so it costs only itself, as without
batching; a refused destination costs its run. ENOSYS/EOPNOTSUPP, or
EPERM/EINVAL on a datagram sendto then delivers, switch the socket
back to sendto for good.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 16:10:26 -05:00
Caleb JasikandClaude Opus 5.5 4d191f805f overlay/darwin: drain several tun packets per read wakeup
The darwin tun returned one packet per Read, and listenIn flushes its
UDP send batch after every Read, so each outbound packet went to the
underlay as its own batch. tunQueue.Read now keeps calling readv until
the utun would block, up to tunReadBatch packets, so a burst reaches the
send batch together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 12:44:00 -05:00
Caleb JasikandClaude Opus 5.5 61947ff1c2 overlay/darwin: batch tun writes with sendmsg_x
Passthrough.Flush hands its whole lane to a new tio.BatchWriter in one
WriteBatch call when the queue implements it, and NewSingleQueue
exposes WriteBatch when its source does. On darwin, tun.WriteBatch
writes up to 64 packets per sendmsg_x on the utun control socket,
which is a connected datagram socket, so xnu takes the batched
sendit_x path rather than one sendit per message. utun registers no
send_list handler, so the kernel still passes each packet to utun on
its own: this saves syscalls and fd locking, not per-packet kernel work.

Error handling follows what sendmsg_x reports:

- A short count with no error (a packet past the send buffer
  mid-call, or a later send failing with ENOBUFS): the remainder goes
  out in the next sendmsg_x.
- EMSGSIZE: only returned when the first packet is too large, before
  the kernel takes any, so that packet alone goes through Write, which
  reports it, and the rest are batched again.
- ENOSYS, EPERM, EOPNOTSUPP: sendmsg_x is private API, so WriteBatch
  logs once and falls back to writev per packet for good.
- Anything else: sendmsg_x returns the error without the count it
  took, so the call is reported rather than retried, which could
  deliver packets twice.

One loss mode has no userspace fix. sendit_x counts each packet as
sent once it is copied into an mbuf; if a later allocation fails, it
frees the copied prefix unsent, returns its length, and sendmsg_x
drops the ENOBUFS because the count is nonzero. That looks exactly
like a real short count, so under mbuf exhaustion WriteBatch silently
drops packets that writev would have delivered.

The iOS tun path (tun_ios.go) is unchanged.

Inside a macOS Network Extension, download goes from 640 to 832 Mbps
against the recvmsg_x change alone in three paired rounds, at 15% less
CPU per Mbps; the likely cause is that its utun charges more per write
than a raw one. Raw dnclient already runs download near line rate,
and there the CPU difference is within noise.

TestTunWriteBatch* round-trip through a connected AF_UNIX datagram
socketpair, which takes sendmsg_x the same way, over both the batched
and the fallback path: order, AF prefixes, skipped invalid packets,
and an oversize packet mid-call and first-in-call. Dropping Namelen
from msghdrX fails them. Mutation testing with gremlins leaves only
equivalent survivors in WriteBatch, plus which error is reported first
when several fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 10:04:03 -05:00
Caleb JasikandClaude Opus 5.5 a33a6356c6 udp/darwin: batch receives with recvmsg_x
ListenOut drains the socket with recvmsg_x, xnu's private batched
recvmsg (SYS_RECVMSG_X, bsd/sys/socket_private.h), up to 64 datagrams
per syscall, and calls flush once per batch instead of once per packet.
Each msghdr_x carries its own sockaddr, so this works on the shared
unconnected socket. The send side stays on sendto: xnu only batches
sendmsg_x through the stack on connected sockets, and on an unconnected
socket like this one it falls back to one sendit per message, with
msg_name documented as unsupported.

recvmsg_x is private, so ListenOut falls back to one
ReadFromUDPAddrPort per datagram, logging once, if the kernel or a
sandbox refuses it (ENOSYS, EPERM, EOPNOTSUPP) or if a batch comes back
in a shape intact msghdr_x entries can't produce: more datagrams than
headers, a length past MTU, or a sender that isn't a matching
AF_INET/AF_INET6 sockaddr. That batch is dropped rather than delivered. IPv6 sources now arrive without their
zone, matching the Linux recvmmsg path; WriteTo already drops it.

macOS raw dnclient to a Windows peer on the same gigabit LAN, iperf3
10 s, median of 3 alternating rounds against 7cfa47d:

  download, 1/4/8 streams  606/600/604 -> 869/873/863 Mbps
  download CPU             ~159% -> ~123%
  upload                   340 -> 323 Mbps
  UDP 500M loss            0.32% -> 0.31%

Download is now close to the 902 Mbps the LAN gives without a tunnel.

TestMsghdrXKernelRoundTrip and FuzzMsghdrXKernelRoundTrip pin msghdrX
against the kernel's struct msghdr_x, which the SDK doesn't ship: they
round-trip bursts through recvmsg_x on loopback and check every field
the kernel writes. recvmsg_x never reports MSG_TRUNC (soreceive_m_list
doesn't set it), so a datagram past MTU arrives cut to MTU with Flags
0, the same as ReadFromUDP. TestListenOutSingleFallback covers the
fallback loop, TestCheckMsghdrX the shape check, and the round trip
asserts the check accepts every batch the kernel returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 16:30:55 -05:00
Caleb JasikandNate Brown e20569ce51 Let the darwin tun adopt an fd the host already configured (#1890)
newTunFromFd was a stub on darwin, so an embedder handed a utun by the OS had
no way to use it. A darwin network extension is given exactly that:
NEPacketTunnelProvider creates the device and applies addressing, mtu and
routes from its network settings, and a sandboxed extension cannot change them
afterwards. Adopting that fd means everything about moving packets is shared
with newTun and only the setup differs, so the tun tracks whether the host owns
it and leaves the system routing table and interface config alone when it does.

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-09-22 11:54:30 -05:00
Caleb JasikandNate Brown 9651ffdb0d Cancel the context when Main hands back no Control (#1889)
The goroutines Main starts stop only when its context does, and only a caller
holding the Control can arrange that. Cancelling solely on error left a config
test, which returns no Control and no error, with the lighthouse query worker
and a hostname resolver per dns named static host running for the life of the
process. Every embedder that validates a config in process rather than by
exec'ing pays for that on each config load, dnclient and the apple clients
included.

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-09-22 11:52:28 -05:00
Caleb Jasik ec3304e3a9 Recompute the transport checksum on self-forwarded packets (#1862)
smoke-extra / freebsd-amd64 (push) Failing after 16s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 14s
smoke-extra / netbsd-amd64 (push) Failing after 16s
smoke-extra / openbsd-amd64 (push) Failing after 15s
smoke-extra / linux-386 (push) Failing after 16s
smoke / Run multi node smoke test (push) Failing after 1m34s
Build and test / Static checks (push) Successful in 43s
Build and test / Test linux (push) Failing after 1m29s
Build and test / Test linux-pkcs11 (push) Failing after 2m2s
Build and test / Test linux-boringcrypto (push) Failing after 2m46s
Build and test / Test linux-fips140 (push) Failing after 2m41s
Build and test / Cross-build linux-arm (push) Successful in 3m0s
Build and test / Cross-build linux-mips (push) Successful in 3m46s
Build and test / Cross-build linux-other (push) Successful in 3m8s
Build and test / Cross-build windows (push) Successful in 1m1s
Build and test / Cross-build freebsd (push) Successful in 1m33s
Build and test / Cross-build netbsd (push) Successful in 1m32s
Build and test / Cross-build openbsd (push) Successful in 1m33s
Build and test / Cross-build mobile (push) Successful in 3m16s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
2026-08-27 16:10:32 -05:00
Caleb Jasik 02d8bcac68 Remove lighthouse goroutine leaks in lighthouse_test.go (#1589)
gofmt / Run gofmt (push) Failing after 3s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 2s
Build and test / Build all and test on ubuntu-linux (push) Failing after 3s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
Using <https://go.dev/doc/go1.26#goroutineleak-profiles> + Claude, I was able to run nebula's unit tests and e2e tests with the leak detector enabled.

Added a TestMain that queries pprof to see if there are any reported goroutine leaks.
I'd love to get some form of this in CI whenever go 1.26 comes out, though I'd also like to prove this is properly useful past the just five detections it got here.

<details>
<summary>TestMain</summary>


```go
package nebula

import (
    "fmt"
    "os"
    "runtime/pprof"
    "strings"
    "testing"
)

// TestMain runs after all tests and checks for goroutine leaks
func TestMain(m *testing.M) {
    // Run all tests
    exitCode := m.Run()

    // Check for goroutine leaks after all tests complete
    prof := pprof.Lookup("goroutineleak")
    if prof != nil {
        var sb strings.Builder
        if err := prof.WriteTo(&sb, 2); err != nil {
            fmt.Fprintf(os.Stderr, "Failed to write goroutineleak profile: %v\n", err)
            os.Exit(1)
        }

        content := sb.String()
        leakedCount := strings.Count(content, "(leaked)")

        if leakedCount > 0 {
            fmt.Fprintf(os.Stderr, "\n=== GOROUTINE LEAK DETECTED ===\n")
            fmt.Fprintf(os.Stderr, "Found %d leaked goroutine(s) in package nebula\n\n", leakedCount)

            goros := strings.Split(content, "\n\n")
            for _, goro := range goros {
                if strings.Contains(goro, "(leaked)") {
                    fmt.Fprintln(os.Stderr, goro)
                    fmt.Fprintln(os.Stderr)
                }
            }
            os.Exit(1)
        } else {
            fmt.Println("✓ No goroutine leaks detected in package nebula")
        }
    }

    os.Exit(exitCode)
}
```

</details>

Also had to install go1.26rc2 and update the makefile to use that go binary + set ex:

```makefile
test-goroutineleak:
	GOEXPERIMENT=goroutineleakprofile go1.26rc2 test -v ./...
```
2026-01-27 23:44:43 -06:00
Caleb Jasik 4444ed166a Add certVersion field to logs when logging the cert name in handshakes (#1359)
gofmt / Run gofmt (push) Successful in 10s
smoke-extra / Run extra smoke tests (push) Failing after 19s
smoke / Run multi node smoke test (push) Failing after 1m24s
Build and test / Build all and test on ubuntu-linux (push) Failing after 19m24s
Build and test / Build and test on linux with boringcrypto (push) Failing after 4m13s
Build and test / Build and test on linux with pkcs11 (push) Failing after 3m27s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
2025-03-25 16:08:36 -05:00
Caleb Jasik 50473bd2a8 Update example config to listen on :: by default (#1351)
gofmt / Run gofmt (push) Successful in 10s
smoke-extra / Run extra smoke tests (push) Failing after 19s
smoke / Run multi node smoke test (push) Failing after 1m27s
Build and test / Build all and test on ubuntu-linux (push) Failing after 19m16s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2m41s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2m56s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
2025-03-12 22:53:16 -05:00
Caleb Jasik 088af8edb2 Enable running testifylint in CI (#1350)
gofmt / Run gofmt (push) Successful in 10s
smoke-extra / Run extra smoke tests (push) Failing after 18s
smoke / Run multi node smoke test (push) Failing after 1m28s
Build and test / Build all and test on ubuntu-linux (push) Failing after 19m44s
Build and test / Build and test on linux with boringcrypto (push) Failing after 3m1s
Build and test / Build and test on linux with pkcs11 (push) Failing after 3m6s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
2025-03-10 17:38:14 -05:00
Caleb Jasik 612637f529 Fix testifylint lint errors (#1321)
gofmt / Run gofmt (push) Successful in 11s
smoke-extra / Run extra smoke tests (push) Failing after 19s
smoke / Run multi node smoke test (push) Failing after 1m28s
Build and test / Build all and test on ubuntu-linux (push) Failing after 19m3s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2m44s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2m54s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
* Fix bool-compare

* Fix empty

* Fix encoded-compare

* Fix error-is-as

* Fix error-nil

* Fix expected-actual

* Fix len
2025-03-10 10:18:34 -04:00
Caleb Jasik f7540ad355 Remove commented out metadata.go (#1320)
gofmt / Run gofmt (push) Successful in 27s
smoke-extra / Run extra smoke tests (push) Failing after 20s
smoke / Run multi node smoke test (push) Failing after 1m26s
Build and test / Build all and test on ubuntu-linux (push) Failing after 18m30s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2m52s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2m35s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
2025-03-07 14:37:07 -06:00
Caleb Jasik 8109cf2170 Add puncuation to doc comment (#1164)
* Add puncuation to doc comment

* Fix list formatting inside `EncryptDanger` doc comment
2024-06-24 14:50:17 -04:00
Caleb Jasik 8e94eb974e Add suggested filenames for collected profiles in the ssh commands (#1109) 2024-04-29 15:20:46 -05:00
Caleb Jasik ed00f5d530 Remove unused config code (last edited 4yrs ago) (#938) 2023-07-31 15:59:20 -05:00
Caleb Jasik 5da79e2a4c Run make vet in CI (#693) 2023-03-13 15:35:12 -04:00
Caleb Jasik f0ac61c1f0 Add nebula.plist based on the homebrew nebula LaunchDaemon plist (#762) 2023-03-13 13:16:46 -05:00
Caleb Jasik 469ae78748 Add homebrew install method to readme (#630) 2023-02-13 14:42:58 -06:00
Caleb Jasik 12dbbd3dd3 Fix typos found by https://github.com/crate-ci/typos (#735) 2022-12-19 11:28:27 -06:00
Caleb Jasik 527f953c2c Remove x509 config loading code (#685) 2022-06-23 10:27:34 -05:00
Caleb Jasik 730a5c4a23 Update link to nebula docs (#655) 2022-03-18 11:15:16 -04:00