On darwin an unconnected sendto runs the address and MAC policy checks
and the EndpointSecurity send check for every datagram, and while
xprotectd throttles it each send takes about 185us, which caps a
tunnel's upload and, through its ACKs, its download. A connected socket
runs those checks once, at connect, and it is the only UDP send that
reports the interface queue's flow control, as ENOBUFS.
Only the current direct remote of an established tunnel is eligible.
udp.EstablishedPeerConn is an optional Conn interface, implemented on
darwin; establishedRemotes refcounts remotes over the main hostmap and
tells each writer on the first and last reference: handshake completion
adds one, deleting or retiring a hostinfo removes it, SetRemote (roaming,
SetRemoteIfPreferred, control, ssh) moves it. Pending and relayed
hostinfos hold none, and a rehandshake's two hostinfos share one.
An eligible peer that Nebula sends NEBULA_CONNSOCK_RUN (64) datagrams to
within NEBULA_CONNSOCK_WINDOW (1s) gets a connected socket: SO_REUSEPORT,
bound to the source address the listener's route to the peer picks on
the listener's port, carrying the listener's IP_BOUND_IF. The peer sees
the same address and port as from the listener, and the socket's reader
delivers through the listener's reader. The listener itself never sets
SO_REUSEPORT (listen routines still do), so no other uid can bind the
port; with a specific listen.host and no routines connected sockets stay
off, since sharing an exact address needs SO_REUSEPORT on both.
All routines listening on one address share one set of connected
sockets: at most NEBULA_CONNSOCKS (8), one per peer, with opens from a
bucket that refills one per NEBULA_CONNSOCK_OPEN_EVERY (1s). A socket
closes when its peer stops being eligible, after NEBULA_CONNSOCK_IDLE
(30s) without traffic, on Rebind (of every routine) and Close, and on an
error that means the connected path is gone. Closing never runs under
the hostmap's or the eligibility lock; the peer can't reopen until the
close finishes. A send that gets ENOBUFS or EAGAIN waits within one
10ms budget shared by every writer since the last send that went
through, then drops; nothing waits in the netpoller without a deadline.
When nothing is eligible or open, a send costs one atomic load before
the listener path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
newTunFromFd was a stub on darwin, so an embedder handed a utun by the OS had
no way to use it. A darwin network extension is given exactly that:
NEPacketTunnelProvider creates the device and applies addressing, mtu and
routes from its network settings, and a sandboxed extension cannot change them
afterwards. Adopting that fd means everything about moving packets is shared
with newTun and only the setup differs, so the tun tracks whether the host owns
it and leaves the system routing table and interface config alone when it does.
Co-authored-by: Nate Brown <nbrown.us@gmail.com>
The goroutines Main starts stop only when its context does, and only a caller
holding the Control can arrange that. Cancelling solely on error left a config
test, which returns no Control and no error, with the lighthouse query worker
and a hostname resolver per dns named static host running for the life of the
process. Every embedder that validates a config in process rather than by
exec'ing pays for that on each config load, dnclient and the apple clients
included.
Co-authored-by: Nate Brown <nbrown.us@gmail.com>
Using <https://go.dev/doc/go1.26#goroutineleak-profiles> + Claude, I was able to run nebula's unit tests and e2e tests with the leak detector enabled.
Added a TestMain that queries pprof to see if there are any reported goroutine leaks.
I'd love to get some form of this in CI whenever go 1.26 comes out, though I'd also like to prove this is properly useful past the just five detections it got here.
<details>
<summary>TestMain</summary>
```go
package nebula
import (
"fmt"
"os"
"runtime/pprof"
"strings"
"testing"
)
// TestMain runs after all tests and checks for goroutine leaks
func TestMain(m *testing.M) {
// Run all tests
exitCode := m.Run()
// Check for goroutine leaks after all tests complete
prof := pprof.Lookup("goroutineleak")
if prof != nil {
var sb strings.Builder
if err := prof.WriteTo(&sb, 2); err != nil {
fmt.Fprintf(os.Stderr, "Failed to write goroutineleak profile: %v\n", err)
os.Exit(1)
}
content := sb.String()
leakedCount := strings.Count(content, "(leaked)")
if leakedCount > 0 {
fmt.Fprintf(os.Stderr, "\n=== GOROUTINE LEAK DETECTED ===\n")
fmt.Fprintf(os.Stderr, "Found %d leaked goroutine(s) in package nebula\n\n", leakedCount)
goros := strings.Split(content, "\n\n")
for _, goro := range goros {
if strings.Contains(goro, "(leaked)") {
fmt.Fprintln(os.Stderr, goro)
fmt.Fprintln(os.Stderr)
}
}
os.Exit(1)
} else {
fmt.Println("✓ No goroutine leaks detected in package nebula")
}
}
os.Exit(exitCode)
}
```
</details>
Also had to install go1.26rc2 and update the makefile to use that go binary + set ex:
```makefile
test-goroutineleak:
GOEXPERIMENT=goroutineleakprofile go1.26rc2 test -v ./...
```