Compare commits

..
Author SHA1 Message Date
JackDoan 7aa6f6c7da firewall: Key conntrack by the peer's first vpn address 2026-10-16 16:35:06 -05:00
JackDoan 56a1a94a54 firewall: Let a reply come back through another of our gateways for an unsafe network 2026-10-16 16:32:46 -05:00
JackDoan a0045aef09 firewall: Keep conntrack entries per peer and recheck them when a peer reconnects 2026-10-16 13:41:25 -05:00
Jack Doan bbc49669b7 fix noisy log line (#1961) 2026-10-16 12:08:55 -05:00
Nate Brown 915495d5f4 Fail the reload on a startup only value that would fail a restart, and stop asking for tun.dev restarts that change nothing (#1953)
smoke-extra / freebsd-amd64 (push) Failing after 25s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 13s
smoke-extra / netbsd-amd64 (push) Failing after 11s
smoke-extra / openbsd-amd64 (push) Failing after 12s
smoke-extra / linux-386 (push) Failing after 14s
smoke / Run multi node smoke test (push) Failing after 1m43s
Build and test / Static checks (push) Successful in 2m25s
Build and test / Test linux (push) Failing after 1m28s
Build and test / Test linux-pkcs11 (push) Failing after 2m5s
Build and test / Test linux-boringcrypto (push) Failing after 2m56s
Build and test / Test linux-fips140 (push) Failing after 3m3s
Build and test / Cross-build linux-arm (push) Successful in 3m14s
Build and test / Cross-build linux-mips (push) Successful in 3m48s
Build and test / Cross-build linux-other (push) Successful in 3m13s
Build and test / Cross-build windows (push) Successful in 1m3s
Build and test / Cross-build freebsd (push) Successful in 1m34s
Build and test / Cross-build netbsd (push) Successful in 1m36s
Build and test / Cross-build openbsd (push) Successful in 1m37s
Build and test / Cross-build mobile (push) Successful in 3m33s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
smoke-extra / Run windows smoke test (push) Canceled after 0s
2026-10-15 23:34:09 -05:00
Nate Brown a203d044bb Fail the reload when a config entry is skipped with a warning (#1952) 2026-10-15 23:22:47 -05:00
Nate Brown 316bf87780 Don't restart sshd when only sshd.sandbox_dir changes (#1948) 2026-10-15 22:12:12 -05:00
Nate Brown 1a0ac25db0 Report how a config reload went: failures, restarts required, and the keys that need one (#1935) 2026-10-15 15:36:39 -05:00
John Maguire 42597dda7d service: Don't hold listener lock while handing off a connection (#1939) 2026-10-15 15:14:39 -04:00
John Maguire afbdc87ed9 Update gvisor and bump to go1.27 (#1940)
gvisor's arm64 race-build compare-and-swap (pkg/sync/race_arm64.s)
loaded the new value into the register meant for the old one (R1
instead of R2), so it never succeeded. Sleepers could not commit to
parking and spun: under -race on arm64 every idle service netstack
busied about 14 cores, and TestService's static host lookup of
localhost missed its 250ms timeout and stalled for the 30s static_map
cadence. Fixed in gvisor 1b2946c4fa.

Go 1.27.2 is now the minimum because gvisor's go branch requires it.
golangci-lint is bumped to v2.14, the first release that can read
go1.27.2 export data.
2026-10-14 21:30:59 -04:00
Nate Brown 94372f4f51 Don't delete routes we never installed on darwin and the BSDs (#1933) 2026-10-13 16:34:38 -05:00
John Maguire 42280f04ce darwin: Assign host address to the utun and route vpn explicitly (#1931)
Tailscale with an exit node treats every interface address prefix as an
attached LAN and either claims it or excludes it, and configd then rewrites
the matching route away from nebula's utun. Interfaces holding a single
address are skipped, so assign /32 and /128 and add the network route
ourselves for both families.
2026-10-13 13:50:37 -04:00
Nate Brown 11c1917387 Warn when an unsafe_route is already routed through another interface on darwin (#1925) 2026-10-12 16:57:14 -05:00
Nate Brown 5ba7aff4b3 ssh: Only restart sshd on reload when its config or host key changed (#1923) 2026-10-12 16:56:51 -05:00
Ryan Huber 8a2de9b1b5 examples: set socket buffers and raise tun tx_queue (#1928)
* examples: set socket buffers and raise tun tx_queue

* examples: use the default tcp max window for socket buffers

* examples: note buffers are linux only, clarify tx_queue default

* tun: default tx_queue to 5000
2026-10-11 00:30:07 -04:00
Nate Brown a8c52f5c13 overlay: Handle routes Windows already has, and parse quoted route numbers (#1924) 2026-10-08 18:52:53 -05:00
Jack Doan 13af8dd6f3 Cap configured MTUs at 8937 so relayed sends never outgrow their buffers (#1899) 2026-10-08 15:41:08 -05:00
Nate Brown 5bb0622192 stats: Follow stats.interval on reload, interface gauges ride the capture loop (#1919) 2026-10-08 10:00:48 -05:00
Nate Brown f3705cbb67 graphite: Time out a send, and abandon it on stop (#1920) 2026-10-07 18:01:09 -05:00
Nate Brown 94be454f4f lighthouse: Recompute relays for me when relay.am_relay changes (#1917) 2026-10-07 14:06:27 -05:00
Nate Brown 5f26c1a7e5 Add a simple rate limiter with a helper to use it in logging (#1915) 2026-10-07 10:53:37 -05:00
Nate Brown d47c8c71f1 lighthouse: Send an update when the relays we advertise change on reload (#1916) 2026-10-07 10:52:42 -05:00
Nate Brown f57539a163 overlay/linux: Keep unsafe routes when a reload only changes tun.mtu (#1918) 2026-10-07 10:43:11 -05:00
John Maguire 8d8ae0f744 Disable DNS registration on the Windows tunnel adapter (#1914)
On a domain-joined Windows host the DNS Client registers every adapter's
addresses under the host name in AD DNS when "Register this connection's
addresses in DNS" is on, which is the default for a new adapter. When the
Nebula adapter registers, LAN clients that are not on the overlay resolve
the host name to its overlay IP and RDP, SMB and PowerShell Remoting to it
fail intermittently, with nothing pointing at the tunnel as the cause.
Windows registers on address assignment, periodically, and on every
ipconfig /registerdns.

Right after the wintun adapter is created and before Activate assigns any
address, write RegistrationEnabled=0, DisableDynamicUpdate=1 and
MaxNumberOfAddressesToRegister=0 under both the Tcpip and Tcpip6
Parameters\Interfaces\{GUID} keys. The adapter checkbox alone has proven
insufficient in practice, as has disabling dynamic updates alone, so all
three are set, matching what Tailscale and NetBird do. A failure to write
the keys is logged as a warning and does not fail adapter creation.
2026-10-07 11:42:08 -04:00
Matt Richardson f9b0853476 Attach a pending handshake's RemoteList when the handshake starts (#1909)
StartHandshake returned a hostinfo with no RemoteList; handleOutbound attached
one lazily on its first tick. Anything that touched the pending host before
then, notably create-tunnel -address, dereferenced nil and took the daemon
down. StartHandshake now attaches the lighthouse's list as it builds the
hostinfo. To keep the surrounding locking sound, a lighthouse now resolves a
punch target before taking the querier's RemoteList lock, and closeTunnel
deletes the host and evicts its RemoteList under a single hostmap write lock so
a completing handshake cannot land in between. The lock order, now documented
on the handshake manager mutex, is hostmap, then handshake manager, then
lighthouse, then RemoteList; no RemoteList lock is ever held across a hostmap
lookup.
2026-10-07 11:22:50 -04:00
Nate Brown d5062dacf0 lighthouse: Never block in QueryServer, drop queries when the channel is full (#1910) 2026-09-30 15:26:34 -05:00
Jack Doan 7cfa47d569 tweaks to error handling in iputil.IPv6FindUpperProtocol (#1893) 2026-09-22 15:45:01 -05:00
Caleb JasikandNate Brown e20569ce51 Let the darwin tun adopt an fd the host already configured (#1890)
newTunFromFd was a stub on darwin, so an embedder handed a utun by the OS had
no way to use it. A darwin network extension is given exactly that:
NEPacketTunnelProvider creates the device and applies addressing, mtu and
routes from its network settings, and a sandboxed extension cannot change them
afterwards. Adopting that fd means everything about moving packets is shared
with newTun and only the setup differs, so the tun tracks whether the host owns
it and leaves the system routing table and interface config alone when it does.

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-09-22 11:54:30 -05:00
Caleb JasikandNate Brown 9651ffdb0d Cancel the context when Main hands back no Control (#1889)
The goroutines Main starts stop only when its context does, and only a caller
holding the Control can arrange that. Cancelling solely on error left a config
test, which returns no Control and no error, with the lighthouse query worker
and a hostname resolver per dns named static host running for the life of the
process. Every embedder that validates a config in process rather than by
exec'ing pays for that on each config load, dnclient and the apple clients
included.

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-09-22 11:52:28 -05:00
Jack Doan 3a45c01656 Advertise underlay addresses on Android (#1881) 2026-09-21 14:59:56 -05:00
Jack Doan 019cc46a7f smoke: Prove windows honors NlMtu with a 9000 byte ping (#1873) 2026-09-21 12:59:31 -05:00
Nate Brown cf3f315cd3 IPv6FindUpperProtocol: fail closed when the extension header walk runs out of budget (#1883) 2026-09-21 12:09:51 -05:00
Wade Simmons 3fc627e08d Add script to backport a merged PR to a release branch (#1851)
* Add script to backport a merged PR to a release branch

.github/backport.sh cherry-picks a merged PR's commit onto a release
branch and opens a backport PR mirroring the shape of #1842.

- Infers the target release from the PR's milestone (v1.12.0 -> release-1.11);
  an explicit target-version argument overrides it.
- Requires a clean working tree (untracked files allowed) before starting.
- Prompts with the exact push and gh pr create commands before running them.
- Sets the new PR's milestone to the highest open v<target>.* milestone.
- Removes the needs-backport label from the original PR once the backport opens.
- Supports --continue to resume after resolving cherry-pick conflicts.

* cleanup

* just fetch all of origin

We need to ensure we have MERGE_SHA available too
2026-09-14 16:14:43 -04:00
dependabot[bot] abff1a4555 Bump Vampire/setup-wsl from 3 to 7 (#1720)
Bumps [Vampire/setup-wsl](https://github.com/vampire/setup-wsl) from 3 to 7.
- [Release notes](https://github.com/vampire/setup-wsl/releases)
- [Commits](https://github.com/vampire/setup-wsl/compare/v3...v7)

---
updated-dependencies:
- dependency-name: Vampire/setup-wsl
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 10:11:27 -04:00
dependabot[bot] 227134a1d5 Bump golang.org/x/net in the golang-x-dependencies group (#1876)
Bumps the golang-x-dependencies group with 1 update: [golang.org/x/net](https://github.com/golang/net).


Updates `golang.org/x/net` from 0.58.0 to 0.59.0
- [Commits](https://github.com/golang/net/compare/v0.58.0...v0.59.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.59.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 10:01:25 -04:00
dependabot[bot] 232bacfdd0 Bump github.com/gaissmai/bart from 0.28.0 to 0.29.0 (#1829)
Bumps [github.com/gaissmai/bart](https://github.com/gaissmai/bart) from 0.28.0 to 0.29.0.
- [Release notes](https://github.com/gaissmai/bart/releases)
- [Commits](https://github.com/gaissmai/bart/compare/v0.28.0...v0.29.0)

---
updated-dependencies:
- dependency-name: github.com/gaissmai/bart
  dependency-version: 0.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:39:13 -04:00
dependabot[bot] 4e1ef9cd73 Bump the golang-x-dependencies group across 1 directory with 5 updates (#1875)
Bumps the golang-x-dependencies group with 2 updates in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto) and [golang.org/x/sync](https://github.com/golang/sync).


Updates `golang.org/x/crypto` from 0.54.0 to 0.57.0
- [Commits](https://github.com/golang/crypto/compare/v0.54.0...v0.57.0)

Updates `golang.org/x/net` from 0.57.0 to 0.58.0
- [Commits](https://github.com/golang/net/compare/v0.57.0...v0.58.0)

Updates `golang.org/x/sync` from 0.22.0 to 0.23.0
- [Commits](https://github.com/golang/sync/compare/v0.22.0...v0.23.0)

Updates `golang.org/x/sys` from 0.47.0 to 0.48.0
- [Commits](https://github.com/golang/sys/compare/v0.47.0...v0.48.0)

Updates `golang.org/x/term` from 0.45.0 to 0.46.0
- [Commits](https://github.com/golang/term/compare/v0.45.0...v0.46.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.57.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.58.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/sync
  dependency-version: 0.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/sys
  dependency-version: 0.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/term
  dependency-version: 0.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:37:37 -04:00
dependabot[bot] 614e0ea3aa Bump google.golang.org/protobuf (#1846)
Bumps the protobuf-dependencies group with 1 update in the / directory: google.golang.org/protobuf.


Updates `google.golang.org/protobuf` from 1.36.11 to 1.36.12

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-version: 1.36.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: protobuf-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:31:44 -04:00
dependabot[bot] 2999fc0b6a Bump github.com/stretchr/testify from 1.12.0 to 1.12.1 (#1859)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.12.0 to 1.12.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.12.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-version: 1.12.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:29:03 -04:00
dependabot[bot] f474b534e1 Bump github.com/miekg/dns from 1.1.72 to 1.1.73 (#1860)
Bumps [github.com/miekg/dns](https://github.com/miekg/dns) from 1.1.72 to 1.1.73.
- [Commits](https://github.com/miekg/dns/compare/v1.1.72...v1.1.73)

---
updated-dependencies:
- dependency-name: github.com/miekg/dns
  dependency-version: 1.1.73
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:28:03 -04:00
Wade Simmons 0dc8a28d1f apply go fix for Go 1.26 (#1867)
* apply `go fix` for Go 1.26

This applies the `go fix` recommendations for go1.26

* for all of ./...
2026-09-14 09:25:35 -04:00
Nate Brown 89178f45ba windows: Fix NLMTU for ipv6 (#1871)
smoke-extra / freebsd-amd64 (push) Failing after 16s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 15s
smoke-extra / netbsd-amd64 (push) Failing after 14s
smoke-extra / openbsd-amd64 (push) Failing after 17s
smoke-extra / linux-386 (push) Failing after 16s
smoke / Run multi node smoke test (push) Failing after 1m33s
Build and test / Static checks (push) Successful in 21s
Build and test / Test linux (push) Failing after 59s
Build and test / Test linux-pkcs11 (push) Failing after 1m49s
Build and test / Test linux-boringcrypto (push) Failing after 2m38s
Build and test / Test linux-fips140 (push) Failing after 2m52s
Build and test / Cross-build linux-arm (push) Successful in 2m54s
Build and test / Cross-build linux-mips (push) Successful in 3m35s
Build and test / Cross-build linux-other (push) Successful in 2m58s
Build and test / Cross-build windows (push) Successful in 59s
Build and test / Cross-build freebsd (push) Successful in 1m28s
Build and test / Cross-build netbsd (push) Successful in 1m28s
Build and test / Cross-build openbsd (push) Successful in 1m28s
Build and test / Cross-build mobile (push) Successful in 3m7s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
2026-09-08 18:30:16 -05:00
Wade Simmons 6a72e1c304 v1.11.1 Changelog (#1857)
smoke-extra / freebsd-amd64 (push) Failing after 18s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 17s
smoke-extra / netbsd-amd64 (push) Failing after 14s
smoke-extra / openbsd-amd64 (push) Failing after 16s
smoke-extra / linux-386 (push) Failing after 17s
smoke / Run multi node smoke test (push) Failing after 1m33s
Build and test / Static checks (push) Successful in 23s
Build and test / Test linux (push) Failing after 58s
Build and test / Test linux-pkcs11 (push) Failing after 1m50s
Build and test / Test linux-boringcrypto (push) Failing after 2m36s
Build and test / Test linux-fips140 (push) Failing after 3m0s
Build and test / Cross-build linux-arm (push) Successful in 2m50s
Build and test / Cross-build linux-mips (push) Successful in 3m34s
Build and test / Cross-build linux-other (push) Successful in 2m59s
Build and test / Cross-build windows (push) Successful in 1m0s
Build and test / Cross-build freebsd (push) Successful in 1m28s
Build and test / Cross-build netbsd (push) Successful in 1m28s
Build and test / Cross-build openbsd (push) Successful in 1m28s
Build and test / Cross-build mobile (push) Successful in 3m9s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
Copy the v1.11.1 CHANGELOG updates from release-v1.11
2026-09-08 13:50:27 -04:00
Matt Richardson e50f8128f4 Drop hostQueries to our overlay addresses (#1866)
smoke-extra / freebsd-amd64 (push) Failing after 24s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 23s
smoke-extra / netbsd-amd64 (push) Failing after 16s
smoke-extra / openbsd-amd64 (push) Failing after 11s
smoke-extra / linux-386 (push) Failing after 22s
smoke / Run multi node smoke test (push) Failing after 1m37s
Build and test / Static checks (push) Successful in 2m16s
Build and test / Test linux (push) Failing after 1m20s
Build and test / Test linux-pkcs11 (push) Failing after 1m50s
Build and test / Test linux-boringcrypto (push) Failing after 2m39s
Build and test / Test linux-fips140 (push) Failing after 2m51s
Build and test / Cross-build linux-arm (push) Successful in 2m55s
Build and test / Cross-build linux-mips (push) Successful in 3m41s
Build and test / Cross-build linux-other (push) Successful in 3m0s
Build and test / Cross-build windows (push) Successful in 1m1s
Build and test / Cross-build freebsd (push) Successful in 1m30s
Build and test / Cross-build netbsd (push) Successful in 1m29s
Build and test / Cross-build openbsd (push) Successful in 1m30s
Build and test / Cross-build mobile (push) Successful in 3m10s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
* Drop hostQueries to our overlay addresses

Don't bother responding to hostQuery's for our overlay addresses.
2026-09-04 11:26:08 -04:00
176 changed files with 5830 additions and 4669 deletions

No files matched your search

+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env bash
#
# Backport a merged PR to a release branch.
#
# Cherry-picks the merge commit of a merged PR onto release-1.11 and opens a
# backport PR, mirroring the shape of #1842.
#
# Usage: ./backport.sh <pr-number> [target-version] [--continue]
# pr-number The merged PR to backport
# target-version Release version to target (default: 1.11)
# --continue Resume after resolving cherry-pick conflicts: assumes the
# fixes are staged, runs `git cherry-pick --continue`, and
# proceeds to push and open the backport PR.
#
# Requires: gh (authenticated), git, jq.
set -euo pipefail
PR_NUMBER=${1:?usage: backport.sh <pr-number> [target-version] [--continue]}
shift
CONTINUE=0
TARGET_VERSION=""
for arg in "$@"; do
case "$arg" in
--continue) CONTINUE=1 ;;
*) TARGET_VERSION=$arg ;;
esac
done
for cmd in gh git jq; do
command -v "$cmd" >/dev/null || { echo "error: $cmd is required" >&2; exit 1; }
done
# Pull the source PR's metadata. Refuse to backport a PR that never merged.
pr_json=$(gh pr view "$PR_NUMBER" --json title,body,mergedAt,mergeCommit,milestone)
PR_TITLE=$(jq -r '.title' <<<"$pr_json")
PR_BODY=$(jq -r '.body' <<<"$pr_json")
MERGE_SHA=$(jq -r '.mergeCommit.oid // empty' <<<"$pr_json")
PR_MILESTONE=$(jq -r '.milestone.title // empty' <<<"$pr_json")
if [ "$(jq -r '.mergedAt // empty' <<<"$pr_json")" = "" ] || [ -z "$MERGE_SHA" ]; then
echo "error: PR #${PR_NUMBER} is not merged (no merge commit to cherry-pick)" >&2
exit 1
fi
# Default the target to one minor below the PR's milestone: a PR landing in
# v1.12.0 backports to release-1.11. An explicit target-version arg overrides.
if [ -z "$TARGET_VERSION" ]; then
if [[ $PR_MILESTONE =~ ^v?([0-9]+)\.([0-9]+) ]]; then
TARGET_VERSION="${BASH_REMATCH[1]}.$(( BASH_REMATCH[2] - 1 ))"
else
echo "error: PR #${PR_NUMBER} has no v<major>.<minor>.* milestone to infer the target from." >&2
echo "Pass the target version explicitly, e.g. $0 ${PR_NUMBER} 1.11" >&2
exit 1
fi
fi
TARGET_BRANCH="release-${TARGET_VERSION}"
# Slug from the PR title: lower-case, non-alphanumerics to dashes, trimmed.
slug=$(printf '%s' "$PR_TITLE" \
| tr '[:upper:]' '[:lower:]' \
| sed -E 's/[^a-z0-9]+/-/g; s/^-+//; s/-+$//' \
| cut -c1-50 \
| sed -E 's/-+$//')
BRANCH="backport-${TARGET_VERSION//./-}-${slug}"
if [ "$CONTINUE" -eq 1 ]; then
# Resuming: the branch already exists and a cherry-pick is mid-conflict.
current=$(git rev-parse --abbrev-ref HEAD)
if [ "$current" != "$BRANCH" ]; then
echo "error: --continue expects to be on ${BRANCH}, but HEAD is ${current}" >&2
exit 1
fi
echo "Resuming backport of #${PR_NUMBER} on ${BRANCH}"
# Conflicts assumed resolved and staged; core.editor=true keeps the commit message.
git -c core.editor=true cherry-pick --continue
else
# A fresh run switches branches and cherry-picks, so tracked changes must be
# clean. Untracked files are fine, and --continue is exempt (its resolved
# conflicts are meant to be staged).
if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
echo "error: working tree has uncommitted changes to tracked files; commit or stash them first." >&2
exit 1
fi
echo "Backporting #${PR_NUMBER} (${MERGE_SHA}) onto ${TARGET_BRANCH} as ${BRANCH}"
git fetch origin
git checkout -b "$BRANCH" "origin/${TARGET_BRANCH}"
# -m 1 handles a real merge commit; plain cherry-pick handles a squash merge.
if ! { git cherry-pick -x -m 1 "$MERGE_SHA" 2>/dev/null || git cherry-pick -x "$MERGE_SHA"; }; then
echo "error: ${MERGE_SHA} did not cherry-pick cleanly onto ${TARGET_BRANCH}." >&2
echo "Resolve the conflicts, 'git add' them, then re-run:" >&2
echo " $0 ${PR_NUMBER} ${TARGET_VERSION} --continue" >&2
exit 1
fi
fi
title="backport v${TARGET_VERSION}: ${PR_TITLE}"
body=$(printf 'Backport from #%s to release v%s\n\n---\n\n%s' \
"$PR_NUMBER" "$TARGET_VERSION" "$PR_BODY")
# Highest open milestone matching v<version>.* (e.g. v1.11.1), if any.
MILESTONE=$(gh api "repos/{owner}/{repo}/milestones?state=open" \
--jq ".[].title | select(startswith(\"v${TARGET_VERSION}.\"))" \
| sort -V | tail -n1)
milestone_args=()
if [ -n "$MILESTONE" ]; then
milestone_args=(--milestone "$MILESTONE")
else
echo "warning: no open milestone matching v${TARGET_VERSION}.* found; PR will have no milestone" >&2
fi
echo
echo "About to run:"
printf ' git push -u --force-with-lease origin %q\n' "$BRANCH"
printf ' gh pr create --base %q --head %q --title %q --body %q' \
"$TARGET_BRANCH" "$BRANCH" "$title" "$body"
[ -n "$MILESTONE" ] && printf ' --milestone %q' "$MILESTONE"
printf '\n'
echo
read -r -p "Open this pull request? [y/N] " reply
case "$reply" in
[yY] | [yY][eE][sS]) ;;
*) echo "Aborted. The cherry-pick is on local branch ${BRANCH}; push it manually if desired."; exit 0 ;;
esac
git push -u --force-with-lease origin "$BRANCH"
gh pr create \
--base "$TARGET_BRANCH" \
--head "$BRANCH" \
--title "$title" \
--body "$body" \
${milestone_args[@]+"${milestone_args[@]}"}
# The backport now exists, so drop the label that flagged this PR for one.
gh pr edit "$PR_NUMBER" --remove-label needs-backport
# Switch back to the original branch we were on before doing the backport
git checkout -
+3 -3
View File
@@ -14,7 +14,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build
@@ -40,7 +40,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build
@@ -83,7 +83,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
# GitHub holds ARNs, not credentials, and ARNs outlive a rotation
+4 -4
View File
@@ -34,7 +34,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: add hashicorp source
@@ -66,7 +66,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: add hashicorp source
@@ -92,13 +92,13 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
# WSL2 + Ubuntu so the smoke can run a real linux peer with its own
# netns. iputils-ping is needed for the in-WSL ping check. WSL1 has no
# real kernel and would lack /dev/net/tun, so we have to force WSL2.
- uses: Vampire/setup-wsl@v3
- uses: Vampire/setup-wsl@v7
with:
distribution: Ubuntu-24.04
additional-packages: iputils-ping iproute2
+2 -2
View File
@@ -22,7 +22,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Smoke Docker
@@ -51,7 +51,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: build
+48 -5
View File
@@ -51,15 +51,19 @@ wsl -d $Distro -- bash -c "rm -rf $WslDir && mkdir -p $WslDir" | Out-Null
$DevName = 'nebula-smoke'
$Ip1 = '192.168.241.1'
$Ip2 = '192.168.241.2'
# Dual stack on purpose: a v4-only overlay never exercises the v6 side of tun.mtu.
$Ip6_1 = 'fd42:4242:241::1'
$Ip6_2 = 'fd42:4242:241::2'
$Mtu = 1300
$Port = 4242
& $NebulaCert ca -name 'smoke-ca' -out-crt "$WorkDir\ca.crt" -out-key "$WorkDir\ca.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert ca failed (exit $LASTEXITCODE)" }
& $NebulaCert sign -name 'lighthouse' -networks "$Ip1/24" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\lighthouse.crt" -out-key "$WorkDir\lighthouse.key"
& $NebulaCert sign -name 'lighthouse' -networks "$Ip1/24,$Ip6_1/64" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\lighthouse.crt" -out-key "$WorkDir\lighthouse.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert sign lighthouse failed (exit $LASTEXITCODE)" }
& $NebulaCert sign -name 'peer' -networks "$Ip2/24" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\peer.crt" -out-key "$WorkDir\peer.key"
& $NebulaCert sign -name 'peer' -networks "$Ip2/24,$Ip6_2/64" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\peer.crt" -out-key "$WorkDir\peer.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert sign peer failed (exit $LASTEXITCODE)" }
# Windows lighthouse config.
@@ -82,7 +86,7 @@ tun:
drop_local_broadcast: false
drop_multicast: false
tx_queue: 500
mtu: 1300
mtu: $Mtu
network_category: private
logging:
level: info
@@ -126,7 +130,7 @@ tun:
drop_local_broadcast: false
drop_multicast: false
tx_queue: 500
mtu: 1300
mtu: $Mtu
logging:
level: info
format: text
@@ -169,7 +173,7 @@ Write-Host '=== WSL diagnostic ==='
wsl --version 2>&1 | Out-Host
wsl --list --verbose 2>&1 | Out-Host
wsl -d $Distro -u root -- uname -a | Out-Host
wsl -d $Distro -u root -- bash -c "modprobe tun 2>&1 || true; mkdir -p /dev/net; [ -c /dev/net/tun ] || mknod /dev/net/tun c 10 200; chmod 600 /dev/net/tun; ls -l /dev/net/tun"
wsl -d $Distro -u root -- bash -c "modprobe tun 2>&1 || true; mkdir -p /dev/net; [ -c /dev/net/tun ] || mknod /dev/net/tun c 10 200; chmod 600 /dev/net/tun; { echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6; echo 0 > /proc/sys/net/ipv6/conf/default/disable_ipv6; } 2>/dev/null || true; ls -l /dev/net/tun"
if ($LASTEXITCODE -ne 0) { throw "failed to prepare /dev/net/tun in WSL (TUN support missing?)" }
# Deliberately no New-NetFirewallRule calls here -- nebula's windows_bypass_wdf
@@ -214,6 +218,16 @@ try {
}
Write-Host "OK: $DevName NetworkCategory=Private"
# v6 silently kept the adapter default of 65535 while v4 was correct.
foreach ($family in @('IPv4', 'IPv6')) {
Wait-Until -TimeoutSec 30 -What "$DevName $family NlMtu=$Mtu" -Predicate {
if ($lhProc.HasExited) { throw "lighthouse exited (code $($lhProc.ExitCode)) before $family mtu was set" }
$rows = @(Get-NetIPInterface -InterfaceAlias $DevName -AddressFamily $family -ErrorAction SilentlyContinue)
$rows.Count -gt 0 -and -not ($rows | Where-Object { $_.NlMtu -ne $Mtu })
}
Write-Host "OK: $DevName $family NlMtu=$Mtu"
}
Wait-Until -TimeoutSec 30 -What "WSL nebula1 with $Ip2" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before tun was ready" }
$r = wsl -d $Distro -u root -- bash -c "ip -o addr show nebula1 2>/dev/null | grep -q 'inet $Ip2' && echo yes"
@@ -221,6 +235,13 @@ try {
}
Write-Host "OK: WSL nebula1 has $Ip2"
Wait-Until -TimeoutSec 30 -What "WSL nebula1 with $Ip6_2" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the v6 address was up" }
$r = wsl -d $Distro -u root -- bash -c "ip -o addr show nebula1 2>/dev/null | grep -q 'inet6 $Ip6_2' && echo yes"
("$r").Trim() -eq 'yes'
}
Write-Host "OK: WSL nebula1 has $Ip6_2"
Wait-Until -TimeoutSec 30 -What "ping from WSL peer to windows lighthouse ($Ip1)" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before ping succeeded" }
$r = wsl -d $Distro -u root -- bash -c "ping -c1 -W1 $Ip1 >/dev/null 2>&1 && echo OK"
@@ -234,6 +255,28 @@ try {
}
Write-Host "OK: windows lighthouse -> WSL peer"
# Otherwise the v6 networks only prove the interface exists, not that it forwards.
Wait-Until -TimeoutSec 30 -What "v6 ping from WSL peer to windows lighthouse ($Ip6_1)" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the v6 ping succeeded" }
$r = wsl -d $Distro -u root -- bash -c "ping -6 -c1 -W1 $Ip6_1 >/dev/null 2>&1 && echo OK"
("$r").Trim() -eq 'OK'
}
Write-Host "OK: WSL peer -> windows lighthouse over v6"
# The NlMtu check above proves the value was written, not that windows honors it. A payload this size only
# crosses the tunnel as fragments cut at NlMtu on the way out of the adapter. Left at the adapter default it
# reaches nebula whole, overflows the udp.MTU (9001) write buffer and is dropped, so the ping never answers.
# Anything smaller would ride out as one oversized datagram and survive on IP fragmentation of the underlay.
$BigPayload = 9000
foreach ($target in @(@{ Family = 'v4'; Ip = $Ip2 }, @{ Family = 'v6'; Ip = $Ip6_2 })) {
Wait-Until -TimeoutSec 30 -What "$($target.Family) ping with a $BigPayload byte payload from windows lighthouse to WSL peer ($($target.Ip))" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the large $($target.Family) ping succeeded" }
$null = & ping.exe -n 1 -w 1000 -l $BigPayload $target.Ip
$LASTEXITCODE -eq 0
}
Write-Host "OK: windows lighthouse -> WSL peer, $BigPayload byte $($target.Family) payload"
}
Write-Host ''
Write-Host 'All smoke checks passed.'
}
+4 -4
View File
@@ -22,7 +22,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Install goimports
@@ -42,7 +42,7 @@ jobs:
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.12
version: v2.14
test:
name: Test ${{ matrix.name }}
@@ -87,7 +87,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build
@@ -132,7 +132,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build ${{ matrix.name }}
+21 -1
View File
@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- `create-tunnel -address <ip:port>` panicked on a nil RemoteList and, over the ssh debug server, took nebula
down with it. A hostinfo returned by StartHandshake had no RemoteList until the handshake loop's first
pass attached one, and `create-tunnel -address` set the remote before that. The list is now attached when
the handshake starts, so a pending hostinfo never exists without one.
## [1.11.1] - 2026-08-21
See the [v1.11.1](https://github.com/slackhq/nebula/milestone/30?closed=1) milestone for a complete list of changes.
### Changed
- IPv6 packets whose next header is a protocol Nebula does not parse (SCTP, GRE, IP-in-IP, etc.) are now
@@ -15,11 +26,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
their true protocol, so only a `proto: any` rule allows them. If you carry one of these protocols over the
overlay, confirm a `proto: any` rule covers it before upgrading, it may have been passing only through this
bypass. (#1840)
- Drop the dependency on `github.com/cyberdelia/go-metrics-graphite`, which has been unmaintained for over ten
years, by inlining the small amount of code Nebula used. (#1832)
### Fixed
- The ICMPv6 type was read from the wrong byte when classifying IPv6 packets, so the echo identifier used
for conntrack was never picked up. (#1840)
- Enforce outbound message counter limits so a tunnel is rehandshaked before the counter can wrap, preventing
nonce reuse. This is unreachable in practice, but is enforced as a defense-in-depth measure. (#1841)
- Prevent `nebula-cert ca` from running out of memory on 32bit systems when generating encrypted private keys. (#1834)
- Tolerate `ErrDumpInterrupted` when listing tun addresses on Linux, so a transient interrupted netlink dump
no longer aborts startup. (#1835)
## [1.11.0] - 2026-07-23
@@ -884,7 +902,9 @@ created.)
- Initial public release.
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.10.3...HEAD
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.11.1...HEAD
[1.11.1]: https://github.com/slackhq/nebula/releases/tag/v1.11.1
[1.11.0]: https://github.com/slackhq/nebula/releases/tag/v1.11.0
[1.10.3]: https://github.com/slackhq/nebula/releases/tag/v1.10.3
[1.10.2]: https://github.com/slackhq/nebula/releases/tag/v1.10.2
[1.10.1]: https://github.com/slackhq/nebula/releases/tag/v1.10.1
+3
View File
@@ -242,11 +242,14 @@ test-cov-html:
go test -coverprofile=coverage.out
go tool cover -html=coverage.out
# The package builds only compile. The final line links an android binary so a linker-only failure
# cannot pass CI.
build-test-mobile:
GOARCH=amd64 GOOS=ios go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=ios go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=amd64 GOOS=android go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=android go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=android go build -o /dev/null ${NEBULA_CMD_PATH}
bench:
go test -bench=.
+2 -11
View File
@@ -79,13 +79,7 @@ func (b *Bits) clearRange(startPos, count uint64) uint64 {
// handle the potential partial word before pos becomes u64 aligned
word := pos >> 6
bit := pos & 63
take := uint64(64) - bit
if take > remaining {
take = remaining
}
if take > b.length-pos {
take = b.length - pos
}
take := min(min(uint64(64)-bit, remaining), b.length-pos)
var mask uint64
if take == 64 {
mask = math.MaxUint64
@@ -189,10 +183,7 @@ func (b *Bits) Update(l *slog.Logger, i uint64) bool {
func (b *Bits) updateSlow(l *slog.Logger, i uint64) bool {
// If i is a jump, adjust the window, record lost, update current, and return true
if i > b.current {
end := i
if end > b.current+b.length {
end = b.current + b.length
}
end := min(i, b.current+b.length)
count := end - b.current
startPos := (b.current + 1) & b.lengthMask
+1 -1
View File
@@ -120,7 +120,7 @@ func TestCertificate_SignP256_AlwaysNormalized(t *testing.T) {
pub := elliptic.Marshal(elliptic.P256(), priv.PublicKey.X, priv.PublicKey.Y)
rawPriv := priv.D.FillBytes(make([]byte, 32))
for i := 0; i < 1000; i++ {
for i := range 1000 {
if i&1 == 1 {
tbs.Version = Version1
} else {
+4 -4
View File
@@ -151,7 +151,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
var groups []string
if *cf.groups != "" {
for _, rg := range strings.Split(*cf.groups, ",") {
for rg := range strings.SplitSeq(*cf.groups, ",") {
g := strings.TrimSpace(rg)
if g != "" {
groups = append(groups, g)
@@ -171,7 +171,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
}
if *cf.networks != "" {
for _, rs := range strings.Split(*cf.networks, ",") {
for rs := range strings.SplitSeq(*cf.networks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -193,7 +193,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
}
if *cf.unsafeNetworks != "" {
for _, rs := range strings.Split(*cf.unsafeNetworks, ",") {
for rs := range strings.SplitSeq(*cf.unsafeNetworks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -221,7 +221,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
if !isP11 && *cf.encryption {
passphrase = []byte(os.Getenv("NEBULA_CA_PASSPHRASE"))
if len(passphrase) == 0 {
for i := 0; i < 5; i++ {
for range 5 {
errOut.Write([]byte("Enter passphrase: "))
passphrase, err = pr.ReadPassword()
-1
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package main
+4 -4
View File
@@ -146,7 +146,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
passphrase = []byte(os.Getenv("NEBULA_CA_PASSPHRASE"))
if len(passphrase) == 0 {
// ask for a passphrase until we get one
for i := 0; i < 5; i++ {
for range 5 {
errOut.Write([]byte("Enter passphrase: "))
passphrase, err = pr.ReadPassword()
@@ -203,7 +203,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
}
if *sf.networks != "" {
for _, rs := range strings.Split(*sf.networks, ",") {
for rs := range strings.SplitSeq(*sf.networks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -228,7 +228,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
}
if *sf.unsafeNetworks != "" {
for _, rs := range strings.Split(*sf.unsafeNetworks, ",") {
for rs := range strings.SplitSeq(*sf.unsafeNetworks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -247,7 +247,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
var groups []string
if *sf.groups != "" {
for _, rg := range strings.Split(*sf.groups, ",") {
for rg := range strings.SplitSeq(*sf.groups, ",") {
g := strings.TrimSpace(rg)
if g != "" {
groups = append(groups, g)
-1
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package main
-1
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package main
+2 -1
View File
@@ -85,9 +85,10 @@ func main() {
fmt.Printf("failed to apply logging config: %s", err)
os.Exit(1)
}
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := logging.ApplyConfig(l, c); err != nil {
l.Error("Failed to reconfigure logger on reload", "error", err)
r.Failed()
}
})
+2 -1
View File
@@ -34,9 +34,10 @@ func (p *program) Start(s service.Service) error {
if err := logging.ApplyConfig(l, c); err != nil {
return fmt.Errorf("failed to apply logging config: %s", err)
}
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := logging.ApplyConfig(l, c); err != nil {
l.Error("Failed to reconfigure logger on reload", "error", err)
r.Failed()
}
})
+2 -1
View File
@@ -71,9 +71,10 @@ func main() {
fmt.Printf("failed to apply logging config: %s", err)
os.Exit(1)
}
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := logging.ApplyConfig(l, c); err != nil {
l.Error("Failed to reconfigure logger on reload", "error", err)
r.Failed()
}
})
-1
View File
@@ -1,5 +1,4 @@
//go:build !linux
// +build !linux
package main
+38 -32
View File
@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"log/slog"
"maps"
"math"
"os"
"os/signal"
@@ -25,9 +26,12 @@ type C struct {
files []string
Settings map[string]any
oldSettings map[string]any
callbacks []func(*C)
callbacks []func(*C, *Reload)
l *slog.Logger
reloadLock sync.Mutex
// Set by the first reload that needs a restart and never cleared. Guarded by reloadLock
needsRestart bool
}
func NewC(l *slog.Logger) *C {
@@ -71,8 +75,9 @@ func (c *C) LoadString(raw string) error {
// RegisterReloadCallback stores a function to be called when a config reload is triggered. The functions registered
// here should decide if they need to make a change to the current process before making the change. HasChanged can be
// used to help decide if a change is necessary.
// These functions should return quickly or spawn their own go routine if they will take a while
func (c *C) RegisterReloadCallback(f func(*C)) {
// These functions should return quickly or spawn their own go routine if they will take a while.
// Use r to report anything that failed or needs a restart to apply.
func (c *C) RegisterReloadCallback(f func(c *C, r *Reload)) {
c.callbacks = append(c.callbacks, f)
}
@@ -143,54 +148,55 @@ func (c *C) CatchHUP(ctx context.Context) {
return
case <-ch:
c.l.Info("Caught HUP, reloading config")
c.ReloadConfig()
c.ReloadConfig().Log(c.l)
}
}
}()
}
func (c *C) ReloadConfig() {
// ReloadConfig reloads the config from the path provided to Load and runs every registered callback. A config that
// fails to load is logged here, like any other failure, and the result carries the error for the summary.
func (c *C) ReloadConfig() ReloadResult {
c.reloadLock.Lock()
defer c.reloadLock.Unlock()
c.oldSettings = make(map[string]any)
for k, v := range c.Settings {
c.oldSettings[k] = v
maps.Copy(c.oldSettings, c.Settings)
if err := c.Load(c.path); err != nil {
c.l.Error("Error occurred while reloading config", "config_path", c.path, "error", err)
return ReloadResult{Err: err, RestartRequired: c.needsRestart}
}
err := c.Load(c.path)
if err != nil {
c.l.Error("Error occurred while reloading config",
"config_path", c.path,
"error", err,
)
return
}
for _, v := range c.callbacks {
v(c)
}
return c.runCallbacks()
}
func (c *C) ReloadConfigString(raw string) error {
// ReloadConfigString is ReloadConfig for a config held in memory rather than on disk.
func (c *C) ReloadConfigString(raw string) ReloadResult {
c.reloadLock.Lock()
defer c.reloadLock.Unlock()
c.oldSettings = make(map[string]any)
for k, v := range c.Settings {
c.oldSettings[k] = v
maps.Copy(c.oldSettings, c.Settings)
if err := c.LoadString(raw); err != nil {
c.l.Error("Error occurred while reloading config", "error", err)
return ReloadResult{Err: err, RestartRequired: c.needsRestart}
}
err := c.LoadString(raw)
if err != nil {
return err
}
return c.runCallbacks()
}
func (c *C) runCallbacks() ReloadResult {
r := &Reload{c: c}
for _, v := range c.callbacks {
v(c)
v(c, r)
}
return nil
if r.restart {
c.needsRestart = true
}
return ReloadResult{Failed: r.failed, RestartRequired: c.needsRestart, RestartKeys: r.keys}
}
// GetString will get the string for k or return the default d if not found or invalid
@@ -216,7 +222,7 @@ func (c *C) GetStringSlice(k string, d []string) []string {
}
v := make([]string, len(rv))
for i := 0; i < len(v); i++ {
for i := range v {
v[i] = fmt.Sprintf("%v", rv[i])
}
@@ -310,8 +316,8 @@ func (c *C) IsSet(k string) bool {
}
func (c *C) get(k string, v any) any {
parts := strings.Split(k, ".")
for _, p := range parts {
parts := strings.SplitSeq(k, ".")
for p := range parts {
m, ok := v.(map[string]any)
if !ok {
return nil
@@ -394,7 +400,7 @@ func (c *C) parse() error {
var nm map[string]any
err = yaml.Unmarshal(b, &nm)
if err != nil {
return err
return fmt.Errorf("%s: %w", path, err)
}
// We need to use WithAppendSlice so that firewall rules in separate
+2 -2
View File
@@ -19,7 +19,7 @@ func TestConfig_Load(t *testing.T) {
// invalid yaml
c := NewC(l)
os.WriteFile(filepath.Join(dir, "01.yaml"), []byte(" invalid yaml"), 0644)
require.EqualError(t, c.Load(dir), "yaml: unmarshal errors:\n line 1: cannot unmarshal !!str `invalid...` into map[string]interface {}")
require.EqualError(t, c.Load(dir), filepath.Join(dir, "01.yaml")+": yaml: unmarshal errors:\n line 1: cannot unmarshal !!str `invalid...` into map[string]interface {}")
// simple multi config merge
c = NewC(l)
@@ -129,7 +129,7 @@ func TestConfig_ReloadConfig(t *testing.T) {
os.WriteFile(filepath.Join(dir, "01.yaml"), []byte("outer:\n inner: ho"), 0644)
c.RegisterReloadCallback(func(c *C) {
c.RegisterReloadCallback(func(c *C, _ *Reload) {
done <- true
})
+98
View File
@@ -0,0 +1,98 @@
package config
import (
"context"
"fmt"
"log/slog"
"slices"
)
// ReloadResult describes how a reload went. Problems are logged where they happen, Log writes a one line summary.
type ReloadResult struct {
// Err is set when the new config could not be read or parsed, no callbacks ran
Err error
// Failed is true when a callback could not apply part of the new config
Failed bool
// RestartRequired is true once any reload has needed a restart, and stays true until the process restarts
RestartRequired bool
// RestartKeys are the keys this reload changed that only apply on a restart
RestartKeys []string
}
// Summary returns a one line summary of the reload and the level to log it at.
func (r ReloadResult) Summary() (slog.Level, string) {
switch {
case r.Err != nil && r.RestartRequired:
return slog.LevelError, "Config reload failed, the new config did not load and a restart is still required"
case r.Err != nil:
return slog.LevelError, "Config reload failed, the new config did not load"
case r.Failed && r.RestartRequired:
return slog.LevelError, "Config reload completed with errors and requires a restart to fully apply, review the logs"
case r.Failed:
return slog.LevelError, "Config reload completed with errors, review the logs"
case r.RestartRequired:
return slog.LevelWarn, "Config reload was completed but requires a restart to fully apply"
default:
return slog.LevelInfo, "Config reload complete"
}
}
// String is the summary with Err and RestartKeys included, for output other than the log.
func (r ReloadResult) String() string {
_, msg := r.Summary()
if r.Err != nil {
msg = fmt.Sprintf("%s: %v", msg, r.Err)
}
if len(r.RestartKeys) > 0 {
msg = fmt.Sprintf("%s, keys %v", msg, r.RestartKeys)
}
return msg
}
// Log writes Summary to l.
func (r ReloadResult) Log(l *slog.Logger) {
level, msg := r.Summary()
if len(r.RestartKeys) > 0 {
l.Log(context.Background(), level, msg, "keys", r.RestartKeys)
return
}
l.Log(context.Background(), level, msg)
}
// Reload is passed to each reload callback to report how applying the new config went.
type Reload struct {
c *C
failed bool
restart bool
// Changed keys that only apply on a restart, from every callback
keys []string
}
// Failed marks the reload as failed, the caller logs why. Safe to call on a nil Reload, which is what the initial load
// passes.
func (r *Reload) Failed() {
if r != nil {
r.failed = true
}
}
// RestartRequired marks the reload as needing a restart, the caller logs why. Safe to call on a nil Reload.
func (r *Reload) RestartRequired() {
if r != nil {
r.restart = true
}
}
// RestartRequiredIfChanged marks the reload as needing a restart if any of keys changed, and adds them to RestartKeys.
// Use it for keys that are only read at startup. Safe to call on a nil Reload.
func (r *Reload) RestartRequiredIfChanged(keys ...string) {
if r == nil {
return
}
for _, k := range keys {
if r.c.HasChanged(k) && !slices.Contains(r.keys, k) {
r.keys = append(r.keys, k)
r.restart = true
}
}
}
+108
View File
@@ -0,0 +1,108 @@
package config
import (
"errors"
"log/slog"
"testing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestReloadConfigString(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1"))
fail := false
ran := 0
c.RegisterReloadCallback(func(c *C, r *Reload) {
ran++
if fail {
r.Failed()
}
})
assert.Equal(t, ReloadResult{}, c.ReloadConfigString("a: 2"))
fail = true
assert.Equal(t, ReloadResult{Failed: true}, c.ReloadConfigString("a: 3"))
assert.Equal(t, 2, ran)
// A config that fails to load runs no callbacks and returns the error
res := c.ReloadConfigString("a: [")
require.Error(t, res.Err)
assert.False(t, res.Failed)
assert.Equal(t, 2, ran)
assert.Equal(t, 3, c.GetInt("a", 0))
}
func TestReloadRestartRequiredLatches(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1\nb: 1"))
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequiredIfChanged("a", "b") })
// Nothing that needs a restart changed
assert.Equal(t, ReloadResult{}, c.ReloadConfigString("a: 1\nb: 1\nc: 1"))
assert.Equal(t, ReloadResult{RestartRequired: true, RestartKeys: []string{"a"}}, c.ReloadConfigString("a: 2\nb: 1\nc: 1"))
// The restart sticks through a reload with no changes, reverting the key and a config that fails to load. Only the
// reload that changed the key lists it
assert.Equal(t, ReloadResult{RestartRequired: true}, c.ReloadConfigString("a: 2\nb: 1\nc: 1"))
assert.Equal(t, ReloadResult{RestartRequired: true, RestartKeys: []string{"a"}}, c.ReloadConfigString("a: 1\nb: 1\nc: 1"))
res := c.ReloadConfigString("a: [")
require.Error(t, res.Err)
assert.True(t, res.RestartRequired)
}
// A key reported by more than one callback is listed once
func TestReloadRestartKeysDeduped(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1\nb: 1"))
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequiredIfChanged("a") })
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequiredIfChanged("a", "b") })
assert.Equal(t, []string{"a", "b"}, c.ReloadConfigString("a: 2\nb: 2").RestartKeys)
}
// A callback can require a restart without a key changing, a file the config points at may have
func TestReloadRestartRequiredWithoutAChange(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1"))
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequired() })
assert.Equal(t, ReloadResult{RestartRequired: true}, c.ReloadConfigString("a: 1"))
}
func TestReloadResultSummary(t *testing.T) {
cases := []struct {
res ReloadResult
level slog.Level
msg string
}{
{ReloadResult{}, slog.LevelInfo, "Config reload complete"},
{ReloadResult{RestartRequired: true}, slog.LevelWarn, "Config reload was completed but requires a restart to fully apply"},
{ReloadResult{Failed: true}, slog.LevelError, "Config reload completed with errors, review the logs"},
{ReloadResult{Failed: true, RestartRequired: true}, slog.LevelError, "Config reload completed with errors and requires a restart to fully apply, review the logs"},
{ReloadResult{Err: errors.New("bad yaml")}, slog.LevelError, "Config reload failed, the new config did not load"},
{ReloadResult{Err: errors.New("bad yaml"), RestartRequired: true}, slog.LevelError, "Config reload failed, the new config did not load and a restart is still required"},
}
for _, tc := range cases {
level, msg := tc.res.Summary()
assert.Equal(t, tc.level, level, tc.msg)
assert.Equal(t, tc.msg, msg)
}
res := ReloadResult{RestartRequired: true, RestartKeys: []string{"a", "b"}}
assert.Equal(t, "Config reload was completed but requires a restart to fully apply, keys [a b]", res.String())
// String includes the error
assert.Equal(t, "Config reload failed, the new config did not load: bad yaml", ReloadResult{Err: errors.New("bad yaml")}.String())
}
// The initial load passes a nil Reload
func TestReloadNilIsSafe(t *testing.T) {
var r *Reload
r.Failed()
r.RestartRequired()
r.RestartRequiredIfChanged("a")
}
+7 -25
View File
@@ -57,7 +57,8 @@ func newConnectionManagerFromConfig(l *slog.Logger, c *config.C, hm *HostMap, p
}
cm.reload(c, true)
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("timers.connection_alive_interval", "timers.pending_deletion_interval")
cm.reload(c, false)
})
@@ -174,10 +175,8 @@ func (cm *connectionManager) doTrafficCheck(localIndex uint32, p, nb, out []byte
switch decision {
case deleteTunnel:
if cm.hostMap.DeleteHostInfo(hostinfo) {
// Only clearing the lighthouse cache if this is the last hostinfo for this vpn ip in the hostmap
cm.intf.lightHouse.DeleteVpnAddrs(hostinfo.vpnAddrs)
}
// Local delete only, clearing the lighthouse cache if this was the last hostinfo for the vpn addr
cm.intf.closeTunnel(hostinfo)
case closeTunnel:
cm.intf.sendCloseTunnel(hostinfo)
@@ -197,19 +196,6 @@ func (cm *connectionManager) doTrafficCheck(localIndex uint32, p, nb, out []byte
}
cm.resetRelayTrafficCheck(hostinfo)
cm.maintainLanes(decision, hostinfo, now, nb, out)
}
// maintainLanes piggybacks multiport lane probing on the per-tunnel traffic
// tick. makeTrafficDecision returns a nil hostinfo for a tunnel that is idle in
// both directions, and that tunnel is skipped: probing it would cost a probe per
// lane per keepalive for a tunnel nothing is using. Its lanes are probed again
// on the first tick after traffic resumes.
func (cm *connectionManager) maintainLanes(decision trafficDecision, hostinfo *HostInfo, now time.Time, nb, out []byte) {
if hostinfo == nil || decision == deleteTunnel || decision == closeTunnel {
return
}
cm.intf.probeLanes(hostinfo, now, nb, out)
}
func (cm *connectionManager) resetRelayTrafficCheck(hostinfo *HostInfo) {
@@ -342,10 +328,7 @@ func (cm *connectionManager) makeTrafficDecision(localIndex uint32, now time.Tim
return closeTunnel, hostinfo, nil
}
// The highest counter across the base session and its lanes: the lanes carry
// the data, so the base counter alone would sit near zero while a lane runs
// its keys past the nonce ceiling.
if hostinfo.maxMessageCounter() >= RejectAfterMessages {
if hostinfo.ConnectionState != nil && hostinfo.ConnectionState.messageCounter.Load() >= RejectAfterMessages {
// Send path can't encrypt a CloseTunnel notify, so just delete locally; the peer recovers via recv_error.
hostinfo.logger(cm.l).Error("Dropping tunnel, message counter is exhausted")
return deleteTunnel, hostinfo, nil
@@ -453,7 +436,6 @@ func (cm *connectionManager) isInactive(hostinfo *HostInfo, now time.Time) (time
return 0, false
}
// Lane traffic is this hostinfo's traffic, so lastUsed already covers it.
inactiveDuration := now.Sub(hostinfo.lastUsed)
if inactiveDuration < cm.getInactivityTimeout() {
// It's not considered inactive
@@ -477,7 +459,7 @@ func (cm *connectionManager) shouldSwapPrimary(current *HostInfo) bool {
return false
}
if current.maxMessageCounter() >= RehandshakeAfterMessages {
if current.ConnectionState.messageCounter.Load() >= RehandshakeAfterMessages {
// This tunnel is being rolled for counter exhaustion, never swap back onto its spent key.
return false
}
@@ -581,7 +563,7 @@ func (cm *connectionManager) tryRehandshake(hostinfo *HostInfo) {
cm.intf.handshakeManager.StartHandshake(hostinfo.vpnAddrs[0], nil)
return
}
if hostinfo.maxMessageCounter() >= RehandshakeAfterMessages {
if hostinfo.ConnectionState.messageCounter.Load() >= RehandshakeAfterMessages {
cm.l.Info("Re-handshaking with remote",
"vpnAddrs", hostinfo.vpnAddrs,
"reason", "message counter rehandshake threshold reached",
+3
View File
@@ -7,6 +7,7 @@ import (
"testing"
"time"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/overlay/overlaytest"
@@ -21,6 +22,8 @@ func newTestLighthouse() *LightHouse {
l: test.NewLogger(),
addrMap: map[netip.Addr]*RemoteList{},
queryChan: make(chan netip.Addr, 10),
// Nothing drains queryChan here, so an 11th query is dropped.
queryDropped: metrics.NilCounter{},
}
lighthouses := []netip.Addr{}
staticList := map[netip.Addr]struct{}{}
-59
View File
@@ -1,12 +1,9 @@
package nebula
import (
"crypto/hkdf"
"crypto/sha256"
"encoding/json"
"fmt"
"log/slog"
"strconv"
"sync"
"sync/atomic"
@@ -77,53 +74,6 @@ func newConnectionStateFromResult(r *handshake.Result) (*ConnectionState, error)
return ci, nil
}
// newLaneConnectionState derives multiport lane s's session from the base
// tunnel's material. Each key is an HKDF expansion of the base tunnel's matching
// key, labelled with the lane index, so the pair stays matched with no extra
// negotiation: Noise leaves our send key equal to the peer's receive key, and
// expanding both with the same label preserves that.
//
// The lane gets its own counter and replay window starting from zero. No
// handshake messages were spent on it, so unlike the base session there is
// nothing to seed.
func newLaneConnectionState(m *laneMaterial, lane uint8) (*ConnectionState, error) {
if lane == 0 {
return nil, fmt.Errorf("lane 0 is the base session")
}
eKey, err := deriveLaneKey(m.eKey, lane)
if err != nil {
return nil, err
}
dKey, err := deriveLaneKey(m.dKey, lane)
if err != nil {
return nil, err
}
return &ConnectionState{
myCert: m.myCert,
initiator: m.initiator,
peerCert: m.peerCert,
eKey: noiseutil.NewCipherStateFromKey(eKey, m.cipher),
dKey: noiseutil.NewCipherStateFromKey(dKey, m.cipher),
window: NewBits(ReplayWindow),
epoch: sessionEpoch.Add(1),
}, nil
}
// deriveLaneKey expands a base tunnel key into the key for one lane.
func deriveLaneKey(base [32]byte, lane uint8) ([32]byte, error) {
var out [32]byte
// The base key is already unique to this tunnel and direction, so the lane
// index is the only thing that needs to vary; no salt is required.
k, err := hkdf.Key(sha256.New, base[:], nil, laneKeyInfo+" "+strconv.Itoa(int(lane)), len(out))
if err != nil {
return out, err
}
copy(out[:], k)
return out, nil
}
func (cs *ConnectionState) MarshalJSON() ([]byte, error) {
return json.Marshal(m{
"certificate": cs.peerCert,
@@ -168,15 +118,6 @@ func (cs *ConnectionState) Decrypt(l *slog.Logger, messageCounter uint64, packet
return out, nil
}
// noteSeen records a counter that some other session for the same keys already
// accepted, so a packet doesn't become replayable just because the session that
// decrypted it was thrown away. See laneSet.installSession, its only caller.
func (cs *ConnectionState) noteSeen(l *slog.Logger, messageCounter uint64) {
cs.decryptLock.Lock()
cs.window.Update(l, messageCounter)
cs.decryptLock.Unlock()
}
func (cs *ConnectionState) VerifyRelay(l *slog.Logger, messageCounter uint64, packet []byte, nb []byte) error {
cs.decryptLock.Lock()
result := cs.window.Check(l, messageCounter)
+120 -3
View File
@@ -2,6 +2,8 @@ package nebula
import (
"net/netip"
"strconv"
"sync"
"testing"
"time"
@@ -11,6 +13,7 @@ import (
ct "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/overlay"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
@@ -58,7 +61,6 @@ func runTestHandshake(t *testing.T) (initR, respR *handshake.Result) {
cert.Version2, initCreds, verifier,
func() (uint32, error) { return 1000, nil },
true, header.HandshakeIXPSK0,
nil,
)
require.NoError(t, err)
@@ -66,7 +68,6 @@ func runTestHandshake(t *testing.T) (initR, respR *handshake.Result) {
cert.Version2, respCreds, verifier,
func() (uint32, error) { return 2000, nil },
false, header.HandshakeIXPSK0,
nil,
)
require.NoError(t, err)
@@ -99,7 +100,7 @@ func TestConnectionState_NextMessageCounter(t *testing.T) {
assert.Equal(t, RejectAfterMessages, cs.messageCounter.Load())
// Continued send attempts stay refused and the counter never wraps
for i := 0; i < 10; i++ {
for range 10 {
_, ok = cs.NextMessageCounter()
assert.False(t, ok)
}
@@ -149,6 +150,122 @@ func TestSendNoMetricsCloseTunnelKeepsRebindEpoch(t *testing.T) {
assert.True(t, hostinfo.markOut(1), "a CloseTunnel send must not consume the rebind epoch")
}
// capturingConn is a udp.Conn that records every WriteTo.
type capturingConn struct {
udp.NoopConn
writes [][]byte
addrs []netip.AddrPort
}
func (c *capturingConn) WriteTo(b []byte, addr netip.AddrPort) error {
c.writes = append(c.writes, append([]byte(nil), b...))
c.addrs = append(c.addrs, addr)
return nil
}
// TestSendNoMetricsViaRelay sends through a relay from the mtu sized buffer the cached packet flush uses. Payloads up
// to overlay.MaxMTU go out intact, anything bigger is dropped rather than outgrowing the buffer, which once panicked.
func TestSendNoMetricsViaRelay(t *testing.T) {
tests := []struct {
n int
sent bool
}{
{0, true},
{1300, true},
{overlay.MaxMTU, true},
{overlay.MaxMTU + 1, false}, // too big for the relay's tag
{mtu - 47, false}, // too big for the inner tag, this panicked
{9000, false}, // a full packet at tun.mtu 9000
}
for _, tt := range tests {
t.Run(strconv.Itoa(tt.n), func(t *testing.T) {
// One tunnel to the target, carried end to end, and one to the relay that wraps it
targetInit, targetResp := runTestHandshake(t)
relayInit, relayResp := runTestHandshake(t)
ci, err := newConnectionStateFromResult(targetInit)
require.NoError(t, err)
relayCI, err := newConnectionStateFromResult(relayInit)
require.NoError(t, err)
targetAddr := netip.MustParseAddr("10.0.0.2")
relayAddr := netip.MustParseAddr("10.0.0.3")
relayRemote := netip.MustParseAddrPort("192.0.2.1:4242")
relay := &Relay{Type: TerminalType, State: Established, LocalIndex: 5, RemoteIndex: 6, PeerAddr: targetAddr}
relayHI := &HostInfo{
vpnAddrs: []netip.Addr{relayAddr},
ConnectionState: relayCI,
relayState: RelayState{
relayForByAddr: map[netip.Addr]*Relay{targetAddr: relay},
relayForByIdx: map[uint32]*Relay{relay.LocalIndex: relay},
},
}
relayHI.remote.Store(&relayRemote)
// No remote of its own, so the send has to go via the relay
hostinfo := &HostInfo{
vpnAddrs: []netip.Addr{targetAddr},
ConnectionState: ci,
remoteIndexId: 7,
relayState: RelayState{
relays: []netip.Addr{relayAddr},
relayForByAddr: map[netip.Addr]*Relay{},
relayForByIdx: map[uint32]*Relay{},
},
}
l := test.NewLogger()
hm := newHostMap(l)
hm.Hosts[relayAddr] = relayHI
conn := &capturingConn{}
f := &Interface{
l: l,
hostMap: hm,
messageMetrics: &MessageMetrics{txExhausted: metrics.NewCounter()},
writers: []udp.Conn{conn},
}
f.connectionManager = &connectionManager{intf: f, relayUsed: map[uint32]struct{}{}, relayUsedLock: &sync.RWMutex{}}
payload := make([]byte, tt.n)
for i := range payload {
payload[i] = byte(i)
}
counter := ci.messageCounter.Load()
f.sendNoMetrics(header.Message, 0, ci, hostinfo, netip.AddrPort{}, payload, make([]byte, 12), make([]byte, mtu), 0)
if !tt.sent {
assert.Empty(t, conn.writes)
assert.Equal(t, counter, ci.messageCounter.Load(), "a dropped packet must not spend a counter")
return
}
require.Len(t, conn.writes, 1)
assert.Equal(t, relayRemote, conn.addrs[0])
pkt := conn.writes[0]
// The relay authenticates the whole packet under its own header
outer := &header.H{}
require.NoError(t, outer.Parse(pkt))
assert.Equal(t, header.MessageRelay, outer.Subtype)
assert.Equal(t, relay.RemoteIndex, outer.RemoteIndex)
relayPeer, err := newConnectionStateFromResult(relayResp)
require.NoError(t, err)
require.NoError(t, relayPeer.VerifyRelay(l, outer.MessageCounter, pkt, make([]byte, 12)))
// Inside it, the target decrypts the end to end packet back to the payload
inner := pkt[header.Len : len(pkt)-relayPeer.dKey.Overhead()]
h := &header.H{}
require.NoError(t, h.Parse(inner))
assert.Equal(t, header.Message, h.Type)
assert.Equal(t, hostinfo.remoteIndexId, h.RemoteIndex)
targetPeer, err := newConnectionStateFromResult(targetResp)
require.NoError(t, err)
got, err := targetPeer.Decrypt(l, h.MessageCounter, inner, make([]byte, 12))
require.NoError(t, err)
assert.Equal(t, payload, got)
})
}
}
func TestNewConnectionStateFromResult(t *testing.T) {
initR, respR := runTestHandshake(t)
+2 -42
View File
@@ -67,16 +67,6 @@ type ControlHostInfo struct {
CurrentRemote netip.AddrPort `json:"currentRemote"`
CurrentRelaysToMe []netip.Addr `json:"currentRelaysToMe"`
CurrentRelaysThroughMe []netip.Addr `json:"currentRelaysThroughMe"`
Lanes []ControlLane `json:"lanes,omitempty"`
}
// ControlLane reports one multiport lane of a tunnel. Only lanes we may send on
// are listed; receive-only lanes have no state worth showing.
type ControlLane struct {
Index uint8 `json:"index"`
Up bool `json:"up"`
Remote netip.AddrPort `json:"remote,omitempty"`
MessageCounter uint64 `json:"messageCounter"`
}
// Start actually runs nebula, this is a nonblocking call.
@@ -212,15 +202,10 @@ func (c *Control) RebindUDPServer() {
return
}
// Every socket needs rebinding, not just the base: with multiport the sockets are spread over the lane ports, and
// even without it the surplus SO_REUSEPORT sockets stay pinned to the interface we came up on otherwise.
//
// A failure here means we are likely still pinned to the interface we came up on, so the rest of this is
// unlikely to help. Say so instead of silently carrying on as if we rebound.
for i, w := range c.f.writers {
if err := w.Rebind(); err != nil {
c.l.Error("Failed to rebind udp socket", "error", err, "writer", i)
}
if err := c.f.outside.Rebind(); err != nil {
c.l.Error("Failed to rebind udp socket", "error", err)
}
// Trigger a lighthouse update, useful for mobile clients that should have an update interval of 0
@@ -400,7 +385,6 @@ func copyHostInfo(h *HostInfo, preferredRanges []netip.Prefix) ControlHostInfo {
CurrentRelaysToMe: h.relayState.CopyRelayIps(),
CurrentRelaysThroughMe: h.relayState.CopyRelayForIps(),
CurrentRemote: h.GetRemote(),
Lanes: copyLanes(h),
}
for i, a := range h.vpnAddrs {
@@ -418,30 +402,6 @@ func copyHostInfo(h *HostInfo, preferredRanges []netip.Prefix) ControlHostInfo {
return chi
}
// copyLanes snapshots the sendable multiport lanes of a tunnel, or nil when it
// has none. txAddr is the lane's gate as well as its destination, so a nil load
// is exactly "this lane is down and its flows are riding the base tunnel".
func copyLanes(h *HostInfo) []ControlLane {
ls := h.lanes
if ls == nil || ls.txLanes < 2 {
return nil
}
lanes := make([]ControlLane, 0, ls.txLanes-1)
for s := 1; s < ls.txLanes; s++ {
l := ControlLane{Index: uint8(s)}
if addr := ls.txAddr[s].Load(); addr != nil {
l.Up = true
l.Remote = *addr
}
if cs := ls.sessions[s].Load(); cs != nil {
l.MessageCounter = cs.messageCounter.Load()
}
lanes = append(lanes, l)
}
return lanes
}
func listHostMapHosts(hl controlHostLister) []ControlHostInfo {
hosts := make([]ControlHostInfo, 0)
pr := hl.GetPreferredRanges()
+4 -4
View File
@@ -151,9 +151,9 @@ func (c *fakeConn) WriteTo(_ []byte, _ netip.AddrPort) error { return nil }
func (c *fakeConn) WriteBatch(bufs [][]byte, _ []netip.AddrPort) (int, error) {
return len(bufs), nil
}
func (c *fakeConn) ReloadConfig(_ *config.C) {}
func (c *fakeConn) SupportsMultipleReaders() bool { return true }
func (c *fakeConn) Close() error { c.closed = true; return nil }
func (c *fakeConn) ReloadConfig(_ *config.C, _ *config.Reload) {}
func (c *fakeConn) SupportsMultipleReaders() bool { return true }
func (c *fakeConn) Close() error { c.closed = true; return nil }
type multiqueueDevice struct {
*fakeDevice
@@ -247,7 +247,7 @@ func TestControl_ConcurrentStopAndStart(t *testing.T) {
c, _, _ := newReadyControl(t)
var wg sync.WaitGroup
for i := 0; i < 2; i++ {
for range 2 {
wg.Go(func() { c.Stop() })
}
wg.Go(func() { _ = c.Start() })
+1 -1
View File
@@ -105,7 +105,7 @@ func TestControl_GetHostInfoByVpnIp(t *testing.T) {
}
// Make sure we don't have any unexpected fields
assertFields(t, []string{"VpnAddrs", "LocalIndex", "RemoteIndex", "RemoteAddrs", "Cert", "MessageCounter", "CurrentRemote", "CurrentRelaysToMe", "CurrentRelaysThroughMe", "Lanes"}, thi)
assertFields(t, []string{"VpnAddrs", "LocalIndex", "RemoteIndex", "RemoteAddrs", "Cert", "MessageCounter", "CurrentRemote", "CurrentRelaysToMe", "CurrentRelaysThroughMe"}, thi)
assert.Equal(t, &expectedInfo, thi)
test.AssertDeepCopyEqual(t, &expectedInfo, thi)
-544
View File
@@ -1,544 +0,0 @@
# Multiport lanes
Status: experimental. Linux only. Off unless `multiport.ports` is set.
## The problem
A nebula tunnel is one UDP 4-tuple. Everything between the two hosts that makes
a decision per flow makes it once, for the whole tunnel:
- **ECMP / LAG** hashes the 4-tuple and picks one path. A tunnel gets one path's
worth of bandwidth no matter how many exist.
- **NIC RSS** hashes the 4-tuple to one receive queue, so one CPU takes every
interrupt for the tunnel and receive is capped by a single core.
- **Per-flow policers and shapers** see one flow and rate-limit it as one.
- **Cloud per-flow bandwidth caps** are the hard version of that. AWS EC2 meters
each 5-tuple separately and caps a single flow well below what the instance can
do in aggregate -- on the order of 5 Gbps for a single flow within a VPC (more
inside a cluster placement group, or with ENA Express; check the current EC2
network-limits docs for exact numbers) on instances whose aggregate allowance
is many times that. Other providers do the same. Nothing about the path is
saturated when this fires, and no amount of retuning changes it: the hypervisor
is metering *the flow*, so the only way to get more is to be more than one
flow.
Running `routines: N` does not help. Those N sockets share one port through
`SO_REUSEPORT`, and the group is keyed on the exact `(addr, port)` pair, so the
kernel's reuseport hash is the *only* thing spreading the work -- every hash
outside this host still sees a single flow.
Multiport gives one tunnel several underlay 4-tuples, so all of those per-flow
decisions get made several times, independently. A tunnel with N lanes is N
flows to everything counting flows: N ECMP hashes, N receive queues, N of the
cloud provider's per-flow buckets.
There is a second bottleneck, and it is inside this host rather than out on the
network: in FIPS 140 mode the AES-GCM implementation refuses to seal twice under
the same nonce, which forces every encrypt on a session to be serialized behind
one mutex. `routines: N` does not help there either. Because each lane is a
separate session, multiport splits that mutex as well -- see
[The FIPS 140 encrypt lock](#the-fips-140-encrypt-lock).
## What a lane is
A lane is **not** a second tunnel. It is an extra session on the same
`HostInfo`.
Noise leaves both sides with `A.eKey == B.dKey` and `A.dKey == B.eKey`. Expanding
both keys through HKDF-SHA256 with the same per-lane label preserves that
equality, so both sides land on a matched key pair having exchanged nothing:
```
lane s send key = HKDF(base eKey, info: "nebula multiport lane v1 <s>")
lane s recv key = HKDF(base dKey, info: "nebula multiport lane v1 <s>")
```
(`connection_state.go:deriveLaneKey`. The base key is already unique per tunnel
and per direction, so the lane index is the only thing that needs to vary and no
salt is required.)
Consequences worth stating plainly:
- A lane costs **no handshake** and has no half-established state.
- A lane dies exactly when its base tunnel dies. There is no independent
lifetime to reason about, no second teardown path.
- Each lane is a **full** session: its own message counter, its own replay
window, its own cipher states, its own encrypt lock. Flows on different paths
never contend for shared replay state, which is what makes reordering across
lanes harmless, and under FIPS 140 the separate encrypt locks are what let one
tunnel encrypt on more than one core.
- Rolling the base tunnel replaces every lane key, because lanes are derived
from it. `maxMessageCounter` therefore reports the max across the base and all
lane counters, so rehandshake and exhaustion thresholds see the real data
volume rather than the base session's small share.
Which lane a packet belongs to travels in the nebula header, in the low 8 bits
of what used to be `Reserved` (see `header/header.go`). It is part of the AEAD's
associated data, so a lane index cannot be altered in flight -- a packet
decrypts on the lane it claims or not at all.
**Lane 0 is the base tunnel itself**: `HostInfo.ConnectionState`, the base port,
the peer's real remote address. It is not a special case reserved for control
traffic; it carries its share of data flows like any other lane.
## Socket layout
`routines` is **per port**. Each of `multiport.ports` consecutive ports gets its
own full group of `routines` sockets sharing it through `SO_REUSEPORT`, so total
sockets = total routines = total tun queues = `routines * multiport.ports`.
```
routines: 2, multiport.ports: 3, listen.port: 4242
port 4242 port 4243 port 4244
(lane 0/base) (lane 1) (lane 2)
+-------------+ +-------------+ +-------------+
writers[] | 0 | 1 | | 2 | 3 | | 4 | 5 |
+-------------+ +-------------+ +-------------+
routine 0 1 0 1 0 1
```
Sockets are laid out **port-major**: `writers[s*routinesPerPort + r]` is the
r'th socket on port `listen.port+s`. So:
```go
laneSock(q, s) = s*routinesPerPort + q%routinesPerPort // inside.go
egressSock(q) = laneSock(q, 0) // base traffic
```
Two properties fall out of this, and the rest of the design depends on both:
1. **Every routine owns exactly one socket.** Routine `i`'s UDP reader
(`listenOut`) blocks in `recvmmsg` on socket `i` and owns its batcher; its tun
reader (`listenIn`) owns its `txQueue` and conntrack cache. All of it is
single-owner and lock-free. More sockets than routines would need epoll or
locks.
2. **Every routine has a sibling socket at the same group position on every
port.** So socket selection is a pure function of `(queue, lane)` with no
borrowing and no shared state, and each port's traffic spreads across its
whole group rather than funnelling into one socket.
The second property is why `routines` is per port rather than a total to divide
up. With one socket per port, each port would be served by a single core -- worst
of all the base port, which carries every handshake, every lighthouse and punch
packet, every peer without multiport, and every tunnel whose lanes are down.
## Negotiation
Multiport capability rides the existing handshake payload as two new protobuf
fields (`handshake/payload.go`): `InitiatorLanes` field 9 and `ResponderLanes`
field 10, each a `LaneDetails{PortCount, BasePort, TxLanes}`.
- `PortCount` / `BasePort` -- the contiguous port range the sender bound, so the
peer knows where to aim its lanes.
- `TxLanes` -- how many lanes the sender may send on, so the peer knows how many
lane sessions it must be prepared to receive on.
Both are `nil` when multiport is off, which keeps the encoded payload
**byte-identical** to a vanilla one. A peer that has never heard of lanes skips
unknown fields as protobuf requires, and gets a plain tunnel.
From the result, `newLaneSet` computes:
```go
sessions = min(max(myLanes, peerTxLanes), 256) // we must be able to RECEIVE all of theirs
txLanes = min(myLanes, peerPortCount, sessions) // we may only SEND on ports they bound
```
Sizing RX by the peer's count and TX by our own is what lets asymmetric hosts
work: a 4-port laptop talking to a 32-port server sends on 4 lanes, and keeps
room for 32 lane sessions so it can receive on anything the server advertised it
might send on. The server, bounded by the laptop's 4 ports, sends on 4.
### Port pairing
Lane `s` targets `peerBasePort + ((s + portOffset) % peerPortCount)`.
`portOffset` is a per-pair FNV hash of the sorted vpn-address pair. Without it,
every small peer would aim its few lanes at a big peer's first few ports and
concentrate that peer's receive work on a couple of sockets.
The rotation has to cancel, though, or the two directions of one flow would take
unrelated 4-tuples and neither side's traffic would arrive through the conntrack
or NAT entry the other's probe opened. So both sides hash the *same* sorted pair
and the higher-addressed side **negates** the result. When the port counts match,
the two rotations cancel exactly on ports: our lane `s` runs from our port
`base+s` to the peer's port `base+s+portOffset`, and the peer's lane
`s+portOffset` runs back the other way, so the two are each other's reverse
4-tuple. The lane *indices* don't cancel -- our lane `s` partners the peer's lane
`s+portOffset` (mod the port count) -- and `laneBias` makes up for that: it
rotates the flow hash by the same amount, so both directions of a flow pick
partner lanes.
That pairing only exists when our lane indices map one-to-one onto the peer's
ports, which is exactly the `txLanes == peerPortCount` test `newLaneSet` applies
before setting `laneBias`. If we send on 4 lanes and the peer bound 8 ports, four
of its ports have no lane of ours pointing at them, and no choice of rotation can
make each of our lanes the reverse of one of its lanes. So in that case
`laneBias` stays 0 and each side hashes the flow to a lane on its own. The flow
still works and is still spread; the two directions simply take two unrelated
4-tuples instead of one 4-tuple and its exact reverse, and each direction depends
on its own lane's probe having opened its own conntrack or NAT entry.
## Bringing a lane up
Receiving on a lane needs no permission: the keys are derivable the moment the
base handshake completes. **Sending** on one needs proof the new 4-tuple actually
works, because nothing else would notice a middlebox quietly dropping it. So:
```
lane down --probe (Test/LaneProbe on lane s, from port base+s to peer's lane port)-->
<--ack (Test/LaneProbeAck, on the BASE tunnel)--
lane up
```
- The probe is encrypted with **the lane's own session**, so an ack proves the
whole lane end to end: our source port reached the peer, its reply reached us,
and the keys we derived match the ones it derived.
- The ack rides the **base tunnel** on purpose. A probe proves the peer's lane
works in *its* send direction; answering on our own lane `s` would make the
result depend on a second path that can be broken independently.
- The ack echoes the header's lane, not the payload's, so a peer cannot get us to
vouch for a lane it did not probe.
- A generation byte in the probe is echoed in the ack, so a late ack cannot
promote a lane on the strength of a superseded probe.
`txAddr[s]` is a single `atomic.Pointer[netip.AddrPort]` that is *both* the TX
gate and the destination, so a data-plane routine that loads non-nil has
everything it needs in one atomic read and there is no window where one is set
and the other is not.
Probing is driven by the connection manager's per-tunnel traffic tick
(`maintainLanes` -> `probeLanes`), which skips a tunnel that is idle in both
directions -- a lane is only worth having while there is traffic to put on it.
Probes and their acks are not counted as tunnel traffic, so a lane's own
keepalive cannot keep an otherwise idle tunnel from being dropped by
`tunnels.drop_inactive`. Timers:
| timer | value |
|---|---|
| probe timeout | 2s (shorter than the default 5s tick on purpose) |
| keepalive | 30s |
| retry backoff | 10s, doubling to 60s |
| max failure count | 8 |
Traffic on a lane is not evidence the lane works -- that is the whole reason
lanes need probing -- so a lane that silently breaks is only caught by the
keepalive.
Every lane starts out **demanded**, so the first traffic tick probes all of them
at once. This matters more than it looks: see "flow pinning" below. A lane that
is down also raises demand from the TX path each time a flow wants it, so a
lane that keeps failing is retried only while something still wants it, and an
idle tunnel costs nothing.
A lane aims **only** at its own port. There is no fallback to the peer's base
port when the lane port doesn't answer: a lane sharing the base port's
destination would gain only a source port of its own while costing the peer the
receive spread that is the entire point. A lane that can't reach its port stays
down and its flows ride the base tunnel.
## Flow -> lane -> routine: keeping a flow consistent
This is the part that took the most iterations to get right, so it's worth
spelling out why it is shaped the way it is.
### The kernel's tun queue feedback loop
On Linux, a multiqueue tun device does not simply hash a flow to a queue. It
also *learns*: `tun_flow_update` records "this flow was last seen on queue *q*"
from the packets **we write in**, and `tun_automq_select_queue` prefers what it
learned over the hash for as long as the flow stays busy.
We write an inbound packet to the tun queue with the same index as the UDP
routine that received it (`batchers[rxc.q]`). So the queue a flow's *outbound*
packets arrive on is decided by the socket its *inbound* packets landed on, at
the far end, one RTT ago.
### Why the lane comes from the flow, not the routine
The obvious design -- routine `q` sends on lane `q` -- deadlocks against that
feedback loop:
1. A tunnel comes up. All lanes are down, so all traffic goes out lane 0.
2. The peer receives it all on socket 0 and writes it all to tun queue 0.
3. Both kernels now believe every flow belongs on queue 0.
4. Every flow is read by routine 0, so every flow picks lane 0.
5. Go to 2. The tunnel is pinned to lane 0 for as long as its flows stay busy.
So the lane comes from **the flow's own 5-tuple**, not from the routine index:
```go
s = (laneFlowHash(fwPacket) + laneBias) % txLanes // lanes.go:txLaneForFlow
```
This makes lane spread completely independent of how the kernel steers tun
queues. It also gives the properties you actually want from a flow's point of
view:
- **A flow stays on one lane for its life.** The hash is a pure function of the
5-tuple, so there is no per-packet lane hopping and therefore no reordering
introduced by multiport.
- **One lane's replay window sees one stable set of flows.**
- **Both directions of a flow pick partner lanes.** `laneFlowHash` orders the two
endpoints before hashing, so it returns the same value from either end, and
`laneBias` lines the two sides' choices up. The two directions are exact
reverse 4-tuples, which is what NAT and stateful firewalls need.
`newLaneSet` demanding every lane up front is the other half of this. Lanes have
to be up *before* the flows are: a flow that starts while the lanes are still
down gets its queue pinned by step 2 above and can stay there for its whole
life. Eager demand costs one probe per lane on any tunnel that has traffic, and
nothing at all on one that doesn't.
### The full path
```
outbound inbound (at the peer)
-------- ---------------------
inside flow
| kernel tun hash, or the queue it
| learned from our last write
v
tun queue q -> routine q
|
| s = (laneFlowHash(flow) + laneBias) lane s arrives on the peer's
| % txLanes port for lane s
v |
lane s session | SO_REUSEPORT hash of the
| | 4-tuple picks one socket
| writers[laneSock(q,s)] v
v routine q' (owner of that socket)
port base+s -------------------------> |
v
tun queue q' (teaches the kernel
flow -> q')
```
Note that RX is entirely socket-agnostic: the lane comes from the header byte,
not from the port the packet arrived on, and roaming is skipped for `lane != 0`
(a lane's source address is a per-lane 4-tuple, not the tunnel's remote -- letting
it roam the hostinfo would point every non-lane packet at a lane port). So a
lane packet may legitimately arrive on any socket, which is what makes the
reuseport spread within a port safe.
### Ordering and locking
Several routines can write to one socket, since the routines whose lane
arithmetic lands on the same index share it. Linux's `batchWriter` serializes
`sendmmsg` with a mutex. Per-flow wire order still holds regardless: a flow is
hashed onto one lane and read by one routine, so nothing else is writing that
flow.
Each routine holds one `txQueue` with one shared arena (~1.16 MB) and builds a
`SendBatch` per lane lazily, on the first packet that picks it, so a routine that
never sends on a lane never pays for one.
### The FIPS 140 encrypt lock
In FIPS 140 mode -- a `boringcrypto` build, or `GODEBUG=fips140=on` -- nebula
uses `noiseutil.CipherAESGCMFIPS140` instead of the plain AES-GCM cipher. That
cipher is the TLS 1.3 GCM (`GCMWithXORCounterNonce`), which **panics** if it is
asked to seal with a counter that is not strictly greater than the last one. That
check is the point: it is the nonce-reuse protection FIPS 140 requires, and
`noiseutil`'s startup self-test refuses to run if the check has gone missing.
The check means encrypts on one session cannot overlap, so
`noiseutil.EncryptLockNeeded` is true and every send path takes that session's
`ConnectionState.writeLock`:
```go
if noiseutil.EncryptLockNeeded {
ci.writeLock.Lock()
}
c := ci.messageCounter.Add(1)
out = header.EncodeLane(scratch, ..., c, lane)
out, encErr := ci.eKey.EncryptDanger(out, out, seg, c, nb)
if noiseutil.EncryptLockNeeded {
ci.writeLock.Unlock()
}
```
The lock has to cover the seal and not just the counter increment. Reserving
counters atomically is easy; the requirement is that the seals *arrive at the
AEAD in counter order*, and only holding the lock across both gives that.
So under FIPS 140 the encrypt cost of a tunnel is pinned to one core. `routines:
N` gives N readers, but every one of them that has a packet for the same peer
queues on the same mutex, and with TSO/USO the lock is taken and released once
per segment -- up to ~45 times for a single superpacket
(`sendInsideEncrypt`). Receive is not affected: `extractFIPSAEAD` deliberately
pulls the inner FIPS AEAD out of the `crypto/tls` wrapper because that inner
implementation is safe to `Open` concurrently, and `decryptLock` is only ever
held around the replay-window check and update.
Lanes split the lock because a lane is a separate `ConnectionState`: its own
`writeLock`, its own message counter, its own nonce sequence. A peer we send to
on `txLanes` lanes has `txLanes` independent encrypt locks, so encrypt for that
one tunnel can run on that many cores at once.
The flow hash is what makes this work rather than merely legal. A flow maps to
one lane, so all of a flow's packets take one lock in counter order -- exactly
what the FIPS AEAD demands -- while different flows to the same peer land on
different locks. Contention is divided, not eliminated: any routine may send on
any lane, so two routines whose flows hash to the same lane still serialize
against each other.
Two caveats. This is only a FIPS 140 benefit -- in a normal build
`EncryptLockNeeded` is false, encrypt is lock-free, and lanes buy path and queue
spread only. And it is not unilateral: `txLanes` is bounded by the peer's port
count, so a peer that binds one port leaves us back on one encrypt lock however
many ports we bound ourselves.
## Falling back to the old behavior
Multiport degrades rather than failing, at every level. This is deliberate:
managed deployments can't be hard-errored on config they don't control.
### Whole-node: back to one port
Any of these turns multiport off, logs why, and leaves a node that binds one
port with `routines` `SO_REUSEPORT` sockets -- bit-for-bit the pre-multiport
configuration:
| condition | log |
|---|---|
| `multiport.enabled: false` | - |
| `multiport.ports` unset, 0, or 1 | `multiport disabled: set multiport.ports > 1 ...` |
| `listen.port + ports - 1 > 65535` | `multiport disabled: would bind ports beyond 65535` |
| platform can't run multiple UDP readers | `multiport disabled: this platform does not support multiple udp readers` |
| that capability couldn't be probed | `multiport disabled: could not probe udp reader support` |
`multiport.ports` is also clamped to 256 (the lane header limit) and to
`maxRoutines / routines`, with a warning, rather than being rejected.
With a dynamic `listen.port: 0`, the first socket binds dynamically and the
range is claimed above it; a partially-occupied range re-rolls with a fresh
dynamic port, for up to 6 attempts in all.
`laneSock` collapses to the identity when multiport is off, so the send path is
unchanged: a routine writes to its own socket and no lane batches are built at
all.
### Per-tunnel: back to a single session
The tunnel gets no lane set and is an ordinary one when:
- the peer advertised no port count (vanilla peer, or multiport off there),
- the peer's advertised range is invalid -- a zero base port, or a range running
past 65535. `maybeAllocLanes` logs `Ignoring multiport advert with an invalid
port range` rather than aiming lanes at ports that wrapped, or
- `sessions < 2`, i.e. neither side offers a lane (`newLaneSet` returns `nil`).
`txLanes` can also land at 1 -- we bound ports but the peer bound only one -- in
which case the set exists for RX but we never send on a lane.
### Per-packet: back to the base tunnel
`txLaneForFlow` returns a nil session and the packet rides the base tunnel,
decided per packet with no state to unwind:
- the flow hashed onto lane 0 (its fair share of flows);
- the lane it hashed onto has never come up;
- the lane was **demoted** -- a probe or keepalive went unanswered. Fallback is
immediate, on the very next packet, and the miss re-raises demand so the lane
is re-probed;
- the tunnel is **relayed** -- lanes are direct-only, so `probeLanes` drops them
all when there is no direct path and rebuilds when one returns;
- the peer **roamed** -- a new NAT mapping has no derivable relationship to the
old lane ports, so every lane is torn down and re-probed from a clean backoff.
Demand is kept across a reset, and the first packet of a flow that hashes onto
a lane that is now down raises it again, so lanes still carrying flows are
re-probed on the next tick.
### Always on the base tunnel
Handshakes, lighthouse traffic, punching, relay carriers, close packets, rejects,
and lane probe *acks* all use the base session and a socket on the base port
(`egressSock`). Base traffic must keep the base source port or a vanilla peer
would see the tunnel's address move and roam-thrash.
`recv_error` is the one deliberate exception: it replies from the socket the
offending packet arrived on, so the reply to lane traffic travels the reverse of
the lane's own 4-tuple, the only one the peer's NAT and stateful firewalls are
known to pass. The receiving side's spoof guard accepts a `recv_error` from the
base remote or from the target of any of its up lanes (`isLaneRemote`).
### Wire compatibility
- A vanilla sender emits lane 0 in a field it thinks is reserved, and lane 0 is
the base tunnel, so it reads correctly with no version check.
- We always send the upper 8 reserved bits as zero.
- A lane index above what this tunnel has (a stale lane from a rolled tunnel, or
a peer sending above what it advertised) is dropped **silently** -- a
`recv_error` would tear down a perfectly good base tunnel on the strength of
one odd packet.
- Lane ciphertext wrapped in a relay carrier is refused before the session
lookup, so a junk relay packet can't make us derive a session.
## Security notes
- The lane index is in the AEAD's associated data, so it is authenticated, not
just carried.
- On RX, a lane session derived for an unrecognized lane is **not installed**
until the packet actually decrypts. Anyone who can spoof this tunnel's local
index can name any lane; installing on sight would let them make us hold a
replay window and two cipher states per lane, per tunnel, for lanes they never
send on.
- Two routines racing on a lane's first packet each derive a session. The loser's
is dropped, and its replay-window entry for the packet it just accepted is
handed to the winner (`installSession`) -- the keys are identical, so it is the
same window in every respect that matters.
## Configuration
```yaml
routines: 8 # PER PORT under multiport; total workers = routines * ports
multiport:
enabled: true # default true, but inert without ports
ports: 4 # consecutive ports from listen.port; must be > 1, no default
lanes: 0 # 0 = one per bound port; lower to send on a subset
```
`multiport.ports` has no default on purpose: under these semantics a default
would silently multiply the worker count. Nothing here is reloadable.
Both sides need a port range, and the range
`[listen.port, listen.port+ports-1]` must be open in both directions. Opening
only the base port is the common failure and gives you exactly one working lane
-- the one whose rotation happens to land on the base port.
## Observability
```
nebula-ssh> print-tunnel -vpn-addr <peer>
```
`lanes[]` gives per-lane `up`, `remote` and `messageCounter`, which is the fastest
way to tell "no lanes negotiated" (key absent) from "lanes up but traffic on one"
(counters).
Metrics, registered only when multiport is running so they don't sit at zero on
nodes without it:
- `multiport.lanes.up` -- lanes currently proven usable for sending
- `multiport.lanes.tunnels` -- tunnels with a lane set, including ones with no
lane up and receive-only ones
Both are counted by walking the hostmap rather than kept at promotion/demotion,
because a counter would drift upward forever: a tunnel torn down with its lanes
up never demotes them.
Logs worth grepping: `multiport enabled` and `multiport routines` at startup, the
`lanes` attr on handshake completion (`tx`, `sessions`, `peerBasePort`,
`peerPorts`, `portOffset`), and `Multiport lane up` / `Multiport lane demoted`,
both of which name the `udpAddr` involved. A lane that has never come up logs
`Multiport lane not coming up` at Warn, once, after 3 unanswered probes in a
row -- usually a sign that only the base port is open.
## Known gaps
- No `readOutsidePackets`-level test for the RX lane drop paths.
- No multiport coverage in the e2e suite.
- `routines * ports > 256` fails at startup from the kernel's tun queue limit
rather than being clamped with a warning.
+2 -1
View File
@@ -69,9 +69,10 @@ func newDnsServerFromConfig(ctx context.Context, l *slog.Logger, pki *PKI, hostM
ds.mux = dns.NewServeMux()
ds.mux.HandleFunc(".", ds.handleDnsRequest)
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := ds.reload(c, false); err != nil {
ds.l.Error("Failed to reload DNS responder from config", "error", err)
r.Failed()
}
})
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
+1 -2
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
@@ -1514,7 +1513,7 @@ func TestLighthouseUpdateOnReload(t *testing.T) {
// Reload the config. The lighthouse.hosts change triggers TriggerUpdate,
// which wakes the update worker. It calls SendUpdate, initiating a
// handshake to the new lighthouse and caching the HostUpdateNotification.
require.NoError(t, myConfig.ReloadConfigString(string(newCfg)))
require.NoError(t, myConfig.ReloadConfigString(string(newCfg)).Err)
// Route until the lighthouse receives the HostUpdateNotification.
// This covers: handshake stage 1, stage 2, then the cached update.
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package router
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package router
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
+2 -3
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
@@ -130,7 +129,7 @@ func TestCertUpgrade(t *testing.T) {
}
r.Logf("reload new v2-only config")
err = myC.ReloadConfigString(string(cb))
err = myC.ReloadConfigString(string(cb)).Err
assert.NoError(t, err)
r.Log("yay, spin until their sees it")
waitStart := time.Now()
@@ -223,7 +222,7 @@ func TestCertDowngrade(t *testing.T) {
}
r.Logf("reload new v1-only config")
err = myC.ReloadConfigString(string(cb))
err = myC.ReloadConfigString(string(cb)).Err
assert.NoError(t, err)
r.Log("yay, spin until their sees it")
waitStart := time.Now()
+26 -60
View File
@@ -43,7 +43,7 @@ static_host_map:
lighthouse:
# am_lighthouse is used to enable lighthouse functionality for a node. This should ONLY be true on nodes
# you have configured to be lighthouses in your network
# you have configured to be lighthouses in your network. Not reloadable.
am_lighthouse: false
# serve_dns optionally starts a dns listener that responds to various queries and can even be
# delegated to for resolution
@@ -126,8 +126,9 @@ lighthouse:
# Port Nebula will be listening on. The default here is 4242. For a lighthouse node, the port should be defined,
# however using port 0 will dynamically assign a port and is recommended for roaming nodes.
listen:
# To listen on only ipv4, use "0.0.0.0"
# To listen on only ipv4, use "0.0.0.0". Not reloadable.
host: "::"
# Not reloadable.
port: 4242
# Sets the max number of packets to pull from the kernel for each syscall (under systems that support recvmmsg)
# default is 64, does not support reload
@@ -135,12 +136,13 @@ listen:
# superpacket, so the receive scratch is batch * 64KiB per listening socket (~4MiB per routine at the
# default of 64). Lower this to trade peak per-syscall throughput for memory on constrained hosts.
#batch: 64
# Linux only. Reloadable, but removing a value leaves the socket at the old size until Nebula restarts.
# Configure socket buffers for the udp side (outside), leave unset to use the system defaults. Values will be doubled by the kernel
# Default is net.core.rmem_default and net.core.wmem_default (/proc/sys/net/core/rmem_default and /proc/sys/net/core/rmem_default)
# Maximum is limited by memory in the system, SO_RCVBUFFORCE and SO_SNDBUFFORCE is used to avoid having to raise the system wide
# max, net.core.rmem_max and net.core.wmem_max
#read_buffer: 10485760
#write_buffer: 10485760
read_buffer: 3412512
write_buffer: 3412512
# On Windows only
# When true, Nebula installs a WFP (Windows Filtering Platform) PERMIT filter scoped to UDP at the listener port.
@@ -179,56 +181,9 @@ listen:
# Currently, this defaults to 1 which means we have 1 tun queue reader and 1
# UDP queue reader. Setting this above one will set IFF_MULTI_QUEUE on the tun
# device and SO_REUSEPORT on the UDP socket to allow multiple queues.
# With multiport enabled this is the number of routines *per port*, so the total
# is routines * multiport.ports.
# This option is only supported on Linux.
# This option is only supported on Linux. Not reloadable.
#routines: 1
# EXPERIMENTAL: multiport lanes give each pair of hosts multiple underlay UDP
# flows so overlay traffic is no longer bottlenecked by a single 5-tuple
# (one ECMP path, one NIC RSS queue, one per-flow policer). Instead of every
# socket sharing listen.port, multiport.ports consecutive ports are bound and
# each gets its own group of `routines` sockets sharing it via SO_REUSEPORT, so
# no port (the base port above all, which carries every handshake and every
# vanilla peer) depends on a single core. Each port carries one lane: lane i
# sends from local port listen.port+i to the peer's advertised
# base+((i + pair_offset) mod peer_ports), where pair_offset is a per-pair hash
# that spreads many small peers across a big peer's whole port range. Lane 0 is
# the base tunnel itself.
# A lane is not a separate tunnel and has no handshake of its own. Both sides
# derive its keys from the base tunnel's, so it comes and goes with the base
# tunnel, but it has its own nonce counter and replay window, so flows taking
# different paths never fight over shared replay state. Each inside flow is
# hashed onto one lane for its whole life, so multiport adds no reordering.
#
# Peers negotiate lanes in the handshake; vanilla peers get a single normal
# tunnel. All control traffic (handshakes, lighthouse, punching, relays) and
# the data fallback stay on the base tunnel/port. A lane only carries data once
# a probe on it has been answered, and is re-probed every 30s while it is up;
# traffic falls back to the base tunnel while a lane is down or not yet up.
# Lanes are only probed on tunnels with traffic, so an idle tunnel costs
# nothing extra.
#
# Requirements: multiport.ports > 1, Linux, and the port range
# [listen.port, listen.port+multiport.ports-1] reachable through firewalls on
# both sides. A lane whose port is unreachable stays down and its traffic rides
# the base tunnel. With listen.port 0 the base port is dynamic and the next
# ports-1 ports above it are claimed. Degrades to a single port when the
# requirements don't hold. Not reloadable.
#multiport:
#enabled: true
# How many consecutive UDP ports to bind, starting at listen.port. Must be
# greater than 1 for multiport to do anything; there is no default, since
# each port costs a full set of `routines` threads and sockets. Capped at 256
# (the lane header limit). 4-8 is plenty to escape a single ECMP path.
#ports: 0
# How many lanes to send on, counting the base tunnel as lane 0. 0 (default)
# means one per bound port. Lowering this sends on a subset of the range
# (useful on a big server with many peers, to bound how many lane probes and
# sessions each peer costs). It only limits sending: the ports are bound and
# read either way, and we still receive on every lane a peer sends on.
#lanes: 0
punchy:
# Continues to punch inbound/outbound at a regular interval to avoid expiration of firewall nat mappings
# This setting is reloadable.
@@ -250,6 +205,7 @@ punchy:
# Cipher allows you to choose between the available ciphers for your network. Options are chachapoly or aes
# IMPORTANT: this value must be identical on ALL NODES/LIGHTHOUSES. We do not/will not support use of different ciphers simultaneously!
# Not reloadable.
#cipher: aes
# Preferred ranges is used to define a hint about the local network ranges, which speeds up discovering the fastest
@@ -281,6 +237,7 @@ punchy:
# and absolute paths outside of it will be rejected. Default is $TMP/nebula-debug.
# The directory is NOT automatically created.
# Overriding this to "" is the same as "/" and will allow overwriting any path on the host.
# Not reloadable.
#sandbox_dir: /var/tmp/nebula-debug
# EXPERIMENTAL: relay support for networks that can't establish direct connections.
@@ -300,18 +257,25 @@ relay:
# Configure the private interface. Note: addr is baked into the nebula certificate
tun:
# When tun is disabled, a lighthouse can be started without a local tun interface (and therefore without root)
# Not reloadable.
disabled: false
# Name of the device. If not set, a default will be chosen by the OS.
# For macOS: if set, must be in the form `utun[0-9]+`.
# For NetBSD: Required to be set, must be in the form `tun[0-9]+`
# For OpenBSD and NetBSD: if set, must be in the form `tun[0-9]+`, otherwise the first free one is used.
# Not reloadable.
dev: nebula1
# Toggles forwarding of local broadcast packets, the address of which depends on the ip/mask encoded in pki.cert
# Not reloadable.
drop_local_broadcast: false
# Toggles forwarding of multicast packets
# Toggles forwarding of multicast packets. Not reloadable.
drop_multicast: false
# Sets the transmit queue length, if you notice lots of transmit drops on the tun it may help to raise this number. Default is 500
tx_queue: 500
# Sets the transmit queue length, if you notice lots of transmit drops on the tun it may help to raise this number. Default is 5000
# Not reloadable.
tx_queue: 5000
# Default MTU for every packet, safe setting is (and the default) 1300 for internet based traffic
# Anything above 8937, here or on a route below, is capped to 8937 with a warning
# Reloadable on Linux only. Where the tun is handed to nebula by a host app (iOS, Android, macOS network extensions)
# the host app owns the MTU.
mtu: 1300
# the use_offloads setting controls if Nebula will attempt to enable GSO and GRO for the tun device. Linux only, not reloadable.
@@ -404,7 +368,7 @@ tun:
#use_system_route_table: false
# Buffer size for reading routes updates. 0 means default system buffer size. (/proc/sys/net/core/rmem_default).
# If using massive routes updates, for example BGP, you may need to increase this value to avoid packet loss.
# SO_RCVBUFFORCE is used to avoid having to raise the system wide max
# SO_RCVBUFFORCE is used to avoid having to raise the system wide max. Not reloadable.
#use_system_route_table_buffer_size: 0
# Configure logging level
@@ -454,14 +418,16 @@ logging:
# Handshakes are sent to all known addresses at each interval with a linear backoff,
# Wait try_interval after the 1st attempt, 2 * try_interval after the 2nd, etc, until the handshake is older than timeout
# A 100ms interval with the default 10 retries will give a handshake 5.5 seconds to resolve before timing out
# Not reloadable.
#try_interval: 100ms
#retries: 20
# query_buffer is the size of the buffer channel for querying lighthouses
#query_buffer: 64
# query_buffer is the size of the buffer channel for querying lighthouses, drops are recorded in the lighthouse.query.dropped metric.
# Not reloadable.
#query_buffer: 1024
# trigger_buffer is the size of the buffer channel for quickly sending handshakes
# after receiving the response for lighthouse queries
# after receiving the response for lighthouse queries. Not reloadable.
#trigger_buffer: 64
# Tunnel manager settings
+101 -34
View File
@@ -32,16 +32,21 @@ type conn struct {
Expires time.Time // Time when this conntrack entry will expire
// record why the original connection passed the firewall, so we can re-validate
// after ruleset changes. Note, rulesVersion is a uint16 so that these two
// fields pack for free after the uint32 above
// after ruleset changes or when the peer handshakes again, which may present a
// cert with different groups. These pack into the word after Expires.
incoming bool
rulesVersion uint16
localIndex uint32
}
// TODO: need conntrack max tracked connections handling
type Firewall struct {
Conntrack *FirewallConntrack
// siblings finds the other gateways we route an unsafe network through, so a reply that comes back through a
// different gateway than the flow went out through can be matched to it. Nil turns that off.
siblings gatewaySiblings
InRules *FirewallTable
OutRules *FirewallTable
@@ -82,8 +87,8 @@ type firewallMetrics struct {
type FirewallConntrack struct {
sync.Mutex
Conns map[firewall.Packet]*conn
TimerWheel *TimerWheel[firewall.Packet]
Conns map[firewall.ConntrackKey]*conn
TimerWheel *TimerWheel[firewall.ConntrackKey]
}
// FirewallTable is the entry point for a rule, the evaluation order is:
@@ -167,8 +172,8 @@ func NewFirewall(l *slog.Logger, tcpTimeout, UDPTimeout, defaultTimeout time.Dur
return &Firewall{
Conntrack: &FirewallConntrack{
Conns: make(map[firewall.Packet]*conn),
TimerWheel: NewTimerWheel[firewall.Packet](tmin, tmax),
Conns: make(map[firewall.ConntrackKey]*conn),
TimerWheel: NewTimerWheel[firewall.ConntrackKey](tmin, tmax),
},
InRules: newFirewallTable(),
OutRules: newFirewallTable(),
@@ -425,6 +430,7 @@ var ErrNoMatchingRule = errors.New("no matching rule in firewall table")
// returns nil if the packet should not be dropped.
func (f *Firewall) Drop(fp firewall.Packet, incoming bool, h *HostInfo, caPool *cert.CAPool, localCache firewall.ConntrackCache) error {
// Make sure remote address matches nebula certificate, and determine how to treat it
unsafe := false
if h.networks == nil {
// Simple case: Certificate has one address and no unsafe networks
if h.vpnAddrs[0] != fp.RemoteAddr {
@@ -444,7 +450,7 @@ func (f *Firewall) Drop(fp firewall.Packet, incoming bool, h *HostInfo, caPool *
f.metrics(incoming).droppedRemoteAddr.Inc(1)
return ErrPeerRejected // reject for now, one day this may have different FW rules
case NetworkTypeUnsafe:
break // nothing special, one day this may have different FW rules
unsafe = true // nothing else special, one day this may have different FW rules
default:
f.metrics(incoming).droppedRemoteAddr.Inc(1)
return ErrUnknownNetworkType //should never happen
@@ -457,8 +463,11 @@ func (f *Firewall) Drop(fp firewall.Packet, incoming bool, h *HostInfo, caPool *
return ErrInvalidLocalIP
}
// The first vpn address identifies the peer, as it does elsewhere. A host's v1 and v2 certs must agree on it
key := firewall.ConntrackKey{Packet: fp, Peer: h.vpnAddrs[0]}
// Check if we spoke to this tuple, if we did then allow this packet
if f.inConns(fp, h, caPool, localCache) {
if f.inConns(key, h, caPool, localCache) {
return nil
}
@@ -469,12 +478,15 @@ func (f *Firewall) Drop(fp firewall.Packet, incoming bool, h *HostInfo, caPool *
// We now know which firewall table to check against
if !table.match(fp, incoming, h.ConnectionState.peerCert, caPool) {
if unsafe && f.inheritConn(key, incoming, h, caPool) {
return nil
}
f.metrics(incoming).droppedNoRule.Inc(1)
return ErrNoMatchingRule
}
// We always want to conntrack since it is a faster operation
f.addConn(fp, incoming)
f.addConn(key, incoming, h)
return nil
}
@@ -503,9 +515,9 @@ func (f *Firewall) EmitStats() {
metrics.GetOrRegisterGauge("firewall.rules.hash", nil).Update(int64(f.GetRuleHashFNV()))
}
func (f *Firewall) inConns(fp firewall.Packet, h *HostInfo, caPool *cert.CAPool, localCache firewall.ConntrackCache) bool {
func (f *Firewall) inConns(key firewall.ConntrackKey, h *HostInfo, caPool *cert.CAPool, localCache firewall.ConntrackCache) bool {
if localCache != nil {
if _, ok := localCache[fp]; ok {
if _, ok := localCache[key]; ok {
return true
}
}
@@ -518,49 +530,52 @@ func (f *Firewall) inConns(fp firewall.Packet, h *HostInfo, caPool *cert.CAPool,
f.evict(ep)
}
c, ok := conntrack.Conns[fp]
c, ok := conntrack.Conns[key]
if !ok {
conntrack.Unlock()
return false
}
if c.rulesVersion != f.rulesVersion {
// This conntrack entry was for an older rule set, validate
// it still passes with the current rule set
if c.rulesVersion != f.rulesVersion || c.localIndex != h.localIndexId {
// This conntrack entry was allowed by an older rule set or for an earlier tunnel to the peer, validate
// it still passes with the current rule set and the peer's current certificate
table := f.OutRules
if c.incoming {
table = f.InRules
}
// We now know which firewall table to check against
if !table.match(fp, c.incoming, h.ConnectionState.peerCert, caPool) {
if !table.match(key.Packet, c.incoming, h.ConnectionState.peerCert, caPool) {
if f.l.Enabled(context.Background(), slog.LevelDebug) {
h.logger(f.l).Debug("dropping old conntrack entry, does not match new ruleset",
"fwPacket", fp,
h.logger(f.l).Debug("dropping old conntrack entry, does not match current ruleset and certificate",
"fwPacket", key.Packet,
"incoming", c.incoming,
"rulesVersion", f.rulesVersion,
"oldRulesVersion", c.rulesVersion,
"oldLocalIndex", c.localIndex,
)
}
delete(conntrack.Conns, fp)
delete(conntrack.Conns, key)
conntrack.Unlock()
return false
}
if f.l.Enabled(context.Background(), slog.LevelDebug) {
h.logger(f.l).Debug("keeping old conntrack entry, does match new ruleset",
"fwPacket", fp,
h.logger(f.l).Debug("keeping old conntrack entry, does match current ruleset and certificate",
"fwPacket", key.Packet,
"incoming", c.incoming,
"rulesVersion", f.rulesVersion,
"oldRulesVersion", c.rulesVersion,
"oldLocalIndex", c.localIndex,
)
}
c.rulesVersion = f.rulesVersion
c.localIndex = h.localIndexId
}
switch fp.Protocol {
switch key.Protocol {
case iputil.IPProtocolTCP:
c.Expires = time.Now().Add(f.TCPTimeout)
case iputil.IPProtocolUDP:
@@ -572,17 +587,17 @@ func (f *Firewall) inConns(fp firewall.Packet, h *HostInfo, caPool *cert.CAPool,
conntrack.Unlock()
if localCache != nil {
localCache[fp] = struct{}{}
localCache[key] = struct{}{}
}
return true
}
func (f *Firewall) addConn(fp firewall.Packet, incoming bool) {
func (f *Firewall) addConn(key firewall.ConntrackKey, incoming bool, h *HostInfo) {
var timeout time.Duration
c := &conn{}
switch fp.Protocol {
switch key.Protocol {
case iputil.IPProtocolTCP:
timeout = f.TCPTimeout
case iputil.IPProtocolUDP:
@@ -593,26 +608,78 @@ func (f *Firewall) addConn(fp firewall.Packet, incoming bool) {
conntrack := f.Conntrack
conntrack.Lock()
if _, ok := conntrack.Conns[fp]; !ok {
if _, ok := conntrack.Conns[key]; !ok {
conntrack.TimerWheel.Advance(time.Now())
conntrack.TimerWheel.Add(fp, timeout)
conntrack.TimerWheel.Add(key, timeout)
}
// Record which rulesVersion allowed this connection, so we can retest after
// firewall reload
// Record which rulesVersion and tunnel allowed this connection, so we can retest after
// firewall reload or when the peer handshakes again
c.incoming = incoming
c.rulesVersion = f.rulesVersion
c.localIndex = h.localIndexId
c.Expires = time.Now().Add(timeout)
conntrack.Conns[fp] = c
conntrack.Conns[key] = c
conntrack.Unlock()
}
// gatewaySiblings finds the other gateways we route an unsafe network through
type gatewaySiblings interface {
// siblingPeers appends to buf the conntrack peer of every other gateway we route remote through, if h is one of
// those gateways
siblingPeers(remote netip.Addr, h *HostInfo, buf []netip.Addr) []netip.Addr
}
// inheritConn lets h carry a flow that another of our gateways for the same unsafe network already carries. We
// spread the flows we open across the gateways, but the hosts behind them answer through whichever gateway their own
// routing picks, so a reply can come back through a different gateway than the flow went out through. The flow is
// checked against the rules that allowed it in the first place with h's certificate, the same check an entry gets
// when its peer reconnects. Peers that aren't one of our gateways for the network never inherit a flow.
func (f *Firewall) inheritConn(key firewall.ConntrackKey, incoming bool, h *HostInfo, caPool *cert.CAPool) bool {
if f.siblings == nil {
return false
}
var buf [4]netip.Addr
now := time.Now()
for _, peer := range f.siblings.siblingPeers(key.RemoteAddr, h, buf[:0]) {
sk := key
sk.Peer = peer
conntrack := f.Conntrack
conntrack.Lock()
c, ok := conntrack.Conns[sk]
live := ok && c.Expires.After(now)
var allowedIncoming bool
if live {
allowedIncoming = c.incoming
}
conntrack.Unlock()
// The rules for this packet's own direction already said no, only a flow allowed the other way can help
if !live || allowedIncoming == incoming {
continue
}
table := f.OutRules
if allowedIncoming {
table = f.InRules
}
if !table.match(key.Packet, allowedIncoming, h.ConnectionState.peerCert, caPool) {
continue
}
f.addConn(key, allowedIncoming, h)
return true
}
return false
}
// Evict checks if a conntrack entry has expired, if so it is removed, if not it is re-added to the wheel
// Caller must own the connMutex lock!
func (f *Firewall) evict(p firewall.Packet) {
func (f *Firewall) evict(key firewall.ConntrackKey) {
// Are we still tracking this conn?
conntrack := f.Conntrack
t, ok := conntrack.Conns[p]
t, ok := conntrack.Conns[key]
if !ok {
return
}
@@ -622,12 +689,12 @@ func (f *Firewall) evict(p firewall.Packet) {
// Timeout is in the future, re-add the timer
if newT > 0 {
conntrack.TimerWheel.Advance(time.Now())
conntrack.TimerWheel.Add(p, newT)
conntrack.TimerWheel.Add(key, newT)
return
}
// This conn is done
delete(conntrack.Conns, p)
delete(conntrack.Conns, key)
}
func (ft *FirewallTable) match(p firewall.Packet, incoming bool, c *cert.CachedCertificate, caPool *cert.CAPool) bool {
+9 -1
View File
@@ -3,15 +3,23 @@ package firewall
import (
"context"
"log/slog"
"net/netip"
"sync/atomic"
"time"
"github.com/slackhq/nebula/logging"
)
// ConntrackKey identifies a tracked flow by its packet and the peer it was allowed for, so peers that route
// the same unsafe network never share a conntrack entry.
type ConntrackKey struct {
Peer netip.Addr
Packet
}
// ConntrackCache is used as a local routine cache to know if a given flow
// has been seen in the conntrack table.
type ConntrackCache map[Packet]struct{}
type ConntrackCache map[ConntrackKey]struct{}
type ConntrackCacheTicker struct {
cacheV uint64
+2 -2
View File
@@ -22,8 +22,8 @@ func newFixedTicker(t *testing.T, l *slog.Logger, cacheLen int) *ConntrackCacheT
l: l,
cache: make(ConntrackCache, cacheLen),
}
for i := 0; i < cacheLen; i++ {
c.cache[Packet{LocalPort: uint16(i) + 1}] = struct{}{}
for i := range cacheLen {
c.cache[ConntrackKey{Packet: Packet{LocalPort: uint16(i) + 1}}] = struct{}{}
}
c.cacheTick.Store(1) // cacheV starts at 0, so Get() takes the reset path
return c
+4
View File
@@ -26,6 +26,10 @@ type Packet struct {
RemotePort uint16
Protocol uint8
Fragment bool
// pad fills what would otherwise be trailing padding so maps hash and compare a ConntrackKey as one
// block of memory instead of field by field. It must always be zero.
pad [2]byte
}
func (fp *Packet) Copy() *Packet {
+248 -1
View File
@@ -3,6 +3,7 @@ package nebula
import (
"bytes"
"errors"
"fmt"
"log/slog"
"math"
"net/netip"
@@ -979,6 +980,252 @@ func TestFirewall_ConntrackSourceSpoofingAcrossPeers(t *testing.T) {
"SECURITY: attacker spoofed victim's overlay source IP (192.0.2.2) by reusing an existing conntrack 4-tuple; Drop returned %v instead of rejecting", got)
}
func TestFirewall_ConntrackUnsafeNetworkPeers(t *testing.T) {
l := test.NewLoggerWithOutput(&bytes.Buffer{})
myVpnNetworksTable := new(bart.Lite)
myVpnNetworksTable.Insert(netip.MustParsePrefix("192.0.2.1/24"))
owner := &dummyCert{
name: "owner",
networks: []netip.Prefix{netip.MustParsePrefix("192.0.2.1/24")},
}
// Two routers for the same unsafe network, only one of them is trusted inbound
router := func(name string, addr string, groups ...string) *HostInfo {
inverted := map[string]struct{}{}
for _, g := range groups {
inverted[g] = struct{}{}
}
c := &cert.CachedCertificate{
Certificate: &dummyCert{
name: name,
groups: groups,
networks: []netip.Prefix{netip.MustParsePrefix(addr + "/24")},
unsafeNetworks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
},
InvertedGroups: inverted,
}
h := &HostInfo{
ConnectionState: &ConnectionState{peerCert: c},
vpnAddrs: []netip.Addr{netip.MustParseAddr(addr)},
}
h.buildNetworks(myVpnNetworksTable, c.Certificate)
return h
}
trusted := router("trusted", "192.0.2.2", "trusted")
other := router("other", "192.0.2.3")
cp := cert.NewCAPool()
flow := firewall.Packet{
LocalAddr: netip.MustParseAddr("192.0.2.1"),
RemoteAddr: netip.MustParseAddr("198.51.100.5"),
LocalPort: 443,
RemotePort: 55000,
Protocol: iputil.IPProtocolTCP,
}
for _, useCache := range []bool{false, true} {
var cache firewall.ConntrackCache
if useCache {
cache = firewall.ConntrackCache{}
}
t.Run(fmt.Sprintf("inbound flow allowed for one peer/cache=%v", useCache), func(t *testing.T) {
fw := NewFirewall(l, time.Second, time.Minute, time.Hour, owner)
require.NoError(t, fw.AddRule(true, firewall.ProtoAny, 0, 0, []string{"trusted"}, "", "", "", "", ""))
clear(cache)
require.NoError(t, fw.Drop(flow, true, trusted, cp, cache))
assert.Equal(t, ErrNoMatchingRule, fw.Drop(flow, true, other, cp, cache))
require.NoError(t, fw.Drop(flow, true, trusted, cp, cache))
})
t.Run(fmt.Sprintf("reply to a flow opened via one peer/cache=%v", useCache), func(t *testing.T) {
fw := NewFirewall(l, time.Second, time.Minute, time.Hour, owner)
require.NoError(t, fw.AddRule(false, firewall.ProtoAny, 0, 0, []string{"any"}, "", "", "", "", ""))
clear(cache)
require.NoError(t, fw.Drop(flow, false, trusted, cp, cache))
require.NoError(t, fw.Drop(flow, true, trusted, cp, cache))
assert.Equal(t, ErrNoMatchingRule, fw.Drop(flow, true, other, cp, cache))
})
}
}
// fakeSiblings maps each gateway to the conntrack peers of the gateways it shares a route with
type fakeSiblings map[*HostInfo][]netip.Addr
func (s fakeSiblings) siblingPeers(_ netip.Addr, h *HostInfo, buf []netip.Addr) []netip.Addr {
return append(buf, s[h]...)
}
func TestFirewall_ConntrackGatewaySiblings(t *testing.T) {
l := test.NewLoggerWithOutput(&bytes.Buffer{})
myVpnNetworksTable := new(bart.Lite)
myVpnNetworksTable.Insert(netip.MustParsePrefix("192.0.2.1/24"))
owner := &dummyCert{
name: "owner",
networks: []netip.Prefix{netip.MustParsePrefix("192.0.2.1/24")},
}
// Routers for the same unsafe network. a and b are both our gateways for it, c only claims it in its cert
router := func(name string, addr string, groups ...string) *HostInfo {
inverted := map[string]struct{}{}
for _, g := range groups {
inverted[g] = struct{}{}
}
c := &cert.CachedCertificate{
Certificate: &dummyCert{
name: name,
groups: groups,
networks: []netip.Prefix{netip.MustParsePrefix(addr + "/24")},
unsafeNetworks: []netip.Prefix{netip.MustParsePrefix("198.51.100.0/24")},
},
InvertedGroups: inverted,
}
h := &HostInfo{
ConnectionState: &ConnectionState{peerCert: c},
vpnAddrs: []netip.Addr{netip.MustParseAddr(addr)},
}
h.buildNetworks(myVpnNetworksTable, c.Certificate)
return h
}
a := router("a", "192.0.2.2", "trusted")
b := router("b", "192.0.2.3", "trusted")
c := router("c", "192.0.2.4", "trusted")
siblings := fakeSiblings{a: {b.vpnAddrs[0]}, b: {a.vpnAddrs[0]}}
cp := cert.NewCAPool()
flow := firewall.Packet{
LocalAddr: netip.MustParseAddr("192.0.2.1"),
RemoteAddr: netip.MustParseAddr("198.51.100.5"),
LocalPort: 55000,
RemotePort: 443,
Protocol: iputil.IPProtocolTCP,
}
// We open flows through our gateways, nothing is allowed in on its own
newFirewall := func(outGroup string) *Firewall {
fw := NewFirewall(l, time.Second, time.Minute, time.Hour, owner)
require.NoError(t, fw.AddRule(false, firewall.ProtoAny, 0, 0, []string{outGroup}, "", "", "", "", ""))
fw.siblings = siblings
return fw
}
t.Run("reply through the other gateway", func(t *testing.T) {
fw := newFirewall("trusted")
require.NoError(t, fw.Drop(flow, false, b, cp, nil))
require.NoError(t, fw.Drop(flow, true, a, cp, nil))
// a has its own entry now, and keeps the flow when b's is gone
delete(fw.Conntrack.Conns, firewall.ConntrackKey{Packet: flow, Peer: b.vpnAddrs[0]})
require.NoError(t, fw.Drop(flow, true, a, cp, nil))
})
t.Run("not our gateway", func(t *testing.T) {
fw := newFirewall("trusted")
require.NoError(t, fw.Drop(flow, false, b, cp, nil))
assert.Equal(t, ErrNoMatchingRule, fw.Drop(flow, true, c, cp, nil))
})
t.Run("off without siblings", func(t *testing.T) {
fw := newFirewall("trusted")
fw.siblings = nil
require.NoError(t, fw.Drop(flow, false, b, cp, nil))
assert.Equal(t, ErrNoMatchingRule, fw.Drop(flow, true, a, cp, nil))
})
t.Run("expired sibling entry", func(t *testing.T) {
fw := newFirewall("trusted")
require.NoError(t, fw.Drop(flow, false, b, cp, nil))
fw.Conntrack.Conns[firewall.ConntrackKey{Packet: flow, Peer: b.vpnAddrs[0]}].Expires = time.Now().Add(-time.Second)
assert.Equal(t, ErrNoMatchingRule, fw.Drop(flow, true, a, cp, nil))
})
t.Run("the original rules must allow the new gateway", func(t *testing.T) {
fw := newFirewall("b-only")
b.ConnectionState.peerCert.InvertedGroups["b-only"] = struct{}{}
defer delete(b.ConnectionState.peerCert.InvertedGroups, "b-only")
require.NoError(t, fw.Drop(flow, false, b, cp, nil))
assert.Equal(t, ErrNoMatchingRule, fw.Drop(flow, true, a, cp, nil))
})
}
func TestFirewall_ConntrackPeerRehandshake(t *testing.T) {
l := test.NewLoggerWithOutput(&bytes.Buffer{})
myVpnNetworksTable := new(bart.Lite)
myVpnNetworksTable.Insert(netip.MustParsePrefix("192.0.2.1/24"))
owner := &dummyCert{
name: "owner",
networks: []netip.Prefix{netip.MustParsePrefix("192.0.2.1/24")},
}
// Each call is a new tunnel to the same peer, presenting a cert with the given groups
tunnel := func(localIndex uint32, groups ...string) *HostInfo {
inverted := map[string]struct{}{}
for _, g := range groups {
inverted[g] = struct{}{}
}
c := &cert.CachedCertificate{
Certificate: &dummyCert{
name: "peer",
groups: groups,
networks: []netip.Prefix{netip.MustParsePrefix("192.0.2.2/24"), netip.MustParsePrefix("2001:db8::2/64")},
},
InvertedGroups: inverted,
}
h := &HostInfo{
ConnectionState: &ConnectionState{peerCert: c},
vpnAddrs: []netip.Addr{netip.MustParseAddr("192.0.2.2"), netip.MustParseAddr("2001:db8::2")},
localIndexId: localIndex,
}
h.buildNetworks(myVpnNetworksTable, c.Certificate)
return h
}
cp := cert.NewCAPool()
flow := firewall.Packet{
LocalAddr: netip.MustParseAddr("192.0.2.1"),
RemoteAddr: netip.MustParseAddr("192.0.2.2"),
LocalPort: 443,
RemotePort: 55000,
Protocol: iputil.IPProtocolTCP,
}
key := firewall.ConntrackKey{Packet: flow, Peer: netip.MustParseAddr("192.0.2.2")}
t.Run("inbound flow follows the peer's groups", func(t *testing.T) {
fw := NewFirewall(l, time.Second, time.Minute, time.Hour, owner)
require.NoError(t, fw.AddRule(true, firewall.ProtoAny, 0, 0, []string{"trusted"}, "", "", "", "", ""))
require.NoError(t, fw.Drop(flow, true, tunnel(1, "trusted"), cp, nil))
// Same groups on a new tunnel keeps the entry and moves it to the new tunnel
require.NoError(t, fw.Drop(flow, true, tunnel(2, "trusted"), cp, nil))
require.Contains(t, fw.Conntrack.Conns, key)
assert.Equal(t, uint32(2), fw.Conntrack.Conns[key].localIndex)
// Losing the group on a new tunnel removes the entry
assert.Equal(t, ErrNoMatchingRule, fw.Drop(flow, true, tunnel(3), cp, nil))
assert.NotContains(t, fw.Conntrack.Conns, key)
})
t.Run("reply to a flow we opened survives a new tunnel", func(t *testing.T) {
fw := NewFirewall(l, time.Second, time.Minute, time.Hour, owner)
require.NoError(t, fw.AddRule(false, firewall.ProtoAny, 0, 0, []string{"any"}, "", "", "", "", ""))
require.NoError(t, fw.Drop(flow, false, tunnel(1, "trusted"), cp, nil))
require.NoError(t, fw.Drop(flow, true, tunnel(2), cp, nil))
assert.Equal(t, uint32(2), fw.Conntrack.Conns[key].localIndex)
})
}
// BenchmarkFirewallDropConntrackHit measures Drop on an already-established flow
// (a conntrack hit). This is the fast path that the source-IP<->cert binding
// reordering adds work to, so it quantifies the cost of moving the address checks
@@ -1742,6 +1989,6 @@ func (mf *mockFirewall) AddRule(incoming bool, proto uint8, startPort int32, end
func resetConntrack(fw *Firewall) {
fw.Conntrack.Lock()
fw.Conntrack.Conns = map[firewall.Packet]*conn{}
fw.Conntrack.Conns = map[firewall.ConntrackKey]*conn{}
fw.Conntrack.Unlock()
}
+14 -15
View File
@@ -1,39 +1,39 @@
module github.com/slackhq/nebula
go 1.26.0
go 1.27.2
require (
dario.cat/mergo v1.0.2
filippo.io/bigmod v0.1.0
github.com/DefinedNet/netlink v1.3.2-dn.1
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
github.com/armon/go-radix v1.0.0
github.com/flynn/noise v1.1.0
github.com/gaissmai/bart v0.28.0
github.com/gaissmai/bart v0.29.0
github.com/gogo/protobuf v1.3.2
github.com/google/gopacket v1.1.19
github.com/kardianos/service v1.3.0
github.com/miekg/dns v1.1.72
github.com/miekg/dns v1.1.73
github.com/miekg/pkcs11 v1.1.2
github.com/nbrownus/go-metrics-prometheus v0.0.0-20210712211119-974a6260965f
github.com/prometheus/client_golang v1.24.1
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6
github.com/stretchr/testify v1.12.0
github.com/vishvananda/netlink v1.3.1
github.com/stretchr/testify v1.12.1
go.uber.org/goleak v1.3.0
go.yaml.in/yaml/v3 v3.0.5
golang.org/x/crypto v0.54.0
golang.org/x/net v0.57.0
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0
golang.org/x/term v0.45.0
golang.org/x/crypto v0.57.0
golang.org/x/net v0.59.0
golang.org/x/sync v0.23.0
golang.org/x/sys v0.48.0
golang.org/x/term v0.46.0
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2
golang.zx2c4.com/wireguard v0.0.0-20230325221338-052af4a8072b
golang.zx2c4.com/wireguard/windows v1.0.1
google.golang.org/protobuf v1.36.11
google.golang.org/protobuf v1.36.12
gopkg.in/yaml.v3 v3.0.1
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe
gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980
)
require (
@@ -45,7 +45,6 @@ require (
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/vishvananda/netns v0.0.5 // indirect
golang.org/x/mod v0.36.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.45.0 // indirect
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc // indirect
golang.org/x/time v0.15.0 // indirect
)
+26 -30
View File
@@ -3,6 +3,8 @@ dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
filippo.io/bigmod v0.1.0 h1:UNzDk7y9ADKST+axd9skUpBQeW7fG2KrTZyOE4uGQy8=
filippo.io/bigmod v0.1.0/go.mod h1:OjOXDNlClLblvXdwgFFOQFJEocLhhtai8vGLy0JCZlI=
github.com/DefinedNet/netlink v1.3.2-dn.1 h1:Md2PsUE1Xuy1Zxs/Bm6356n1kt5bq2HspXlPJ6pEspg=
github.com/DefinedNet/netlink v1.3.2-dn.1/go.mod h1:o4YSn8Jw7zoDcYXDmV77TRd4eS5sSLuE+vu2zAN67C8=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
@@ -23,8 +25,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/flynn/noise v1.1.0 h1:KjPQoQCEFdZDiP03phOvGi11+SVVhBG2wOWAorLsstg=
github.com/flynn/noise v1.1.0/go.mod h1:xbMo+0i6+IGbYdJhF31t2eR1BIU0CYc12+BNAKwUTag=
github.com/gaissmai/bart v0.28.0 h1:89yZLo8NmyqD0RYgJ3QO9HhqqGGw+oWhf90cZm69Lko=
github.com/gaissmai/bart v0.28.0/go.mod h1:GREWQfTLRWz/c5FTOsIw+KkscuFkIV5t8Rp7Nd1Td5c=
github.com/gaissmai/bart v0.29.0 h1:wO6HGE8g9YE0Wm0bCpYxwRzfQ4+fbJKOhL64e5ACGCI=
github.com/gaissmai/bart v0.29.0/go.mod h1:GREWQfTLRWz/c5FTOsIw+KkscuFkIV5t8Rp7Nd1Td5c=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
@@ -81,8 +83,8 @@ github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/miekg/dns v1.1.72 h1:vhmr+TF2A3tuoGNkLDFK9zi36F2LS+hKTRW0Uf8kbzI=
github.com/miekg/dns v1.1.72/go.mod h1:+EuEPhdHOsfk6Wk5TT2CzssZdqkmFhf8r+aVyDEToIs=
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
github.com/miekg/pkcs11 v1.1.2 h1:/VxmeAX5qU6Q3EwafypogwWbYryHFmF2RpkJmw3m4MQ=
github.com/miekg/pkcs11 v1.1.2/go.mod h1:XsNlhZGX73bx86s2hdc/FuaLm2CPZJemRLMA+WTFxgs=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -136,10 +138,8 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.12.0 h1:K6Mr6jO9JICuend/5xzTM03ydSV3vdNRYAdPSukj8uI=
github.com/stretchr/testify v1.12.0/go.mod h1:bOYBZb5qJ00vPzWfIqBUZPaxK8jWiXc6d3ErP4Ca9Gw=
github.com/vishvananda/netlink v1.3.1 h1:3AEMt62VKqz90r0tmNhog0r/PpWKmrEShJU0wJW6bV0=
github.com/vishvananda/netlink v1.3.1/go.mod h1:ARtKouGSTGchR8aMwmkzC0qiNPrrWO5JS/XMVl45+b4=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/vishvananda/netns v0.0.5 h1:DfiHV+j8bA32MFM7bfEunvT8IAqQ/NzSJHtcmW5zdEY=
github.com/vishvananda/netns v0.0.5/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -155,14 +155,14 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc h1:TS73t7x3KarrNd5qAipmspBDS1rkMcgVG/fS1aRb4Rc=
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc/go.mod h1:A+z0yzpGtvnG90cToK5n2tu8UJVP2XUATh+r+sfOOOc=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -173,8 +173,8 @@ golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -182,8 +182,8 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -197,25 +197,21 @@ golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=
golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -234,8 +230,8 @@ google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miE
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -249,5 +245,5 @@ gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe h1:fre4i6mv4iBuz5lCMOzHD1rH1ljqHWSICFmZRbbgp3g=
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe/go.mod h1:sxc3Uvk/vHcd3tj7/DHVBoR5wvWT/MmRq2pj7HRJnwU=
gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980 h1:ribyrTYQ/QYeUIs8x8bVLYwGyepa7RnHLqJzegT5YYk=
gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980/go.mod h1:pK/dapF5Bf3KGW7/kd5CI5IC4lYh9FMFY1qETAIGLXk=
+18 -5
View File
@@ -31,6 +31,7 @@ package nebula
import (
"bufio"
"context"
"fmt"
"net"
"strconv"
@@ -51,17 +52,29 @@ type graphiteConfigExport struct {
Percentiles []float64 // Percentiles to export from timers and histograms
}
// graphiteTimeout bounds one submission, the connect and the write together. An export is a few KB, a host that can't
// take that in this long isn't going to.
const graphiteTimeout = 30 * time.Second
// graphiteOnce performs a single submission to Graphite, returning a non-nil
// error on failed connections.
func graphiteOnce(c graphiteConfigExport) error {
// error if it could not connect or write all of it within timeout. It gives up
// as soon as ctx is done.
func graphiteOnce(ctx context.Context, c graphiteConfigExport, timeout time.Duration) error {
now := time.Now().Unix()
du := float64(c.DurationUnit)
flushSeconds := float64(c.FlushInterval) / float64(time.Second)
conn, err := net.DialTCP("tcp", nil, c.Addr)
deadline := time.Now().Add(timeout)
d := net.Dialer{Deadline: deadline}
conn, err := d.DialContext(ctx, "tcp", c.Addr.String())
if err != nil {
return err
}
defer conn.Close()
if err := conn.SetDeadline(deadline); err != nil {
return err
}
stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
defer stop()
w := bufio.NewWriter(conn)
c.Registry.Each(func(name string, i any) {
switch metric := i.(type) {
@@ -111,7 +124,7 @@ func graphiteOnce(c graphiteConfigExport) error {
fmt.Fprintf(w, "%s.%s.fifteen-minute %.2f %d\n", c.Prefix, name, t.Rate15(), now)
fmt.Fprintf(w, "%s.%s.mean-rate %.2f %d\n", c.Prefix, name, t.RateMean(), now)
}
w.Flush()
})
return nil
// A failed write sticks, so this reports the first one
return w.Flush()
}
+315
View File
@@ -0,0 +1,315 @@
package nebula
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net"
"strings"
"sync"
"testing"
"time"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// graphiteHost accepts connections and hands each one to the test
type graphiteHost struct {
ln net.Listener
conns chan net.Conn
mu sync.Mutex
closed bool
open []net.Conn
}
// newGraphiteHost accepts connections for the test. A non-zero window shrinks each connection's receive buffer, so a
// sender that is not read from blocks after a little whatever the host's TCP tuning. Only for tests that need a stall,
// a small window makes reading very slow on Linux.
func newGraphiteHost(t *testing.T, window int) *graphiteHost {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
h := &graphiteHost{ln: ln, conns: make(chan net.Conn, 8)}
done := make(chan struct{})
t.Cleanup(func() {
_ = ln.Close()
h.mu.Lock()
h.closed = true
for _, c := range h.open {
_ = c.Close()
}
h.mu.Unlock()
close(done)
})
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
h.mu.Lock()
if h.closed {
h.mu.Unlock()
_ = c.Close()
return
}
h.open = append(h.open, c)
h.mu.Unlock()
if tc, ok := c.(*net.TCPConn); ok && window > 0 {
_ = tc.SetReadBuffer(window)
}
select {
case h.conns <- c:
case <-done:
return
}
}
}()
return h
}
func (h *graphiteHost) addr(t *testing.T) *net.TCPAddr {
a, err := net.ResolveTCPAddr("tcp", h.ln.Addr().String())
require.NoError(t, err)
return a
}
// readExport reads what one send wrote, bounded so a send that never finishes fails the test rather than hanging it
func readExport(t *testing.T, c net.Conn) []byte {
t.Helper()
require.NoError(t, c.SetReadDeadline(time.Now().Add(10*time.Second)))
b, err := io.ReadAll(c)
require.NoError(t, err)
return b
}
func (h *graphiteHost) next(t *testing.T, wait time.Duration) net.Conn {
t.Helper()
select {
case c := <-h.conns:
return c
case <-time.After(wait):
t.Fatal("no connection")
return nil
}
}
// lockedBuffer is a log destination a test can read while the code under test writes to it from another goroutine
type lockedBuffer struct {
mu sync.Mutex
buf bytes.Buffer
}
func (b *lockedBuffer) Write(p []byte) (int, error) {
b.mu.Lock()
defer b.mu.Unlock()
return b.buf.Write(p)
}
func (b *lockedBuffer) String() string {
b.mu.Lock()
defer b.mu.Unlock()
return b.buf.String()
}
func testGraphiteConfig(addr *net.TCPAddr, r metrics.Registry) graphiteConfigExport {
return graphiteConfigExport{Addr: addr, Registry: r, FlushInterval: time.Second, DurationUnit: time.Nanosecond, Prefix: "nebula"}
}
// Enough metrics that one export overflows the socket buffers on any platform, so a send to a host that stops
// reading blocks
func bigGraphiteRegistry(r metrics.Registry, t *testing.T) {
for i := range 200000 {
name := fmt.Sprintf("big.gauge.%d", i)
metrics.GetOrRegisterGauge(name, r).Update(int64(i))
t.Cleanup(func() { r.Unregister(name) })
}
}
func TestGraphiteOnce_sends(t *testing.T) {
r := metrics.NewRegistry()
metrics.GetOrRegisterGauge("g", r).Update(7)
metrics.GetOrRegisterCounter("c", r).Inc(3)
h := newGraphiteHost(t, 0)
require.NoError(t, graphiteOnce(t.Context(), testGraphiteConfig(h.addr(t), r), time.Minute))
b := readExport(t, h.next(t, 5*time.Second))
assert.Contains(t, string(b), "nebula.g.value 7 ")
assert.Contains(t, string(b), "nebula.c.count 3 ")
assert.Contains(t, string(b), "nebula.c.count_ps 3.00 ")
}
// A host that accepts and never reads is given up on, and the failed write is reported
func TestGraphiteOnce_stalledHostTimesOut(t *testing.T) {
r := metrics.NewRegistry()
bigGraphiteRegistry(r, t)
h := newGraphiteHost(t, 4096)
done := make(chan error, 1)
cfg := testGraphiteConfig(h.addr(t), r)
go func() { done <- graphiteOnce(t.Context(), cfg, 300*time.Millisecond) }()
h.next(t, 5*time.Second) // accepted, never read
select {
case err := <-done:
require.Error(t, err)
case <-time.After(10 * time.Second):
t.Fatal("the send did not give up on a host that stopped reading")
}
}
// The timeout covers the whole send, a host that keeps reading but too slowly is still given up on
func TestGraphiteOnce_timeoutCoversTheWholeSend(t *testing.T) {
r := metrics.NewRegistry()
bigGraphiteRegistry(r, t)
h := newGraphiteHost(t, 0)
done := make(chan error, 1)
start := time.Now()
cfg := testGraphiteConfig(h.addr(t), r)
go func() { done <- graphiteOnce(t.Context(), cfg, 500*time.Millisecond) }()
c := h.next(t, 5*time.Second)
stopReading := make(chan struct{})
defer close(stopReading)
// 32KB every 10ms keeps every write well inside the timeout, the whole export takes seconds
go func() {
buf := make([]byte, 32<<10)
for {
select {
case <-stopReading:
return
case <-time.After(10 * time.Millisecond):
}
if _, err := c.Read(buf); err != nil {
return
}
}
}()
select {
case err := <-done:
require.Error(t, err)
assert.Less(t, time.Since(start), 5*time.Second)
case <-time.After(10 * time.Second):
t.Fatal("a host that kept reading slowly held the send past its timeout")
}
}
func TestGraphiteOnce_cancelledCtxDoesNotConnect(t *testing.T) {
h := newGraphiteHost(t, 0)
ctx, cancel := context.WithCancel(t.Context())
cancel()
require.Error(t, graphiteOnce(ctx, testGraphiteConfig(h.addr(t), metrics.NewRegistry()), time.Minute))
select {
case <-h.conns:
t.Fatal("connected on a cancelled ctx")
case <-time.After(200 * time.Millisecond):
}
}
// A stop or reload abandons a stuck send rather than waiting out its timeout
func TestGraphiteOnce_cancelAbandonsSend(t *testing.T) {
r := metrics.NewRegistry()
bigGraphiteRegistry(r, t)
h := newGraphiteHost(t, 4096)
ctx, cancel := context.WithCancel(t.Context())
done := make(chan error, 1)
cfg := testGraphiteConfig(h.addr(t), r)
go func() { done <- graphiteOnce(ctx, cfg, time.Minute) }()
h.next(t, 5*time.Second)
select {
case err := <-done:
t.Fatalf("the send should be stuck on a host that is not reading, it ended with %v", err)
case <-time.After(200 * time.Millisecond):
}
cancel()
select {
case err := <-done:
require.Error(t, err)
case <-time.After(5 * time.Second):
t.Fatal("the send is still stuck after its ctx was cancelled")
}
}
// buildRuntime hands graphite the configured prefix and interval
func TestStatsServer_graphiteExportUsesConfig(t *testing.T) {
const name = "graphite.wiring.test"
metrics.GetOrRegisterCounter(name, nil).Inc(8)
t.Cleanup(func() { metrics.DefaultRegistry.Unregister(name) })
h := newGraphiteHost(t, 0)
s, c := newTestStatsServer(t)
setStatsConfig(c, map[string]any{"type": "graphite", "interval": "4s", "prefix": "pfx", "host": h.ln.Addr().String()})
cfg, err := loadStatsConfig(c)
require.NoError(t, err)
s.runMu.Lock()
fns, _ := s.buildRuntime(t.Context(), cfg)
s.runMu.Unlock()
for _, fn := range fns {
fn()
}
b := readExport(t, h.next(t, 5*time.Second))
assert.Contains(t, string(b), "pfx."+name+".count 8 ")
assert.Contains(t, string(b), "pfx."+name+".count_ps 2.00 ")
}
func TestStatsServer_graphiteFailureIsLogged(t *testing.T) {
logs := &lockedBuffer{}
s, c := newTestStatsServer(t)
s.l = test.NewLoggerWithOutput(logs)
setStatsConfig(c, map[string]any{"type": "graphite", "interval": "1h", "host": "127.0.0.1:" + freeTCPPort(t)})
cfg, err := loadStatsConfig(c)
require.NoError(t, err)
s.runMu.Lock()
fns, _ := s.buildRuntime(t.Context(), cfg)
s.runMu.Unlock()
for _, fn := range fns {
fn()
}
assert.Contains(t, logs.String(), "Graphite export failed")
}
// Stop abandons the running export, so it goes through Start rather than handing buildRuntime a ctx, and a stop is
// not logged as a failure
func TestStatsServer_stopAbandonsGraphiteSend(t *testing.T) {
bigGraphiteRegistry(metrics.DefaultRegistry, t)
h := newGraphiteHost(t, 4096)
logs := &lockedBuffer{}
s, c := newTestStatsServer(t)
s.l = test.NewLoggerWithOutput(logs)
setStatsConfig(c, map[string]any{"type": "graphite", "interval": "50ms", "host": h.ln.Addr().String()})
require.NoError(t, s.reload(c, true))
started := make(chan struct{})
go func() {
s.Start()
close(started)
}()
stuck := h.next(t, 30*time.Second)
s.Stop()
select {
case <-started:
case <-time.After(5 * time.Second):
t.Fatal("Start did not return after Stop")
}
// A closed sender resets the connection once it sees this, one still sending just buffers it. Either way the
// read below doesn't have to drain the export through the small window to find out
_, err := stuck.Write([]byte("x"))
require.NoError(t, err)
require.NoError(t, stuck.SetReadDeadline(time.Now().Add(10*time.Second)))
n, err := io.Copy(io.Discard, stuck)
var ne net.Error
if errors.As(err, &ne) && ne.Timeout() {
t.Fatalf("the send kept the connection after Stop, read %d bytes", n)
}
assert.Less(t, n, int64(1<<20), "the send kept going after Stop")
// The capture loop logs on its own goroutine after Start has returned, give it the chance
assert.Never(t, func() bool { return strings.Contains(logs.String(), "Graphite export failed") }, 500*time.Millisecond, 10*time.Millisecond, "a stop is not a failure")
}
-16
View File
@@ -26,20 +26,4 @@ message NebulaHandshakeDetails {
uint32 CertVersion = 8;
// reserved for WIP multiport
reserved 6, 7;
// Multiport lane negotiation. Absent on hosts without multiport enabled;
// vanilla nebula treats 9 and 10 as unknown fields and skips them.
LaneDetails InitiatorLanes = 9;
LaneDetails ResponderLanes = 10;
}
// LaneDetails advertises a host's multiport lane capability. PortCount/BasePort
// describe the sender's consecutively bound UDP ports, and TxLanes is how many
// lanes the sender may send on, so the peer knows how many lane sessions it must
// be prepared to receive on.
message LaneDetails {
uint32 PortCount = 1;
uint32 BasePort = 2;
// was a per-lane handshake index
reserved 3;
uint32 TxLanes = 4;
}
-1
View File
@@ -71,7 +71,6 @@ func newTestMachine(
cs.version, cs.getCredential,
verifier, func() (uint32, error) { return localIndex, nil },
initiator, header.HandshakeIXPSK0,
nil,
)
require.NoError(t, err)
return m
+1 -34
View File
@@ -39,14 +39,6 @@ type Result struct {
HandshakeTime uint64
MessageIndex uint64 // number of messages exchanged during the handshake
Initiator bool
// Multiport lane negotiation, from the peer's LaneDetails. All zero when
// the peer did not advertise (vanilla peer or multiport disabled).
// PeerTxLanes is how many lanes the peer may send on, which is how many lane
// sessions we need in order to receive everything it sends.
PeerPortCount uint32
PeerBasePort uint32
PeerTxLanes uint32
}
// Machine drives a Noise handshake through N messages. It handles Noise
@@ -69,7 +61,6 @@ type Machine struct {
verifier CertVerifier
result *Result
msgs []msgFlags
lanes *LaneDetails // our multiport advert; nil emits a vanilla payload
myVersion cert.Version
subtype header.MessageSubType
indexAllocated bool
@@ -82,8 +73,6 @@ type Machine struct {
// the noise pattern and the per-message content layout. The credential for
// `version` is fetched via getCred and used to seed the noise.HandshakeState.
// IndexAllocator is called lazily when the first outgoing payload is built.
// lanes, when non-nil, is emitted as this side's multiport advert on every
// payload-bearing message; nil produces byte-identical vanilla payloads.
func NewMachine(
version cert.Version,
getCred GetCredentialFunc,
@@ -91,7 +80,6 @@ func NewMachine(
allocIndex IndexAllocator,
initiator bool,
subtype header.MessageSubType,
lanes *LaneDetails,
) (*Machine, error) {
info, err := subtypeInfoFor(subtype)
if err != nil {
@@ -115,7 +103,6 @@ func NewMachine(
getCred: getCred,
allocIndex: allocIndex,
verifier: verifier,
lanes: lanes,
myVersion: version,
result: &Result{
Initiator: initiator,
@@ -311,8 +298,7 @@ func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
}
// Assert the payload contains exactly what we expect
hasPayloadData := payload.InitiatorIndex != 0 || payload.ResponderIndex != 0 || payload.Time != 0 ||
payload.InitiatorLanes != nil || payload.ResponderLanes != nil
hasPayloadData := payload.InitiatorIndex != 0 || payload.ResponderIndex != 0 || payload.Time != 0
if hasPayloadData != flags.expectsPayload {
m.failed = true
return ErrUnexpectedContent
@@ -341,23 +327,6 @@ func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
m.result.RemoteIndex = remoteIndex
m.result.HandshakeTime = payload.Time
m.payloadSet = true
// Multiport advert from the peer's side of the exchange. Out-of-range
// values mean a peer we can't pair lanes with; ignore the advert
// rather than failing the handshake — the tunnel itself is fine, it
// just won't get lanes. Semantic policing (the port range, lane
// clamping) belongs to the handshake manager and newLaneSet.
var peerLanes *LaneDetails
if m.result.Initiator {
peerLanes = payload.ResponderLanes
} else {
peerLanes = payload.InitiatorLanes
}
if peerLanes != nil && peerLanes.BasePort <= 0xffff && peerLanes.PortCount <= 0xffff && peerLanes.TxLanes <= 0xffff {
m.result.PeerPortCount = peerLanes.PortCount
m.result.PeerBasePort = peerLanes.BasePort
m.result.PeerTxLanes = peerLanes.TxLanes
}
}
// Process certificate
@@ -428,11 +397,9 @@ func (m *Machine) marshalOutgoing(flags msgFlags) ([]byte, error) {
if m.result.Initiator {
p.InitiatorIndex = m.result.LocalIndex
p.InitiatorLanes = m.lanes
} else {
p.ResponderIndex = m.result.LocalIndex
p.InitiatorIndex = m.result.RemoteIndex
p.ResponderLanes = m.lanes
}
p.Time = uint64(time.Now().UnixNano())
}
-111
View File
@@ -1,111 +0,0 @@
package handshake
import (
"net/netip"
"testing"
"time"
"github.com/slackhq/nebula/cert"
ct "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/header"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// newTestLaneMachine is newTestMachine with a lane advert attached.
func newTestLaneMachine(
t *testing.T,
cs *testCertState,
verifier CertVerifier,
initiator bool,
localIndex uint32,
lanes *LaneDetails,
) *Machine {
t.Helper()
m, err := NewMachine(
cs.version, cs.getCredential,
verifier, func() (uint32, error) { return localIndex, nil },
initiator, header.HandshakeIXPSK0,
lanes,
)
require.NoError(t, err)
return m
}
func doFullLaneHandshake(t *testing.T, initLanes, respLanes *LaneDetails) (initR, respR *Result) {
t.Helper()
ca, _, caKey, _ := ct.NewTestCaCert(
cert.Version2, cert.Curve_CURVE25519, time.Time{}, time.Time{}, nil, nil, nil,
)
caPool := ct.NewTestCAPool(ca)
v := testVerifier(caPool)
initCS := newTestCertState(t, ca, caKey, "initiator", []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")})
respCS := newTestCertState(t, ca, caKey, "responder", []netip.Prefix{netip.MustParsePrefix("10.0.0.2/24")})
initM := newTestLaneMachine(t, initCS, v, true, 1000, initLanes)
respM := newTestLaneMachine(t, respCS, v, false, 2000, respLanes)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
resp, respR, err := respM.ProcessPacket(nil, msg1)
require.NoError(t, err)
require.NotNil(t, respR)
_, initR, err = initM.ProcessPacket(nil, resp)
require.NoError(t, err)
require.NotNil(t, initR)
return initR, respR
}
func TestMachineLaneAdvertBothSides(t *testing.T) {
initR, respR := doFullLaneHandshake(t,
&LaneDetails{PortCount: 8, BasePort: 4242, TxLanes: 8},
&LaneDetails{PortCount: 4, BasePort: 5353, TxLanes: 3},
)
// Each side's Result carries the peer's advert.
assert.Equal(t, uint32(4), initR.PeerPortCount)
assert.Equal(t, uint32(5353), initR.PeerBasePort)
assert.Equal(t, uint32(3), initR.PeerTxLanes)
assert.Equal(t, uint32(8), respR.PeerPortCount)
assert.Equal(t, uint32(4242), respR.PeerBasePort)
assert.Equal(t, uint32(8), respR.PeerTxLanes)
}
func TestMachineLaneAdvertAsymmetric(t *testing.T) {
// Vanilla initiator, multiport responder and vice versa: the nil side
// yields all-zero peer fields on the other end.
initR, respR := doFullLaneHandshake(t, nil, &LaneDetails{PortCount: 4, BasePort: 5353})
assert.Equal(t, uint32(4), initR.PeerPortCount)
assert.Equal(t, uint32(0), respR.PeerPortCount)
assert.Equal(t, uint32(0), respR.PeerBasePort)
initR, respR = doFullLaneHandshake(t, &LaneDetails{PortCount: 8, BasePort: 4242}, nil)
assert.Equal(t, uint32(0), initR.PeerPortCount)
assert.Equal(t, uint32(8), respR.PeerPortCount)
}
func TestMachineLaneAdvertOutOfRangeIgnored(t *testing.T) {
// A BasePort that can't be a real UDP port is ignored, not fatal.
initR, respR := doFullLaneHandshake(t,
&LaneDetails{PortCount: 8, BasePort: 70000, TxLanes: 8},
&LaneDetails{PortCount: 4, BasePort: 5353, TxLanes: 4},
)
assert.Equal(t, uint32(0), respR.PeerPortCount)
assert.Equal(t, uint32(0), respR.PeerBasePort)
assert.Equal(t, uint32(0), respR.PeerTxLanes)
// The sane side still negotiates.
assert.Equal(t, uint32(4), initR.PeerPortCount)
// An out-of-range TxLanes drops the whole advert the same way: a lane index
// that does not fit the header is as unusable as an impossible port.
initR, respR = doFullLaneHandshake(t,
&LaneDetails{PortCount: 8, BasePort: 4242, TxLanes: 0x10000},
&LaneDetails{PortCount: 4, BasePort: 5353, TxLanes: 4},
)
assert.Equal(t, uint32(0), respR.PeerPortCount)
assert.Equal(t, uint32(4), initR.PeerPortCount)
}
-2
View File
@@ -444,7 +444,6 @@ func TestMachineThreeMessagePattern(t *testing.T) {
initCS.getCredential, v,
func() (uint32, error) { return 1000, nil },
true, header.HandshakeXXPSK0,
nil,
)
require.NoError(t, err)
@@ -453,7 +452,6 @@ func TestMachineThreeMessagePattern(t *testing.T) {
respCS.getCredential, v,
func() (uint32, error) { return 2000, nil },
false, header.HandshakeXXPSK0,
nil,
)
require.NoError(t, err)
+5 -120
View File
@@ -20,40 +20,15 @@ type Payload struct {
ResponderIndex uint32
Time uint64
CertVersion uint32
// Multiport lane negotiation; nil when the sender has multiport disabled
// (which keeps the encoded payload byte-identical to a vanilla one).
InitiatorLanes *LaneDetails
ResponderLanes *LaneDetails
}
// LaneDetails advertises multiport lane capability: the contiguous UDP port
// range the sender bound, and how many lanes it may send on. The receiver needs
// PortCount/BasePort to aim its own lanes and TxLanes to know how many lane
// sessions to derive for receiving.
type LaneDetails struct {
PortCount uint32
BasePort uint32
TxLanes uint32
}
// Proto field numbers for NebulaHandshakeDetails
const (
fieldCert = 1 // bytes
fieldInitiatorIndex = 2 // uint32
fieldResponderIndex = 3 // uint32
fieldTime = 5 // uint64
fieldCertVersion = 8 // uint32
fieldInitiatorLanes = 9 // LaneDetails
fieldResponderLanes = 10 // LaneDetails
)
// Proto field numbers for LaneDetails.
// Field 3 was a per-lane handshake index and is permanently reserved.
const (
fieldLanePortCount = 1 // uint32
fieldLaneBasePort = 2 // uint32
fieldLaneTxLanes = 4 // uint32
fieldCert = 1 // bytes
fieldInitiatorIndex = 2 // uint32
fieldResponderIndex = 3 // uint32
fieldTime = 5 // uint64
fieldCertVersion = 8 // uint32
)
// MarshalPayload encodes a handshake payload in protobuf wire format compatible
@@ -82,16 +57,6 @@ func MarshalPayload(out []byte, p Payload) []byte {
details = protowire.AppendTag(details, fieldCertVersion, protowire.VarintType)
details = protowire.AppendVarint(details, uint64(p.CertVersion))
}
// Emitted last to keep the encoding in ascending field-number order, which
// is what protoc-gen-go would produce for the same message.
if p.InitiatorLanes != nil {
details = protowire.AppendTag(details, fieldInitiatorLanes, protowire.BytesType)
details = protowire.AppendBytes(details, p.InitiatorLanes.marshal(nil))
}
if p.ResponderLanes != nil {
details = protowire.AppendTag(details, fieldResponderLanes, protowire.BytesType)
details = protowire.AppendBytes(details, p.ResponderLanes.marshal(nil))
}
out = protowire.AppendTag(out, 1, protowire.BytesType)
out = protowire.AppendBytes(out, details)
@@ -99,20 +64,6 @@ func MarshalPayload(out []byte, p Payload) []byte {
return out
}
// marshal appends the LaneDetails submessage fields to out. All fields are
// emitted unconditionally: a LaneDetails is only present at all when multiport
// is negotiating, and explicit zeros keep the parser's presence semantics
// trivial.
func (d *LaneDetails) marshal(out []byte) []byte {
out = protowire.AppendTag(out, fieldLanePortCount, protowire.VarintType)
out = protowire.AppendVarint(out, uint64(d.PortCount))
out = protowire.AppendTag(out, fieldLaneBasePort, protowire.VarintType)
out = protowire.AppendVarint(out, uint64(d.BasePort))
out = protowire.AppendTag(out, fieldLaneTxLanes, protowire.VarintType)
out = protowire.AppendVarint(out, uint64(d.TxLanes))
return out
}
// UnmarshalPayload decodes a protobuf-encoded NebulaHandshake message.
func UnmarshalPayload(b []byte) (Payload, error) {
var p Payload
@@ -210,72 +161,6 @@ func unmarshalPayloadDetails(p *Payload, b []byte) error {
}
p.CertVersion = uint32(v)
b = b[n:]
case fieldInitiatorLanes:
if typ != protowire.BytesType {
return errInvalidHandshakeDetails
}
v, n := protowire.ConsumeBytes(b)
if n < 0 {
return errInvalidHandshakeDetails
}
p.InitiatorLanes = new(LaneDetails)
if err := unmarshalLaneDetails(p.InitiatorLanes, v); err != nil {
return err
}
b = b[n:]
case fieldResponderLanes:
if typ != protowire.BytesType {
return errInvalidHandshakeDetails
}
v, n := protowire.ConsumeBytes(b)
if n < 0 {
return errInvalidHandshakeDetails
}
p.ResponderLanes = new(LaneDetails)
if err := unmarshalLaneDetails(p.ResponderLanes, v); err != nil {
return err
}
b = b[n:]
default:
n := protowire.ConsumeFieldValue(num, typ, b)
if n < 0 {
return errInvalidHandshakeDetails
}
b = b[n:]
}
}
return nil
}
func unmarshalLaneDetails(d *LaneDetails, b []byte) error {
for len(b) > 0 {
num, typ, n := protowire.ConsumeTag(b)
if n < 0 {
return errInvalidHandshakeDetails
}
b = b[n:]
// Same contract as the details parser: known fields hard-fail on a
// wire-type mismatch, unknown fields are skipped, repeated singular
// fields follow proto3 last-wins.
switch num {
case fieldLanePortCount, fieldLaneBasePort, fieldLaneTxLanes:
if typ != protowire.VarintType {
return errInvalidHandshakeDetails
}
v, n := protowire.ConsumeVarint(b)
if n < 0 || v > math.MaxUint32 {
return errInvalidHandshakeDetails
}
switch num {
case fieldLanePortCount:
d.PortCount = uint32(v)
case fieldLaneBasePort:
d.BasePort = uint32(v)
case fieldLaneTxLanes:
d.TxLanes = uint32(v)
}
b = b[n:]
default:
n := protowire.ConsumeFieldValue(num, typ, b)
if n < 0 {
+12 -138
View File
@@ -117,134 +117,23 @@ func TestPayloadUnknownFields(t *testing.T) {
assert.Equal(t, uint32(88), got.ResponderIndex)
})
t.Run("unknown field inside LaneDetails is skipped", func(t *testing.T) {
var lane []byte
lane = protowire.AppendTag(lane, fieldLanePortCount, protowire.VarintType)
lane = protowire.AppendVarint(lane, 4)
lane = protowire.AppendTag(lane, 50, protowire.VarintType) // unknown subfield
lane = protowire.AppendVarint(lane, 9999)
lane = protowire.AppendTag(lane, fieldLaneBasePort, protowire.VarintType)
lane = protowire.AppendVarint(lane, 4242)
t.Run("reserved fields 6 and 7 are skipped", func(t *testing.T) {
// Fields 6 and 7 are reserved in the proto definition
var details []byte
details = protowire.AppendTag(details, fieldInitiatorIndex, protowire.VarintType)
details = protowire.AppendVarint(details, 100)
details = protowire.AppendTag(details, fieldInitiatorLanes, protowire.BytesType)
details = protowire.AppendBytes(details, lane)
details = protowire.AppendTag(details, 6, protowire.VarintType)
details = protowire.AppendVarint(details, 1)
details = protowire.AppendTag(details, 7, protowire.VarintType)
details = protowire.AppendVarint(details, 2)
got, err := UnmarshalPayload(wrapDetails(details))
var data []byte
data = protowire.AppendTag(data, 1, protowire.BytesType)
data = protowire.AppendBytes(data, details)
got, err := UnmarshalPayload(data)
require.NoError(t, err)
assert.Equal(t, uint32(100), got.InitiatorIndex)
require.NotNil(t, got.InitiatorLanes)
assert.Equal(t, uint32(4), got.InitiatorLanes.PortCount)
assert.Equal(t, uint32(4242), got.InitiatorLanes.BasePort)
})
}
func TestPayloadLaneDetails(t *testing.T) {
t.Run("round trip both sides", func(t *testing.T) {
data := MarshalPayload(nil, Payload{
InitiatorIndex: 12345,
Time: 999,
InitiatorLanes: &LaneDetails{PortCount: 8, BasePort: 4242, TxLanes: 3},
ResponderLanes: &LaneDetails{PortCount: 4, BasePort: 5353},
})
got, err := UnmarshalPayload(data)
require.NoError(t, err)
require.NotNil(t, got.InitiatorLanes)
assert.Equal(t, LaneDetails{PortCount: 8, BasePort: 4242, TxLanes: 3}, *got.InitiatorLanes)
require.NotNil(t, got.ResponderLanes)
assert.Equal(t, LaneDetails{PortCount: 4, BasePort: 5353}, *got.ResponderLanes)
})
t.Run("zero-valued LaneDetails survives the round trip", func(t *testing.T) {
// Presence is what negotiation keys on; an all-zero advert must not
// decay to nil.
data := MarshalPayload(nil, Payload{
InitiatorIndex: 1,
InitiatorLanes: &LaneDetails{},
})
got, err := UnmarshalPayload(data)
require.NoError(t, err)
require.NotNil(t, got.InitiatorLanes)
assert.Equal(t, LaneDetails{}, *got.InitiatorLanes)
assert.Nil(t, got.ResponderLanes)
})
t.Run("nil lanes marshal byte-identical to a vanilla payload", func(t *testing.T) {
p := Payload{
Cert: []byte("cert"),
CertVersion: 2,
InitiatorIndex: 100,
Time: 999,
}
// The vanilla encoding of the same fields, built by hand in field order.
var details []byte
details = protowire.AppendTag(details, fieldCert, protowire.BytesType)
details = protowire.AppendBytes(details, p.Cert)
details = protowire.AppendTag(details, fieldInitiatorIndex, protowire.VarintType)
details = protowire.AppendVarint(details, uint64(p.InitiatorIndex))
details = protowire.AppendTag(details, fieldTime, protowire.VarintType)
details = protowire.AppendVarint(details, p.Time)
details = protowire.AppendTag(details, fieldCertVersion, protowire.VarintType)
details = protowire.AppendVarint(details, uint64(p.CertVersion))
assert.Equal(t, wrapDetails(details), MarshalPayload(nil, p))
})
t.Run("lane field with wrong wire type rejected", func(t *testing.T) {
for _, field := range []protowire.Number{fieldInitiatorLanes, fieldResponderLanes} {
var details []byte
details = protowire.AppendTag(details, field, protowire.VarintType)
details = protowire.AppendVarint(details, 1)
_, err := UnmarshalPayload(wrapDetails(details))
assert.Error(t, err)
}
})
t.Run("lane subfield with wrong wire type rejected", func(t *testing.T) {
var lane []byte
lane = protowire.AppendTag(lane, fieldLanePortCount, protowire.BytesType)
lane = protowire.AppendBytes(lane, []byte{1, 2, 3})
var details []byte
details = protowire.AppendTag(details, fieldInitiatorLanes, protowire.BytesType)
details = protowire.AppendBytes(details, lane)
_, err := UnmarshalPayload(wrapDetails(details))
assert.Error(t, err)
})
t.Run("truncated LaneDetails submessage rejected", func(t *testing.T) {
var details []byte
details = protowire.AppendTag(details, fieldInitiatorLanes, protowire.BytesType)
details = append(details, 0x0a, 0x01, 0x02) // length 10, only 2 bytes
_, err := UnmarshalPayload(wrapDetails(details))
assert.Error(t, err)
})
t.Run("truncated varint inside LaneDetails rejected", func(t *testing.T) {
var lane []byte
lane = protowire.AppendTag(lane, fieldLaneBasePort, protowire.VarintType)
lane = append(lane, 0x80) // incomplete varint
var details []byte
details = protowire.AppendTag(details, fieldResponderLanes, protowire.BytesType)
details = protowire.AppendBytes(details, lane)
_, err := UnmarshalPayload(wrapDetails(details))
assert.Error(t, err)
})
t.Run("lane subfield varint overflow rejected", func(t *testing.T) {
var lane []byte
lane = protowire.AppendTag(lane, fieldLaneTxLanes, protowire.VarintType)
lane = protowire.AppendVarint(lane, math.MaxUint32+1)
var details []byte
details = protowire.AppendTag(details, fieldInitiatorLanes, protowire.BytesType)
details = protowire.AppendBytes(details, lane)
_, err := UnmarshalPayload(wrapDetails(details))
assert.Error(t, err)
})
}
@@ -439,12 +328,6 @@ func FuzzPayload(f *testing.F) {
Time: 3,
CertVersion: 2,
}))
f.Add(MarshalPayload(nil, Payload{
InitiatorIndex: 1,
Time: 3,
InitiatorLanes: &LaneDetails{PortCount: 8, BasePort: 4242, TxLanes: 2},
ResponderLanes: &LaneDetails{PortCount: 4, BasePort: 5353},
}))
f.Add([]byte{})
f.Add([]byte{0xff})
@@ -474,14 +357,5 @@ func payloadsEqual(a, b Payload) bool {
a.InitiatorIndex == b.InitiatorIndex &&
a.ResponderIndex == b.ResponderIndex &&
a.Time == b.Time &&
a.CertVersion == b.CertVersion &&
laneDetailsEqual(a.InitiatorLanes, b.InitiatorLanes) &&
laneDetailsEqual(a.ResponderLanes, b.ResponderLanes)
}
func laneDetailsEqual(a, b *LaneDetails) bool {
if a == nil || b == nil {
return a == b
}
return *a == *b
a.CertVersion == b.CertVersion
}
+13 -80
View File
@@ -50,19 +50,12 @@ type HandshakeConfig struct {
retries int64
triggerBuffer int
// Multiport lane parameters; laneCount == 0 means multiport is disabled.
// laneCount includes implicit lane 0 (the base tunnel), so lanes
// 1..laneCount-1 may carry traffic. lanePortCount/laneBasePort describe our
// own bound port range and are advertised in every handshake payload.
laneCount int
lanePortCount uint16
laneBasePort uint16
messageMetrics *MessageMetrics
}
type HandshakeManager struct {
// Mutex for interacting with the vpnIps and indexes maps
// Mutex for interacting with the vpnIps and indexes maps.
// Lock order is hostmap, then this lock, then the lighthouse, then a RemoteList.
sync.RWMutex
vpnIps map[netip.Addr]*HandshakeHostInfo
@@ -267,14 +260,6 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
"style": header.SubTypeName(header.Handshake, hh.machine.Subtype()),
}
// Get a remotes object if we don't already have one.
// This is mainly to protect us as this should never be the case
// NB ^ This comment doesn't jive. It's how the thing gets initialized.
// It's the common path. Should it update every time, in case a future LH query/queries give us more info?
if hostinfo.remotes == nil {
hostinfo.remotes = hm.lightHouse.QueryCache([]netip.Addr{vpnIp})
}
remotes := hostinfo.remotes.CopyAddrs(hm.mainHostMap.GetPreferredRanges())
remotesHaveChanged := !slices.Equal(remotes, hh.lastRemotes)
@@ -379,6 +364,9 @@ func (hm *HandshakeManager) StartHandshake(vpnAddr netip.Addr, cacheCb func(*Han
hostinfo := &HostInfo{
vpnAddrs: []netip.Addr{vpnAddr},
HandshakePacket: make(map[uint8][]byte, 0),
// Attached here so a pending hostinfo never exists without one: SetRemote dereferences
// it, and create-tunnel -address can reach SetRemote before the handshake loop runs.
remotes: hm.lightHouse.QueryCache([]netip.Addr{vpnAddr}),
relayState: RelayState{
relays: nil,
relayForByAddr: map[netip.Addr]*Relay{},
@@ -543,9 +531,6 @@ func (hm *HandshakeManager) DeleteHostInfo(hostinfo *HostInfo) {
func (hm *HandshakeManager) unlockedDeleteHostInfo(hostinfo *HostInfo) {
for _, addr := range hostinfo.vpnAddrs {
// Only delete the pending entry if it is actually ours: an
// unconditional delete could evict a concurrently pending handshake for
// the same address.
if cur, ok := hm.vpnIps[addr]; ok && cur.hostinfo == hostinfo {
delete(hm.vpnIps, addr)
}
@@ -683,7 +668,6 @@ func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
v, cs.GetCredential,
hm.certVerifier(), func() (uint32, error) { return hm.allocateIndex(hh) },
true, header.HandshakeIXPSK0,
hm.laneAdvert(),
)
if err != nil {
hm.f.l.Error("Failed to create handshake machine",
@@ -707,42 +691,6 @@ func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
return true
}
// laneAdvert returns our multiport advert for a handshake payload, or nil
// when multiport is disabled (which keeps the payload byte-identical to
// vanilla).
func (hm *HandshakeManager) laneAdvert() *handshake.LaneDetails {
if hm.config.laneCount == 0 {
return nil
}
return &handshake.LaneDetails{
PortCount: uint32(hm.config.lanePortCount),
BasePort: uint32(hm.config.laneBasePort),
TxLanes: uint32(hm.config.laneCount),
}
}
// maybeAllocLanes sets up the multiport lanes for a just-completed tunnel. Must
// run before the hostinfo becomes visible in the hostmap: the data plane reads
// hostinfo.lanes without synchronizing on it. The sessions themselves are derived
// later, on the first packet that needs each one.
func (hm *HandshakeManager) maybeAllocLanes(hostinfo *HostInfo, result *handshake.Result) {
if hm.config.laneCount == 0 || result.PeerPortCount == 0 {
return
}
if len(hm.f.myVpnAddrs) == 0 || len(hostinfo.vpnAddrs) == 0 {
return
}
if !validLaneAdvert(result.PeerBasePort, result.PeerPortCount) {
// Lane targets are computed in uint16, so a range like this would wrap
// onto low ports. The tunnel itself is fine; it just gets no lanes.
hostinfo.logger(hm.l).Warn("Ignoring multiport advert with an invalid port range",
"peerBasePort", result.PeerBasePort, "peerPorts", result.PeerPortCount)
return
}
hostinfo.lanes = newLaneSet(result, hm.config.laneCount, hm.f.myVpnAddrs[0], hostinfo.vpnAddrs[0])
}
// beginHandshake handles an incoming handshake packet that doesn't match any
// existing pending handshake. It creates a new responder Machine and processes
// the first message.
@@ -761,7 +709,6 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
v, cs.GetCredential,
hm.certVerifier(), func() (uint32, error) { return generateIndex(f.l) },
false, header.HandshakeIXPSK0,
hm.laneAdvert(),
)
if err != nil {
f.l.Error("Failed to create handshake machine", "from", via, "error", err)
@@ -818,10 +765,6 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
},
}
// Lanes are allocated before the log line so it can report what was actually
// negotiated, and must in any case be in place before CheckAndComplete below.
hm.maybeAllocLanes(hostinfo, result)
msg := "Handshake message received"
if !anyVpnAddrsInCommon {
msg = "Handshake message received, but no vpnNetworks in common."
@@ -836,7 +779,6 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
"initiatorIndex", result.RemoteIndex,
"responderIndex", result.LocalIndex,
"handshake", m{"stage": uint64(machine.MessageIndex()), "style": header.SubTypeName(header.Handshake, machine.Subtype())},
laneLogAttr(hm.config.laneCount, hostinfo.lanes),
)
// packet aliases the listener's incoming buffer, so this copy must stay.
@@ -1000,25 +942,18 @@ func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostIn
hm.DeleteHostInfo(hostinfo)
hm.StartHandshake(hostinfo.vpnAddrs[0], func(newHH *HandshakeHostInfo) {
newHH.hostinfo.remotes = hostinfo.remotes
newHH.hostinfo.remotes.BlockRemote(via)
newHH.packetStore = hh.packetStore
hh.packetStore = []*cachedPacket{}
hostinfo.vpnAddrs = vpnAddrs
f.sendCloseTunnel(hostinfo)
})
// sendCloseTunnel may take the hostmap lock, so it must not run under the handshake
// manager lock that StartHandshake holds around the callback.
hostinfo.vpnAddrs = vpnAddrs
f.sendCloseTunnel(hostinfo)
return
}
duration := time.Since(hh.startTime).Nanoseconds()
hostinfo.vpnAddrs = vpnAddrs
hostinfo.buildNetworks(f.myVpnNetworksTable, remoteCert.Certificate)
// Lanes are allocated before the log line so it can report what was actually
// negotiated, and must in any case be in place before Complete below.
hm.maybeAllocLanes(hostinfo, result)
msg := "Handshake message received"
if !anyVpnAddrsInCommon {
msg = "Handshake message received, but no vpnNetworks in common."
@@ -1035,9 +970,11 @@ func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostIn
"handshake", m{"stage": uint64(machine.MessageIndex()), "style": header.SubTypeName(header.Handshake, machine.Subtype())},
"durationNs", duration,
"sentCachedPackets", len(hh.packetStore),
laneLogAttr(hm.config.laneCount, hostinfo.lanes),
)
hostinfo.vpnAddrs = vpnAddrs
hostinfo.buildNetworks(f.myVpnNetworksTable, remoteCert.Certificate)
hm.Complete(hostinfo, f)
if len(hh.packetStore) > 0 {
@@ -1146,11 +1083,7 @@ func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hos
if !via.IsRelayed {
fields := append(logFields, "from", via)
// Reply from the socket the handshake arrived on so the initiator sees
// the source port it targeted. Identical to f.outside under vanilla
// config (all writers share one port); under multiport it keeps the reply on
// the base port, from whichever socket of its group the handshake reached.
err := f.writers[via.SockIdx].WriteTo(msg, via.UdpAddr)
err := f.outside.WriteTo(msg, via.UdpAddr)
if err != nil {
f.l.Error("Failed to send handshake message", append(fields, "error", err)...)
} else {
+68 -2
View File
@@ -11,6 +11,7 @@ import (
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func Test_NewHandshakeManagerVpnIp(t *testing.T) {
@@ -42,8 +43,6 @@ func Test_NewHandshakeManagerVpnIp(t *testing.T) {
i2 := blah.StartHandshake(ip, nil)
assert.Same(t, i, i2)
i.remotes = NewRemoteList([]netip.Addr{}, nil)
// Adding something to pending should not affect the main hostmap
assert.Empty(t, mainHM.Hosts)
@@ -235,3 +234,70 @@ func TestHandleIncomingDispatch(t *testing.T) {
assert.Empty(t, hm.indexes, "orphan stage-2 must not create state")
})
}
// Test_PendingHandshakeHasRemotes covers create-tunnel -address: a pending hostinfo must already
// carry its RemoteList when StartHandshake runs its callback, so SetRemote there, before the
// handshake loop has run, is safe and the address becomes one the handshake is sent to.
func Test_PendingHandshakeHasRemotes(t *testing.T) {
l := test.NewLogger()
mainHM := newHostMap(l)
preferredRanges := []netip.Prefix{}
mainHM.preferredRanges.Store(&preferredRanges)
hm := NewHandshakeManager(l, mainHM, newTestLighthouse(), &udp.NoopConn{}, defaultHandshakeConfig)
vpnIp := netip.MustParseAddr("172.1.1.2")
remote := netip.MustParseAddrPort("192.168.1.9:4242")
hi := hm.StartHandshake(vpnIp, func(hh *HandshakeHostInfo) {
require.NotNil(t, hh.hostinfo.remotes, "a pending handshake must never be without its RemoteList")
hh.hostinfo.SetRemote(remote)
})
assert.Contains(t, hi.remotes.CopyAddrs(preferredRanges), remote, "the handshake must be sent to the supplied remote")
assert.Equal(t, remote, hi.GetRemote())
// A RemoteList keeps one learned address per host, so a second remote replaces the
// first — the same behaviour change-remote has on an established tunnel.
other := netip.MustParseAddrPort("192.168.1.10:4242")
hi.SetRemote(other)
assert.Equal(t, []netip.AddrPort{other}, hi.remotes.CopyAddrs(preferredRanges))
assert.Equal(t, other, hi.GetRemote())
}
// newRemotesRaceFixture builds a handshake manager with an established tunnel to vpnAddr whose
// RemoteList is the lighthouse's cached list, the state every teardown-during-handshake race starts from.
func newRemotesRaceFixture(t *testing.T, vpnAddr netip.Addr) (*HandshakeManager, *Interface, *HostInfo) {
t.Helper()
l := test.NewLogger()
mainHM := newHostMap(l)
preferredRanges := []netip.Prefix{}
mainHM.preferredRanges.Store(&preferredRanges)
lh := newTestLighthouse()
hm := NewHandshakeManager(l, mainHM, lh, &udp.NoopConn{}, defaultHandshakeConfig)
f := &Interface{hostMap: mainHM, lightHouse: lh, handshakeManager: hm, l: l}
hm.f = f
established := &HostInfo{
vpnAddrs: []netip.Addr{vpnAddr},
localIndexId: 1,
remoteIndexId: 101,
remotes: lh.QueryCache([]netip.Addr{vpnAddr}),
HandshakePacket: map[uint8][]byte{},
}
mainHM.Lock()
mainHM.unlockedAddHostInfo(established, f)
mainHM.Unlock()
return hm, f, established
}
// With no handshake pending, tearing down the last tunnel still evicts the list.
func Test_CloseTunnelEvictsWithoutPendingHandshake(t *testing.T) {
vpnAddr := netip.MustParseAddr("172.1.1.2")
_, f, established := newRemotesRaceFixture(t, vpnAddr)
f.closeTunnel(established)
assert.NotContains(t, f.hostMap.Hosts, vpnAddr)
assert.NotContains(t, f.lightHouse.addrMap, vpnAddr)
}
+20 -51
View File
@@ -8,34 +8,27 @@ import (
)
//Version 1 header:
// 0 31
// |------------------------------------------------------------------------------------|
// | Version (uint4) | Type (uint4) | Subtype (uint8) | Reserved (uint8) | Lane (uint8) | 32
// |------------------------------------------------------------------------------------|
// | Remote index (uint32) | 64
// |------------------------------------------------------------------------------------|
// | Message counter | 96
// | (uint64) | 128
// |------------------------------------------------------------------------------------|
// | payload... |
//
// Lane is the multiport lane index, carved out of the low 8 bits of what was a
// single Reserved (uint16) before multiport. Lane 0 is the base tunnel, which is
// what every sender that does not know about lanes emits and what every non-lane
// packet — handshakes, lighthouse, relays, close — carries, so the field is
// compatible in both directions with a peer that has never heard of it. The
// remaining 8 bits stay reserved and are always sent as zero. The H struct still
// holds the pair as one Reserved field; use H.Lane and EncodeLane to reach the
// low byte.
//
// Lane is part of the AEAD's associated data, so a lane index cannot be altered
// in flight: a packet decrypts on the lane it claims or not at all.
// 0 31
// |-----------------------------------------------------------------------|
// | Version (uint4) | Type (uint4) | Subtype (uint8) | Reserved (uint16) | 32
// |-----------------------------------------------------------------------|
// | Remote index (uint32) | 64
// |-----------------------------------------------------------------------|
// | Message counter | 96
// | (uint64) | 128
// |-----------------------------------------------------------------------|
// | payload... |
type m = map[string]any
const (
Version uint8 = 1
Len = 16
// MaxOverhead is the most a payload grows on its way to the underlay.
// A relayed packet has two headers and two AEAD tags, one for the inner tunnel and another for the relay tunnel.
// Every supported cipher (today) has a 16 byte tag.
MaxOverhead = 2 * (Len + 16)
)
type MessageType uint8
@@ -69,18 +62,8 @@ const (
const (
TestRequest MessageSubType = 0
TestReply MessageSubType = 1
// LaneProbe is sent on a multiport lane to prove the lane's 5-tuple is
// usable; LaneProbeAck answers it on the base tunnel.
LaneProbe MessageSubType = 2
LaneProbeAck MessageSubType = 3
)
// MaxLane is the largest lane index the header can carry.
const MaxLane = 0xff
// laneMask covers the bits of Reserved that hold the lane index.
const laneMask uint16 = 0x00ff
const (
HandshakeIXPSK0 MessageSubType = 0
HandshakeXXPSK0 MessageSubType = 1
@@ -89,10 +72,8 @@ const (
var ErrHeaderTooShort = errors.New("header is too short")
var subTypeTestMap = map[MessageSubType]string{
TestRequest: "testRequest",
TestReply: "testReply",
LaneProbe: "laneProbe",
LaneProbeAck: "laneProbeAck",
TestRequest: "testRequest",
TestReply: "testReply",
}
var subTypeNoneMap = map[MessageSubType]string{0: "none"}
@@ -124,16 +105,10 @@ type H struct {
// Encode uses the provided byte array to encode the provided header values into.
// Byte array must be capped higher than HeaderLen or this will panic
func Encode(b []byte, v uint8, t MessageType, st MessageSubType, ri uint32, c uint64) []byte {
return EncodeLane(b, v, t, st, ri, c, 0)
}
// EncodeLane is Encode with an explicit multiport lane index, which is carried
// in the low 8 bits of Reserved.
func EncodeLane(b []byte, v uint8, t MessageType, st MessageSubType, ri uint32, c uint64, lane uint8) []byte {
b = b[:Len]
b[0] = v<<4 | byte(t&0x0f)
b[1] = byte(st)
binary.BigEndian.PutUint16(b[2:4], uint16(lane))
binary.BigEndian.PutUint16(b[2:4], 0)
binary.BigEndian.PutUint32(b[4:8], ri)
binary.BigEndian.PutUint64(b[8:16], c)
return b
@@ -166,13 +141,7 @@ func (h *H) Encode(b []byte) ([]byte, error) {
return nil, errors.New("nil header")
}
return EncodeLane(b, h.Version, h.Type, h.Subtype, h.RemoteIndex, h.MessageCounter, h.Lane()), nil
}
// Lane returns the multiport lane index carried in Reserved. Lane 0 is the base
// tunnel, which is what any sender that does not know about lanes will report.
func (h *H) Lane() uint8 {
return uint8(h.Reserved & laneMask)
return Encode(b, h.Version, h.Type, h.Subtype, h.RemoteIndex, h.MessageCounter), nil
}
// Parse is a helper function to parses given bytes into new Header struct
@@ -232,7 +201,7 @@ func IsValidSubType(t MessageType, s MessageSubType) bool {
case Handshake:
return s == HandshakeIXPSK0
case Test:
return s == TestReply || s == TestRequest || s == LaneProbe || s == LaneProbeAck
return s == TestReply || s == TestRequest
case Control, CloseTunnel, RecvError, LightHouse:
return s == 0
default:
+1 -25
View File
@@ -52,28 +52,6 @@ func TestParse(t *testing.T) {
}
}
func TestEncodeLane(t *testing.T) {
b := EncodeLane(make([]byte, Len), Version, Message, MessageNone, 10, 9, 3)
assert.Equal(t, []byte{0x11, 0x0, 0x0, 0x3}, b[:4])
h := &H{}
require.NoError(t, h.Parse(b))
assert.Equal(t, uint16(3), h.Reserved)
assert.Equal(t, uint8(3), h.Lane())
// Encode is EncodeLane on the base tunnel, and the H method round trips the lane.
assert.Equal(t,
EncodeLane(make([]byte, Len), Version, Message, MessageNone, 10, 9, 0),
Encode(make([]byte, Len), Version, Message, MessageNone, 10, 9))
rt, err := h.Encode(make([]byte, Len))
require.NoError(t, err)
assert.Equal(t, b, rt)
// Only the low 8 bits of Reserved are the lane.
assert.Equal(t, uint8(0x2a), (&H{Reserved: 0xff2a}).Lane())
}
func TestTypeName(t *testing.T) {
assert.Equal(t, "test", TypeName(Test))
assert.Equal(t, "test", (&H{Type: Test}).TypeName())
@@ -149,9 +127,7 @@ func TestIsValidSubType(t *testing.T) {
assert.True(t, IsValidSubType(Test, TestRequest))
assert.True(t, IsValidSubType(Test, TestReply))
assert.True(t, IsValidSubType(Test, LaneProbe))
assert.True(t, IsValidSubType(Test, LaneProbeAck))
assert.False(t, IsValidSubType(Test, 4))
assert.False(t, IsValidSubType(Test, 2))
// These types only ever carry subtype 0.
for _, mt := range []MessageType{Control, CloseTunnel, RecvError, LightHouse} {
+7 -76
View File
@@ -6,7 +6,6 @@ import (
"errors"
"fmt"
"log/slog"
"net"
"net/netip"
"slices"
"sync"
@@ -18,7 +17,6 @@ import (
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/logging"
)
const defaultPromoteEvery = 1000 // Count of packets sent before we try moving a tunnel to a preferred underlay ip address
@@ -278,11 +276,6 @@ type HostInfo struct {
// This value will be behind against actual tunnel utilization in the hot path.
// This should only be used by the ConnectionManagers ticker routine.
lastUsed time.Time
// lanes holds this tunnel's multiport lane sessions. Allocated when the
// handshake completes if both sides advertised multiport, nil otherwise.
// Immutable once the hostinfo is published to the data plane.
lanes *laneSet
}
type ViaSender struct {
@@ -290,11 +283,6 @@ type ViaSender struct {
relayHI *HostInfo // relayHI is the host info object of the relay
relay *Relay // relay contains the rest of the relay information, including the PeerIP of the host trying to communicate with us.
IsRelayed bool // IsRelayed is true if the packet was sent through a relay
// SockIdx is the local socket (Interface.writers index) the packet
// arrived on. Replies that must originate from the same 4-tuple egress
// f.writers[SockIdx].
SockIdx int
}
func (v ViaSender) String() string {
@@ -328,9 +316,9 @@ type cachedPacketMetrics struct {
func NewHostMapFromConfig(l *slog.Logger, c *config.C) *HostMap {
hm := newHostMap(l)
hm.reload(c, true)
c.RegisterReloadCallback(func(c *config.C) {
hm.reload(c, false)
hm.reload(c, true, nil)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
hm.reload(c, false, r)
})
l.Info("Main HostMap created", "preferredRanges", hm.GetPreferredRanges())
@@ -349,7 +337,8 @@ func newHostMap(l *slog.Logger) *HostMap {
}
}
func (hm *HostMap) reload(c *config.C, initial bool) {
// r is nil on the initial load
func (hm *HostMap) reload(c *config.C, initial bool, r *config.Reload) {
if initial || c.HasChanged("preferred_ranges") {
var preferredRanges []netip.Prefix
rawPreferredRanges := c.GetStringSlice("preferred_ranges", []string{})
@@ -360,8 +349,9 @@ func (hm *HostMap) reload(c *config.C, initial bool) {
if err != nil {
hm.l.Warn("Failed to parse preferred ranges, ignoring",
"error", err,
"range", rawPreferredRanges,
"range", rawPreferredRange,
)
r.Failed()
continue
}
@@ -484,9 +474,6 @@ func (hm *HostMap) unlockedMakePrimary(hostinfo *HostInfo) bool {
// any tunnel to the peer), which the caller uses to decide whether to clear learned lighthouse
// state and disestablish relays.
func (hm *HostMap) unlockedDeleteHostInfo(hostinfo *HostInfo) bool {
// Lane sessions hang off this hostinfo, so deleting it takes them with it
// and there is nothing extra to unwind here.
// Remove this hostinfo from each of its address lists. The lists are independent, so a
// sibling is never promoted to an address it does not own and no other list is touched.
final := true
@@ -943,59 +930,3 @@ func (i *HostInfo) logger(l *slog.Logger) *slog.Logger {
return li
}
// Utility functions
func localAddrs(l *slog.Logger, allowList *LocalAllowList) []netip.Addr {
//FIXME: This function is pretty garbage
var finalAddrs []netip.Addr
ifaces, _ := net.Interfaces()
for _, i := range ifaces {
allow := allowList.AllowName(i.Name)
if l.Enabled(context.Background(), logging.LevelTrace) {
l.Log(context.Background(), logging.LevelTrace, "localAllowList.AllowName",
"interfaceName", i.Name,
"allow", allow,
)
}
if !allow {
continue
}
addrs, _ := i.Addrs()
for _, rawAddr := range addrs {
var addr netip.Addr
switch v := rawAddr.(type) {
case *net.IPNet:
//continue
addr, _ = netip.AddrFromSlice(v.IP)
case *net.IPAddr:
addr, _ = netip.AddrFromSlice(v.IP)
}
if !addr.IsValid() {
if l.Enabled(context.Background(), slog.LevelDebug) {
l.Debug("addr was invalid", "localAddr", rawAddr)
}
continue
}
addr = addr.Unmap()
if addr.IsLoopback() == false && addr.IsLinkLocalUnicast() == false {
isAllowed := allowList.Allow(addr)
if l.Enabled(context.Background(), logging.LevelTrace) {
l.Log(context.Background(), logging.LevelTrace, "localAllowList.Allow",
"localAddr", addr,
"allowed", isAllowed,
)
}
if !isAllowed {
continue
}
finalAddrs = append(finalAddrs, addr)
}
}
}
return finalAddrs
}
+13
View File
@@ -447,3 +447,16 @@ func TestHostInfo_markOutOnly(t *testing.T) {
assert.True(t, h.markOut(6), "a relayed send must not consume the epoch edge")
assert.False(t, h.markOut(6))
}
// A preferred range that doesn't parse is skipped and the rest are used, the reload says one was skipped
func TestHostMap_reloadBadPreferredRangeFails(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("preferred_ranges: [10.0.0.0/8]\n"))
hm := NewHostMapFromConfig(l, c)
res := c.ReloadConfigString("preferred_ranges: [10.0.0.0/8, nope]\n")
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Equal(t, []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")}, hm.GetPreferredRanges())
}
+46 -86
View File
@@ -10,12 +10,14 @@ import (
"github.com/slackhq/nebula/firewall"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/iputil"
"github.com/slackhq/nebula/logging"
"github.com/slackhq/nebula/noiseutil"
"github.com/slackhq/nebula/overlay/batch"
"github.com/slackhq/nebula/overlay/tio"
"github.com/slackhq/nebula/routing"
)
func (f *Interface) consumeInsidePacket(pkt tio.Packet, fwPacket *firewall.ParsedPacket, nb []byte, tx *txQueue, rejectBuf []byte, q int, localCache firewall.ConntrackCache) {
func (f *Interface) consumeInsidePacket(pkt tio.Packet, fwPacket *firewall.ParsedPacket, nb []byte, sendBatch *batch.SendBatch, rejectBuf []byte, q int, localCache firewall.ConntrackCache) {
// borrowed: pkt.Bytes is owned by the originating tio.Queue and is
// only valid until the next Read on that queue. Every consumer below
// (parse, self-forward, handshake cache, sendInsideMessage) reads it
@@ -110,7 +112,7 @@ func (f *Interface) consumeInsidePacket(pkt tio.Packet, fwPacket *firewall.Parse
dropReason := f.firewall.Drop(fwPacket.Packet, false, hostinfo, f.pki.GetCAPool(), localCache)
if dropReason == nil {
f.sendInsideMessage(hostinfo, pkt, &fwPacket.Packet, nb, tx)
f.sendInsideMessage(hostinfo, pkt, nb, sendBatch)
} else {
f.rejectInside(packet, rejectBuf, q)
if f.l.Enabled(context.Background(), slog.LevelDebug) {
@@ -122,13 +124,13 @@ func (f *Interface) consumeInsidePacket(pkt tio.Packet, fwPacket *firewall.Parse
}
}
func (f *Interface) sendInsideEncrypt(hostinfo *HostInfo, ci *ConnectionState, lane uint8, seg, scratch, nb []byte) []byte {
func (f *Interface) sendInsideEncrypt(hostinfo *HostInfo, ci *ConnectionState, seg, scratch, nb []byte) []byte {
if noiseutil.EncryptLockNeeded {
ci.writeLock.Lock()
}
c := ci.messageCounter.Add(1)
out := header.EncodeLane(scratch, header.Version, header.Message, 0, hostinfo.remoteIndexId, c, lane)
out := header.Encode(scratch, header.Version, header.Message, 0, hostinfo.remoteIndexId, c)
out, encErr := ci.eKey.EncryptDanger(out, out, seg, c, nb)
if noiseutil.EncryptLockNeeded {
@@ -153,19 +155,12 @@ func (f *Interface) sendInsideEncrypt(hostinfo *HostInfo, ci *ConnectionState, l
// kernel-supplied superpacket bytes never get written into a separate
// scratch arena: SegmentSuperpacket builds each segment's plaintext in
// segScratch[:segLen] in turn, and we encrypt directly into a fresh SendBatch slot.
//
// When this flow has a usable multiport lane to this peer, the direct path swaps
// to that lane's session and socket below. Relay and base traffic stays on
// tx.base, this routine's socket on the base port.
func (f *Interface) sendInsideMessage(hostinfo *HostInfo, pkt tio.Packet, fwPacket *firewall.Packet, nb []byte, tx *txQueue) {
func (f *Interface) sendInsideMessage(hostinfo *HostInfo, pkt tio.Packet, nb []byte, sendBatch *batch.SendBatch) {
ci := hostinfo.ConnectionState
if ci.eKey == nil {
return
}
// Base and relay traffic stays on the base port; the direct path may swap to tx.lane below.
sendBatch := tx.base
// One traffic-out mark covers every segment of the superpacket; doing it
// per segment in sendInsideEncrypt paid an atomic store up to ~45 extra
// times per TSO packet, inside writeLock under boring crypto.
@@ -204,10 +199,10 @@ func (f *Interface) sendInsideMessage(hostinfo *HostInfo, pkt tio.Packet, fwPack
}
err = tio.SegmentSuperpacket(pkt, func(seg []byte) error {
//relay header + header + plaintext + AEAD tag (16 bytes for both AES-GCM and ChaCha20-Poly1305) + relay tag
scratch := sendBatch.Reserve(header.Len + header.Len + len(seg) + 16 + 16)
//relay header + header + plaintext + AEAD tag + relay tag
scratch := sendBatch.Reserve(len(seg) + header.MaxOverhead)
innerPacket := f.sendInsideEncrypt(hostinfo, ci, 0, seg, scratch[header.Len:], nb)
innerPacket := f.sendInsideEncrypt(hostinfo, ci, seg, scratch[header.Len:], nb)
if innerPacket == nil {
return nil
}
@@ -228,28 +223,11 @@ func (f *Interface) sendInsideMessage(hostinfo *HostInfo, pkt tio.Packet, fwPack
return
}
// Direct path: prefer this flow's multiport lane once it is proven usable.
// txLaneForFlow hands back the lane's session and destination together, so
// there is no window where one is set and the other is not, and a demotion
// drops us back onto the base tunnel on the very next packet.
//
// A miss is also how a lane gets re-probed after a demotion: txLane raises
// demand, which the connection manager's next tick on this tunnel picks up.
// Until the lane is up the traffic rides the base tunnel, the same fallback
// a demoted lane uses.
lane := uint8(0)
if s, lci, laneRemote := hostinfo.lanes.txLaneForFlow(fwPacket); lci != nil {
lane = uint8(s)
ci = lci
remote = laneRemote
sendBatch = tx.laneBatch(f, s)
}
err := tio.SegmentSuperpacket(pkt, func(seg []byte) error {
// header + plaintext + AEAD tag (16 bytes for both AES-GCM and ChaCha20-Poly1305)
scratch := sendBatch.Reserve(header.Len + len(seg) + 16)
out := f.sendInsideEncrypt(hostinfo, ci, lane, seg, scratch, nb)
out := f.sendInsideEncrypt(hostinfo, ci, seg, scratch, nb)
if out == nil {
return nil
}
@@ -538,61 +516,39 @@ func (f *Interface) SendVia(via *HostInfo, relay *Relay, ad, nb, out []byte, noc
return
}
err = f.writers[f.egressSock(q)].WriteTo(toSend, via.GetRemote())
err = f.writers[q].WriteTo(toSend, via.GetRemote())
if err != nil {
via.logger(f.l).Info("Failed to WriteTo in sendVia", "error", err)
if l, ok := logging.RateLimited(f.l, f.udpRelayTxErrLimiter); ok {
via.logger(l).Info("Failed to WriteTo in sendVia", "error", err)
}
}
}
// egressSock picks the socket a tunnel packet leaves from.
//
// Everything that is not lane data plane leaves from the base port: handshakes, keepalives, close packets, rejects and
// relay carriers all belong to the base tunnel's 4-tuple, which is the only one a peer's spoof/roam checks and a
// vanilla peer's expectations know about. Lane data goes through laneSock instead and never comes here.
//
// Which socket on the base port doesn't matter — they share an address, so they produce identical packets — so keep to
// this routine's own share of the group and leave the rest of it uncontended. Without multiport that is q itself, since
// every socket is on the base port.
func (f *Interface) egressSock(q int) int {
return f.laneSock(q, 0)
}
// laneSock returns the index in writers of a socket bound to lane s's port, for a
// routine that reads queue q.
//
// Under multiport the sockets are laid out port-major — writers[s*routinesPerPort
// + r] is the r'th socket on port listen.port+s — so every routine has a sibling
// socket on every port and the arithmetic is a lane index away. Routines pick the
// sibling matching their own position in their group, which spreads the writers
// for one port over that port's whole group rather than funnelling them onto its
// first socket. It is a pure function of (q, s), so a flow always leaves from the
// same socket and cannot reorder itself across two of them.
//
// Without multiport there is one port and every socket is on it, so any lane
// resolves to q's own socket.
func (f *Interface) laneSock(q, s int) int {
if !f.multiport {
return q
}
return s*f.routinesPerPort + q%f.routinesPerPort
}
func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType, ci *ConnectionState, hostinfo *HostInfo, remote netip.AddrPort, p, nb, out []byte, q int) {
if ci.eKey == nil {
return
}
q = f.egressSock(q)
useRelay := !remote.IsValid() && !hostinfo.GetRemote().IsValid()
fullOut := out
// The encrypt appends to out in place and a relayed packet is then wrapped in place,
// so out has to fit the whole packet up front or the encrypt reallocates out from under SendVia.
// We cap the MTU of tun devices and routes we create ourselves, so only a host that set a bigger MTU than we allow gets here.
need := header.Len + len(p) + ci.eKey.Overhead()
if useRelay {
if len(out) < header.Len {
// out always has a capacity of mtu, but not always a length greater than the header.Len.
// Grow it to make sure the next operation works.
out = out[:header.Len]
// The relay tunnel runs the same cipher, and adds its own header and tag
need += header.Len + ci.eKey.Overhead()
}
if cap(out) < need {
if l, ok := logging.RateLimited(f.l, f.txOversizeLimiter); ok {
hostinfo.logger(l).Error("Dropping outbound packet, out buffer not large enough", "outCap", cap(out), "payloadLen", len(p), "relayed", useRelay)
}
return
}
fullOut := out
if useRelay {
// Save a header's worth of data at the front of the 'out' buffer.
out = out[header.Len:]
out = out[header.Len:header.Len]
}
if noiseutil.EncryptLockNeeded {
@@ -611,9 +567,9 @@ func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType
//l.WithField("trace", string(debug.Stack())).Error("out Header ", &Header{Version, t, st, 0, hostinfo.remoteIndexId, c}, p)
out = header.Encode(out, header.Version, t, st, hostinfo.remoteIndexId, c)
// A closing tunnel is torn down right after this, so skip the connection manager entirely: no point recording
// traffic or asking the lighthouse for a punch. Lane probe acks are not traffic either. Otherwise, if we rebound
// since this tunnel last sent, ask the lighthouse to get the far side punching at us again.
if t != header.CloseTunnel && !isLaneProbe(t, st) && f.connectionManager.Out(hostinfo) {
// traffic or asking the lighthouse for a punch. Otherwise, if we rebound since this tunnel last sent, ask the
// lighthouse to get the far side punching at us again.
if t != header.CloseTunnel && f.connectionManager.Out(hostinfo) {
f.lightHouse.QueryServer(hostinfo.vpnAddrs[0])
if f.l.Enabled(context.Background(), slog.LevelDebug) {
f.l.Debug("Lighthouse update triggered for punch due to rebind epoch",
@@ -639,18 +595,22 @@ func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType
if remote.IsValid() {
err = f.writers[q].WriteTo(out, remote)
if err != nil {
hostinfo.logger(f.l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", remote,
)
if l, ok := logging.RateLimited(f.l, f.udpTxErrLimiter); ok {
hostinfo.logger(l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", remote,
)
}
}
} else if hr := hostinfo.GetRemote(); hr.IsValid() {
err = f.writers[q].WriteTo(out, hr)
if err != nil {
hostinfo.logger(f.l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", hr,
)
if l, ok := logging.RateLimited(f.l, f.udpTxErrLimiter); ok {
hostinfo.logger(l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", hr,
)
}
}
} else {
// Try to send via a relay
+69 -179
View File
@@ -24,6 +24,8 @@ import (
"github.com/slackhq/nebula/overlay"
"github.com/slackhq/nebula/overlay/batch"
"github.com/slackhq/nebula/overlay/tio"
"github.com/slackhq/nebula/ratelimit"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/udp"
)
@@ -43,22 +45,10 @@ type InterfaceConfig struct {
DropLocalBroadcast bool
DropMulticast bool
routines int
// Multiport means the sockets are spread over a range of ports
// (listen.port+slot) rather than all sharing listen.port, and that lane
// sessions are negotiated with capable peers.
Multiport bool
// RoutinesPerPort is how many sockets share each port under multiport, and so
// the stride between port slots in writers: writers[s*RoutinesPerPort+r] is
// the r'th socket bound to listen.port+s. It is `routines` as configured,
// while routines above is that times the number of ports.
RoutinesPerPort int
// LaneCount is the number of lanes counting the base tunnel as lane 0
// (multiport.lanes, clamped to the number of ports bound).
LaneCount int
MessageMetrics *MessageMetrics
version string
relayManager *relayManager
punchy *Punchy
MessageMetrics *MessageMetrics
version string
relayManager *relayManager
punchy *Punchy
tryPromoteEvery uint32
reQueryEvery uint32
@@ -99,9 +89,6 @@ type Interface struct {
dropLocalBroadcast bool
dropMulticast bool
routines int
multiport bool
routinesPerPort int
laneCount int
disconnectInvalid atomic.Bool
closed atomic.Bool
// cpuAffinity, when non-empty, names the CPUs each TUN reader goroutine
@@ -139,6 +126,10 @@ type Interface struct {
batchers []*batch.MultiCoalescer
wg sync.WaitGroup
udpTxErrLimiter *ratelimit.Limiter // udp direct transmit error log limiter
udpRelayTxErrLimiter *ratelimit.Limiter // udp relay transmit error log limiter
txOversizeLimiter *ratelimit.Limiter // oversized outbound packet drop log limiter
// fatalErr holds the first unexpected reader error that caused shutdown.
// nil means "no fatal error" (yet)
fatalErr atomic.Pointer[error]
@@ -213,6 +204,19 @@ func NewInterface(ctx context.Context, c *InterfaceConfig) (*Interface, error) {
return nil, errors.New("no connection manager")
}
udpTxErrLimiter, err := ratelimit.New(1, time.Second)
if err != nil {
return nil, err
}
udpRelayTxErrLimiter, err := ratelimit.New(1, time.Second)
if err != nil {
return nil, err
}
txOversizeLimiter, err := ratelimit.New(1, time.Second)
if err != nil {
return nil, err
}
if c.routines <= 1 {
c.PinThreads = false //pinning is not useful unless there's more than one tun reader
}
@@ -232,9 +236,6 @@ func NewInterface(ctx context.Context, c *InterfaceConfig) (*Interface, error) {
dropLocalBroadcast: c.DropLocalBroadcast,
dropMulticast: c.DropMulticast,
routines: c.routines,
multiport: c.Multiport,
routinesPerPort: max(c.RoutinesPerPort, 1),
laneCount: c.LaneCount,
version: c.version,
writers: make([]udp.Conn, c.routines),
batchers: make([]*batch.MultiCoalescer, c.routines),
@@ -248,6 +249,9 @@ func NewInterface(ctx context.Context, c *InterfaceConfig) (*Interface, error) {
conntrackCacheTimeout: c.ConntrackCacheTimeout,
cpuAffinity: c.CpuAffinity,
pinThreads: c.PinThreads,
udpTxErrLimiter: udpTxErrLimiter,
udpRelayTxErrLimiter: udpRelayTxErrLimiter,
txOversizeLimiter: txOversizeLimiter,
metricHandshakes: metrics.GetOrRegisterHistogram("handshakes", nil, metrics.NewExpDecaySample(1028, 0.015)),
metricTxDropped: metrics.GetOrRegisterCounter("udp.tx.dropped", nil),
@@ -265,6 +269,9 @@ func NewInterface(ctx context.Context, c *InterfaceConfig) (*Interface, error) {
ifce.reQueryWait.Store(int64(c.reQueryWait))
ifce.connectionManager.intf = ifce
if ifce.firewall != nil {
ifce.firewall.siblings = ifce
}
// Held until Close so waiting on the interface blocks until the resources are actually released
ifce.wg.Add(1)
@@ -294,9 +301,7 @@ func (f *Interface) activate() error {
"fips140Enforced", fips140.Enforced(),
)
// Under multiport, main.go already probed this capability and turned
// multiport off where it is missing, so there is nothing to fall back from.
if f.routines > 1 && !f.multiport && !f.outside.SupportsMultipleReaders() {
if f.routines > 1 && !f.outside.SupportsMultipleReaders() {
f.routines = 1
f.l.Warn("multiple udp readers are not supported on this platform, falling back to a single routine")
}
@@ -309,11 +314,6 @@ func (f *Interface) activate() error {
return err
}
if len(queues) < f.routines {
if f.multiport {
// The lane sockets are already bound one-per-routine; shrinking
// the routine count would leave bound ports with no reader.
return fmt.Errorf("multiport requires %d tun queues, device provided %d", f.routines, len(queues))
}
// TODO: this clamp is only safe because it is unreachable when the
// udp side has multiple readers (linux Queues opens exactly n or
// errors; every other platform already clamped routines to 1 above).
@@ -414,7 +414,7 @@ func (f *Interface) listenOut(i int) {
rxc := newRxContext(f, i)
listener := func(fromUdpAddr netip.AddrPort, payload []byte) {
f.readOutsidePackets(ViaSender{UdpAddr: fromUdpAddr, SockIdx: i}, payload, rxc)
f.readOutsidePackets(ViaSender{UdpAddr: fromUdpAddr}, payload, rxc)
}
flusher := func() {
@@ -456,114 +456,6 @@ func (f *Interface) pinThisThread(i int) {
}
}
// txQueue is the per-routine TX state owned by one listenIn goroutine.
//
// base carries base-session data, relay carriers, and everything on a tunnel
// without lanes. It goes out a socket on the base port — egressSock's pick — since
// base traffic must keep the base source port or a vanilla peer would see it move
// and roam-thrash. lane[s] goes out a socket on listen.port+s and carries traffic
// encrypted with lane s's session; lane[0] is base, and the rest are built on the
// first packet that picks them, since a routine that never sends on a lane should
// not hold a batch for it. Both come from laneSock, so a routine writes to its own
// share of each port's socket group.
//
// Which lane a packet rides comes from its own flow hash, not from this
// routine's index. That is deliberate. Which routine reads a flow is the
// kernel's decision: it hashes the flow to a tun queue, but it also *learns*
// the queue we write that flow's inbound packets to, and prefers what it
// learned. So if the lane followed the routine, a peer whose lanes were still
// down — every peer, for the first moments of a tunnel — would write all of its
// inbound traffic to queue 0, teaching both kernels to steer every flow to
// queue 0, and every tunnel would collapse onto lane 0 and stay there for as
// long as its flows kept busy. Hashing here makes lane spread independent of
// tun steering entirely.
//
// Every batch borrows arena, so a routine holding a batch per lane still costs
// one slab. The arena is reset by flush once every batch over it is drained.
//
// Several routines can still write to one socket — the sockets on a port are
// shared by the routines whose lane arithmetic lands on them — which the underlay
// serializes (see batchWriter). Per-flow wire order still holds: a flow is hashed
// onto one lane and read by one routine, so nothing else is writing it.
type txQueue struct {
// q is the queue this state belongs to, which laneSock needs to resolve a lane
// to one of its port's sockets.
q int
base *batch.SendBatch
lane []*batch.SendBatch
arena *batch.Arena
// live is every batch built so far, in build order, so base is first: see
// flush. Kept as its own slice because lane is mostly nil holes and both
// full and flush walk this per read batch.
live []txBatch
}
// txBatch is a live batch and the index in writers of the socket it flushes to.
type txBatch struct {
sb *batch.SendBatch
sock int
}
func (f *Interface) newTxQueue(q int) *txQueue {
baseSock := f.egressSock(q)
arena := batch.NewArena(batch.SendBatchCap * (udp.MTU + 32))
base := batch.NewSendBatchSharedArena(f.writers[baseSock], batch.SendBatchCap, arena)
tx := &txQueue{
q: q,
base: base,
arena: arena,
live: []txBatch{{sb: base, sock: baseSock}},
}
if f.multiport && f.laneCount > 1 {
tx.lane = make([]*batch.SendBatch, f.laneCount)
tx.lane[0] = base
}
return tx
}
// laneBatch returns the batch for lane s, building it the first time this
// routine sends on that lane. Lanes this queue doesn't cover fall back to base,
// which is also lane 0's batch.
func (tx *txQueue) laneBatch(f *Interface, s int) *batch.SendBatch {
if s <= 0 || s >= len(tx.lane) {
return tx.base
}
sb := tx.lane[s]
if sb == nil {
sock := f.laneSock(tx.q, s)
sb = batch.NewSendBatchSharedArena(f.writers[sock], batch.SendBatchCap, tx.arena)
tx.lane[s] = sb
tx.live = append(tx.live, txBatch{sb: sb, sock: sock})
}
return sb
}
// full reports a full sendmmsg worth of work queued across every lane, rather
// than on any one of them: the arena is shared, so it is the total that bounds
// how much is outstanding.
func (tx *txQueue) full() bool {
n := 0
for _, b := range tx.live {
n += b.sb.Len()
}
return n >= batch.SendBatchCap
}
// flush drains base before the lanes so that when a flow moves from the base
// session onto a freshly promoted lane mid-window, its packets still leave this
// host in encryption order. Resetting the shared arena is this queue's job,
// since no single batch's Flush can know the others are done with it.
func (tx *txQueue) flush(f *Interface) {
for _, b := range tx.live {
if b.sb.Len() > 0 {
f.flushSendBatch(b.sb, b.sock)
}
}
tx.arena.Reset()
}
func (f *Interface) listenIn(queue tio.Queue, i int) {
// Pinning this thread (and goroutine) to a single CPU keeps every sendmmsg from this goroutine going through the
// same TX ring on the nic, so the wire sees per-flow order. Skip entirely when tun.pin_threads is false.
@@ -572,7 +464,8 @@ func (f *Interface) listenIn(queue tio.Queue, i int) {
}
rejectBuf := make([]byte, mtu)
tx := f.newTxQueue(i)
arenaSize := batch.SendBatchCap * (udp.MTU + 32)
sb := batch.NewSendBatch(f.writers[i], batch.SendBatchCap, arenaSize)
fwPacket := &firewall.ParsedPacket{}
nb := make([]byte, 12, 12)
@@ -590,15 +483,15 @@ func (f *Interface) listenIn(queue tio.Queue, i int) {
}
for _, pkt := range pkts {
f.consumeInsidePacket(pkt, fwPacket, nb, tx, rejectBuf, i, conntrackCache.Get())
f.consumeInsidePacket(pkt, fwPacket, nb, sb, rejectBuf, i, conntrackCache.Get())
// Flush incrementally once a full sendmmsg batch has
// accumulated so the first packets of a deep read drain
// hit the wire while the rest are still being encrypted.
if tx.full() {
tx.flush(f)
if sb.Len() >= batch.SendBatchCap {
f.flushSendBatch(sb, i)
}
}
tx.flush(f)
f.flushSendBatch(sb, i)
}
f.l.Debug("overlay reader is done", "reader", i)
@@ -620,10 +513,11 @@ func (f *Interface) flushSendBatch(sb *batch.SendBatch, q int) {
func (f *Interface) RegisterConfigChangeCallbacks(c *config.C) {
c.RegisterReloadCallback(f.reloadFirewall)
c.RegisterReloadCallback(f.reloadSendRecvError)
c.RegisterReloadCallback(f.reloadAcceptRecvError)
c.RegisterReloadCallback(f.reloadDisconnectInvalid)
c.RegisterReloadCallback(f.reloadMisc)
for _, reload := range []func(*config.C){
f.reloadSendRecvError, f.reloadAcceptRecvError, f.reloadDisconnectInvalid, f.reloadMisc,
} {
c.RegisterReloadCallback(func(c *config.C, _ *config.Reload) { reload(c) })
}
for _, udpConn := range f.writers {
c.RegisterReloadCallback(udpConn.ReloadConfig)
@@ -640,7 +534,7 @@ func (f *Interface) reloadDisconnectInvalid(c *config.C) {
}
}
func (f *Interface) reloadFirewall(c *config.C) {
func (f *Interface) reloadFirewall(c *config.C, r *config.Reload) {
cs := f.pki.getCertState()
curCert := cs.getCertificate(cert.Version2)
if curCert == nil {
@@ -659,10 +553,12 @@ func (f *Interface) reloadFirewall(c *config.C) {
fw, err := NewFirewallFromConfig(f.l, cs, c)
if err != nil {
f.l.Error("Error while creating firewall during reload", "error", err)
r.Failed()
return
}
oldFw := f.firewall
fw.siblings = f
conntrack := oldFw.Conntrack
conntrack.Lock()
defer conntrack.Unlock()
@@ -756,33 +652,19 @@ func (f *Interface) reloadMisc(c *config.C) {
}
}
func (f *Interface) emitStats(ctx context.Context, i time.Duration) {
ticker := time.NewTicker(i)
defer ticker.Stop()
// statsEmitter updates the interface's gauges, the stats server runs it on its capture loop.
func (f *Interface) statsEmitter() func() {
udpStats := udp.NewUDPStatsEmitter(f.writers)
certExpirationGauge := metrics.GetOrRegisterGauge("certificate.ttl_seconds", nil)
certInitiatingVersion := metrics.GetOrRegisterGauge("certificate.initiating_version", nil)
certMaxVersion := metrics.GetOrRegisterGauge("certificate.max_version", nil)
// Registered only when we run multiport, so these don't sit at zero on a node
// that was never going to have a lane and read as a broken feature.
var lanesUpGauge, laneTunnelsGauge metrics.Gauge
if f.multiport && f.laneCount > 1 {
lanesUpGauge = metrics.GetOrRegisterGauge("multiport.lanes.up", nil)
laneTunnelsGauge = metrics.GetOrRegisterGauge("multiport.lanes.tunnels", nil)
}
emit := func() {
return func() {
f.firewall.EmitStats()
f.handshakeManager.EmitStats()
udpStats()
if lanesUpGauge != nil {
f.emitLaneStats(lanesUpGauge, laneTunnelsGauge)
}
certState := f.pki.getCertState()
defaultCrt := certState.GetDefaultCertificate()
certExpirationGauge.Update(int64(defaultCrt.NotAfter().Sub(time.Now()) / time.Second))
@@ -795,19 +677,6 @@ func (f *Interface) emitStats(ctx context.Context, i time.Duration) {
certMaxVersion.Update(int64(certState.v1Cert.Version()))
}
}
// Prime gauges so a Prometheus scrape that lands before the first tick
// sees real values instead of the zero defaults (issue #907).
emit()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
emit()
}
}
}
func (f *Interface) GetHostInfo(vpnIp netip.Addr) *HostInfo {
@@ -847,3 +716,24 @@ func (f *Interface) Close() error {
f.wg.Done()
return errors.Join(errs...)
}
// siblingPeers implements gatewaySiblings from the routes and tunnels we have now. Routes name a gateway by any of its
// vpn addresses, conntrack keys a peer by its first one, so each gateway is resolved through its tunnel. A gateway
// we have no tunnel to can't be carrying a live flow.
func (f *Interface) siblingPeers(remote netip.Addr, h *HostInfo, buf []netip.Addr) []netip.Addr {
gateways := f.inside.RoutesFor(remote)
isGateway := func(g routing.Gateway) bool { return slices.Contains(h.vpnAddrs, g.Addr()) }
if len(gateways) < 2 || !slices.ContainsFunc(gateways, isGateway) {
return buf
}
for _, g := range gateways {
if isGateway(g) {
continue
}
if s := f.hostMap.QueryVpnAddr(g.Addr()); s != nil && s != h {
buf = append(buf, s.vpnAddrs[0])
}
}
return buf
}
+6 -15
View File
@@ -3,7 +3,6 @@
package nebula
import (
"context"
"net/netip"
"testing"
"time"
@@ -18,11 +17,8 @@ import (
"github.com/stretchr/testify/require"
)
// Test_emitStats_primesGauges covers issue #907: a Prometheus scrape that
// landed before the first ticker fire used to read 0 for the cert gauges.
// emitStats now primes the gauges before entering the ticker loop. We assert
// the gauge is zero before the first call and non-zero after.
func Test_emitStats_primesGauges(t *testing.T) {
// Priming before the first tick (issue #907) is Start's job, see TestStatsServer_Start_primes.
func Test_statsEmitter_setsGauges(t *testing.T) {
defer metrics.DefaultRegistry.UnregisterAll()
l := test.NewLogger()
@@ -43,7 +39,7 @@ func Test_emitStats_primesGauges(t *testing.T) {
hostMap: hostMap,
inside: &overlaytest.NoopTun{},
outside: &udp.NoopConn{},
firewall: &Firewall{Conntrack: &FirewallConntrack{Conns: map[firewall.Packet]*conn{}}},
firewall: &Firewall{Conntrack: &FirewallConntrack{Conns: map[firewall.ConntrackKey]*conn{}}},
lightHouse: lh,
pki: &PKI{},
handshakeManager: NewHandshakeManager(l, hostMap, lh, &udp.NoopConn{}, defaultHandshakeConfig),
@@ -56,17 +52,12 @@ func Test_emitStats_primesGauges(t *testing.T) {
ifce.pki.cs.Store(cs)
ttlGauge := metrics.GetOrRegisterGauge("certificate.ttl_seconds", nil)
require.Zero(t, ttlGauge.Value(), "gauge should be zero before emitStats runs")
require.Zero(t, ttlGauge.Value(), "gauge should be zero before the emitter runs")
// Pre-cancel the context so emitStats returns after priming the gauges
// without ever reading from ticker.C. The one hour interval is just a
// belt-and-suspenders, the test does not expect the ticker to fire.
ctx, cancel := context.WithCancel(context.Background())
cancel()
ifce.emitStats(ctx, time.Hour)
ifce.statsEmitter()()
ttl := ttlGauge.Value()
assert.Positive(t, ttl, "ttl gauge should be primed by emitStats before its first tick")
assert.Positive(t, ttl, "ttl gauge should be set by one run of the emitter")
assert.LessOrEqual(t, ttl, int64(3600))
assert.Equal(t, int64(cert.Version1), metrics.GetOrRegisterGauge("certificate.initiating_version", nil).Value())
assert.Equal(t, int64(cert.Version1), metrics.GetOrRegisterGauge("certificate.max_version", nil).Value())
+68 -4
View File
@@ -4,8 +4,11 @@ import (
"net/netip"
"testing"
"github.com/gaissmai/bart"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/overlay/overlaytest"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -65,10 +68,10 @@ func TestReloadFirewall_CertUnsafeNetworksChanged(t *testing.T) {
pki.cs.Store(&CertState{v2Cert: c2, initiatingVersion: cert.Version2})
// Reload with the same YAML so HasChanged("firewall") reports false.
require.NoError(t, cfg.ReloadConfigString(rawYAML))
require.NoError(t, cfg.ReloadConfigString(rawYAML).Err)
require.False(t, cfg.HasChanged("firewall"))
f.reloadFirewall(cfg)
f.reloadFirewall(cfg, nil)
assert.NotSame(t, fw, f.firewall, "firewall pointer should have been replaced")
assert.Equal(t, newUnsafe, f.firewall.unsafeNetworks)
@@ -113,8 +116,69 @@ func TestReloadFirewall_NoChange(t *testing.T) {
l: l,
}
require.NoError(t, cfg.ReloadConfigString(rawYAML))
f.reloadFirewall(cfg)
require.NoError(t, cfg.ReloadConfigString(rawYAML).Err)
f.reloadFirewall(cfg, nil)
assert.Same(t, fw, f.firewall, "firewall should not have been replaced")
}
// Rules that don't parse keep the old firewall, and the reload says it could not apply them
func TestReloadFirewall_BadRulesFailTheReload(t *testing.T) {
l := test.NewLogger()
pki := &PKI{}
pki.cs.Store(&CertState{v2Cert: &dummyCert{version: cert.Version2, networks: []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")}}, initiatingVersion: cert.Version2})
rules := func(proto string) string {
return "firewall:\n outbound:\n - port: any\n proto: any\n host: any\n inbound:\n - port: any\n proto: " + proto + "\n host: any\n"
}
cfg := config.NewC(l)
require.NoError(t, cfg.LoadString(rules("any")))
fw, err := NewFirewallFromConfig(l, pki.getCertState(), cfg)
require.NoError(t, err)
f := &Interface{pki: pki, firewall: fw, l: l}
cfg.RegisterReloadCallback(f.reloadFirewall)
res := cfg.ReloadConfigString(rules("nope"))
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Same(t, fw, f.firewall)
}
// routesTun is a NoopTun with a route table
type routesTun struct {
overlaytest.NoopTun
routes *bart.Table[routing.Gateways]
}
func (t routesTun) RoutesFor(ip netip.Addr) routing.Gateways {
r, _ := t.routes.Lookup(ip)
return r
}
func TestInterface_SiblingPeers(t *testing.T) {
addr := netip.MustParseAddr
routes := new(bart.Table[routing.Gateways])
// a and b carry 198.51.100.0/24, the route names b by its v6 address
routes.Insert(netip.MustParsePrefix("198.51.100.0/24"), routing.Gateways{
routing.NewGateway(addr("192.0.2.2"), 1),
routing.NewGateway(addr("fd00::3"), 1),
routing.NewGateway(addr("192.0.2.5"), 1), // no tunnel to this one
})
routes.Insert(netip.MustParsePrefix("203.0.113.0/24"), routing.Gateways{routing.NewGateway(addr("192.0.2.2"), 1)})
hm := newHostMap(test.NewLogger())
a := &HostInfo{vpnAddrs: []netip.Addr{addr("192.0.2.2"), addr("fd00::2")}, localIndexId: 1}
b := &HostInfo{vpnAddrs: []netip.Addr{addr("192.0.2.3"), addr("fd00::3")}, localIndexId: 2}
c := &HostInfo{vpnAddrs: []netip.Addr{addr("192.0.2.4")}, localIndexId: 3}
for _, h := range []*HostInfo{a, b, c} {
hm.unlockedAddHostInfo(h, &Interface{})
}
f := &Interface{inside: routesTun{routes: routes}, hostMap: hm}
// Siblings come back as their first vpn address, the one conntrack keys on, whichever address the route names
assert.Equal(t, []netip.Addr{addr("192.0.2.3")}, f.siblingPeers(addr("198.51.100.5"), a, nil))
assert.Equal(t, []netip.Addr{addr("192.0.2.2")}, f.siblingPeers(addr("198.51.100.5"), b, nil))
assert.Empty(t, f.siblingPeers(addr("198.51.100.5"), c, nil), "not one of our gateways for the network")
assert.Empty(t, f.siblingPeers(addr("203.0.113.5"), a, nil), "a single gateway route has no siblings")
assert.Empty(t, f.siblingPeers(addr("192.0.2.3"), a, nil), "no route")
}
+3 -13
View File
@@ -45,8 +45,7 @@ func setTransportChecksum4(packet []byte) {
if ihl < ipv4.HeaderLen || end < ihl || end > len(packet) {
return
}
// The checksum covers the whole datagram, which a fragment (MF set or a
// non-zero offset) does not carry.
// do not attempt to set checksums for fragments
if binary.BigEndian.Uint16(packet[6:8])&0x3fff != 0 {
return
}
@@ -68,10 +67,7 @@ func setTransportChecksum6(packet []byte) {
return
}
// The checksum covers the whole datagram, which a fragment does not carry.
// An unknown extension header hides where the transport header starts. A
// chain longer than the walk's budget ends it early, at an offset that was
// never checked against the packet.
// we must find the transport header in order to write a checksum
proto, offset, _, anyFragment, err := IPv6FindUpperProtocol(packet[:end])
if err != nil || anyFragment || offset >= end {
return
@@ -109,8 +105,7 @@ func transportExtent(transport []byte, proto uint8) ([]byte, bool) {
// writeTransportChecksum stores the checksum of transport, taken over the
// pseudo-header sum csum, in the header's checksum field. A UDP checksum that
// computes to zero goes on the wire as 0xffff: zero means no checksum was
// computed (RFC 768), and over IPv6 the checksum is mandatory (RFC 8200
// section 8.1).
// computed (RFC 768), and over IPv6 the checksum is mandatory (RFC 8200 section 8.1).
func writeTransportChecksum(transport []byte, proto uint8, csum uint32) {
var at, minLen int
switch proto {
@@ -133,11 +128,6 @@ func writeTransportChecksum(transport []byte, proto uint8, csum uint32) {
binary.BigEndian.PutUint16(transport[at:], sum)
}
// fold reduces a pseudo-header sum to the 16 bit seed Checksum takes. Carrying
// the high half back into the low half is what keeps the reduction lossless, so
// the seed sums exactly as the wider value would; 0xffff is its fixed point.
// Every term of that sum comes from a 16 bit field, so it stays far below the
// width at which the accumulator would wrap.
func fold(csum uint32) uint16 {
for csum > 0xffff {
csum = (csum >> 16) + (csum & 0xffff)
+13 -16
View File
@@ -345,14 +345,11 @@ func ipv6CreateRejectTCPPacket(packet []byte, out []byte, offset int) []byte {
return out
}
// IPv6FindUpperProtocol walks the ipv6 extension header chain and returns the upper layer protocol, the
// offset it begins at, and whether the packet is a non-first fragment. Only the RFC 8200 and IANA extension
// headers below are walked. Everything else, including Mobility (135), HIP (139), Shim6 (140), experimental
// 253/254, and real upper layer protocols like SCTP or GRE, is terminal. Walking those as extension headers
// is a firewall bypass, so they fail closed. For a non-first fragment the returned protocol is the fragmented
// protocol and offset points at the fragment header, there is no transport header to locate. Returns
// ErrIPv6CouldNotFindPayload if packet is smaller than an ipv6 header or the chain is truncated before a
// terminal protocol is reached.
// IPv6FindUpperProtocol walks the ipv6 extension header chain and returns the upper layer protocol, the offset it begins at, and whether the packet is a non-first fragment.
// Only the RFC 8200 and IANA extension headers below are walked.
// Everything else, including Mobility (135), HIP (139), Shim6 (140), experimental 253/254, and real upper layer protocols like SCTP or GRE, is terminal.
// For a non-first fragment the returned protocol is the fragmented protocol and offset points at the fragment header, there is no transport header to locate.
// Returns ErrIPv6CouldNotFindPayload if packet is smaller than an ipv6 header, or the chain is truncated, or if more than 8 extension headers are present.
func IPv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragment bool, anyFragment bool, err error) {
const maxIPv6ExtHeaders = 8
if len(packet) < ipv6.HeaderLen {
@@ -365,14 +362,14 @@ func IPv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragm
switch nextHeader {
case 0, 43, 60: // Hop-by-Hop, Routing, Destination
if len(packet) < offset+2 {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
nextHeader = packet[offset]
offset += (int(packet[offset+1]) + 1) << 3
case 44: // Fragment
if len(packet) < offset+8 {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
anyFragment = true
// Non-first fragments carry no transport header, report the fragmented protocol and stop
@@ -384,21 +381,21 @@ func IPv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragm
case 51: // AH
if len(packet) < offset+2 {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
nextHeader = packet[offset]
offset += (int(packet[offset+1]) + 2) << 2
default:
// A prior extension header can declare a length that advances offset past the packet. The terminal
// protocol's header isn't actually here, so treat the chain as truncated rather than classifying it.
if offset > len(packet) {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
// A prior extension header can declare a length that advances offset past the packet, reject this.
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
return nextHeader, offset, isFragment, anyFragment, nil
return nextHeader, offset, isFragment, anyFragment, nil //success!
}
}
return nextHeader, offset, isFragment, anyFragment, nil
// Fail closed
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
func CreateICMPEchoResponse(packet, out []byte) []byte {
+17 -5
View File
@@ -529,6 +529,17 @@ func Test_IPv6FindUpperProtocol(t *testing.T) {
firstFragToUDP := []byte{17, 0, 0, 1, 0, 0, 0, 1} // frag offset 0, M=1, next = UDP
nonFirstFrag := []byte{17, 0, 0, 9, 0, 0, 0, 1} // frag offset non-zero, next = UDP
transport := []byte{0, 80, 1, 187, 0, 0, 0, 0} // stand-in bytes, IPv6FindUpperProtocol never reads ports
extToHbH := []byte{0, 0, 0, 0, 0, 0, 0, 0} // len 0 -> 8 bytes, next = Hop-by-Hop
// 8 walked headers spend the budget before the 9th slot's terminal is ever classified. This is the
// report 4040546 shape: without failing closed the walk hands back offset 104, and a caller reads ports
// from the attacker's bytes there.
var budgetChain []byte
for range 7 {
budgetChain = append(budgetChain, extToHbH...)
}
budgetChain = append(budgetChain, extToTCP...)
budgetChain = append(budgetChain, transport...)
tests := []struct {
name string
@@ -549,9 +560,10 @@ func Test_IPv6FindUpperProtocol(t *testing.T) {
{"first fragment walks to transport", 44, append(firstFragToUDP, transport...), 17, ipv6.HeaderLen + 8, false, true, nil},
{"non-first fragment stops", 44, append(nonFirstFrag, transport...), 17, ipv6.HeaderLen, true, true, nil},
{"unknown protocol is terminal", 132, transport, 132, ipv6.HeaderLen, false, false, nil}, // SCTP
{"truncated extension header", 0, nil, 0, ipv6.HeaderLen, false, false, ErrIPv6CouldNotFindPayload},
{"truncated extension header", 0, nil, 0, 0, false, false, ErrIPv6CouldNotFindPayload},
// Destination Options with a declared length (255+1)*8 = 2048 that runs past the 48 byte buffer, next = SCTP
{"extension length past buffer", 60, []byte{132, 255, 0, 0, 0, 0, 0, 0}, 132, ipv6.HeaderLen + 2048, false, false, ErrIPv6CouldNotFindPayload},
{"extension length past buffer", 0, []byte{132, 255, 0, 0, 0, 0, 0, 0}, 0, 0, false, false, ErrIPv6CouldNotFindPayload},
{"budget exhausted fails closed", 0, budgetChain, 0, 0, false, false, ErrIPv6CouldNotFindPayload},
}
for _, tt := range tests {
@@ -559,10 +571,10 @@ func Test_IPv6FindUpperProtocol(t *testing.T) {
packet := makeIPv6Packet(src, dst, tt.nextHeader, tt.payload)
proto, offset, isFragment, anyFragment, err := IPv6FindUpperProtocol(packet)
if tt.wantErr != nil {
assert.ErrorIs(t, err, tt.wantErr)
return
require.ErrorIs(t, err, tt.wantErr)
} else {
require.NoError(t, err)
}
require.NoError(t, err)
assert.Equal(t, tt.wantProto, proto)
assert.Equal(t, tt.wantOffset, offset)
assert.Equal(t, tt.wantFragment, isFragment)
-759
View File
@@ -1,759 +0,0 @@
package nebula
import (
"context"
"hash/fnv"
"log/slog"
"net/netip"
"sync"
"sync/atomic"
"time"
"github.com/flynn/noise"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/firewall"
"github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/noiseutil"
)
// Multiport lanes give one tunnel several underlay 5-tuples, so its traffic
// spreads over ECMP paths, NIC receive queues and per-flow policers instead of
// funnelling through a single flow. Each inside flow picks a lane by hashing its
// own 5-tuple, so the spread doesn't depend on how a kernel steers tun queues
// (see txQueue).
//
// A lane is not a second tunnel: it is an extra session on the same HostInfo.
// Noise leaves us with A.eKey == B.dKey, so both sides expand the same two keys
// with the same per-lane label and land on a matched pair without exchanging
// anything. A lane therefore costs no handshake, has no half-established state,
// and dies exactly when its base tunnel does. Which lane a packet belongs to
// travels in the nebula header, inside the AEAD's associated data.
//
// Lane 0 is the base tunnel itself: HostInfo.ConnectionState, the base port, and
// the peer's real remote address, and it carries its share of flows like any
// other. Lane s > 0 egresses a socket on listen.port+s (see laneSock) toward the
// peer's advertised port range. Receiving on a lane
// needs no permission — the keys are derivable the moment the base handshake
// completes — but sending on one needs proof the new 5-tuple actually works,
// since nothing else would notice a middlebox quietly dropping it. So a lane
// stays down until a probe on it is acked, and falls back to the base tunnel the
// moment it stops being acked.
//
// Receive is pay-per-use. A lane we only receive on gets its session on the
// first packet that arrives on it, because how many lanes exist is partly the
// peer's call: it advertises how many it sends on, and we have to be able to
// receive all of them. Deriving them all up front would let a peer advertising
// the maximum cost us a replay window and two cipher states per lane, per
// tunnel, for lanes it may never send on. A lane we send on gets its session
// from its first probe, and every such lane is probed on the tunnel's first
// tick with traffic.
const (
// laneKeyInfo is the HKDF label prefix for lane key expansion. Changing it
// means older builds derive different keys and drop our lane traffic; the
// base tunnel would keep working, so the failure would be a silent loss of
// lanes rather than of connectivity.
laneKeyInfo = "nebula multiport lane v1"
// laneRetryBase and laneRetryMax bound the backoff between probes of a lane
// that will not come up, so a peer whose lane ports are firewalled costs one
// packet a minute rather than one per traffic tick.
laneRetryBase = 5 * time.Second
laneRetryMax = 60 * time.Second
// laneMaxFails caps the failure counter; the backoff saturates well before.
laneMaxFails = 8
// laneWarnFails is how many unanswered probes a lane that has never come up
// takes before it is worth a warning. One or two can be loss; this many in a
// row is usually a firewall that only opens the base port.
laneWarnFails = 3
// laneProbeTimeout is how long a probe may go unacked before it counts as a
// failure. It is shorter than the connection manager's check interval on
// purpose: an outstanding probe is judged on the next tick either way, and a
// longer timeout would only delay that by a whole tick.
laneProbeTimeout = 2 * time.Second
// laneKeepalive is how often a lane that is up re-proves its path. Traffic
// on a lane is not evidence the lane works — that is the whole reason lanes
// need probing — so a lane that silently breaks is only caught here.
laneKeepalive = 30 * time.Second
)
// laneSet holds a peer's lane sessions and the state deciding which lanes may
// carry traffic. It is built when the base handshake completes and never
// resized, so the slices and their lengths are immutable; mu guards the fields
// under it, and sessions/txAddr/demand are atomics read by the data plane
// without it.
type laneSet struct {
// sessions[s] holds lane s's session once something has needed it, and nil
// until then. sessions[0] is never populated: lane 0 is the base tunnel's own
// ConnectionState. The length is immutable, so the data plane bounds-checks
// and loads with no locking.
sessions []atomic.Pointer[ConnectionState]
// material is what a lane session is derived from, kept because sessions are
// derived lazily and the handshake result is long gone by then.
material laneMaterial
// txAddr[s] holds lane s's remote address while the lane is proven usable
// and nil otherwise. This single atomic is both the TX gate and the
// destination, so a routine that loads non-nil has everything it needs.
// Sized txLanes: lanes above that never send.
txAddr []atomic.Pointer[netip.AddrPort]
// demand[s] is raised at creation, and again by the TX path whenever a flow
// hashes onto lane s while it is down. Probing is demand-driven, so a peer we
// never send to costs nothing beyond its base tunnel no matter how many lanes
// are configured, and a lane that keeps failing is only retried while
// something still wants it. Sized txLanes.
demand []atomic.Bool
// txLanes is how many lanes we may send on — our lane count clamped to the
// ports the peer bound — and so the modulus a flow's hash is reduced by.
// Lanes from txLanes up can only receive, which is how a peer with more
// ports than us still spreads its own traffic.
// Immutable, and the length of every TX-side slice here.
txLanes int
mu sync.Mutex
// peerPortCount and peerBasePort are the peer's advertised port range and
// portOffset is this pair's rotation within it: lane s targets
// peerBasePort + ((s + portOffset) % peerPortCount).
peerPortCount uint16
peerBasePort uint16
portOffset uint16
// laneBias rotates the flow hash before it picks a lane, so the two sides of
// a flow land on lanes that are each other's partner rather than at
// independent points in the range. See newLaneSet.
laneBias uint16
// peerAddr is the address the current lane targets were built from. The
// peer's lane ports have no derivable relationship to a new NAT mapping, so
// a roam invalidates every lane rather than moving it.
peerAddr netip.Addr
// probe[s] is lane s's probe and backoff state. Sized txLanes.
probe []laneProbeState
}
// laneMaterial is everything a lane session is derived from. The two base keys
// are the same secret the base tunnel's own cipher states already hold — the
// noiseutil.CipherState interface just doesn't hand them back, so a lane set
// keeps its own copy rather than a reference to the session.
type laneMaterial struct {
eKey, dKey [32]byte
cipher noise.CipherFunc
myCert cert.Certificate
peerCert *cert.CachedCertificate
initiator bool
}
type laneProbeState struct {
// gen is the generation of the last probe sent, echoed in the ack so a late
// ack cannot promote a lane on the strength of a superseded probe.
gen uint8
// fails is the consecutive failure count driving retryAt.
fails uint8
// sentAt is when the outstanding probe went out, zero when none is pending.
sentAt time.Time
// target is where the outstanding probe went, promoted to txAddr on ack. It
// outlives the probe, which is what lets a demotion log the address that
// stopped answering.
target netip.AddrPort
// lastAck is when the lane was last confirmed usable, driving the keepalive.
lastAck time.Time
// retryAt is the earliest we may probe this lane again.
retryAt time.Time
}
// validLaneAdvert reports whether a peer's advertised port range is one we can
// aim lanes at: a real base port, and every port in the range a valid one.
func validLaneAdvert(basePort, portCount uint32) bool {
return basePort != 0 && portCount != 0 && basePort+portCount-1 <= 0xffff
}
// newLaneSet sets up the lanes for a freshly completed base handshake. It
// returns nil when the pair has no lane beyond the base tunnel, which is the
// normal answer for a peer running without multiport. No session is derived
// here; each is derived on the first packet that needs it.
func newLaneSet(r *handshake.Result, myLanes int, myAddr, peerAddr netip.Addr) *laneSet {
// PeerPortCount and PeerBasePort are already bounded to uint16 by the
// handshake payload parser. A zero port count is a peer that did not
// advertise multiport at all, so there is no lane to be had in either
// direction.
peerPorts := uint16(r.PeerPortCount)
if peerPorts == 0 {
return nil
}
// Sessions have to cover both directions: we send on our lanes and receive
// on the peer's, and one derived session serves both ends of a lane index.
// Only the session table is sized by the peer's advertised count; the TX-side
// state is sized by what we will actually send on.
n := min(max(myLanes, int(r.PeerTxLanes)), header.MaxLane+1)
if n < 2 {
return nil
}
txLanes := min(myLanes, int(peerPorts), n)
offset := lanePortOffset(myAddr, peerAddr, peerPorts)
// A flow picks its lane from a hash both sides compute identically, so with
// the ranges lined up the two directions of a flow would pick the same lane
// index — and lane s targets the peer's lane (s + portOffset), not lane s. The
// high-addressed side rotates its choice by the low side's offset, which is
// its own negated, so the two directions land on partner lanes and their
// 4-tuples are exact reverses: each side's traffic then arrives through the
// conntrack entry the other's probe opened. With mismatched ranges there are
// no partner lanes to find, so don't pretend: hash straight.
bias := uint16(0)
if txLanes == int(peerPorts) && peerAddr.Less(myAddr) {
bias = (peerPorts - offset) % peerPorts
}
ls := &laneSet{
sessions: make([]atomic.Pointer[ConnectionState], n),
material: laneMaterial{
eKey: r.EKey.UnsafeKey(),
dKey: r.DKey.UnsafeKey(),
cipher: r.Cipher,
myCert: r.MyCert,
peerCert: r.RemoteCert,
initiator: r.Initiator,
},
txAddr: make([]atomic.Pointer[netip.AddrPort], txLanes),
demand: make([]atomic.Bool, txLanes),
probe: make([]laneProbeState, txLanes),
txLanes: txLanes,
peerPortCount: peerPorts,
peerBasePort: uint16(r.PeerBasePort),
portOffset: offset,
laneBias: bias,
}
// Every lane starts out demanded, so the first traffic tick on this tunnel
// probes all of them at once instead of waiting for a flow to hash onto each.
// Lanes have to be up *before* the flows are, not after: the peer writes a
// flow's inbound packets to the tun queue matching the socket they arrived on,
// and the kernel remembers that for as long as the flow stays busy. A flow that
// starts while the lanes are still down therefore gets pinned to queue 0 on
// both hosts for its whole life. This costs one probe per lane on any tunnel
// with traffic; an idle tunnel is never probed, so it still costs nothing.
for s := 1; s < txLanes; s++ {
ls.demand[s].Store(true)
}
return ls
}
// laneLogAttr summarizes what a tunnel negotiated, for the handshake log lines.
// It is an empty attr, which slog drops, on a node not running multiport. A peer
// that negotiated no lanes still logs, with zeros: "we offered and got nothing"
// is exactly what you want to see when you expected lanes and have none.
func laneLogAttr(myLanes int, ls *laneSet) slog.Attr {
if myLanes == 0 {
return slog.Attr{}
}
if ls == nil {
return slog.Any("lanes", m{"tx": 0, "sessions": 0})
}
// Every field read here is immutable once the set is built.
return slog.Any("lanes", m{
"tx": ls.txLanes,
"sessions": len(ls.sessions),
"peerBasePort": ls.peerBasePort,
"peerPorts": ls.peerPortCount,
"portOffset": ls.portOffset,
})
}
// emitLaneStats reports how many lanes are carrying traffic and how many tunnels
// have any. Both are counted by walking the hostmap, because a counter kept at
// promotion and demotion would drift upward forever: a tunnel torn down while its
// lanes are up never demotes them. Only a node running multiport pays for the
// walk.
func (f *Interface) emitLaneStats(up, tunnels metrics.Gauge) {
var nUp, nTunnels int64
f.hostMap.ForEachIndex(func(hostinfo *HostInfo) {
ls := hostinfo.lanes
if ls == nil {
return
}
nTunnels++
for s := 1; s < ls.txLanes; s++ {
if ls.txAddr[s].Load() != nil {
nUp++
}
}
})
up.Update(nUp)
tunnels.Update(nTunnels)
}
// lanePortOffset returns the rotation applied to this pair's lane target ports,
// in [0, peerPortCount). Without it every peer with few ports would aim its few
// lanes at a big peer's first few ports, concentrating the big peer's receive
// work on a couple of sockets; the hash spreads pairs across the whole range.
//
// Both sides hash the same sorted vpn-address pair and the higher address
// negates the result, so when port counts match the two sides' rotations
// cancel on ports: our lane s's 4-tuple is the reverse of the peer's lane
// s+offset, and each side's probe opens the conntrack entry the other's arrives
// through. The lane indices still differ by offset; laneBias lines those up. (The one
// lane a nonzero rotation lands on the peer's base port has no partner lane, so
// behind a port-restricted NAT it is the one lane that may never come up.)
func lanePortOffset(myAddr, peerAddr netip.Addr, peerPortCount uint16) uint16 {
if peerPortCount == 0 {
return 0
}
lo, hi := myAddr, peerAddr
if hi.Less(lo) {
lo, hi = hi, lo
}
h := fnv.New32a()
b := lo.As16()
h.Write(b[:])
b = hi.As16()
h.Write(b[:])
o := uint16(h.Sum32() % uint32(peerPortCount))
if myAddr == hi {
o = (peerPortCount - o) % peerPortCount
}
return o
}
// laneSession returns the session to decrypt a lane s packet with, deriving one
// if this is the first packet to claim that lane. A nil session with no error
// means this tunnel has no lane s at all.
//
// cached reports whether the session was already in the table. A fresh one is
// deliberately left out of it: anyone who can spoof this tunnel's local index
// can name any lane, and installing on sight would let them make us hold a
// replay window and two cipher states per lane without authenticating anything.
// The caller must install with installSession once the packet decrypts, which is
// the first moment the lane is known to be real.
func (i *HostInfo) laneSession(s uint8) (ci *ConnectionState, cached bool, err error) {
ls := i.lanes
if ls == nil || s == 0 || int(s) >= len(ls.sessions) {
return nil, false, nil
}
if cs := ls.sessions[s].Load(); cs != nil {
return cs, true, nil
}
cs, err := newLaneConnectionState(&ls.material, s)
if err != nil {
return nil, false, err
}
return cs, false, nil
}
// installSession publishes a session derived by laneSession, so the next packet
// on the lane doesn't have to derive it again. cs must have already decrypted the
// packet at messageCounter.
//
// Two routines can race on a lane's first packet and derive a session each. The
// loser's is dropped, and with it the replay-window entry for the packet it just
// accepted, so hand that counter to the session that survives — the keys are
// identical, so it is the same window in every respect that matters.
func (ls *laneSet) installSession(l *slog.Logger, s uint8, cs *ConnectionState, messageCounter uint64) {
if ls.sessions[s].CompareAndSwap(nil, cs) {
return
}
ls.sessions[s].Load().noteSeen(l, messageCounter)
}
// session returns lane s's session for our own use, deriving and installing it
// if it doesn't exist yet. Unlike the RX path this needs no proof the lane is
// real: we only ask for lanes we chose to send on. s must be a lane this set
// covers.
func (ls *laneSet) session(s int) (*ConnectionState, error) {
if cs := ls.sessions[s].Load(); cs != nil {
return cs, nil
}
cs, err := newLaneConnectionState(&ls.material, uint8(s))
if err != nil {
return nil, err
}
if !ls.sessions[s].CompareAndSwap(nil, cs) {
return ls.sessions[s].Load(), nil
}
return cs, nil
}
// maxMessageCounter returns the highest counter across the base session and
// every lane session. Data rides the lanes, so the base counter alone would
// never reach the rehandshake or exhaustion thresholds and the lane keys would
// be used past their data-volume margin. Rolling the base tunnel replaces the
// lane keys with it, since lanes are derived from it.
func (i *HostInfo) maxMessageCounter() uint64 {
if i.ConnectionState == nil {
return 0
}
c := i.ConnectionState.messageCounter.Load()
if ls := i.lanes; ls != nil {
for s := range ls.sessions {
cs := ls.sessions[s].Load()
if cs == nil {
// Never derived, so it has never sent anything either.
continue
}
if lc := cs.messageCounter.Load(); lc > c {
c = lc
}
}
}
return c
}
// txLane returns the session and destination for lane s, or a nil session when
// the lane is down and the caller must use the base tunnel. A miss raises
// demand, which is what gets a down lane probed again, so we pay for a lane
// exactly where real traffic wanted one. Callers on the data plane come through
// txLaneForFlow.
func (ls *laneSet) txLane(s int) (*ConnectionState, netip.AddrPort) {
if ls == nil || s <= 0 || s >= ls.txLanes {
return nil, netip.AddrPort{}
}
if addr := ls.txAddr[s].Load(); addr != nil {
// The lane is only up because a probe was acked on it, and that probe
// derived the session, so this load cannot miss. Fall back rather than
// derive here anyway: this is the hot path and a nil is not worth an HKDF.
if cs := ls.sessions[s].Load(); cs != nil {
return cs, *addr
}
return nil, netip.AddrPort{}
}
// Load-guarded so the common case of a lane that will not come up is a
// plain read and cannot ping-pong the cache line these flags share.
if !ls.demand[s].Load() {
ls.demand[s].Store(true)
}
return nil, netip.AddrPort{}
}
// txLaneForFlow picks the lane a flow rides and returns it with its session and
// destination, or lane 0 and a nil session when the flow belongs on the base
// tunnel — either because the hash landed on lane 0 or because the lane it
// landed on is down.
//
// The lane comes from the flow rather than from the sending routine so that lane
// use doesn't depend on how the kernel steers tun queues; see txQueue. It is a
// pure function of the 5-tuple, so a flow stays on one lane for its life: no
// per-packet reordering, and one lane's replay window sees one set of flows.
func (ls *laneSet) txLaneForFlow(p *firewall.Packet) (int, *ConnectionState, netip.AddrPort) {
if ls == nil || ls.txLanes < 2 {
return 0, nil, netip.AddrPort{}
}
s := int((laneFlowHash(p) + uint32(ls.laneBias)) % uint32(ls.txLanes))
if s == 0 {
// Lane 0 is the base tunnel, and a full share of flows belongs on it.
return 0, nil, netip.AddrPort{}
}
cs, addr := ls.txLane(s)
return s, cs, addr
}
// laneFlowHash hashes a 5-tuple to the same value from either end of the flow,
// which is what lets both peers pick partner lanes for it (see laneBias). FNV-1a
// by hand rather than through hash/fnv: this runs per packet, and the interface
// there would escape the addresses to the heap.
func laneFlowHash(p *firewall.Packet) uint32 {
// Order the two endpoints so the direction of travel cannot change the hash.
aAddr, aPort := p.LocalAddr, p.LocalPort
bAddr, bPort := p.RemoteAddr, p.RemotePort
if bAddr.Less(aAddr) || (aAddr == bAddr && bPort < aPort) {
aAddr, aPort, bAddr, bPort = bAddr, bPort, aAddr, aPort
}
const prime = 16777619
h := uint32(2166136261)
x, y := aAddr.As16(), bAddr.As16()
for i := range x {
h = (h ^ uint32(x[i])) * prime
h = (h ^ uint32(y[i])) * prime
}
for _, b := range [5]byte{byte(aPort >> 8), byte(aPort), byte(bPort >> 8), byte(bPort), p.Protocol} {
h = (h ^ uint32(b)) * prime
}
return h
}
// laneTargetPortLocked returns the peer port lane s aims at. Only meaningful
// when peerPortCount is nonzero, which txLanes > 0 guarantees.
func (ls *laneSet) laneTargetPortLocked(s int) uint16 {
return ls.peerBasePort + uint16((s+int(ls.portOffset))%int(ls.peerPortCount))
}
// laneTargetLocked returns where lane s's probes go: the peer port this lane is
// paired with, on the peer's current direct address.
//
// A lane only ever aims at its own port. There is no fallback to the peer's base
// port when the lane port doesn't answer — a lane sharing the base port's
// destination gains only a source port of its own, while costing the peer the
// receive spread that is the whole point, so a lane that can't reach its port
// stays down and its traffic rides the base tunnel.
func (ls *laneSet) laneTargetLocked(s int, addr netip.Addr) netip.AddrPort {
return netip.AddrPortFrom(addr, ls.laneTargetPortLocked(s))
}
// laneRetryDelay is the backoff after fails consecutive probe failures.
func laneRetryDelay(fails uint8) time.Duration {
d := laneRetryBase << min(fails, 4)
if d > laneRetryMax {
d = laneRetryMax
}
return d
}
// noteAck records an acked probe for lane s, promoting the lane if it was down.
// gen must match the outstanding probe. Reports the target the lane came up on,
// and whether this ack is what promoted it.
func (ls *laneSet) noteAck(s int, gen uint8, now time.Time) (netip.AddrPort, bool) {
if s <= 0 || s >= len(ls.probe) {
return netip.AddrPort{}, false
}
ls.mu.Lock()
defer ls.mu.Unlock()
p := &ls.probe[s]
if p.sentAt.IsZero() || p.gen != gen {
// No probe outstanding, or an ack for a probe we have already given up
// on. Either way it says nothing about the lane's current path.
return netip.AddrPort{}, false
}
p.sentAt = time.Time{}
p.lastAck = now
p.fails = 0
p.retryAt = time.Time{}
target := p.target
if ls.txAddr[s].Load() != nil {
// Keepalive for a lane already up.
return target, false
}
ls.txAddr[s].Store(&target)
return target, true
}
// probeLanes runs one lane maintenance pass for a peer: it demotes lanes whose
// probe went unanswered, re-proves lanes that have been up a while without one,
// and probes down lanes the data plane asked for. Driven by the connection
// manager's per-tunnel traffic tick, which skips a tunnel that is idle in both
// directions.
func (f *Interface) probeLanes(hostinfo *HostInfo, now time.Time, nb, out []byte) {
ls := hostinfo.lanes
if ls == nil || ls.txLanes < 2 {
return
}
remote := hostinfo.GetRemote()
ls.mu.Lock()
defer ls.mu.Unlock()
if !remote.IsValid() {
// Relayed, or otherwise without a direct path. Lanes are direct-only,
// so drop them all; a later tick rebuilds if a direct path returns.
ls.resetLocked()
return
}
if ls.peerAddr != remote.Addr() {
if ls.peerAddr.IsValid() {
// A roam is a new path, not a failure: forget the lanes built on
// the old one and let demand re-probe from a clean backoff. On the
// first pass there is nothing built yet, so just record the address.
ls.resetLocked()
}
ls.peerAddr = remote.Addr()
}
for s := 1; s < ls.txLanes; s++ {
p := &ls.probe[s]
up := ls.txAddr[s].Load() != nil
if !p.sentAt.IsZero() {
if now.Sub(p.sentAt) < laneProbeTimeout {
continue
}
// An aged-out probe is a failure whether it was bringing the lane
// up or keeping it up.
p.sentAt = time.Time{}
p.fails = min(p.fails+1, laneMaxFails)
p.retryAt = now.Add(laneRetryDelay(p.fails))
if up {
ls.txAddr[s].Store(nil)
hostinfo.logger(f.l).Info("Multiport lane demoted, probe unanswered", "lane", s, "udpAddr", p.target)
} else if p.lastAck.IsZero() && p.fails == laneWarnFails {
// Said once per lane: fails only passes through this value again
// after an ack or a reset clears it.
hostinfo.logger(f.l).Warn("Multiport lane not coming up, probes unanswered; check that the peer's lane ports are reachable",
"lane", s, "udpAddr", p.target, "fails", p.fails)
}
continue
}
if up {
if now.Sub(p.lastAck) < laneKeepalive {
continue
}
} else if now.Before(p.retryAt) || !ls.demand[s].Swap(false) {
continue
}
p.gen++
p.target = ls.laneTargetLocked(s, remote.Addr())
if f.sendLaneProbe(hostinfo, s, p.gen, p.target, nb, out) {
p.sentAt = now
} else {
p.fails = min(p.fails+1, laneMaxFails)
p.retryAt = now.Add(laneRetryDelay(p.fails))
}
}
}
// resetLocked takes every lane down and clears its probe state, without
// counting it as a failure.
//
// The probe generation survives the reset. Starting it over would let a late ack
// for a probe sent before the reset match the first probe sent after it, and
// promote the lane to a target nothing has proven.
//
// Demand is deliberately left standing: a roam or a relay detour doesn't change
// what the data plane wants. Lanes that were up had their demand spent when they
// were probed, but the first packet of a flow that hashes onto one after the
// reset raises it again, so lanes still carrying flows are re-probed on the next
// tick, while a lane nothing hashes onto stays down.
func (ls *laneSet) resetLocked() {
for s := 1; s < len(ls.probe); s++ {
ls.txAddr[s].Store(nil)
ls.probe[s] = laneProbeState{gen: ls.probe[s].gen}
}
}
// isLaneProbe reports whether t/st is a lane probe or its ack. Probing is upkeep
// the lanes do for themselves, not traffic: counting it would let a lane's own
// keepalive keep an idle tunnel from ever being dropped.
func isLaneProbe(t header.MessageType, st header.MessageSubType) bool {
return t == header.Test && (st == header.LaneProbe || st == header.LaneProbeAck)
}
// isLaneRemote reports whether addr is the target of one of our up lanes. The
// peer answers a lane packet from the socket it arrived on, so a recv_error for
// lane traffic comes from that lane's target rather than the base remote.
func (ls *laneSet) isLaneRemote(addr netip.AddrPort) bool {
if ls == nil {
return false
}
for s := 1; s < ls.txLanes; s++ {
if a := ls.txAddr[s].Load(); a != nil && *a == addr {
return true
}
}
return false
}
// sendLaneProbe sends a probe on lane s to addr from a socket on lane s's port.
// The probe is an ordinary Test packet encrypted with the lane's session, so an
// ack proves the whole lane: our source port reached the peer, its reply reached
// us, and the keys we derived for this lane match the ones it derived. Reports
// whether the probe made it onto the wire.
//
// It goes out the first socket on the port, since the connection manager runs
// this and has no queue of its own; every socket on the port has the same address,
// so the probe proves the path for whichever one the data plane picks.
func (f *Interface) sendLaneProbe(hostinfo *HostInfo, s int, gen uint8, addr netip.AddrPort, nb, out []byte) bool {
// The first probe on a lane is what derives its session.
ci, err := hostinfo.lanes.session(s)
if err != nil {
hostinfo.logger(f.l).Error("Failed to derive multiport lane session", "error", err, "lane", s)
return false
}
if ci == nil || ci.eKey == nil {
return false
}
if noiseutil.EncryptLockNeeded {
ci.writeLock.Lock()
}
c, ok := ci.NextMessageCounter()
if !ok {
if noiseutil.EncryptLockNeeded {
ci.writeLock.Unlock()
}
f.dropExhausted(hostinfo, c, "Dropping multiport lane probe, lane message counter is exhausted")
return false
}
b := header.EncodeLane(out[:0], header.Version, header.Test, header.LaneProbe, hostinfo.remoteIndexId, c, uint8(s))
b, err = ci.eKey.EncryptDanger(b, b, []byte{uint8(s), gen}, c, nb)
if noiseutil.EncryptLockNeeded {
ci.writeLock.Unlock()
}
if err != nil {
hostinfo.logger(f.l).Error("Failed to encrypt multiport lane probe", "error", err, "lane", s)
return false
}
f.messageMetrics.Tx(header.Test, header.LaneProbe, 1)
if err := f.writers[f.laneSock(0, s)].WriteTo(b, addr); err != nil {
hostinfo.logger(f.l).Error("Failed to send multiport lane probe", "error", err, "lane", s, "udpAddr", addr)
return false
}
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("Multiport lane probe sent", "lane", s, "gen", gen, "udpAddr", addr)
}
return true
}
// handleLaneProbe answers a peer's lane probe. The ack rides the base tunnel on
// purpose: a probe proves the peer's lane s works in its send direction, and
// answering on our own lane s would make the result depend on a second path
// that may be broken independently.
func (f *Interface) handleLaneProbe(hostinfo *HostInfo, lane uint8, payload []byte, rxc *rxContext) {
if lane == 0 || len(payload) < 2 {
return
}
// Echo the header's lane rather than the payload's, so a peer cannot get us
// to vouch for a lane it did not actually probe.
f.send(header.Test, header.LaneProbeAck, hostinfo.ConnectionState, hostinfo,
[]byte{lane, payload[1]}, rxc.nb, rxc.scratch[:0])
}
// handleLaneProbeAck promotes the lane a peer just acked.
func (f *Interface) handleLaneProbeAck(hostinfo *HostInfo, payload []byte) {
ls := hostinfo.lanes
if ls == nil || len(payload) < 2 {
return
}
// The target is worth logging next to the demotion that names the same
// address, so a flapping lane can be read off the logs.
if target, promoted := ls.noteAck(int(payload[0]), payload[1], time.Now()); promoted {
hostinfo.logger(f.l).Info("Multiport lane up", "lane", payload[0], "udpAddr", target)
}
}
-1175
View File
File diff suppressed because it is too large. Load diff
+81 -11
View File
@@ -15,10 +15,12 @@ import (
"time"
"github.com/gaissmai/bart"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/logging"
"github.com/slackhq/nebula/nicfinder"
"github.com/slackhq/nebula/udp"
"github.com/slackhq/nebula/util"
)
@@ -34,12 +36,18 @@ type LightHouse struct {
myVpnNetworks []netip.Prefix
myVpnNetworksTable *bart.Lite
punchy *Punchy
// myVpnAddrsTable contains our overlay host addrs, as opposed to the overlay networks
myVpnAddrsTable *bart.Lite
punchy *Punchy
// localAddrsFn enumerates the underlay addresses we advertise. It is a field so tests can supply simulated
// addresses rather than whatever this machine's NICs happen to be. Set it before Start.
localAddrsFn func(*LocalAllowList) []netip.Addr
// lastLocalAddrsErr is the previous localAddrsFn failure. Enumeration runs on every update, so an
// unchanged failure is demoted to Debug rather than warning every lighthouse.interval forever.
lastLocalAddrsErr atomic.Pointer[string]
// Local cache of answers from light houses
// map of vpn addr to answers
addrMap map[netip.Addr]*RemoteList
@@ -74,6 +82,8 @@ type LightHouse struct {
updateTrigger chan struct{}
queryChan chan netip.Addr
// queryDropped counts queries QueryServer dropped because queryChan was full.
queryDropped metrics.Counter
calculatedRemotes atomic.Pointer[bart.Table[[]*calculatedRemote]] // Maps VpnAddr to []*calculatedRemote
@@ -104,15 +114,19 @@ func NewLightHouseFromConfig(ctx context.Context, l *slog.Logger, c *config.C, c
amLighthouse: amLighthouse,
myVpnNetworks: cs.myVpnNetworks,
myVpnNetworksTable: cs.myVpnNetworksTable,
myVpnAddrsTable: cs.myVpnAddrsTable,
addrMap: make(map[netip.Addr]*RemoteList),
nebulaPort: nebulaPort,
punchy: p,
updateTrigger: make(chan struct{}, 1),
queryChan: make(chan netip.Addr, c.GetUint32("handshakes.query_buffer", 64)),
queryChan: make(chan netip.Addr, c.GetUint32("handshakes.query_buffer", 1024)),
queryDropped: metrics.GetOrRegisterCounter("lighthouse.query.dropped", nil),
l: l,
}
h.localAddrsFn = func(al *LocalAllowList) []netip.Addr {
return localAddrs(h.l, al)
addrs, err := nicfinder.Find(ctx, h.l, al)
h.logLocalAddrsErr(err)
return addrs
}
lighthouses := make([]netip.Addr, 0)
@@ -124,18 +138,21 @@ func NewLightHouseFromConfig(ctx context.Context, l *slog.Logger, c *config.C, c
h.metrics = newLighthouseMetrics()
}
err := h.reload(c, true)
err := h.reload(c, true, nil)
if err != nil {
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := h.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("lighthouse.am_lighthouse", "listen.port", "handshakes.query_buffer", "stats.lighthouse_metrics")
err := h.reload(c, false, r)
switch v := err.(type) {
case *util.ContextualError:
v.Log(l)
r.Failed()
case error:
l.Error("failed to reload lighthouse", "error", err)
r.Failed()
}
})
@@ -176,7 +193,8 @@ func (lh *LightHouse) GetUpdateInterval() int64 {
return lh.interval.Load()
}
func (lh *LightHouse) reload(c *config.C, initial bool) error {
// r is nil on the initial load
func (lh *LightHouse) reload(c *config.C, initial bool, r *config.Reload) error {
if initial || c.HasChanged("lighthouse.advertise_addrs") {
rawAdvAddrs := c.GetStringSlice("lighthouse.advertise_addrs", []string{})
advAddrs := make([]netip.AddrPort, 0)
@@ -211,6 +229,7 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
"addr", rawAddr,
"entry", i+1,
)
r.Failed()
continue
}
@@ -345,7 +364,13 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
}
}
if initial || c.HasChanged("relay.relays") {
// am_relay decides whether relay.relays applies to us at all
if initial || c.HasChanged("relay.relays") || c.HasChanged("relay.am_relay") {
var before []netip.Addr
if !initial {
before = lh.GetRelaysForMe()
}
switch c.GetBool("relay.am_relay", false) {
case true:
// Relays aren't allowed to specify other relays
@@ -363,6 +388,7 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
"relay", v,
"error", err,
)
r.Failed()
} else {
lh.l.Info("Read relay from config", "relay", v)
relaysForMe = append(relaysForMe, configRIP)
@@ -370,6 +396,11 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
}
lh.relaysForMe.Store(&relaysForMe)
}
// Tell the lighthouses now rather than at the next interval, but only when what we advertise changed
if !initial && !slices.Equal(before, lh.GetRelaysForMe()) {
lh.TriggerUpdate()
}
}
return nil
@@ -502,14 +533,19 @@ func (lh *LightHouse) Query(vpnAddr netip.Addr) *RemoteList {
return nil
}
// QueryServer is asynchronous so no reply should be expected
// QueryServer is asynchronous and non-blocking, no reply should be expected.
// If the query channel is full, the query is dropped.
func (lh *LightHouse) QueryServer(vpnAddr netip.Addr) {
// Don't put lighthouse addrs in the query channel because we can't query lighthouses about lighthouses
if lh.amLighthouse || lh.IsLighthouseAddr(vpnAddr) {
return
}
lh.queryChan <- vpnAddr
select {
case lh.queryChan <- vpnAddr:
default:
lh.queryDropped.Inc(1)
}
}
func (lh *LightHouse) QueryCache(vpnAddrs []netip.Addr) *RemoteList {
@@ -913,6 +949,26 @@ func (lh *LightHouse) TriggerUpdate() {
}
}
// logLocalAddrsErr reports a localAddrs failure at Warn the first time it is seen and at Debug while
// it persists unchanged, so a permanent failure does not warn on every update forever.
func (lh *LightHouse) logLocalAddrsErr(err error) {
if err == nil {
lh.lastLocalAddrsErr.Store(nil)
return
}
msg := err.Error()
prev := lh.lastLocalAddrsErr.Swap(&msg)
if prev != nil && *prev == msg {
if lh.l.Enabled(context.Background(), slog.LevelDebug) {
lh.l.Debug("Failed to collect local addresses to advertise", "error", err)
}
return
}
lh.l.Warn("Failed to collect local addresses to advertise", "error", err)
}
func (lh *LightHouse) SendUpdate() {
var v4 []*V4AddrPort
var v6 []*V6AddrPort
@@ -1158,6 +1214,17 @@ func (lhh *LightHouseHandler) handleHostQuery(n *NebulaMeta, fromVpnAddrs []neti
return
}
// Don't respond to requests for us.
if lhh.lh.myVpnAddrsTable.Contains(queryVpnAddr) {
if lhh.l.Enabled(context.Background(), slog.LevelDebug) {
lhh.l.Debug("Ignoring HostQuery for one of my own addresses",
"fromVpnAddrs", fromVpnAddrs,
"queryVpnAddr", queryVpnAddr,
)
}
return
}
found, ln, err := lhh.lh.queryAndPrepMessage(queryVpnAddr, func(c *cache) (int, error) {
n = lhh.resetMeta()
n.Type = NebulaMeta_HostQueryReply
@@ -1194,10 +1261,13 @@ func (lhh *LightHouseHandler) handleHostQuery(n *NebulaMeta, fromVpnAddrs []neti
// sendHostPunchNotification signals the other side to punch some zero byte udp packets
func (lhh *LightHouseHandler) sendHostPunchNotification(n *NebulaMeta, fromVpnAddrs []netip.Addr, punchNotifDest netip.Addr, w EncWriter) {
whereToPunch := fromVpnAddrs[0]
// Resolve the target before taking the querier's RemoteList lock below. GetHostInfo takes the
// hostmap lock, and the hostmap is taken before the lighthouse, which is taken before a
// RemoteList, so looking it up inside the callback would deadlock.
targetHI := lhh.lh.ifce.GetHostInfo(punchNotifDest)
found, ln, err := lhh.lh.queryAndPrepMessage(whereToPunch, func(c *cache) (int, error) {
n = lhh.resetMeta()
n.Type = NebulaMeta_HostPunchNotification
targetHI := lhh.lh.ifce.GetHostInfo(punchNotifDest)
var useVersion cert.Version
if targetHI == nil {
useVersion = lhh.lh.ifce.GetCertState().initiatingVersion
+368 -64
View File
@@ -1,16 +1,21 @@
package nebula
import (
"bytes"
"encoding/binary"
"errors"
"fmt"
"net/netip"
"sync"
"testing"
"time"
"github.com/gaissmai/bart"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.yaml.in/yaml/v3"
@@ -27,15 +32,27 @@ func TestOldIPv4Only(t *testing.T) {
assert.Equal(t, binary.BigEndian.Uint32(bp[:]), m.GetAddr())
}
func testCertState(networks ...netip.Prefix) *CertState {
cs := &CertState{
myVpnNetworks: networks,
myVpnNetworksTable: new(bart.Lite),
myVpnAddrs: make([]netip.Addr, 0, len(networks)),
myVpnAddrsTable: new(bart.Lite),
}
for _, n := range networks {
cs.myVpnNetworksTable.Insert(n)
cs.myVpnAddrs = append(cs.myVpnAddrs, n.Addr())
cs.myVpnAddrsTable.Insert(netip.PrefixFrom(n.Addr(), n.Addr().BitLen()))
}
return cs
}
func Test_lhStaticMapping(t *testing.T) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/16")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh1 := "10.128.0.2"
c := config.NewC(l)
@@ -55,12 +72,7 @@ func Test_lhStaticMapping(t *testing.T) {
func TestReloadLighthouseInterval(t *testing.T) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/16")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh1 := "10.128.0.2"
c := config.NewC(l)
@@ -75,27 +87,22 @@ func TestReloadLighthouseInterval(t *testing.T) {
lh.ifce = &mockEncWriter{}
// The first one routine is kicked off by main.go currently, lets make sure that one dies
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 5"))
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 5").Err)
assert.Equal(t, int64(5), lh.interval.Load())
// Subsequent calls are killed off by the LightHouse.Reload function
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 10"))
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 10").Err)
assert.Equal(t, int64(10), lh.interval.Load())
// If this completes then nothing is stealing our reload routine
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 11"))
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 11").Err)
assert.Equal(t, int64(11), lh.interval.Load())
}
func BenchmarkLighthouseHandleRequest(b *testing.B) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/0")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
c := config.NewC(l)
lh, err := NewLightHouseFromConfig(b.Context(), l, c, cs, nil, nil)
@@ -195,12 +202,7 @@ func TestLighthouse_Memory(t *testing.T) {
c.Settings["listen"] = map[string]any{"port": 4242}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
lh.ifce = &mockEncWriter{}
require.NoError(t, err)
@@ -280,12 +282,7 @@ func TestLighthouse_reload(t *testing.T) {
c.Settings["listen"] = map[string]any{"port": 4242}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
@@ -299,7 +296,7 @@ func TestLighthouse_reload(t *testing.T) {
require.NoError(t, err)
c.ReloadConfigString(string(rc))
err = lh.reload(c, false)
err = lh.reload(c, false, nil)
require.NoError(t, err)
}
@@ -315,12 +312,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
@@ -342,7 +334,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err := yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl := lh.Query(staticHost)
require.NotNil(t, rl)
@@ -358,7 +350,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl = lh.Query(staticHost)
require.NotNil(t, rl)
@@ -366,7 +358,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
assert.Equal(t, []netip.AddrPort{netip.MustParseAddrPort("1.1.1.1:4242")}, rl.CopyAddrs([]netip.Prefix{}))
// Reload with the same config. An unchanged entry must not duplicate.
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl = lh.Query(staticHost)
require.NotNil(t, rl)
@@ -381,7 +373,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
// Add a second host alongside the first. Both should be present, neither duplicated.
nc = map[string]any{
@@ -392,7 +384,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl = lh.Query(staticHost)
require.NotNil(t, rl)
@@ -414,7 +406,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
_, isStatic := lh.GetStaticHostList()[staticHost]
assert.False(t, isStatic)
@@ -429,7 +421,9 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
assert.Equal(t, []netip.AddrPort{netip.MustParseAddrPort("3.3.3.3:4242")}, rl.CopyAddrs([]netip.Prefix{}))
}
func newLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, lhh *LightHouseHandler) testLhReply {
// sendLHHostRequest delivers a HostQuery to lhh and hands back the writer that
// captured what it emitted. Pass a nil filter to see every message.
func sendLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, lhh *LightHouseHandler, filter *NebulaMeta_MessageType) *testEncWriter {
req := &NebulaMeta{
Type: NebulaMeta_HostQuery,
Details: &NebulaMetaDetails{},
@@ -447,12 +441,59 @@ func newLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, l
panic(err)
}
filter := NebulaMeta_HostQueryReply
w := &testEncWriter{
metaFilter: &filter,
}
w := &testEncWriter{metaFilter: filter}
lhh.HandleRequest(fromAddr, []netip.Addr{myVpnIp}, b, w)
return w.lastReply
return w
}
func newLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, lhh *LightHouseHandler) testLhReply {
filter := NebulaMeta_HostQueryReply
return sendLHHostRequest(fromAddr, myVpnIp, queryVpnIp, lhh, &filter).lastReply
}
func TestLighthouse_IgnoresHostQueryForItself(t *testing.T) {
// Validate that we don't answer host queries for our own address.
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
myVpnIp := myVpnNet.Addr()
c := config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"am_lighthouse": true}
c.Settings["listen"] = map[string]any{"port": 4242}
// Add a static_host_map entry for ourselves, so our address
// is in the addrMap.
c.Settings["static_host_map"] = map[string]any{
myVpnIp.String(): []any{"192.168.100.1:4242"},
}
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(myVpnNet), nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
lhh := lh.NewRequestHandler()
peerVpnIp := netip.MustParseAddr("10.128.0.2")
peerUdpAddr := netip.MustParseAddrPort("10.0.0.2:4242")
otherVpnIp := netip.MustParseAddr("10.128.0.3")
otherUdpAddr := netip.MustParseAddrPort("10.0.0.3:4242")
newLHHostUpdate(peerUdpAddr, peerVpnIp, []netip.AddrPort{peerUdpAddr}, lhh)
newLHHostUpdate(otherUdpAddr, otherVpnIp, []netip.AddrPort{otherUdpAddr}, lhh)
// Control: a query about a real peer is still answered, and still ends with
// the punch notification aimed at the host that was asked about.
w := sendLHHostRequest(peerUdpAddr, peerVpnIp, otherVpnIp, lhh, nil)
require.NotNil(t, w.lastReply.msg)
assert.Equal(t, NebulaMeta_HostPunchNotification, w.lastReply.msg.Type)
assert.Equal(t, otherVpnIp, w.lastReply.vpnIp)
// Now validate that we don't send to ourselves.
found, _, err := lh.queryAndPrepMessage(myVpnIp, func(*cache) (int, error) { return 0, nil })
require.NoError(t, err)
require.True(t, found, "the lighthouse should hold a cache entry for its own address")
w = sendLHHostRequest(peerUdpAddr, peerVpnIp, myVpnIp, lhh, nil)
assert.Nil(t, w.lastReply.msg, "a query about our own address must produce no reply and no punch notification")
}
func newLHHostUpdate(fromAddr netip.AddrPort, vpnIp netip.Addr, addrs []netip.AddrPort, lhh *LightHouseHandler) {
@@ -642,12 +683,7 @@ func TestLighthouse_Dont_Delete_Static_Hosts(t *testing.T) {
}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
@@ -708,12 +744,7 @@ func TestLighthouse_DeletesWork(t *testing.T) {
}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
@@ -738,3 +769,276 @@ func TestLighthouse_DeletesWork(t *testing.T) {
out = lh.Query(testHost)
assert.Nil(t, out)
}
func TestLightHouse_logLocalAddrsErr(t *testing.T) {
out := &bytes.Buffer{}
lh := &LightHouse{l: test.NewLoggerWithOutput(out)}
// The first sighting of a failure warns.
lh.logLocalAddrsErr(errors.New("permission denied"))
assert.Contains(t, out.String(), "level=WARN")
assert.Contains(t, out.String(), "permission denied")
// Repeating unchanged does not warn again, which is what keeps a permanent failure from warning
// on every lighthouse.interval for the life of the process.
out.Reset()
lh.logLocalAddrsErr(errors.New("permission denied"))
assert.NotContains(t, out.String(), "level=WARN")
// A different failure is a new event and warns.
out.Reset()
lh.logLocalAddrsErr(errors.New("something else"))
assert.Contains(t, out.String(), "level=WARN")
assert.Contains(t, out.String(), "something else")
// Recovering resets, so the same failure returning later warns again.
out.Reset()
lh.logLocalAddrsErr(nil)
assert.Empty(t, out.String())
lh.logLocalAddrsErr(errors.New("something else"))
assert.Contains(t, out.String(), "level=WARN")
}
// heldEncWriter holds the query worker in its first GetHostInfo until release closes.
type heldEncWriter struct {
mockEncWriter
once sync.Once
held chan struct{}
release chan struct{}
}
func (w *heldEncWriter) GetHostInfo(netip.Addr) *HostInfo {
w.once.Do(func() { close(w.held) })
<-w.release
return nil
}
// QueryServer never waits on the query worker, since the packet readers call it and nothing drains the channel at
// shutdown; a query that finds the channel full is dropped and counted. The worker is held on its first query so the
// channel fills.
func TestLighthouse_QueryServerDoesNotBlock(t *testing.T) {
l := test.NewLogger()
cs := testCertState(netip.MustParsePrefix("10.128.0.1/16"))
lh1 := "10.128.0.2"
c := config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"hosts": []any{lh1}}
c.Settings["static_host_map"] = map[string]any{lh1: []any{"1.1.1.1:4242"}}
c.Settings["handshakes"] = map[string]any{"query_buffer": 1}
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
w := &heldEncWriter{held: make(chan struct{}), release: make(chan struct{})}
lh.ifce = w
t.Cleanup(func() { close(w.release) })
lh.QueryServer(netip.MustParseAddr("10.128.0.10"))
select {
case <-w.held:
case <-time.After(5 * time.Second):
t.Fatal("the query worker never took the first query")
}
dropped := lh.queryDropped.Count()
done := make(chan struct{})
go func() {
lh.QueryServer(netip.MustParseAddr("10.128.0.11"))
lh.QueryServer(netip.MustParseAddr("10.128.0.12"))
close(done)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("QueryServer blocked on a full query channel")
}
assert.EqualValues(t, 1, lh.queryDropped.Count()-dropped)
}
// hostmapEncWriter answers GetHostInfo from a real hostmap, the way Interface does.
type hostmapEncWriter struct {
mockEncWriter
hm *HostMap
}
func (w *hostmapEncWriter) GetHostInfo(a netip.Addr) *HostInfo { return w.hm.QueryVpnAddr(a) }
func (w *hostmapEncWriter) GetCertState() *CertState {
return &CertState{initiatingVersion: cert.Version1}
}
// A lighthouse runs four paths that each hold one lock while waiting for the next: CheckAndComplete
// holds the hostmap and takes the handshake manager, StartHandshake holds the handshake manager and
// takes the lighthouse, a host update holds the lighthouse and takes the sender's RemoteList, and a
// punch notification holds the querier's RemoteList while it resolves the target. If that last step
// reaches the hostmap under the RemoteList lock, the four deadlock. Drive them all at once.
func TestLighthouse_PunchNotificationDoesNotCloseLockCycle(t *testing.T) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
c := config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"am_lighthouse": true}
c.Settings["listen"] = map[string]any{"port": 4242}
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(myVpnNet), nil, nil)
require.NoError(t, err)
mainHM := newHostMap(l)
pr := []netip.Prefix{}
mainHM.preferredRanges.Store(&pr)
lh.ifce = &hostmapEncWriter{hm: mainHM}
hm := NewHandshakeManager(l, mainHM, lh, &udp.NoopConn{}, defaultHandshakeConfig)
f := &Interface{hostMap: mainHM, lightHouse: lh, handshakeManager: hm, l: l}
hm.f = f
peer := netip.MustParseAddr("10.128.0.2")
peerUdp := netip.MustParseAddrPort("10.0.0.2:4242")
target := netip.MustParseAddr("10.128.0.3")
targetUdp := netip.MustParseAddrPort("10.0.0.3:4242")
outbound := netip.MustParseAddr("10.128.0.4")
inbound := netip.MustParseAddr("10.128.0.5")
seed := lh.NewRequestHandler()
newLHHostUpdate(peerUdp, peer, []netip.AddrPort{peerUdp}, seed)
newLHHostUpdate(targetUdp, target, []netip.AddrPort{targetUdp}, seed)
stop := make(chan struct{})
var wg sync.WaitGroup
loop := func(body func()) {
defer wg.Done()
for {
select {
case <-stop:
return
default:
body()
}
}
}
wg.Add(4)
// Outbound handshakes keep starting and being dropped.
go loop(func() {
hm.DeleteHostInfo(hm.StartHandshake(outbound, nil))
})
// Inbound handshakes keep completing and being torn down.
var idx uint32
go loop(func() {
idx++
hi := &HostInfo{
vpnAddrs: []netip.Addr{inbound},
localIndexId: idx,
remoteIndexId: idx,
remotes: lh.QueryCache([]netip.Addr{inbound}),
HandshakePacket: map[uint8][]byte{handshakePacketStage0: {byte(idx)}},
}
if _, err := hm.CheckAndComplete(hi, handshakePacketStage0, f); err == nil {
mainHM.DeleteHostInfo(hi)
}
})
// The peer keeps sending host updates.
updates := lh.NewRequestHandler()
go loop(func() {
newLHHostUpdate(peerUdp, peer, []netip.AddrPort{peerUdp}, updates)
})
// The same peer keeps querying for the target, which ends in a punch notification.
queries := lh.NewRequestHandler()
go loop(func() {
sendLHHostRequest(peerUdp, peer, target, queries, nil)
})
time.Sleep(time.Second)
close(stop)
done := make(chan struct{})
go func() { wg.Wait(); close(done) }()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("deadlock: the lighthouse paths never finished after stop was signalled")
}
}
func TestLighthouse_reloadRelaysTriggersUpdate(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
base := "lighthouse:\n am_lighthouse: true\nlisten:\n port: 4242\n"
require.NoError(t, c.LoadString(base+"relay:\n relays: [10.128.0.5]\n"))
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
assert.Empty(t, lh.updateTrigger)
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6]\n").Err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.6")}, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
<-lh.updateTrigger
// The config changed but what we advertise did not, a bad entry is skipped
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6, nope]\n").Err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.6")}, lh.GetRelaysForMe())
assert.Empty(t, lh.updateTrigger)
}
func TestLighthouse_reloadAmRelayAlone(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
base := "lighthouse:\n am_lighthouse: true\nlisten:\n port: 4242\nrelay:\n relays: [10.128.0.5]\n"
require.NoError(t, c.LoadString(base+" am_relay: false\n"))
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.5")}, lh.GetRelaysForMe())
// Relays aren't allowed to have relays, flipping am_relay alone must drop them and tell the lighthouses
require.NoError(t, c.ReloadConfigString(base+" am_relay: true\n").Err)
assert.Empty(t, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
<-lh.updateTrigger
require.NoError(t, c.ReloadConfigString(base+" am_relay: false\n").Err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.5")}, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
}
func TestLighthouse_reloadRestartRequired(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
base := "lighthouse:\n am_lighthouse: true\n"
require.NoError(t, c.LoadString(base+"listen:\n port: 4242\n"))
_, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
res := c.ReloadConfigString(base + "listen:\n port: 4242\n")
require.NoError(t, res.Err)
assert.False(t, res.RestartRequired)
res = c.ReloadConfigString(base + "listen:\n port: 4243\n")
require.NoError(t, res.Err)
assert.True(t, res.RestartRequired)
}
// A relay that doesn't parse is skipped and the rest are used, the reload says one was skipped
func TestLighthouse_reloadBadRelayFails(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("listen:\n port: 4242\n"))
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
res := c.ReloadConfigString("listen:\n port: 4242\nrelay:\n relays: [10.128.0.2, nope]\n")
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.2")}, lh.GetRelaysForMe())
}
// An advertise_addrs entry inside our own network is skipped and the rest are used, the reload says one was skipped
func TestLighthouse_reloadAdvertiseAddrInNetworkFails(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("listen:\n port: 4242\n"))
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
res := c.ReloadConfigString("listen:\n port: 4242\nlighthouse:\n advertise_addrs: [\"1.2.3.4:4242\", \"10.128.0.5:4242\"]\n")
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Equal(t, []netip.AddrPort{netip.MustParseAddrPort("1.2.3.4:4242")}, lh.GetAdvertiseAddrs())
}
+31
View File
@@ -0,0 +1,31 @@
package logging
import "log/slog"
// RateLimiter is what RateLimited needs from a rate limiter, such as a *ratelimit.Limiter.
// Allow reports whether this call may proceed, and how many calls were denied since the last one that did.
type RateLimiter interface {
Allow() (denied uint64, ok bool)
}
// RateLimited checks lim before anything is built.
// When allowed it returns l, with a "suppressed" count attached if lim denied calls since its last allowed one.
// A nil lim allows everything.
//
// if l, ok := logging.RateLimited(f.l, f.udpTxErrLimiter); ok {
// hostinfo.logger(l).Error("Failed to write outgoing packet", "error", err)
// }
func RateLimited(l *slog.Logger, lim RateLimiter) (*slog.Logger, bool) {
if lim == nil {
return l, true
}
denied, ok := lim.Allow()
if !ok {
return nil, false
}
if denied > 0 {
return l.With("suppressed", denied), true
}
return l, true
}
+106
View File
@@ -0,0 +1,106 @@
package logging
import (
"bytes"
"errors"
"io"
"log/slog"
"net/netip"
"strings"
"testing"
"time"
"github.com/slackhq/nebula/ratelimit"
)
type fakeLimiter struct {
denied uint64
ok bool
}
func (f fakeLimiter) Allow() (uint64, bool) { return f.denied, f.ok }
func TestRateLimited(t *testing.T) {
var buf bytes.Buffer
base := NewLogger(&buf)
if l, ok := RateLimited(base, fakeLimiter{ok: false}); ok || l != nil {
t.Fatalf("denied call: ok=%v logger=%v", ok, l)
}
if l, ok := RateLimited(base, fakeLimiter{ok: true}); !ok || l != base {
t.Fatal("nothing denied should hand back the same logger")
}
if l, ok := RateLimited(base, nil); !ok || l != base {
t.Fatal("a nil limiter should allow with the same logger")
}
l, ok := RateLimited(base, fakeLimiter{denied: 7, ok: true})
if !ok {
t.Fatal("allowed call denied")
}
l.Info("hello", "k", "v")
if !strings.Contains(buf.String(), "suppressed=7") {
t.Fatalf("want suppressed=7, got %q", buf.String())
}
}
// The benchmarks compare a log line through slog directly against the same line behind RateLimited: logged with
// nothing denied, denied, and logged after denials.
var (
benchErr = errors.New("sendto: no buffer space available")
benchAddr = netip.MustParseAddrPort("192.0.2.1:4242")
)
func benchLogger() *slog.Logger {
return NewLogger(io.Discard).With("localIndex", 1234, "remoteIndex", 5678)
}
func BenchmarkRateLimitedBaseline(b *testing.B) {
lg := benchLogger()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
lg.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
func BenchmarkRateLimitedAllowed(b *testing.B) {
lg, lim := benchLogger(), mustLimiter(b, 1, 0)
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if l, ok := RateLimited(lg, lim); ok {
l.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
}
func BenchmarkRateLimitedDenied(b *testing.B) {
lg, lim := benchLogger(), mustLimiter(b, 1, time.Hour)
lim.Allow()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if l, ok := RateLimited(lg, lim); ok {
l.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
}
func BenchmarkRateLimitedAfterDenied(b *testing.B) {
lg, lim := benchLogger(), fakeLimiter{denied: 5, ok: true}
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if l, ok := RateLimited(lg, lim); ok {
l.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
}
func mustLimiter(b *testing.B, burst int, every time.Duration) *ratelimit.Limiter {
l, err := ratelimit.New(burst, every)
if err != nil {
b.Fatal(err)
}
return l
}
+66 -192
View File
@@ -4,7 +4,6 @@ import (
"context"
"fmt"
"log/slog"
"math"
"net"
"net/netip"
"os"
@@ -15,7 +14,6 @@ import (
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/cpupick"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/noiseutil"
"github.com/slackhq/nebula/overlay"
"github.com/slackhq/nebula/sshd"
@@ -34,9 +32,12 @@ const _ = noiseutil.RejectHeadroom - 4*maxRoutines
func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, deviceFactory overlay.DeviceFactory) (retcon *Control, reterr error) {
ctx, cancel := context.WithCancel(context.Background())
// Automatically cancel the context if Main returns an error, to signal all created goroutines to quit.
// The goroutines started below stop only when this context does, and only a caller holding the
// Control can arrange that. Cancel whenever we are not handing one back, which covers an error
// and a config test alike: a config test used to leave the lighthouse query worker, and a
// hostname resolver per dns named static host, running for the life of the process.
defer func() {
if reterr != nil {
if retcon == nil {
cancel()
}
}()
@@ -77,7 +78,7 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
wireSSHReload(l, ssh, c)
var sshStart func()
if c.GetBool("sshd.enabled", false) {
sshStart, err = configSSH(l, ssh, c)
sshStart, err = configSSH(l, ssh, c, nil)
if err != nil {
l.Warn("Failed to configure sshd, ssh debugging will not be available", "error", err)
sshStart = nil
@@ -112,105 +113,6 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
l.Info("Using multiple routines", "routines", routines)
}
port := c.GetInt("listen.port", 0)
batchSize := c.GetInt("listen.batch", 64)
if batchSize < 1 {
oldBatch := batchSize
batchSize = 1
l.Warn("listen.batch size is invalid", "provided", oldBatch, "overridden to", batchSize)
}
offloads := c.GetBool("listen.udp_offloads", false)
var listenHost netip.Addr
if !configTest {
rawListenHost := c.GetString("listen.host", "0.0.0.0")
if rawListenHost == "[::]" {
// Old guidance was to provide the literal `[::]` in `listen.host` but that won't resolve.
listenHost = netip.IPv6Unspecified()
} else {
ips, err := net.DefaultResolver.LookupNetIP(context.Background(), "ip", rawListenHost)
if err != nil {
return nil, util.ContextualizeIfNeeded("Failed to resolve listen.host", err)
}
if len(ips) == 0 {
return nil, util.ContextualizeIfNeeded("Failed to resolve listen.host", err)
}
listenHost = ips[0].Unmap()
}
}
// Multiport lanes: bind a range of consecutive UDP ports (listen.port+p)
// instead of SO_REUSEPORT-sharing one, and derive one extra session per lane
// with capable peers, so a tunnel's inside flows spread over several underlay
// 5-tuples instead of one. Off unless multiport.ports > 1, degrading gracefully when
// preconditions aren't met — managed deployments (dnclient) can't be
// hard-errored on config they don't control.
//
// `routines` is per port here rather than a total to divide up: every port
// gets its own full set, and a port's routines share it through SO_REUSEPORT
// so the kernel hashes each arriving 4-tuple onto one of them. That is what
// keeps a port from being served by a single core — the base port above all,
// since every handshake, every peer without multiport, and every tunnel whose
// lanes are down or firewalled arrives there. A routine still owns exactly one
// socket, which is what lets the read path own its state without locking, so
// the worker count is routines * ports and everything sized by routines from
// here down means that product.
routinesPerPort := routines
multiportPorts := 1
multiport := c.GetBool("multiport.enabled", true)
if multiport {
multiportPorts = c.GetInt("multiport.ports", 0)
if multiportPorts > header.MaxLane+1 {
// A lane index rides in one byte of the nebula header, so a port we
// could never address a lane on is a port we would never send from.
l.Warn("multiport.ports clamped to the lane header limit", "ports", multiportPorts, "limit", header.MaxLane+1)
multiportPorts = header.MaxLane + 1
}
if multiportPorts*routinesPerPort > maxRoutines {
clamped := max(maxRoutines/routinesPerPort, 1)
l.Warn("multiport.ports clamped to the routine limit",
"ports", multiportPorts, "clampedTo", clamped, "routinesPerPort", routinesPerPort, "limit", maxRoutines)
multiportPorts = clamped
}
if multiportPorts < 2 {
// A single port is a plain SO_REUSEPORT listener, which is what a node
// without multiport already runs. Say so rather than claiming lanes.
l.Info("multiport disabled: set multiport.ports > 1 to bind a lane port range")
multiport = false
}
}
if multiport && port != 0 && port+multiportPorts-1 > math.MaxUint16 {
l.Warn("multiport disabled: would bind ports beyond 65535", "listen.port", port, "ports", multiportPorts)
multiport = false
}
if multiport && !configTest {
// Every socket needs its own reader; a platform that can't run multiple
// readers would silently strand all but one of them as blackholes. Probe
// capability before sizing tun queues and routines to the port range.
probe, err := udp.NewListener(l, udp.Settings{
Listen: netip.AddrPortFrom(listenHost, 0),
Batch: 1,
})
if err != nil {
// We could not confirm support, so don't gamble a bound port range on it. The real bind below reports
// the underlying error if it is not transient.
l.Warn("multiport disabled: could not probe udp reader support", "error", err)
multiport = false
} else {
if !probe.SupportsMultipleReaders() {
l.Warn("multiport disabled: this platform does not support multiple udp readers")
multiport = false
}
_ = probe.Close()
}
}
if multiport {
routines = routinesPerPort * multiportPorts
l.Info("multiport routines", "ports", multiportPorts, "routinesPerPort", routinesPerPort, "routines", routines)
}
// EXPERIMENTAL
// Intentionally not documented yet while we do more testing and determine
// a good default value.
@@ -245,6 +147,7 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
// set up our UDP listener
udpConns := make([]udp.Conn, routines)
port := c.GetInt("listen.port", 0)
// Callers get no handle to these until the Control is returned, release them on any error.
defer func() {
@@ -258,72 +161,54 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
}()
if !configTest {
// With a dynamic listen.port, multiport binds the first socket dynamically
// and then claims the next ports-1 above it; if that range turns out to be
// partially occupied, re-roll with a fresh dynamic port.
dynamic := port == 0
var bindErr error
for attempt := 0; attempt < 6; attempt++ {
bindErr = nil
for i := 0; i < routines; i++ {
// Routines are laid out port-major: routine i serves port slot
// i/routinesPerPort, so a port's routines are a contiguous run of
// indices and cpu pinning, which walks the index, spreads each
// port's group across the cores rather than stacking it on one.
slot := 0
if multiport {
slot = i / routinesPerPort
}
udpServer, err := udp.NewListener(l, udp.Settings{
Listen: netip.AddrPortFrom(listenHost, uint16(port+slot)),
// Under multiport the destination port narrows a packet to one
// port's routines and SO_REUSEPORT picks among them by 4-tuple
// hash, so both halves of the steering are load spreading: no
// port depends on a single core, and a lane still has a source
// port of its own to send from.
Multi: routines > 1,
Batch: batchSize,
Offloads: offloads,
})
if err != nil {
bindErr = util.NewContextualError("Failed to open udp listener", m{"queue": i}, err)
break
}
udpServer.ReloadConfig(c)
udpConns[i] = udpServer
rawListenHost := c.GetString("listen.host", "0.0.0.0")
var listenHost netip.Addr
if rawListenHost == "[::]" {
// Old guidance was to provide the literal `[::]` in `listen.host` but that won't resolve.
listenHost = netip.IPv6Unspecified()
// If port is dynamic, discover it before the next pass through the for loop
// This way all routines will use the same port correctly
if port == 0 {
uPort, err := udpServer.LocalAddr()
if err != nil {
return nil, util.NewContextualError("Failed to get listening port", nil, err)
}
port = int(uPort.Port())
if multiport && port+multiportPorts-1 > math.MaxUint16 {
bindErr = util.NewContextualError("multiport dynamic port too close to 65535", m{"port": port}, nil)
break
}
}
l.Info("listening", "addr", netip.AddrPortFrom(listenHost, uint16(port+slot)), "socket", i)
} else {
ips, err := net.DefaultResolver.LookupNetIP(context.Background(), "ip", rawListenHost)
if err != nil {
return nil, util.ContextualizeIfNeeded("Failed to resolve listen.host", err)
}
if bindErr == nil {
break
if len(ips) == 0 {
return nil, util.ContextualizeIfNeeded("Failed to resolve listen.host", err)
}
if !(multiport && dynamic) {
return nil, bindErr
}
for i := range udpConns {
if udpConns[i] != nil {
_ = udpConns[i].Close()
udpConns[i] = nil
}
}
port = 0
l.Debug("multiport dynamic port range collided, retrying", "attempt", attempt+1)
listenHost = ips[0].Unmap()
}
if bindErr != nil {
return nil, bindErr
for i := 0; i < routines; i++ {
listen := netip.AddrPortFrom(listenHost, uint16(port))
l.Info("listening", "addr", listen)
batchSize := c.GetInt("listen.batch", 64)
if batchSize < 1 {
oldBatch := batchSize
batchSize = 1
l.Warn("listen.batch size is invalid", "provided", oldBatch, "overridden to", batchSize)
}
udpSettings := udp.Settings{
Listen: listen,
Multi: routines > 1,
Batch: batchSize,
Offloads: c.GetBool("listen.udp_offloads", false),
}
udpServer, err := udp.NewListener(l, udpSettings)
if err != nil {
return nil, util.NewContextualError("Failed to open udp listener", m{"queue": i}, err)
}
udpServer.ReloadConfig(c, nil)
udpConns[i] = udpServer
// If port is dynamic, discover it before the next pass through the for loop
// This way all routines will use the same port correctly
if port == 0 {
uPort, err := udpServer.LocalAddr()
if err != nil {
return nil, util.NewContextualError("Failed to get listening port", nil, err)
}
port = int(uPort.Port())
}
}
}
@@ -349,26 +234,6 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
messageMetrics: messageMetrics,
}
if multiport {
// A lane needs a port to send from, so we can send on no more lanes than we
// bound ports. multiport.lanes below that sends on a subset of the range,
// which is only interesting for narrowing an experiment: the ports are bound
// and read either way.
lanes := c.GetInt("multiport.lanes", 0)
if lanes <= 0 || lanes > multiportPorts {
lanes = multiportPorts
}
handshakeConfig.laneCount = lanes
handshakeConfig.lanePortCount = uint16(multiportPorts)
handshakeConfig.laneBasePort = uint16(port)
// Every other multiport log line is a reason it turned itself off, so say
// plainly when it is on and with what. A peer only gets lanes if it also
// advertises a port range, so this is our half of the negotiation.
l.Info("multiport enabled", "lanes", lanes, "basePort", port, "ports", multiportPorts,
"routinesPerPort", routinesPerPort)
}
handshakeManager := NewHandshakeManager(l, hostMap, lightHouse, udpConns[0], handshakeConfig)
lightHouse.handshakeTrigger = handshakeManager.trigger
@@ -424,9 +289,6 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
DropLocalBroadcast: c.GetBool("tun.drop_local_broadcast", false),
DropMulticast: c.GetBool("tun.drop_multicast", false),
routines: routines,
Multiport: multiport,
RoutinesPerPort: routinesPerPort,
LaneCount: handshakeConfig.laneCount,
MessageMetrics: messageMetrics,
version: buildVersion,
relayManager: NewRelayManager(ctx, l, hostMap, c),
@@ -438,6 +300,7 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
}
var ifce *Interface
var emitStats func()
if !configTest {
ifce, err = NewInterface(ctx, ifConfig)
if err != nil {
@@ -448,6 +311,16 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
lightHouse.ifce = ifce
ifce.RegisterConfigChangeCallbacks(c)
// These are only read at startup
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged(
"routines", "tun.routines", "listen.routines", "firewall.conntrack.routine_cache_timeout",
"listen.port", "listen.host", "stats.message_metrics",
"handshakes.try_interval", "handshakes.retries", "handshakes.trigger_buffer",
"tun.drop_local_broadcast", "tun.drop_multicast",
)
r.RestartRequiredIfChanged(osStartupOnlyKeys...)
})
ifce.reloadDisconnectInvalid(c)
ifce.reloadSendRecvError(c)
ifce.reloadAcceptRecvError(c)
@@ -456,9 +329,12 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
go handshakeManager.Run(ctx)
punchy.Start(ctx, ifce, hostMap, lightHouse)
// On the stats server's loop, so the gauges follow the interval it accepted
emitStats = ifce.statsEmitter()
}
stats, err := newStatsServerFromConfig(ctx, l, c, buildVersion, configTest)
stats, err := newStatsServerFromConfig(ctx, l, c, buildVersion, configTest, emitStats)
if err != nil {
return nil, util.ContextualizeIfNeeded("Failed to start stats emitter", err)
}
@@ -467,8 +343,6 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
return nil, nil
}
go ifce.emitStats(ctx, c.GetDuration("stats.interval", time.Second*10))
attachCommands(l, c, ssh, ifce)
networkChanges := udp.NewNetworkChangeMonitor(ctx, l, c)
+6
View File
@@ -0,0 +1,6 @@
//go:build linux && !android && !e2e_testing
package nebula
// osStartupOnlyKeys are startup only keys that only matter on linux: udp batch size, udp offloads and thread pinning
var osStartupOnlyKeys = []string{"listen.batch", "listen.udp_offloads", "tun.pin_threads", "tun.pin_threads_key", "tun.cpu_affinity"}
+6
View File
@@ -0,0 +1,6 @@
//go:build !linux || android || e2e_testing
package nebula
// osStartupOnlyKeys is empty, none of the OS specific startup keys matter here
var osStartupOnlyKeys []string
+74
View File
@@ -1,12 +1,21 @@
package nebula
import (
"fmt"
"net/netip"
"os"
"path/filepath"
"testing"
"time"
"github.com/slackhq/nebula/cert"
cert_test "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/util"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/goleak"
)
func TestParseCpuAffinity(t *testing.T) {
@@ -49,3 +58,68 @@ func TestParseCpuAffinity(t *testing.T) {
assert.Nil(t, parseCpuAffinity(newConfig([]any{1 << 20}), l, 1))
}
}
// TestMain_ConfigTestReleasesItsGoroutines pins the rule that Main only leaves goroutines running
// when it hands back a Control to stop them with.
//
// A config test gets no Control, so anything it started had nothing to stop it: the lighthouse
// query worker, and a hostname resolver per dns named static host, ran for the life of the
// process. That matters to every embedder that validates a config in process rather than by
// exec'ing, dnclient and the apple clients included, because they do it on each config load and
// the leak accumulates.
func TestMain_ConfigTestReleasesItsGoroutines(t *testing.T) {
defer goleak.VerifyNone(t, goleak.IgnoreCurrent())
l := test.NewLogger()
dir := t.TempDir()
before := time.Now().Add(-time.Hour)
after := time.Now().Add(time.Hour)
ca, _, caKey, caPEM := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, before, after, nil, nil, nil)
networks := []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")}
_, _, keyPEM, certPEM := cert_test.NewTestCert(
cert.Version2, cert.Curve_CURVE25519, ca, caKey, "config-test", before, after, networks, nil, nil)
caPath := filepath.Join(dir, "ca.pem")
certPath := filepath.Join(dir, "cert.pem")
keyPath := filepath.Join(dir, "key.pem")
require.NoError(t, os.WriteFile(caPath, caPEM, 0o600))
require.NoError(t, os.WriteFile(certPath, certPEM, 0o600))
require.NoError(t, os.WriteFile(keyPath, keyPEM, 0o600))
// A static host by address, not by name: the query worker is the goroutine under test and a
// hostname would drag a real dns lookup into a unit test.
configBody := fmt.Sprintf(`
pki:
ca: %s
cert: %s
key: %s
static_host_map:
"10.0.0.2": ["192.0.2.1:4242"]
lighthouse:
hosts:
- "10.0.0.2"
listen:
host: 127.0.0.1
port: 0
tun:
disabled: true
firewall:
outbound:
- port: any
proto: any
host: any
inbound:
- port: any
proto: any
host: any
`, caPath, certPath, keyPath)
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.yml"), []byte(configBody), 0o600))
c := config.NewC(l)
require.NoError(t, c.Load(dir))
ctrl, err := Main(c, true, "config-test", l, nil)
require.NoError(t, err)
require.Nil(t, ctrl, "a config test hands back nothing to stop, so it must stop itself")
}
+39
View File
@@ -0,0 +1,39 @@
//go:build !android
package nicfinder
import (
"context"
"errors"
"fmt"
"log/slog"
"net"
"net/netip"
)
func localAddrs(ctx context.Context, l *slog.Logger, filter Filter) ([]netip.Addr, error) {
ifaces, err := net.Interfaces()
if err != nil {
return nil, fmt.Errorf("failed to enumerate local interfaces: %w", err)
}
var out []netip.Addr
var errs []error
for _, i := range ifaces {
if !allowName(ctx, l, filter, i.Name) {
continue
}
addrs, err := i.Addrs()
if err != nil {
errs = append(errs, fmt.Errorf("failed to get addresses for %s: %w", i.Name, err))
continue
}
for _, rawAddr := range addrs {
if addr, ok := allowedAddr(ctx, l, filter, rawAddr); ok {
out = append(out, addr)
}
}
}
return out, errors.Join(errs...)
}
+104
View File
@@ -0,0 +1,104 @@
//go:build android
package nicfinder
import (
"context"
"errors"
"fmt"
"log/slog"
"net/netip"
"os"
"github.com/DefinedNet/netlink"
"github.com/DefinedNet/netlink/nl"
"golang.org/x/sys/unix"
)
// Android denies bind() on netlink route sockets and RTM_GETLINK, so net.Interfaces fails with EACCES.
// Enumerate with an RTM_GETADDR dump on an unbound socket instead.
// localInterfaces returns every interface that has an address, with its addresses. It lists
// addresses over a netlink socket that is never bound and names each index with SIOCGIFNAME, the
// same operations Bionic's getifaddrs uses for app UIDs, which the sandbox permits.
func localAddrs(ctx context.Context, l *slog.Logger, filter Filter) ([]netip.Addr, error) {
var errs []error
addrs, err := netlinkAddrList()
if err != nil {
if !errors.Is(err, netlink.ErrDumpInterrupted) {
return nil, fmt.Errorf("failed to enumerate local interfaces: %w", err)
}
// The address table kept changing under the dump and netlink gave up retrying. What it
// returned is still worth advertising.
errs = append(errs, fmt.Errorf("local addresses may be incomplete: %w", err))
}
// Any socket will do for SIOCGIFNAME.
sfd, err := unix.Socket(unix.AF_INET, unix.SOCK_DGRAM|unix.SOCK_CLOEXEC, 0)
if err != nil {
return nil, os.NewSyscallError("socket", err)
}
defer unix.Close(sfd)
var out []netip.Addr
nameAllowed := make(map[int]bool) // by interface index, once named
for _, a := range addrs {
allow, named := nameAllowed[a.LinkIndex]
if (named && !allow) || a.IPNet == nil {
continue
}
addr, ok := allowedAddr(ctx, l, filter, a.IPNet)
if !ok {
continue
}
if !named {
name, err := interfaceNameByIndex(sfd, a.LinkIndex)
// ENODEV means the interface went away after the dump. Any other failure is reported
// and the interface skipped, so the rest are still returned.
if err != nil && !errors.Is(err, unix.ENODEV) {
errs = append(errs, fmt.Errorf("failed to resolve name of interface %d: %w", a.LinkIndex, err))
}
allow = err == nil && allowName(ctx, l, filter, name)
nameAllowed[a.LinkIndex] = allow
}
if allow {
out = append(out, addr)
}
}
return out, errors.Join(errs...)
}
// netlinkAddrList is netlink.AddrList over a socket that is never bound. The kernel autobinds it
// on the first send, which the sandbox permits.
func netlinkAddrList() ([]netlink.Addr, error) {
fd, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW|unix.SOCK_CLOEXEC, unix.NETLINK_ROUTE)
if err != nil {
return nil, os.NewSyscallError("socket", err)
}
s, err := nl.NewNetlinkSocketFromFd(fd)
if err != nil {
unix.Close(fd)
return nil, err
}
h := netlink.NewHandleFromSockets(
map[int]*nl.SocketHandle{unix.NETLINK_ROUTE: {Socket: s}},
netlink.HandleOptions{RetryInterrupted: true},
)
defer h.Close()
if err := h.SetSocketTimeout(netlink.GetSocketTimeout()); err != nil {
return nil, err
}
return h.AddrList(nil, netlink.FAMILY_ALL)
}
// interfaceNameByIndex resolves an interface index with SIOCGIFNAME.
func interfaceNameByIndex(fd int, index int) (string, error) {
var ifr unix.Ifreq
ifr.SetUint32(uint32(index)) // ifr_ifindex is in the ifr_ifru union
if err := unix.IoctlIfreq(fd, unix.SIOCGIFNAME, &ifr); err != nil {
return "", err
}
return ifr.Name(), nil
}
+59
View File
@@ -0,0 +1,59 @@
package nicfinder
import (
"context"
"log/slog"
"net"
"net/netip"
"github.com/slackhq/nebula/logging"
)
// Filter decides which interfaces, by name, and which of their addresses Find returns.
type Filter interface {
AllowName(string) bool
Allow(netip.Addr) bool
}
// Find returns the addresses of this host's network interfaces that allowList accepts, with
// loopback and link-local addresses left out. When only some interfaces could be read it returns
// their addresses alongside an error for the rest. allowList must not be nil; it is called as
// given, so a nil pointer of a type whose methods accept a nil receiver is fine.
func Find(ctx context.Context, l *slog.Logger, allowList Filter) ([]netip.Addr, error) {
return localAddrs(ctx, l, allowList)
}
// allowName asks filter about an interface name and traces the answer.
func allowName(ctx context.Context, l *slog.Logger, filter Filter, name string) bool {
allow := filter.AllowName(name)
if l.Enabled(ctx, logging.LevelTrace) {
l.Log(ctx, logging.LevelTrace, "localAllowList.AllowName", "interfaceName", name, "allow", allow)
}
return allow
}
// allowedAddr converts one address a platform reports and says whether to advertise it: it must
// be a *net.IPNet or *net.IPAddr carrying an IP, not be loopback or link-local, and be accepted by
// filter, whose answer is traced. IPv4 comes back unmapped.
func allowedAddr(ctx context.Context, l *slog.Logger, filter Filter, rawAddr net.Addr) (netip.Addr, bool) {
var ip net.IP
switch v := rawAddr.(type) {
case *net.IPNet:
ip = v.IP
case *net.IPAddr:
ip = v.IP
}
addr, ok := netip.AddrFromSlice(ip)
if !ok {
return netip.Addr{}, false
}
addr = addr.Unmap()
if addr.IsLoopback() || addr.IsLinkLocalUnicast() {
return netip.Addr{}, false
}
isAllowed := filter.Allow(addr)
if l.Enabled(ctx, logging.LevelTrace) {
l.Log(ctx, logging.LevelTrace, "localAllowList.Allow", "localAddr", addr, "allowed", isAllowed)
}
return addr, isAllowed
}
+117
View File
@@ -0,0 +1,117 @@
package nicfinder
import (
"context"
"net"
"net/netip"
"regexp"
"testing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
)
type allowAll struct{}
func (allowAll) Allow(netip.Addr) bool { return true }
func (allowAll) AllowName(string) bool { return true }
type testAllowList struct {
bannedName *regexp.Regexp
bannedAddr netip.Addr
}
func (l *testAllowList) Allow(addr netip.Addr) bool {
return addr != l.bannedAddr
}
func (l *testAllowList) AllowName(n string) bool {
return !l.bannedName.MatchString(n)
}
func newTestAllowList(bannedName string) *testAllowList {
return &testAllowList{
bannedName: regexp.MustCompile(bannedName),
}
}
func TestAllowedAddr(t *testing.T) {
ctx, l := context.Background(), test.NewLogger()
allowed := func(filter Filter, addrs []net.Addr) []netip.Addr {
var out []netip.Addr
for _, rawAddr := range addrs {
if addr, ok := allowedAddr(ctx, l, filter, rawAddr); ok {
out = append(out, addr)
}
}
return out
}
addrs := []net.Addr{
&net.IPNet{IP: net.ParseIP("127.0.0.1"), Mask: net.CIDRMask(8, 32)},
&net.IPNet{IP: net.ParseIP("::1"), Mask: net.CIDRMask(128, 128)},
&net.IPNet{IP: net.ParseIP("10.0.0.5"), Mask: net.CIDRMask(24, 32)},
&net.IPNet{IP: net.ParseIP("169.254.10.10"), Mask: net.CIDRMask(16, 32)},
&net.IPNet{IP: net.ParseIP("fe80::1"), Mask: net.CIDRMask(64, 128)},
&net.IPAddr{IP: net.ParseIP("fd00::5")},
&net.IPNet{},
otherAddr{},
nil,
}
// Loopback, link local and anything without an IP are dropped, and IPv4 comes back unmapped
// even though net.ParseIP hands over a 16 byte slice.
assert.Equal(t, []netip.Addr{
netip.MustParseAddr("10.0.0.5"),
netip.MustParseAddr("fd00::5"),
}, allowed(allowAll{}, addrs))
// The filter sees the unmapped form and can drop a single address.
al := newTestAllowList("none")
al.bannedAddr = netip.MustParseAddr("10.0.0.5")
assert.Equal(t, []netip.Addr{netip.MustParseAddr("fd00::5")}, allowed(al, addrs))
}
// otherAddr is a net.Addr that is neither *net.IPNet nor *net.IPAddr. netlink.Addr is one such
// type: it satisfies net.Addr through an embedded *net.IPNet, and passing it directly once
// silently produced no addresses.
type otherAddr struct{}
func (otherAddr) Network() string { return "other" }
func (otherAddr) String() string { return "other" }
// hostFilter counts AllowName calls and can reject every name or every address.
type hostFilter struct {
rejectNames bool
rejectAddrs bool
asked map[string]int
}
func (f *hostFilter) Allow(netip.Addr) bool { return !f.rejectAddrs }
func (f *hostFilter) AllowName(name string) bool {
f.asked[name]++
return !f.rejectNames
}
// Runs this platform's localAddrs against the host and checks it honors the whole filter.
func TestLocalAddrsAppliesFilter(t *testing.T) {
ctx, l := context.Background(), test.NewLogger()
f := &hostFilter{asked: map[string]int{}}
all, err := localAddrs(ctx, l, f)
if len(f.asked) == 0 {
t.Skipf("no interfaces to enumerate here: %v", err)
}
for name, n := range f.asked {
assert.Equal(t, 1, n, "AllowName asked %d times for %s", n, name)
}
// Loopback and link-local never come back, whatever the filter says.
for _, addr := range all {
assert.False(t, addr.IsLoopback() || addr.IsLinkLocalUnicast(), "%s", addr)
}
none, _ := localAddrs(ctx, l, &hostFilter{rejectNames: true, asked: map[string]int{}})
assert.Empty(t, none, "addresses got past a filter that rejects every interface name")
none, _ = localAddrs(ctx, l, &hostFilter{rejectAddrs: true, asked: map[string]int{}})
assert.Empty(t, none, "addresses got past a filter that rejects every address")
}
-1
View File
@@ -1,5 +1,4 @@
//go:build boringcrypto
// +build boringcrypto
package noiseutil
-1
View File
@@ -1,5 +1,4 @@
//go:build boringcrypto
// +build boringcrypto
package noiseutil
-27
View File
@@ -1,7 +1,6 @@
package noiseutil
import (
"crypto/cipher"
"errors"
"fmt"
"math"
@@ -53,29 +52,3 @@ func NewCipherState(s *noise.CipherState, cipherFunc noise.CipherFunc) CipherSta
panic(fmt.Sprintf("noiseutil: unsupported cipher %q", cipherFunc.CipherName()))
}
}
// NewCipherStateFromKey builds a data-plane CipherState directly from raw key
// bytes, bypassing the noise handshake. Multiport lanes use this to key a
// derived session off the base tunnel's negotiated keys; the caller owns the
// guarantee that key is used with exactly one CipherState so nonces never
// repeat.
func NewCipherStateFromKey(key [32]byte, cipherFunc noise.CipherFunc) CipherState {
c := cipherFunc.Cipher(key)
if cs, ok := c.(CipherState); ok {
return cs
}
aead, ok := c.(cipher.AEAD)
if !ok {
panic(fmt.Sprintf("noiseutil: cipher %q does not expose an AEAD", cipherFunc.CipherName()))
}
switch cipherFunc.CipherName() {
case noise.CipherAESGCM.CipherName():
return &CipherStateAESGCM{c: aead}
case noise.CipherChaChaPoly.CipherName():
return &CipherStateChaChaPoly{c: aead}
default:
panic(fmt.Sprintf("noiseutil: unsupported cipher %q", cipherFunc.CipherName()))
}
}
+30 -85
View File
@@ -102,49 +102,12 @@ func (f *Interface) readOutsidePackets(via ViaSender, packet []byte, rxc *rxCont
// recvError if necessary
if hostinfo == nil || hostinfo.ConnectionState == nil {
if !via.IsRelayed {
f.maybeSendRecvError(via.UdpAddr, h.RemoteIndex, via.SockIdx)
f.maybeSendRecvError(via.UdpAddr, h.RemoteIndex)
}
return
}
// Which session decrypts this packet is the lane index in the header. Lane 0
// is the base tunnel; a higher lane is one of the sessions derived from it.
ci := hostinfo.ConnectionState
lane := h.Lane()
laneCached := true
if lane != 0 {
if isMessageRelay {
// A relay carrier is always the base tunnel, so lane ciphertext can
// never legitimately arrive wrapped in one. Checked before the lookup
// below so a junk relay packet can't make us derive a session.
f.messageMetrics.RxInvalid(1)
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("Refusing relayed multiport lane packet", "from", via, "header", h)
}
return
}
var err error
ci, laneCached, err = hostinfo.laneSession(lane)
if err != nil {
f.messageMetrics.RxInvalid(1)
hostinfo.logger(f.l).Error("Failed to derive multiport lane session", "error", err, "lane", lane)
return
}
if ci == nil {
// A lane this tunnel doesn't have: a stale lane from a tunnel that has
// since rolled, or a peer sending above what it advertised. Dropping
// silently is right for both — a recv_error would tear down a
// perfectly good base tunnel on the strength of one odd packet.
f.messageMetrics.RxInvalid(1)
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("Unknown multiport lane", "from", via, "header", h)
}
return
}
}
if len(packet) < header.Len+ci.dKey.Overhead() {
if len(packet) < header.Len+hostinfo.ConnectionState.dKey.Overhead() {
f.messageMetrics.RxInvalid(1)
if f.l.Enabled(context.Background(), slog.LevelDebug) {
f.l.Debug("packet too small", "from", via, "length", len(packet))
@@ -155,7 +118,7 @@ func (f *Interface) readOutsidePackets(via ViaSender, packet []byte, rxc *rxCont
// All remaining packets are encrypted
if isMessageRelay {
// Relay packets are special, this branch should always early-return
err = ci.VerifyRelay(f.l, h.MessageCounter, packet, rxc.nb)
err = hostinfo.ConnectionState.VerifyRelay(f.l, h.MessageCounter, packet, rxc.nb)
if err != nil {
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("Failed to verify relay packet", "error", err, "from", via, "header", h)
@@ -166,7 +129,7 @@ func (f *Interface) readOutsidePackets(via ViaSender, packet []byte, rxc *rxCont
return
}
out, err := ci.Decrypt(f.l, h.MessageCounter, packet, rxc.nb)
out, err := hostinfo.ConnectionState.Decrypt(f.l, h.MessageCounter, packet, rxc.nb)
if err != nil {
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("Failed to decrypt packet", "error", err, "from", via, "header", h)
@@ -174,27 +137,15 @@ func (f *Interface) readOutsidePackets(via ViaSender, packet []byte, rxc *rxCont
return
}
if !laneCached {
// The packet decrypted, so the peer really is using this lane and the
// session we derived for it is worth keeping.
hostinfo.lanes.installSession(f.l, lane, ci, h.MessageCounter)
}
// Roam before we respond, but only on the base tunnel: a lane's source
// address is a per-lane 4-tuple, not the tunnel's remote, and letting it
// roam the hostinfo would point every non-lane packet at a lane port.
if lane == 0 {
f.handleHostRoaming(hostinfo, via)
}
if !isLaneProbe(h.Type, h.Subtype) {
f.connectionManager.In(hostinfo)
}
// Roam before we respond
f.handleHostRoaming(hostinfo, via)
f.connectionManager.In(hostinfo)
switch h.Type {
case header.Message:
switch h.Subtype {
case header.MessageNone:
f.handleOutsideMessagePacket(hostinfo, ci, h.MessageCounter, out, rxc)
f.handleOutsideMessagePacket(hostinfo, h.MessageCounter, out, rxc)
default:
hostinfo.logger(f.l).Error("IsValidSubType was true, but unexpected message subtype seen", "from", via, "header", h)
return
@@ -209,9 +160,7 @@ func (f *Interface) readOutsidePackets(via ViaSender, packet []byte, rxc *rxCont
case header.TestReply:
// No-op, useful for the Roaming and connectionManager side-effects above
case header.TestRequest:
const maxCipherOverhead = 16 //todo we use this too often, needs a real importable const
const maxOverhead = header.Len + header.Len + maxCipherOverhead + maxCipherOverhead
if maxOverhead+len(out) > len(rxc.scratch) {
if header.MaxOverhead+len(out) > len(rxc.scratch) {
// A reply that cannot fit in scratch is dropped no matter the log level.
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("dropping oversized test request", "payloadLen", len(out), "from", via)
@@ -219,10 +168,6 @@ func (f *Interface) readOutsidePackets(via ViaSender, packet []byte, rxc *rxCont
return
}
f.send(header.Test, header.TestReply, hostinfo.ConnectionState, hostinfo, out, rxc.nb, rxc.scratch[:0])
case header.LaneProbe:
f.handleLaneProbe(hostinfo, lane, out, rxc)
case header.LaneProbeAck:
f.handleLaneProbeAck(hostinfo, out)
default:
hostinfo.logger(f.l).Error("IsValidSubType was true, but unexpected test subtype seen", "from", via, "header", h)
return
@@ -267,7 +212,6 @@ func (f *Interface) handleOutsideRelayPacket(hostinfo *HostInfo, via ViaSender,
relayHI: hostinfo,
relay: relay,
IsRelayed: true,
SockIdx: via.SockIdx,
}
f.readOutsidePackets(via, signedPayload, rxc)
case ForwardingType:
@@ -317,11 +261,16 @@ func (f *Interface) handleOutsideRelayPacket(hostinfo *HostInfo, via ViaSender,
// closeTunnel closes a tunnel locally, it does not send a closeTunnel packet to the remote
func (f *Interface) closeTunnel(hostInfo *HostInfo) {
final := f.hostMap.DeleteHostInfo(hostInfo)
if final {
// We no longer have any tunnels with this vpn addr, clear learned lighthouse state to lower memory usage
f.lightHouse.DeleteVpnAddrs(hostInfo.vpnAddrs)
// Hold the hostmap lock across both the delete and the lighthouse eviction,
// to avoid a race with handshake completion.
f.hostMap.Lock()
defer f.hostMap.Unlock()
if !f.hostMap.unlockedDeleteHostInfo(hostInfo) {
return
}
// We no longer have any tunnels with this vpn addr, clear learned lighthouse state to lower memory usage
f.lightHouse.DeleteVpnAddrs(hostInfo.vpnAddrs)
}
// sendCloseTunnel is a helper function to send a proper close tunnel packet to a remote
@@ -408,7 +357,7 @@ func parseV6(data []byte, incoming bool, fp *firewall.ParsedPacket) error {
// and cannot drift into misreading an unknown protocol (SCTP, GRE, etc.) as a forged transport.
proto, offset, isFragment, anyFragment, err := iputil.IPv6FindUpperProtocol(data)
if err != nil {
return ErrIPv6PacketTooShort
return err
}
fp.Protocol = proto
@@ -524,17 +473,17 @@ func parseV4(data []byte, incoming bool, fp *firewall.ParsedPacket) error {
return nil
}
func (f *Interface) handleOutsideMessagePacket(hostinfo *HostInfo, ci *ConnectionState, messageCounter uint64, out []byte, rxc *rxContext) {
func (f *Interface) handleOutsideMessagePacket(hostinfo *HostInfo, messageCounter uint64, out []byte, rxc *rxContext) {
err := newPacket(out, true, rxc.fwPacket)
if err != nil {
hostinfo.logger(f.l).Warn("Error while validating inbound packet", "error", err, "packet", out)
return
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("Error while validating inbound packet", "error", err, "packet", out)
return
}
}
dropReason := f.firewall.Drop(rxc.fwPacket.Packet, true, hostinfo, f.pki.GetCAPool(), rxc.ctCache.Get())
if dropReason != nil {
// The reject rides the base tunnel: it is a control response, not lane
// data, and the lane it arrived on says nothing about where it belongs.
f.rejectOutside(out, hostinfo.ConnectionState, hostinfo, rxc.nb, rxc.scratch, rxc.q)
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("dropping inbound packet", "fwPacket", rxc.fwPacket, "reason", dropReason)
@@ -542,27 +491,23 @@ func (f *Interface) handleOutsideMessagePacket(hostinfo *HostInfo, ci *Connectio
return
}
err = f.batchers[rxc.q].Commit(out, batch.SortKey{Epoch: ci.epoch, Counter: messageCounter}, rxc.fwPacket)
err = f.batchers[rxc.q].Commit(out, batch.SortKey{Epoch: hostinfo.ConnectionState.epoch, Counter: messageCounter}, rxc.fwPacket)
if err != nil {
f.l.Error("Failed to write to tun", "error", err)
}
}
func (f *Interface) maybeSendRecvError(endpoint netip.AddrPort, index uint32, q int) {
func (f *Interface) maybeSendRecvError(endpoint netip.AddrPort, index uint32) {
if f.sendRecvErrorConfig.ShouldRecvError(endpoint) {
f.sendRecvError(endpoint, index, q)
f.sendRecvError(endpoint, index)
}
}
// sendRecvError replies from the socket the offending packet arrived on (q), so
// the reply to lane traffic rides the reverse of the lane's own 4-tuple: the
// only one the peer's NAT and stateful firewalls are known to let through. The
// peer's handleRecvError accepts it from that lane's target.
func (f *Interface) sendRecvError(endpoint netip.AddrPort, index uint32, q int) {
func (f *Interface) sendRecvError(endpoint netip.AddrPort, index uint32) {
f.messageMetrics.Tx(header.RecvError, 0, 1)
b := header.Encode(make([]byte, header.Len), header.Version, header.RecvError, 0, index, 0)
_ = f.writers[q].WriteTo(b, endpoint)
_ = f.outside.WriteTo(b, endpoint)
if f.l.Enabled(context.Background(), slog.LevelDebug) {
f.l.Debug("Recv error sent",
"index", index,
@@ -594,7 +539,7 @@ func (f *Interface) handleRecvError(addr netip.AddrPort, h *header.H) {
}
hr := hostinfo.GetRemote()
if hr.IsValid() && hr != addr && !hostinfo.lanes.isLaneRemote(addr) {
if hr.IsValid() && hr != addr {
f.l.Info("Someone spoofing recv_errors?",
"addr", addr,
"hostinfoRemote", hr,
+7 -7
View File
@@ -117,7 +117,7 @@ func Test_newPacket_v6(t *testing.T) {
require.NoError(t, err)
err = newPacket(buffer.Bytes(), true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
// A v6 packet with a hop-by-hop extension
// ICMPv6 Payload (Echo Request)
@@ -151,12 +151,12 @@ func Test_newPacket_v6(t *testing.T) {
// A full IPv6 header and 1 byte in the first extension, but missing
// the length byte.
err = newPacket(buffer.Bytes()[:41], true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
// A full IPv6 header plus 1 full extension, but only 1 byte of the
// next layer, missing length byte
err = newPacket(buffer.Bytes()[:49], true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
err = nil
// A good ICMP packet
@@ -354,12 +354,12 @@ func Test_newPacket_v6(t *testing.T) {
// Ensure buffer bounds checking during processing, a truncated AH header can't reach the payload
err = newPacket(b[:41], true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
// Invalid AH header
b = buffer.Bytes()
err = newPacket(b, true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
}
func Test_newPacket_ipv6Fragment(t *testing.T) {
@@ -466,7 +466,7 @@ func Test_newPacket_ipv6Fragment(t *testing.T) {
// Too short of a fragment packet
err = newPacket(secondFrag[:len(secondFrag)-10], false, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
}
func BenchmarkParseV6(b *testing.B) {
@@ -707,7 +707,7 @@ func Test_newPacket_v6ExtHeaderPastBuffer(t *testing.T) {
pkt[40] = byte(layers.IPProtocolSCTP) // Dest Options next header = SCTP
pkt[41] = 255 // declared length (255+1)*8 = 2048, past the 48 byte buffer
require.ErrorIs(t, newPacket(pkt, true, p), ErrIPv6PacketTooShort)
require.ErrorIs(t, newPacket(pkt, true, p), iputil.ErrIPv6CouldNotFindPayload)
}
// Test_newPacket_v6ExtHeaderConfusion is a regression test for parseV6 walking any unrecognized
+2 -2
View File
@@ -127,7 +127,7 @@ func TestPseudoSumIPv6MatchesReference(t *testing.T) {
func TestIPv4HdrChecksumMatchesReference(t *testing.T) {
rng := rand.New(rand.NewSource(0x1791))
for _, hdrLen := range []int{20, 24, 40, 60} {
for trial := 0; trial < 200; trial++ {
for trial := range 200 {
hdr := make([]byte, hdrLen)
rng.Read(hdr)
hdr[0] = 0x40 | byte(hdrLen/4)
@@ -156,7 +156,7 @@ func TestIPv4HdrChecksumMatchesReference(t *testing.T) {
// the way a receiver does (pseudo-header + L4 must sum to all-ones).
func TestChecksumSeedReceiverAcceptance(t *testing.T) {
rng := rand.New(rand.NewSource(0x1826))
for trial := 0; trial < 200; trial++ {
for trial := range 200 {
src := [4]byte{byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256))}
dst := [4]byte{byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256))}
payLen := rng.Intn(1500)
+1 -1
View File
@@ -540,7 +540,7 @@ func TestCoalescerCapBySegments(t *testing.T) {
c := newTestTCPCoalescer(t, w)
pay := make([]byte, 512)
seq := uint32(1000)
for i := 0; i < tcpCoalesceMaxSegs+5; i++ {
for range tcpCoalesceMaxSegs + 5 {
if err := c.Commit(buildTCPv4(seq, tcpAck, pay)); err != nil {
t.Fatal(err)
}
+9 -28
View File
@@ -13,36 +13,19 @@ type batchWriter interface {
// One SendBatch is owned by each listenIn goroutine; no locking is needed.
// Slots are backed by an Arena (see its docs)
type SendBatch struct {
out batchWriter
bufs [][]byte
dsts []netip.AddrPort
// arena backs the slots. borrowed means someone else owns it and Flush must
// leave it alone.
arena *Arena
borrowed bool
out batchWriter
bufs [][]byte
dsts []netip.AddrPort
arena *Arena
}
// NewSendBatch makes a SendBatch with batchCap slots and an arenaSize byte buffer for slices to back those slots
func NewSendBatch(out batchWriter, batchCap, arenaSize int) *SendBatch {
return newSendBatch(out, batchCap, NewArena(arenaSize), false)
}
// NewSendBatchSharedArena makes a SendBatch that borrows arena rather than
// allocating one. Several batches over different sockets can then split a single
// slab, which is what multiport lanes need: the slab is the expensive part of a
// SendBatch and one routine may hold a batch per lane. Flush leaves the arena
// alone, so the owner must Reset it once every batch sharing it is drained.
func NewSendBatchSharedArena(out batchWriter, batchCap int, arena *Arena) *SendBatch {
return newSendBatch(out, batchCap, arena, true)
}
func newSendBatch(out batchWriter, batchCap int, arena *Arena, borrowed bool) *SendBatch {
return &SendBatch{
out: out,
bufs: make([][]byte, 0, batchCap),
dsts: make([]netip.AddrPort, 0, batchCap),
arena: arena,
borrowed: borrowed,
out: out,
bufs: make([][]byte, 0, batchCap),
dsts: make([]netip.AddrPort, 0, batchCap),
arena: NewArena(arenaSize),
}
}
@@ -71,8 +54,6 @@ func (b *SendBatch) Flush() (int, error) {
clear(b.bufs)
b.bufs = b.bufs[:0]
b.dsts = b.dsts[:0]
if !b.borrowed {
b.arena.Reset()
}
b.arena.Reset()
return written, err
}
+2 -2
View File
@@ -28,7 +28,7 @@ func TestSendBatchReserveCommitFlush(t *testing.T) {
b := NewSendBatch(fw, 4, 32)
ap := netip.MustParseAddrPort("10.0.0.1:4242")
for i := 0; i < 4; i++ {
for i := range 4 {
slot := b.Reserve(32)
if cap(slot) != 32 {
t.Fatalf("slot %d: cap=%d want 32", i, cap(slot))
@@ -72,7 +72,7 @@ func TestSendBatchSlotsDoNotOverlap(t *testing.T) {
b := NewSendBatch(fw, 3, 8)
ap := netip.MustParseAddrPort("10.0.0.1:80")
for i := 0; i < 3; i++ {
for i := range 3 {
s := b.Reserve(8)
pkt := append(s[:0], byte(0xA0+i), byte(0xB0+i))
b.Commit(pkt, ap)
+3 -3
View File
@@ -129,7 +129,7 @@ func TestUDPCoalescerCoalescesEqualSized(t *testing.T) {
w := &fakeTunWriter{gsoEnabled: true}
c := newTestUDPCoalescer(t, w)
pay := make([]byte, 1200)
for i := 0; i < 3; i++ {
for range 3 {
if err := c.Commit(buildUDPv4(1000, 53, pay)); err != nil {
t.Fatal(err)
}
@@ -259,7 +259,7 @@ func TestUDPCoalescerCapsAtMaxSegs(t *testing.T) {
w := &fakeTunWriter{gsoEnabled: true}
c := newTestUDPCoalescer(t, w)
pay := make([]byte, 100)
for i := 0; i < udpCoalesceMaxSegs+5; i++ {
for range udpCoalesceMaxSegs + 5 {
if err := c.Commit(buildUDPv4(1000, 53, pay)); err != nil {
t.Fatal(err)
}
@@ -317,7 +317,7 @@ func TestUDPCoalescerIPv6Coalesces(t *testing.T) {
w := &fakeTunWriter{gsoEnabled: true}
c := newTestUDPCoalescer(t, w)
pay := make([]byte, 1200)
for i := 0; i < 3; i++ {
for range 3 {
if err := c.Commit(buildUDPv6(1000, 53, pay)); err != nil {
t.Fatal(err)
}
+1 -1
View File
@@ -151,7 +151,7 @@ func TestChecksumTailPaths(t *testing.T) {
offsets := []int{0, 1, 3, 7, 15} // mix of aligned and odd starts
for k := 0; k <= maxK; k++ {
for tail := 0; tail < 64; tail++ {
for tail := range 64 {
length := 64*k + tail
for _, seed := range seeds {
for _, off := range offsets {
+57
View File
@@ -0,0 +1,57 @@
//go:build !e2e_testing
package overlay
import (
"fmt"
"log/slog"
"golang.org/x/sys/windows"
"golang.org/x/sys/windows/registry"
)
var tcpipInterfaceKeys = []string{
`SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces`,
`SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\Interfaces`,
}
// disableDNSRegistration prevents publishing overlay addresses in the host's
// DNS records, where clients outside the overlay cannot reach them.
// Failures are non-fatal so registry access cannot prevent tunnel creation.
func disableDNSRegistration(l *slog.Logger, guid windows.GUID) {
id := guid.String()
for _, base := range tcpipInterfaceKeys {
path := base + `\` + id
if err := setDNSRegistrationValues(path); err != nil {
l.Warn("Failed to disable DNS registration on the tun adapter", "error", err, "key", path)
continue
}
l.Debug("Disabled DNS registration on the tun adapter", "key", path)
}
}
func setDNSRegistrationValues(path string) error {
// The stack may not have created the adapter's interface key yet.
k, _, err := registry.CreateKey(registry.LOCAL_MACHINE, path, registry.SET_VALUE)
if err != nil {
return fmt.Errorf("open HKLM\\%s: %w", path, err)
}
defer k.Close()
// Neither RegistrationEnabled nor DisableDynamicUpdate alone reliably
// prevents registration, so all three settings are required.
values := []struct {
name string
data uint32
}{
{"RegistrationEnabled", 0},
{"DisableDynamicUpdate", 1},
{"MaxNumberOfAddressesToRegister", 0},
}
for _, v := range values {
if err := k.SetDWordValue(v.name, v.data); err != nil {
return fmt.Errorf("set %s: %w", v.name, err)
}
}
return nil
}
-1
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
Loaded 100 of 176 files, more files were not shown because too many files have changed in this diff. Show more