Compare commits

..
Author SHA1 Message Date
JackDoan 781a590b8a e2e/darwin: Add vpnemu, a configd harness that reproduces exit node route hijacks 2026-10-12 12:49:28 -05:00
Ryan Huber 8a2de9b1b5 examples: set socket buffers and raise tun tx_queue (#1928)
* examples: set socket buffers and raise tun tx_queue

* examples: use the default tcp max window for socket buffers

* examples: note buffers are linux only, clarify tx_queue default

* tun: default tx_queue to 5000
2026-10-11 00:30:07 -04:00
Nate Brown a8c52f5c13 overlay: Handle routes Windows already has, and parse quoted route numbers (#1924) 2026-10-08 18:52:53 -05:00
Jack Doan 13af8dd6f3 Cap configured MTUs at 8937 so relayed sends never outgrow their buffers (#1899) 2026-10-08 15:41:08 -05:00
Nate Brown 5bb0622192 stats: Follow stats.interval on reload, interface gauges ride the capture loop (#1919) 2026-10-08 10:00:48 -05:00
Nate Brown f3705cbb67 graphite: Time out a send, and abandon it on stop (#1920) 2026-10-07 18:01:09 -05:00
Nate Brown 94be454f4f lighthouse: Recompute relays for me when relay.am_relay changes (#1917) 2026-10-07 14:06:27 -05:00
Nate Brown 5f26c1a7e5 Add a simple rate limiter with a helper to use it in logging (#1915) 2026-10-07 10:53:37 -05:00
Nate Brown d47c8c71f1 lighthouse: Send an update when the relays we advertise change on reload (#1916) 2026-10-07 10:52:42 -05:00
Nate Brown f57539a163 overlay/linux: Keep unsafe routes when a reload only changes tun.mtu (#1918) 2026-10-07 10:43:11 -05:00
John Maguire 8d8ae0f744 Disable DNS registration on the Windows tunnel adapter (#1914)
On a domain-joined Windows host the DNS Client registers every adapter's
addresses under the host name in AD DNS when "Register this connection's
addresses in DNS" is on, which is the default for a new adapter. When the
Nebula adapter registers, LAN clients that are not on the overlay resolve
the host name to its overlay IP and RDP, SMB and PowerShell Remoting to it
fail intermittently, with nothing pointing at the tunnel as the cause.
Windows registers on address assignment, periodically, and on every
ipconfig /registerdns.

Right after the wintun adapter is created and before Activate assigns any
address, write RegistrationEnabled=0, DisableDynamicUpdate=1 and
MaxNumberOfAddressesToRegister=0 under both the Tcpip and Tcpip6
Parameters\Interfaces\{GUID} keys. The adapter checkbox alone has proven
insufficient in practice, as has disabling dynamic updates alone, so all
three are set, matching what Tailscale and NetBird do. A failure to write
the keys is logged as a warning and does not fail adapter creation.
2026-10-07 11:42:08 -04:00
Matt Richardson f9b0853476 Attach a pending handshake's RemoteList when the handshake starts (#1909)
StartHandshake returned a hostinfo with no RemoteList; handleOutbound attached
one lazily on its first tick. Anything that touched the pending host before
then, notably create-tunnel -address, dereferenced nil and took the daemon
down. StartHandshake now attaches the lighthouse's list as it builds the
hostinfo. To keep the surrounding locking sound, a lighthouse now resolves a
punch target before taking the querier's RemoteList lock, and closeTunnel
deletes the host and evicts its RemoteList under a single hostmap write lock so
a completing handshake cannot land in between. The lock order, now documented
on the handshake manager mutex, is hostmap, then handshake manager, then
lighthouse, then RemoteList; no RemoteList lock is ever held across a hostmap
lookup.
2026-10-07 11:22:50 -04:00
Nate Brown d5062dacf0 lighthouse: Never block in QueryServer, drop queries when the channel is full (#1910) 2026-09-30 15:26:34 -05:00
Jack Doan 7cfa47d569 tweaks to error handling in iputil.IPv6FindUpperProtocol (#1893) 2026-09-22 15:45:01 -05:00
Caleb JasikandNate Brown e20569ce51 Let the darwin tun adopt an fd the host already configured (#1890)
newTunFromFd was a stub on darwin, so an embedder handed a utun by the OS had
no way to use it. A darwin network extension is given exactly that:
NEPacketTunnelProvider creates the device and applies addressing, mtu and
routes from its network settings, and a sandboxed extension cannot change them
afterwards. Adopting that fd means everything about moving packets is shared
with newTun and only the setup differs, so the tun tracks whether the host owns
it and leaves the system routing table and interface config alone when it does.

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-09-22 11:54:30 -05:00
Caleb JasikandNate Brown 9651ffdb0d Cancel the context when Main hands back no Control (#1889)
The goroutines Main starts stop only when its context does, and only a caller
holding the Control can arrange that. Cancelling solely on error left a config
test, which returns no Control and no error, with the lighthouse query worker
and a hostname resolver per dns named static host running for the life of the
process. Every embedder that validates a config in process rather than by
exec'ing pays for that on each config load, dnclient and the apple clients
included.

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-09-22 11:52:28 -05:00
Jack Doan 3a45c01656 Advertise underlay addresses on Android (#1881) 2026-09-21 14:59:56 -05:00
Jack Doan 019cc46a7f smoke: Prove windows honors NlMtu with a 9000 byte ping (#1873) 2026-09-21 12:59:31 -05:00
Nate Brown cf3f315cd3 IPv6FindUpperProtocol: fail closed when the extension header walk runs out of budget (#1883) 2026-09-21 12:09:51 -05:00
Wade Simmons 3fc627e08d Add script to backport a merged PR to a release branch (#1851)
* Add script to backport a merged PR to a release branch

.github/backport.sh cherry-picks a merged PR's commit onto a release
branch and opens a backport PR mirroring the shape of #1842.

- Infers the target release from the PR's milestone (v1.12.0 -> release-1.11);
  an explicit target-version argument overrides it.
- Requires a clean working tree (untracked files allowed) before starting.
- Prompts with the exact push and gh pr create commands before running them.
- Sets the new PR's milestone to the highest open v<target>.* milestone.
- Removes the needs-backport label from the original PR once the backport opens.
- Supports --continue to resume after resolving cherry-pick conflicts.

* cleanup

* just fetch all of origin

We need to ensure we have MERGE_SHA available too
2026-09-14 16:14:43 -04:00
dependabot[bot] abff1a4555 Bump Vampire/setup-wsl from 3 to 7 (#1720)
Bumps [Vampire/setup-wsl](https://github.com/vampire/setup-wsl) from 3 to 7.
- [Release notes](https://github.com/vampire/setup-wsl/releases)
- [Commits](https://github.com/vampire/setup-wsl/compare/v3...v7)

---
updated-dependencies:
- dependency-name: Vampire/setup-wsl
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 10:11:27 -04:00
dependabot[bot] 227134a1d5 Bump golang.org/x/net in the golang-x-dependencies group (#1876)
Bumps the golang-x-dependencies group with 1 update: [golang.org/x/net](https://github.com/golang/net).


Updates `golang.org/x/net` from 0.58.0 to 0.59.0
- [Commits](https://github.com/golang/net/compare/v0.58.0...v0.59.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.59.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 10:01:25 -04:00
dependabot[bot] 232bacfdd0 Bump github.com/gaissmai/bart from 0.28.0 to 0.29.0 (#1829)
Bumps [github.com/gaissmai/bart](https://github.com/gaissmai/bart) from 0.28.0 to 0.29.0.
- [Release notes](https://github.com/gaissmai/bart/releases)
- [Commits](https://github.com/gaissmai/bart/compare/v0.28.0...v0.29.0)

---
updated-dependencies:
- dependency-name: github.com/gaissmai/bart
  dependency-version: 0.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:39:13 -04:00
dependabot[bot] 4e1ef9cd73 Bump the golang-x-dependencies group across 1 directory with 5 updates (#1875)
Bumps the golang-x-dependencies group with 2 updates in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto) and [golang.org/x/sync](https://github.com/golang/sync).


Updates `golang.org/x/crypto` from 0.54.0 to 0.57.0
- [Commits](https://github.com/golang/crypto/compare/v0.54.0...v0.57.0)

Updates `golang.org/x/net` from 0.57.0 to 0.58.0
- [Commits](https://github.com/golang/net/compare/v0.57.0...v0.58.0)

Updates `golang.org/x/sync` from 0.22.0 to 0.23.0
- [Commits](https://github.com/golang/sync/compare/v0.22.0...v0.23.0)

Updates `golang.org/x/sys` from 0.47.0 to 0.48.0
- [Commits](https://github.com/golang/sys/compare/v0.47.0...v0.48.0)

Updates `golang.org/x/term` from 0.45.0 to 0.46.0
- [Commits](https://github.com/golang/term/compare/v0.45.0...v0.46.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.57.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.58.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/sync
  dependency-version: 0.23.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/sys
  dependency-version: 0.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/term
  dependency-version: 0.46.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:37:37 -04:00
dependabot[bot] 614e0ea3aa Bump google.golang.org/protobuf (#1846)
Bumps the protobuf-dependencies group with 1 update in the / directory: google.golang.org/protobuf.


Updates `google.golang.org/protobuf` from 1.36.11 to 1.36.12

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-version: 1.36.12
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: protobuf-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:31:44 -04:00
dependabot[bot] 2999fc0b6a Bump github.com/stretchr/testify from 1.12.0 to 1.12.1 (#1859)
Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.12.0 to 1.12.1.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.12.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-version: 1.12.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:29:03 -04:00
dependabot[bot] f474b534e1 Bump github.com/miekg/dns from 1.1.72 to 1.1.73 (#1860)
Bumps [github.com/miekg/dns](https://github.com/miekg/dns) from 1.1.72 to 1.1.73.
- [Commits](https://github.com/miekg/dns/compare/v1.1.72...v1.1.73)

---
updated-dependencies:
- dependency-name: github.com/miekg/dns
  dependency-version: 1.1.73
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-14 09:28:03 -04:00
Wade Simmons 0dc8a28d1f apply go fix for Go 1.26 (#1867)
* apply `go fix` for Go 1.26

This applies the `go fix` recommendations for go1.26

* for all of ./...
2026-09-14 09:25:35 -04:00
Nate Brown 89178f45ba windows: Fix NLMTU for ipv6 (#1871)
smoke-extra / freebsd-amd64 (push) Failing after 16s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 15s
smoke-extra / netbsd-amd64 (push) Failing after 14s
smoke-extra / openbsd-amd64 (push) Failing after 17s
smoke-extra / linux-386 (push) Failing after 16s
smoke / Run multi node smoke test (push) Failing after 1m33s
Build and test / Static checks (push) Successful in 21s
Build and test / Test linux (push) Failing after 59s
Build and test / Test linux-pkcs11 (push) Failing after 1m49s
Build and test / Test linux-boringcrypto (push) Failing after 2m38s
Build and test / Test linux-fips140 (push) Failing after 2m52s
Build and test / Cross-build linux-arm (push) Successful in 2m54s
Build and test / Cross-build linux-mips (push) Successful in 3m35s
Build and test / Cross-build linux-other (push) Successful in 2m58s
Build and test / Cross-build windows (push) Successful in 59s
Build and test / Cross-build freebsd (push) Successful in 1m28s
Build and test / Cross-build netbsd (push) Successful in 1m28s
Build and test / Cross-build openbsd (push) Successful in 1m28s
Build and test / Cross-build mobile (push) Successful in 3m7s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
2026-09-08 18:30:16 -05:00
Wade Simmons 6a72e1c304 v1.11.1 Changelog (#1857)
smoke-extra / freebsd-amd64 (push) Failing after 18s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 17s
smoke-extra / netbsd-amd64 (push) Failing after 14s
smoke-extra / openbsd-amd64 (push) Failing after 16s
smoke-extra / linux-386 (push) Failing after 17s
smoke / Run multi node smoke test (push) Failing after 1m33s
Build and test / Static checks (push) Successful in 23s
Build and test / Test linux (push) Failing after 58s
Build and test / Test linux-pkcs11 (push) Failing after 1m50s
Build and test / Test linux-boringcrypto (push) Failing after 2m36s
Build and test / Test linux-fips140 (push) Failing after 3m0s
Build and test / Cross-build linux-arm (push) Successful in 2m50s
Build and test / Cross-build linux-mips (push) Successful in 3m34s
Build and test / Cross-build linux-other (push) Successful in 2m59s
Build and test / Cross-build windows (push) Successful in 1m0s
Build and test / Cross-build freebsd (push) Successful in 1m28s
Build and test / Cross-build netbsd (push) Successful in 1m28s
Build and test / Cross-build openbsd (push) Successful in 1m28s
Build and test / Cross-build mobile (push) Successful in 3m9s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
Copy the v1.11.1 CHANGELOG updates from release-v1.11
2026-09-08 13:50:27 -04:00
Matt Richardson e50f8128f4 Drop hostQueries to our overlay addresses (#1866)
smoke-extra / freebsd-amd64 (push) Failing after 24s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 23s
smoke-extra / netbsd-amd64 (push) Failing after 16s
smoke-extra / openbsd-amd64 (push) Failing after 11s
smoke-extra / linux-386 (push) Failing after 22s
smoke / Run multi node smoke test (push) Failing after 1m37s
Build and test / Static checks (push) Successful in 2m16s
Build and test / Test linux (push) Failing after 1m20s
Build and test / Test linux-pkcs11 (push) Failing after 1m50s
Build and test / Test linux-boringcrypto (push) Failing after 2m39s
Build and test / Test linux-fips140 (push) Failing after 2m51s
Build and test / Cross-build linux-arm (push) Successful in 2m55s
Build and test / Cross-build linux-mips (push) Successful in 3m41s
Build and test / Cross-build linux-other (push) Successful in 3m0s
Build and test / Cross-build windows (push) Successful in 1m1s
Build and test / Cross-build freebsd (push) Successful in 1m30s
Build and test / Cross-build netbsd (push) Successful in 1m29s
Build and test / Cross-build openbsd (push) Successful in 1m30s
Build and test / Cross-build mobile (push) Successful in 3m10s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
* Drop hostQueries to our overlay addresses

Don't bother responding to hostQuery's for our overlay addresses.
2026-09-04 11:26:08 -04:00
126 changed files with 3716 additions and 701 deletions

No files matched your search

+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env bash
#
# Backport a merged PR to a release branch.
#
# Cherry-picks the merge commit of a merged PR onto release-1.11 and opens a
# backport PR, mirroring the shape of #1842.
#
# Usage: ./backport.sh <pr-number> [target-version] [--continue]
# pr-number The merged PR to backport
# target-version Release version to target (default: 1.11)
# --continue Resume after resolving cherry-pick conflicts: assumes the
# fixes are staged, runs `git cherry-pick --continue`, and
# proceeds to push and open the backport PR.
#
# Requires: gh (authenticated), git, jq.
set -euo pipefail
PR_NUMBER=${1:?usage: backport.sh <pr-number> [target-version] [--continue]}
shift
CONTINUE=0
TARGET_VERSION=""
for arg in "$@"; do
case "$arg" in
--continue) CONTINUE=1 ;;
*) TARGET_VERSION=$arg ;;
esac
done
for cmd in gh git jq; do
command -v "$cmd" >/dev/null || { echo "error: $cmd is required" >&2; exit 1; }
done
# Pull the source PR's metadata. Refuse to backport a PR that never merged.
pr_json=$(gh pr view "$PR_NUMBER" --json title,body,mergedAt,mergeCommit,milestone)
PR_TITLE=$(jq -r '.title' <<<"$pr_json")
PR_BODY=$(jq -r '.body' <<<"$pr_json")
MERGE_SHA=$(jq -r '.mergeCommit.oid // empty' <<<"$pr_json")
PR_MILESTONE=$(jq -r '.milestone.title // empty' <<<"$pr_json")
if [ "$(jq -r '.mergedAt // empty' <<<"$pr_json")" = "" ] || [ -z "$MERGE_SHA" ]; then
echo "error: PR #${PR_NUMBER} is not merged (no merge commit to cherry-pick)" >&2
exit 1
fi
# Default the target to one minor below the PR's milestone: a PR landing in
# v1.12.0 backports to release-1.11. An explicit target-version arg overrides.
if [ -z "$TARGET_VERSION" ]; then
if [[ $PR_MILESTONE =~ ^v?([0-9]+)\.([0-9]+) ]]; then
TARGET_VERSION="${BASH_REMATCH[1]}.$(( BASH_REMATCH[2] - 1 ))"
else
echo "error: PR #${PR_NUMBER} has no v<major>.<minor>.* milestone to infer the target from." >&2
echo "Pass the target version explicitly, e.g. $0 ${PR_NUMBER} 1.11" >&2
exit 1
fi
fi
TARGET_BRANCH="release-${TARGET_VERSION}"
# Slug from the PR title: lower-case, non-alphanumerics to dashes, trimmed.
slug=$(printf '%s' "$PR_TITLE" \
| tr '[:upper:]' '[:lower:]' \
| sed -E 's/[^a-z0-9]+/-/g; s/^-+//; s/-+$//' \
| cut -c1-50 \
| sed -E 's/-+$//')
BRANCH="backport-${TARGET_VERSION//./-}-${slug}"
if [ "$CONTINUE" -eq 1 ]; then
# Resuming: the branch already exists and a cherry-pick is mid-conflict.
current=$(git rev-parse --abbrev-ref HEAD)
if [ "$current" != "$BRANCH" ]; then
echo "error: --continue expects to be on ${BRANCH}, but HEAD is ${current}" >&2
exit 1
fi
echo "Resuming backport of #${PR_NUMBER} on ${BRANCH}"
# Conflicts assumed resolved and staged; core.editor=true keeps the commit message.
git -c core.editor=true cherry-pick --continue
else
# A fresh run switches branches and cherry-picks, so tracked changes must be
# clean. Untracked files are fine, and --continue is exempt (its resolved
# conflicts are meant to be staged).
if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
echo "error: working tree has uncommitted changes to tracked files; commit or stash them first." >&2
exit 1
fi
echo "Backporting #${PR_NUMBER} (${MERGE_SHA}) onto ${TARGET_BRANCH} as ${BRANCH}"
git fetch origin
git checkout -b "$BRANCH" "origin/${TARGET_BRANCH}"
# -m 1 handles a real merge commit; plain cherry-pick handles a squash merge.
if ! { git cherry-pick -x -m 1 "$MERGE_SHA" 2>/dev/null || git cherry-pick -x "$MERGE_SHA"; }; then
echo "error: ${MERGE_SHA} did not cherry-pick cleanly onto ${TARGET_BRANCH}." >&2
echo "Resolve the conflicts, 'git add' them, then re-run:" >&2
echo " $0 ${PR_NUMBER} ${TARGET_VERSION} --continue" >&2
exit 1
fi
fi
title="backport v${TARGET_VERSION}: ${PR_TITLE}"
body=$(printf 'Backport from #%s to release v%s\n\n---\n\n%s' \
"$PR_NUMBER" "$TARGET_VERSION" "$PR_BODY")
# Highest open milestone matching v<version>.* (e.g. v1.11.1), if any.
MILESTONE=$(gh api "repos/{owner}/{repo}/milestones?state=open" \
--jq ".[].title | select(startswith(\"v${TARGET_VERSION}.\"))" \
| sort -V | tail -n1)
milestone_args=()
if [ -n "$MILESTONE" ]; then
milestone_args=(--milestone "$MILESTONE")
else
echo "warning: no open milestone matching v${TARGET_VERSION}.* found; PR will have no milestone" >&2
fi
echo
echo "About to run:"
printf ' git push -u --force-with-lease origin %q\n' "$BRANCH"
printf ' gh pr create --base %q --head %q --title %q --body %q' \
"$TARGET_BRANCH" "$BRANCH" "$title" "$body"
[ -n "$MILESTONE" ] && printf ' --milestone %q' "$MILESTONE"
printf '\n'
echo
read -r -p "Open this pull request? [y/N] " reply
case "$reply" in
[yY] | [yY][eE][sS]) ;;
*) echo "Aborted. The cherry-pick is on local branch ${BRANCH}; push it manually if desired."; exit 0 ;;
esac
git push -u --force-with-lease origin "$BRANCH"
gh pr create \
--base "$TARGET_BRANCH" \
--head "$BRANCH" \
--title "$title" \
--body "$body" \
${milestone_args[@]+"${milestone_args[@]}"}
# The backport now exists, so drop the label that flagged this PR for one.
gh pr edit "$PR_NUMBER" --remove-label needs-backport
# Switch back to the original branch we were on before doing the backport
git checkout -
+1 -1
View File
@@ -98,7 +98,7 @@ jobs:
# WSL2 + Ubuntu so the smoke can run a real linux peer with its own
# netns. iputils-ping is needed for the in-WSL ping check. WSL1 has no
# real kernel and would lack /dev/net/tun, so we have to force WSL2.
- uses: Vampire/setup-wsl@v3
- uses: Vampire/setup-wsl@v7
with:
distribution: Ubuntu-24.04
additional-packages: iputils-ping iproute2
+48 -5
View File
@@ -51,15 +51,19 @@ wsl -d $Distro -- bash -c "rm -rf $WslDir && mkdir -p $WslDir" | Out-Null
$DevName = 'nebula-smoke'
$Ip1 = '192.168.241.1'
$Ip2 = '192.168.241.2'
# Dual stack on purpose: a v4-only overlay never exercises the v6 side of tun.mtu.
$Ip6_1 = 'fd42:4242:241::1'
$Ip6_2 = 'fd42:4242:241::2'
$Mtu = 1300
$Port = 4242
& $NebulaCert ca -name 'smoke-ca' -out-crt "$WorkDir\ca.crt" -out-key "$WorkDir\ca.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert ca failed (exit $LASTEXITCODE)" }
& $NebulaCert sign -name 'lighthouse' -networks "$Ip1/24" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\lighthouse.crt" -out-key "$WorkDir\lighthouse.key"
& $NebulaCert sign -name 'lighthouse' -networks "$Ip1/24,$Ip6_1/64" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\lighthouse.crt" -out-key "$WorkDir\lighthouse.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert sign lighthouse failed (exit $LASTEXITCODE)" }
& $NebulaCert sign -name 'peer' -networks "$Ip2/24" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\peer.crt" -out-key "$WorkDir\peer.key"
& $NebulaCert sign -name 'peer' -networks "$Ip2/24,$Ip6_2/64" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\peer.crt" -out-key "$WorkDir\peer.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert sign peer failed (exit $LASTEXITCODE)" }
# Windows lighthouse config.
@@ -82,7 +86,7 @@ tun:
drop_local_broadcast: false
drop_multicast: false
tx_queue: 500
mtu: 1300
mtu: $Mtu
network_category: private
logging:
level: info
@@ -126,7 +130,7 @@ tun:
drop_local_broadcast: false
drop_multicast: false
tx_queue: 500
mtu: 1300
mtu: $Mtu
logging:
level: info
format: text
@@ -169,7 +173,7 @@ Write-Host '=== WSL diagnostic ==='
wsl --version 2>&1 | Out-Host
wsl --list --verbose 2>&1 | Out-Host
wsl -d $Distro -u root -- uname -a | Out-Host
wsl -d $Distro -u root -- bash -c "modprobe tun 2>&1 || true; mkdir -p /dev/net; [ -c /dev/net/tun ] || mknod /dev/net/tun c 10 200; chmod 600 /dev/net/tun; ls -l /dev/net/tun"
wsl -d $Distro -u root -- bash -c "modprobe tun 2>&1 || true; mkdir -p /dev/net; [ -c /dev/net/tun ] || mknod /dev/net/tun c 10 200; chmod 600 /dev/net/tun; { echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6; echo 0 > /proc/sys/net/ipv6/conf/default/disable_ipv6; } 2>/dev/null || true; ls -l /dev/net/tun"
if ($LASTEXITCODE -ne 0) { throw "failed to prepare /dev/net/tun in WSL (TUN support missing?)" }
# Deliberately no New-NetFirewallRule calls here -- nebula's windows_bypass_wdf
@@ -214,6 +218,16 @@ try {
}
Write-Host "OK: $DevName NetworkCategory=Private"
# v6 silently kept the adapter default of 65535 while v4 was correct.
foreach ($family in @('IPv4', 'IPv6')) {
Wait-Until -TimeoutSec 30 -What "$DevName $family NlMtu=$Mtu" -Predicate {
if ($lhProc.HasExited) { throw "lighthouse exited (code $($lhProc.ExitCode)) before $family mtu was set" }
$rows = @(Get-NetIPInterface -InterfaceAlias $DevName -AddressFamily $family -ErrorAction SilentlyContinue)
$rows.Count -gt 0 -and -not ($rows | Where-Object { $_.NlMtu -ne $Mtu })
}
Write-Host "OK: $DevName $family NlMtu=$Mtu"
}
Wait-Until -TimeoutSec 30 -What "WSL nebula1 with $Ip2" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before tun was ready" }
$r = wsl -d $Distro -u root -- bash -c "ip -o addr show nebula1 2>/dev/null | grep -q 'inet $Ip2' && echo yes"
@@ -221,6 +235,13 @@ try {
}
Write-Host "OK: WSL nebula1 has $Ip2"
Wait-Until -TimeoutSec 30 -What "WSL nebula1 with $Ip6_2" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the v6 address was up" }
$r = wsl -d $Distro -u root -- bash -c "ip -o addr show nebula1 2>/dev/null | grep -q 'inet6 $Ip6_2' && echo yes"
("$r").Trim() -eq 'yes'
}
Write-Host "OK: WSL nebula1 has $Ip6_2"
Wait-Until -TimeoutSec 30 -What "ping from WSL peer to windows lighthouse ($Ip1)" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before ping succeeded" }
$r = wsl -d $Distro -u root -- bash -c "ping -c1 -W1 $Ip1 >/dev/null 2>&1 && echo OK"
@@ -234,6 +255,28 @@ try {
}
Write-Host "OK: windows lighthouse -> WSL peer"
# Otherwise the v6 networks only prove the interface exists, not that it forwards.
Wait-Until -TimeoutSec 30 -What "v6 ping from WSL peer to windows lighthouse ($Ip6_1)" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the v6 ping succeeded" }
$r = wsl -d $Distro -u root -- bash -c "ping -6 -c1 -W1 $Ip6_1 >/dev/null 2>&1 && echo OK"
("$r").Trim() -eq 'OK'
}
Write-Host "OK: WSL peer -> windows lighthouse over v6"
# The NlMtu check above proves the value was written, not that windows honors it. A payload this size only
# crosses the tunnel as fragments cut at NlMtu on the way out of the adapter. Left at the adapter default it
# reaches nebula whole, overflows the udp.MTU (9001) write buffer and is dropped, so the ping never answers.
# Anything smaller would ride out as one oversized datagram and survive on IP fragmentation of the underlay.
$BigPayload = 9000
foreach ($target in @(@{ Family = 'v4'; Ip = $Ip2 }, @{ Family = 'v6'; Ip = $Ip6_2 })) {
Wait-Until -TimeoutSec 30 -What "$($target.Family) ping with a $BigPayload byte payload from windows lighthouse to WSL peer ($($target.Ip))" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the large $($target.Family) ping succeeded" }
$null = & ping.exe -n 1 -w 1000 -l $BigPayload $target.Ip
$LASTEXITCODE -eq 0
}
Write-Host "OK: windows lighthouse -> WSL peer, $BigPayload byte $($target.Family) payload"
}
Write-Host ''
Write-Host 'All smoke checks passed.'
}
+21 -1
View File
@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- `create-tunnel -address <ip:port>` panicked on a nil RemoteList and, over the ssh debug server, took nebula
down with it. A hostinfo returned by StartHandshake had no RemoteList until the handshake loop's first
pass attached one, and `create-tunnel -address` set the remote before that. The list is now attached when
the handshake starts, so a pending hostinfo never exists without one.
## [1.11.1] - 2026-08-21
See the [v1.11.1](https://github.com/slackhq/nebula/milestone/30?closed=1) milestone for a complete list of changes.
### Changed
- IPv6 packets whose next header is a protocol Nebula does not parse (SCTP, GRE, IP-in-IP, etc.) are now
@@ -15,11 +26,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
their true protocol, so only a `proto: any` rule allows them. If you carry one of these protocols over the
overlay, confirm a `proto: any` rule covers it before upgrading, it may have been passing only through this
bypass. (#1840)
- Drop the dependency on `github.com/cyberdelia/go-metrics-graphite`, which has been unmaintained for over ten
years, by inlining the small amount of code Nebula used. (#1832)
### Fixed
- The ICMPv6 type was read from the wrong byte when classifying IPv6 packets, so the echo identifier used
for conntrack was never picked up. (#1840)
- Enforce outbound message counter limits so a tunnel is rehandshaked before the counter can wrap, preventing
nonce reuse. This is unreachable in practice, but is enforced as a defense-in-depth measure. (#1841)
- Prevent `nebula-cert ca` from running out of memory on 32bit systems when generating encrypted private keys. (#1834)
- Tolerate `ErrDumpInterrupted` when listing tun addresses on Linux, so a transient interrupted netlink dump
no longer aborts startup. (#1835)
## [1.11.0] - 2026-07-23
@@ -884,7 +902,9 @@ created.)
- Initial public release.
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.10.3...HEAD
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.11.1...HEAD
[1.11.1]: https://github.com/slackhq/nebula/releases/tag/v1.11.1
[1.11.0]: https://github.com/slackhq/nebula/releases/tag/v1.11.0
[1.10.3]: https://github.com/slackhq/nebula/releases/tag/v1.10.3
[1.10.2]: https://github.com/slackhq/nebula/releases/tag/v1.10.2
[1.10.1]: https://github.com/slackhq/nebula/releases/tag/v1.10.1
+3
View File
@@ -242,11 +242,14 @@ test-cov-html:
go test -coverprofile=coverage.out
go tool cover -html=coverage.out
# The package builds only compile. The final line links an android binary so a linker-only failure
# cannot pass CI.
build-test-mobile:
GOARCH=amd64 GOOS=ios go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=ios go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=amd64 GOOS=android go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=android go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=android go build -o /dev/null ${NEBULA_CMD_PATH}
bench:
go test -bench=.
+2 -11
View File
@@ -79,13 +79,7 @@ func (b *Bits) clearRange(startPos, count uint64) uint64 {
// handle the potential partial word before pos becomes u64 aligned
word := pos >> 6
bit := pos & 63
take := uint64(64) - bit
if take > remaining {
take = remaining
}
if take > b.length-pos {
take = b.length - pos
}
take := min(min(uint64(64)-bit, remaining), b.length-pos)
var mask uint64
if take == 64 {
mask = math.MaxUint64
@@ -189,10 +183,7 @@ func (b *Bits) Update(l *slog.Logger, i uint64) bool {
func (b *Bits) updateSlow(l *slog.Logger, i uint64) bool {
// If i is a jump, adjust the window, record lost, update current, and return true
if i > b.current {
end := i
if end > b.current+b.length {
end = b.current + b.length
}
end := min(i, b.current+b.length)
count := end - b.current
startPos := (b.current + 1) & b.lengthMask
+1 -1
View File
@@ -120,7 +120,7 @@ func TestCertificate_SignP256_AlwaysNormalized(t *testing.T) {
pub := elliptic.Marshal(elliptic.P256(), priv.PublicKey.X, priv.PublicKey.Y)
rawPriv := priv.D.FillBytes(make([]byte, 32))
for i := 0; i < 1000; i++ {
for i := range 1000 {
if i&1 == 1 {
tbs.Version = Version1
} else {
+4 -4
View File
@@ -151,7 +151,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
var groups []string
if *cf.groups != "" {
for _, rg := range strings.Split(*cf.groups, ",") {
for rg := range strings.SplitSeq(*cf.groups, ",") {
g := strings.TrimSpace(rg)
if g != "" {
groups = append(groups, g)
@@ -171,7 +171,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
}
if *cf.networks != "" {
for _, rs := range strings.Split(*cf.networks, ",") {
for rs := range strings.SplitSeq(*cf.networks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -193,7 +193,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
}
if *cf.unsafeNetworks != "" {
for _, rs := range strings.Split(*cf.unsafeNetworks, ",") {
for rs := range strings.SplitSeq(*cf.unsafeNetworks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -221,7 +221,7 @@ func ca(args []string, out io.Writer, errOut io.Writer, pr PasswordReader) error
if !isP11 && *cf.encryption {
passphrase = []byte(os.Getenv("NEBULA_CA_PASSPHRASE"))
if len(passphrase) == 0 {
for i := 0; i < 5; i++ {
for range 5 {
errOut.Write([]byte("Enter passphrase: "))
passphrase, err = pr.ReadPassword()
-1
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package main
+4 -4
View File
@@ -146,7 +146,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
passphrase = []byte(os.Getenv("NEBULA_CA_PASSPHRASE"))
if len(passphrase) == 0 {
// ask for a passphrase until we get one
for i := 0; i < 5; i++ {
for range 5 {
errOut.Write([]byte("Enter passphrase: "))
passphrase, err = pr.ReadPassword()
@@ -203,7 +203,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
}
if *sf.networks != "" {
for _, rs := range strings.Split(*sf.networks, ",") {
for rs := range strings.SplitSeq(*sf.networks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -228,7 +228,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
}
if *sf.unsafeNetworks != "" {
for _, rs := range strings.Split(*sf.unsafeNetworks, ",") {
for rs := range strings.SplitSeq(*sf.unsafeNetworks, ",") {
rs := strings.Trim(rs, " ")
if rs != "" {
n, err := netip.ParsePrefix(rs)
@@ -247,7 +247,7 @@ func signCert(args []string, out io.Writer, errOut io.Writer, pr PasswordReader)
var groups []string
if *sf.groups != "" {
for _, rg := range strings.Split(*sf.groups, ",") {
for rg := range strings.SplitSeq(*sf.groups, ",") {
g := strings.TrimSpace(rg)
if g != "" {
groups = append(groups, g)
-1
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package main
-1
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package main
-1
View File
@@ -1,5 +1,4 @@
//go:build !linux
// +build !linux
package main
+6 -9
View File
@@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"log/slog"
"maps"
"math"
"os"
"os/signal"
@@ -154,9 +155,7 @@ func (c *C) ReloadConfig() {
defer c.reloadLock.Unlock()
c.oldSettings = make(map[string]any)
for k, v := range c.Settings {
c.oldSettings[k] = v
}
maps.Copy(c.oldSettings, c.Settings)
err := c.Load(c.path)
if err != nil {
@@ -177,9 +176,7 @@ func (c *C) ReloadConfigString(raw string) error {
defer c.reloadLock.Unlock()
c.oldSettings = make(map[string]any)
for k, v := range c.Settings {
c.oldSettings[k] = v
}
maps.Copy(c.oldSettings, c.Settings)
err := c.LoadString(raw)
if err != nil {
@@ -216,7 +213,7 @@ func (c *C) GetStringSlice(k string, d []string) []string {
}
v := make([]string, len(rv))
for i := 0; i < len(v); i++ {
for i := range v {
v[i] = fmt.Sprintf("%v", rv[i])
}
@@ -310,8 +307,8 @@ func (c *C) IsSet(k string) bool {
}
func (c *C) get(k string, v any) any {
parts := strings.Split(k, ".")
for _, p := range parts {
parts := strings.SplitSeq(k, ".")
for p := range parts {
m, ok := v.(map[string]any)
if !ok {
return nil
+2 -4
View File
@@ -174,10 +174,8 @@ func (cm *connectionManager) doTrafficCheck(localIndex uint32, p, nb, out []byte
switch decision {
case deleteTunnel:
if cm.hostMap.DeleteHostInfo(hostinfo) {
// Only clearing the lighthouse cache if this is the last hostinfo for this vpn ip in the hostmap
cm.intf.lightHouse.DeleteVpnAddrs(hostinfo.vpnAddrs)
}
// Local delete only, clearing the lighthouse cache if this was the last hostinfo for the vpn addr
cm.intf.closeTunnel(hostinfo)
case closeTunnel:
cm.intf.sendCloseTunnel(hostinfo)
+3
View File
@@ -7,6 +7,7 @@ import (
"testing"
"time"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/overlay/overlaytest"
@@ -21,6 +22,8 @@ func newTestLighthouse() *LightHouse {
l: test.NewLogger(),
addrMap: map[netip.Addr]*RemoteList{},
queryChan: make(chan netip.Addr, 10),
// Nothing drains queryChan here, so an 11th query is dropped.
queryDropped: metrics.NilCounter{},
}
lighthouses := []netip.Addr{}
staticList := map[netip.Addr]struct{}{}
+120 -1
View File
@@ -2,6 +2,8 @@ package nebula
import (
"net/netip"
"strconv"
"sync"
"testing"
"time"
@@ -11,6 +13,7 @@ import (
ct "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/overlay"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
@@ -97,7 +100,7 @@ func TestConnectionState_NextMessageCounter(t *testing.T) {
assert.Equal(t, RejectAfterMessages, cs.messageCounter.Load())
// Continued send attempts stay refused and the counter never wraps
for i := 0; i < 10; i++ {
for range 10 {
_, ok = cs.NextMessageCounter()
assert.False(t, ok)
}
@@ -147,6 +150,122 @@ func TestSendNoMetricsCloseTunnelKeepsRebindEpoch(t *testing.T) {
assert.True(t, hostinfo.markOut(1), "a CloseTunnel send must not consume the rebind epoch")
}
// capturingConn is a udp.Conn that records every WriteTo.
type capturingConn struct {
udp.NoopConn
writes [][]byte
addrs []netip.AddrPort
}
func (c *capturingConn) WriteTo(b []byte, addr netip.AddrPort) error {
c.writes = append(c.writes, append([]byte(nil), b...))
c.addrs = append(c.addrs, addr)
return nil
}
// TestSendNoMetricsViaRelay sends through a relay from the mtu sized buffer the cached packet flush uses. Payloads up
// to overlay.MaxMTU go out intact, anything bigger is dropped rather than outgrowing the buffer, which once panicked.
func TestSendNoMetricsViaRelay(t *testing.T) {
tests := []struct {
n int
sent bool
}{
{0, true},
{1300, true},
{overlay.MaxMTU, true},
{overlay.MaxMTU + 1, false}, // too big for the relay's tag
{mtu - 47, false}, // too big for the inner tag, this panicked
{9000, false}, // a full packet at tun.mtu 9000
}
for _, tt := range tests {
t.Run(strconv.Itoa(tt.n), func(t *testing.T) {
// One tunnel to the target, carried end to end, and one to the relay that wraps it
targetInit, targetResp := runTestHandshake(t)
relayInit, relayResp := runTestHandshake(t)
ci, err := newConnectionStateFromResult(targetInit)
require.NoError(t, err)
relayCI, err := newConnectionStateFromResult(relayInit)
require.NoError(t, err)
targetAddr := netip.MustParseAddr("10.0.0.2")
relayAddr := netip.MustParseAddr("10.0.0.3")
relayRemote := netip.MustParseAddrPort("192.0.2.1:4242")
relay := &Relay{Type: TerminalType, State: Established, LocalIndex: 5, RemoteIndex: 6, PeerAddr: targetAddr}
relayHI := &HostInfo{
vpnAddrs: []netip.Addr{relayAddr},
ConnectionState: relayCI,
relayState: RelayState{
relayForByAddr: map[netip.Addr]*Relay{targetAddr: relay},
relayForByIdx: map[uint32]*Relay{relay.LocalIndex: relay},
},
}
relayHI.remote.Store(&relayRemote)
// No remote of its own, so the send has to go via the relay
hostinfo := &HostInfo{
vpnAddrs: []netip.Addr{targetAddr},
ConnectionState: ci,
remoteIndexId: 7,
relayState: RelayState{
relays: []netip.Addr{relayAddr},
relayForByAddr: map[netip.Addr]*Relay{},
relayForByIdx: map[uint32]*Relay{},
},
}
l := test.NewLogger()
hm := newHostMap(l)
hm.Hosts[relayAddr] = relayHI
conn := &capturingConn{}
f := &Interface{
l: l,
hostMap: hm,
messageMetrics: &MessageMetrics{txExhausted: metrics.NewCounter()},
writers: []udp.Conn{conn},
}
f.connectionManager = &connectionManager{intf: f, relayUsed: map[uint32]struct{}{}, relayUsedLock: &sync.RWMutex{}}
payload := make([]byte, tt.n)
for i := range payload {
payload[i] = byte(i)
}
counter := ci.messageCounter.Load()
f.sendNoMetrics(header.Message, 0, ci, hostinfo, netip.AddrPort{}, payload, make([]byte, 12), make([]byte, mtu), 0)
if !tt.sent {
assert.Empty(t, conn.writes)
assert.Equal(t, counter, ci.messageCounter.Load(), "a dropped packet must not spend a counter")
return
}
require.Len(t, conn.writes, 1)
assert.Equal(t, relayRemote, conn.addrs[0])
pkt := conn.writes[0]
// The relay authenticates the whole packet under its own header
outer := &header.H{}
require.NoError(t, outer.Parse(pkt))
assert.Equal(t, header.MessageRelay, outer.Subtype)
assert.Equal(t, relay.RemoteIndex, outer.RemoteIndex)
relayPeer, err := newConnectionStateFromResult(relayResp)
require.NoError(t, err)
require.NoError(t, relayPeer.VerifyRelay(l, outer.MessageCounter, pkt, make([]byte, 12)))
// Inside it, the target decrypts the end to end packet back to the payload
inner := pkt[header.Len : len(pkt)-relayPeer.dKey.Overhead()]
h := &header.H{}
require.NoError(t, h.Parse(inner))
assert.Equal(t, header.Message, h.Type)
assert.Equal(t, hostinfo.remoteIndexId, h.RemoteIndex)
targetPeer, err := newConnectionStateFromResult(targetResp)
require.NoError(t, err)
got, err := targetPeer.Decrypt(l, h.MessageCounter, inner, make([]byte, 12))
require.NoError(t, err)
assert.Equal(t, payload, got)
})
}
}
func TestNewConnectionStateFromResult(t *testing.T) {
initR, respR := runTestHandshake(t)
+1 -1
View File
@@ -247,7 +247,7 @@ func TestControl_ConcurrentStopAndStart(t *testing.T) {
c, _, _ := newReadyControl(t)
var wg sync.WaitGroup
for i := 0; i < 2; i++ {
for range 2 {
wg.Go(func() { c.Stop() })
}
wg.Go(func() { _ = c.Start() })
+68
View File
@@ -0,0 +1,68 @@
# vpnemu
Reproduces, on macOS and without Tailscale, the way a VPN with an exit node takes nebula's vpn network route
away from its utun (#1931).
A macOS network extension never touches the routing table. Its routes are published in configd's dynamic store
under `State:/Network/Service/<id>/IPv4` and `/IPv6`, and configd's IPMonitor installs them. vpnemu creates a
utun and publishes those keys itself, so configd does exactly what it would do for the real app. It publishes
the routes Tailscale would with an exit node: every interface prefix that isn't a single IP, the rule in
Tailscale's `ipnlocal.internalAndExternalInterfacesFrom`.
| mode | Tailscale setting | published as | where nebula's prefix goes |
| --- | --- | --- | --- |
| `claim` | exit node, no LAN access | `AdditionalRoutes` | into vpnemu's utun |
| `exclude` | exit node, LAN access | `ExcludedRoutes`, service overrides primary | out the next interface (en0) |
configd answers `EEXIST` with a delete and an add, so whatever held the prefix loses it. When the service is
withdrawn, configd deletes its routes by prefix alone, whoever holds them by then.
## Build
It is its own module, so `purego` stays out of nebula's `go.mod`. purego reaches CoreFoundation and
SystemConfiguration without cgo, so it cross compiles and runs on a mac with no developer tools.
```sh
cd e2e/darwin/vpnemu
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build -o vpnemu .
# a cross compiled binary copied to a mac may need: codesign -s - -f vpnemu
```
## Run
```sh
sudo ./vpnemu -mode claim -keep en0 # runs until ctrl-c, then withdraws and closes its utun
sudo ./vpnemu -clear # withdraw a service a crashed run left behind
```
`-keep` names interfaces whose prefixes are never published. Keep the one your ssh session arrives on, or
`claim` takes your LAN into a utun nobody reads. `exclude` makes vpnemu primary, so anything off a kept
interface's subnet, internet included, goes into its utun and is dropped while it runs.
`-follow` (default on) republishes when interface prefixes change, as Tailscale does on link changes. With
`-follow=false` a route stays published after its interface is gone, which stages the route configd leaves
behind for a nebula that restarts before the VPN has caught up.
## A/B a nebula build
`repro.sh` starts nebula, then runs vpnemu, and reports egress and ping per peer before, during and after:
```sh
sudo ./repro.sh -n ./nebula -c config.yml -p 192.168.212.1 -p fd19:31::1 -m claim -k en0
```
Before #1931 (utun holds `/24` and `/64`):
```
during 192.168.212.1 egress=utun2 ping=FAIL
after 192.168.212.1 egress=en0 ping=FAIL
```
With #1931 (utun holds `/32` and `/128`, network routes added explicitly) every phase stays on nebula's utun.
### Leftover route
Run an old nebula, start `vpnemu -follow=false`, then restart onto a new nebula while vpnemu is still up. The
new nebula finds the vpn network routed to vpnemu's utun, replaces it and starts. Stop vpnemu and configd deletes
the prefix, taking nebula's route with it until nebula restarts. With real Tailscale this needs nebula to restart
before Tailscale notices the old utun is gone, a one time window when upgrading from a nebula without #1931.
+8
View File
@@ -0,0 +1,8 @@
module github.com/slackhq/nebula/e2e/darwin/vpnemu
go 1.26.0
require (
github.com/ebitengine/purego v0.9.1
golang.org/x/sys v0.48.0
)
+4
View File
@@ -0,0 +1,4 @@
github.com/ebitengine/purego v0.9.1 h1:a/k2f2HQU3Pi399RPW1MOaZyhKJL9w/xFpKAg4q1s0A=
github.com/ebitengine/purego v0.9.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
+292
View File
@@ -0,0 +1,292 @@
//go:build darwin
// vpnemu stands in for a macOS VPN app with an exit node in use, so route hijacks like the one
// Tailscale causes can be reproduced without Tailscale.
//
// A network extension does not touch the routing table itself. It hands its routes to
// nesessionmanager, which publishes them in configd's dynamic store under
// State:/Network/Service/<id>/IPv4 and /IPv6, and configd's IPMonitor installs them. vpnemu
// creates a utun and publishes the same keys directly, so configd does exactly what it would do
// for the real thing.
//
// Which routes it publishes follows Tailscale's rule for exit nodes
// (ipnlocal.internalAndExternalInterfacesFrom): every interface address prefix that is not a
// single IP counts as an attached LAN. With LAN access off those prefixes are claimed into the
// tunnel (-mode claim, published as AdditionalRoutes). With LAN access on they are excluded and
// leave through the next interface (-mode exclude, published as ExcludedRoutes on a service that
// overrides the primary).
//
// It runs until interrupted, then withdraws the service and closes its utun. Needs root.
package main
import (
"flag"
"fmt"
"log"
"net"
"net/netip"
"os"
"os/exec"
"os/signal"
"slices"
"strings"
"syscall"
"time"
"github.com/ebitengine/purego"
"golang.org/x/sys/unix"
)
const (
serviceID = "VPNEMU"
// Tailscale's own addresses, single IPs, so the rule skips them like Tailscale skips its own
addr4 = "100.100.0.1"
addr6 = "fd7a:115c:a1e0::1"
utunControlName = "com.apple.net.utun_control"
utunOptIfname = 2
cfStringUTF8 = 0x08000100
)
var (
cfStringCreateWithCString func(alloc uintptr, s *byte, enc uint32) uintptr
cfDataCreate func(alloc uintptr, b *byte, n int) uintptr
cfPropertyListCreateWithData func(alloc, data uintptr, opts uint, format *int, err *uintptr) uintptr
cfRelease func(ref uintptr)
scDynamicStoreCreate func(alloc, name, callout, ctx uintptr) uintptr
scDynamicStoreSetValue func(store, key, value uintptr) bool
scDynamicStoreRemoveValue func(store, key uintptr) bool
scError func() int32
)
// loadFrameworks binds the few CoreFoundation and SystemConfiguration calls we need. purego does
// it without cgo, so this cross compiles and runs on a mac with no developer tools. JXA can read
// the dynamic store but its writes fail, and scutil cannot build the arrays of dictionaries the
// route keys hold.
func loadFrameworks() {
cf, err := purego.Dlopen("/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation", purego.RTLD_NOW|purego.RTLD_GLOBAL)
if err != nil {
log.Fatal(err)
}
sc, err := purego.Dlopen("/System/Library/Frameworks/SystemConfiguration.framework/SystemConfiguration", purego.RTLD_NOW|purego.RTLD_GLOBAL)
if err != nil {
log.Fatal(err)
}
purego.RegisterLibFunc(&cfStringCreateWithCString, cf, "CFStringCreateWithCString")
purego.RegisterLibFunc(&cfDataCreate, cf, "CFDataCreate")
purego.RegisterLibFunc(&cfPropertyListCreateWithData, cf, "CFPropertyListCreateWithData")
purego.RegisterLibFunc(&cfRelease, cf, "CFRelease")
purego.RegisterLibFunc(&scDynamicStoreCreate, sc, "SCDynamicStoreCreate")
purego.RegisterLibFunc(&scDynamicStoreSetValue, sc, "SCDynamicStoreSetValue")
purego.RegisterLibFunc(&scDynamicStoreRemoveValue, sc, "SCDynamicStoreRemoveValue")
purego.RegisterLibFunc(&scError, sc, "SCError")
}
func cfString(s string) uintptr {
b := append([]byte(s), 0)
return cfStringCreateWithCString(0, &b[0], cfStringUTF8)
}
// cfPlist builds a property list from its xml form, the least code way to make nested CF values
func cfPlist(xml string) uintptr {
b := []byte(xml)
d := cfDataCreate(0, &b[0], len(b))
defer cfRelease(d)
var errRef uintptr
p := cfPropertyListCreateWithData(0, d, 0, nil, &errRef)
if p == 0 {
log.Fatalf("invalid property list:\n%s", xml)
}
return p
}
type store struct {
ref uintptr
k4, k6 uintptr
}
func openStore() *store {
return &store{
ref: scDynamicStoreCreate(0, cfString("vpnemu"), 0, 0),
k4: cfString("State:/Network/Service/" + serviceID + "/IPv4"),
k6: cfString("State:/Network/Service/" + serviceID + "/IPv6"),
}
}
func (s *store) set(key uintptr, xml string) {
v := cfPlist(xml)
defer cfRelease(v)
if !scDynamicStoreSetValue(s.ref, key, v) {
log.Fatalf("SCDynamicStoreSetValue failed: SCError %d", scError())
}
}
func (s *store) withdraw() {
scDynamicStoreRemoveValue(s.ref, s.k4)
scDynamicStoreRemoveValue(s.ref, s.k6)
}
// createUtun opens a utun the kernel numbers for us and gives it Tailscale-like single addresses.
// The interface lives as long as the returned fd.
func createUtun() (int, string) {
fd, err := unix.Socket(unix.AF_SYSTEM, unix.SOCK_DGRAM, unix.AF_SYS_CONTROL)
if err != nil {
log.Fatalf("system socket: %v", err)
}
info := &unix.CtlInfo{}
copy(info.Name[:], utunControlName)
if err = unix.IoctlCtlInfo(fd, info); err != nil {
log.Fatalf("CTLIOCGINFO: %v", err)
}
if err = unix.Connect(fd, &unix.SockaddrCtl{ID: info.Id, Unit: 0}); err != nil {
log.Fatalf("connect utun control: %v", err)
}
name, err := unix.GetsockoptString(fd, unix.AF_SYS_CONTROL, utunOptIfname)
if err != nil {
log.Fatalf("utun name: %v", err)
}
for _, args := range [][]string{
{name, "inet", addr4, addr4, "netmask", "255.255.255.255", "up"},
{name, "inet6", addr6, "prefixlen", "128"},
} {
if out, err := exec.Command("ifconfig", args...).CombinedOutput(); err != nil {
log.Fatalf("ifconfig %s: %v: %s", strings.Join(args, " "), err, out)
}
}
return fd, name
}
// lanPrefixes applies Tailscale's attached-LAN rule to every interface but those in skip
func lanPrefixes(skip []string) (v4, v6 []netip.Prefix) {
ifs, err := net.Interfaces()
if err != nil {
log.Fatal(err)
}
for _, i := range ifs {
if slices.Contains(skip, i.Name) || i.Flags&net.FlagLoopback != 0 {
continue
}
addrs, _ := i.Addrs()
for _, a := range addrs {
n, ok := a.(*net.IPNet)
if !ok {
continue
}
ip, _ := netip.AddrFromSlice(n.IP)
ip = ip.Unmap()
ones, _ := n.Mask.Size()
pfx := netip.PrefixFrom(ip, ones).Masked()
// Tailscale skips single IPs. Link-local is our addition, every interface has the
// same fe80::/64 and claiming it would only break the harness's own host.
if pfx.IsSingleIP() || ip.IsLinkLocalUnicast() {
continue
}
if ip.Is4() {
v4 = append(v4, pfx)
} else {
v6 = append(v6, pfx)
}
}
}
slices.SortFunc(v4, func(a, b netip.Prefix) int { return a.Addr().Compare(b.Addr()) })
slices.SortFunc(v6, func(a, b netip.Prefix) int { return a.Addr().Compare(b.Addr()) })
return v4, v6
}
func routes4(ps []netip.Prefix) string {
var b strings.Builder
for _, p := range ps {
mask := net.IP(net.CIDRMask(p.Bits(), 32)).String()
fmt.Fprintf(&b, "<dict><key>DestinationAddress</key><string>%s</string><key>SubnetMask</key><string>%s</string></dict>", p.Addr(), mask)
}
return b.String()
}
func routes6(ps []netip.Prefix) string {
var b strings.Builder
for _, p := range ps {
fmt.Fprintf(&b, "<dict><key>DestinationAddress</key><string>%s</string><key>PrefixLength</key><integer>%d</integer></dict>", p.Addr(), p.Bits())
}
return b.String()
}
func (s *store) publish(ifName string, exclude bool, v4, v6 []netip.Prefix) {
routesKey, primary := "AdditionalRoutes", ""
if exclude {
// configd only steers excluded routes for a service with a default route that is primary.
// While this runs, everything not on a kept interface's subnet goes into the utun and dies.
routesKey = "ExcludedRoutes"
primary = "<key>Router</key><string>" + addr4 + "</string><key>OverridePrimary</key><integer>1</integer>"
}
const head = `<?xml version="1.0" encoding="UTF-8"?><plist version="1.0"><dict>`
s.set(s.k4, head+
"<key>InterfaceName</key><string>"+ifName+"</string>"+
"<key>Addresses</key><array><string>"+addr4+"</string></array>"+
"<key>SubnetMasks</key><array><string>255.255.255.255</string></array>"+primary+
"<key>"+routesKey+"</key><array>"+routes4(v4)+"</array></dict></plist>")
s.set(s.k6, head+
"<key>InterfaceName</key><string>"+ifName+"</string>"+
"<key>Addresses</key><array><string>"+addr6+"</string></array>"+
"<key>PrefixLength</key><array><integer>128</integer></array>"+
"<key>"+routesKey+"</key><array>"+routes6(v6)+"</array></dict></plist>")
log.Printf("published %s on %s, %s: %v %v", serviceID, ifName, routesKey, v4, v6)
}
func main() {
mode := flag.String("mode", "claim", "claim: exit node without LAN access, the prefixes are routed into the vpn\n"+
"exclude: exit node with LAN access, the prefixes leave via the next interface and the vpn becomes primary")
keep := flag.String("keep", "en0", "comma separated interfaces whose prefixes are never published, keep the one your ssh session uses")
follow := flag.Bool("follow", true, "republish when interface prefixes change, as Tailscale does on link changes.\n"+
"Off, a route stays published after its interface is gone, which is how a leftover route is staged")
clear := flag.Bool("clear", false, "withdraw a service a crashed run left behind, then exit")
flag.Parse()
if *mode != "claim" && *mode != "exclude" {
log.Fatalf("unknown -mode %q", *mode)
}
if os.Geteuid() != 0 {
log.Fatal("needs root, to create the utun and write configd's dynamic store")
}
loadFrameworks()
s := openStore()
if *clear {
s.withdraw()
log.Print("withdrawn")
return
}
fd, ifName := createUtun()
exclude := *mode == "exclude"
skip := append(strings.Split(*keep, ","), ifName)
v4, v6 := lanPrefixes(skip)
s.publish(ifName, exclude, v4, v6)
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
tick := time.NewTicker(time.Second)
defer tick.Stop()
for {
select {
case <-sig:
s.withdraw()
_ = unix.Close(fd)
log.Print("withdrawn")
return
case <-tick.C:
if !*follow {
continue
}
n4, n6 := lanPrefixes(skip)
if !slices.Equal(n4, v4) || !slices.Equal(n6, v6) {
v4, v6 = n4, n6
s.publish(ifName, exclude, v4, v6)
}
}
}
}
+76
View File
@@ -0,0 +1,76 @@
#!/bin/sh
# Reports where traffic to nebula peers goes before, during and after a vpnemu exit node.
#
# usage: sudo ./repro.sh -n <nebula binary> -c <nebula config> -p <peer vpn addr> [-p <peer>...]
# [-m claim|exclude] [-k <keep interfaces>]
#
# Each peer should answer pings over the overlay. A healthy run shows the nebula utun as egress
# and the ping working in all three phases.
set -u
MODE=claim
KEEP=en0
PEERS=""
NEBULA=""
CONFIG=""
while getopts n:c:p:m:k: o; do
case $o in
n) NEBULA=$OPTARG ;;
c) CONFIG=$OPTARG ;;
p) PEERS="$PEERS $OPTARG" ;;
m) MODE=$OPTARG ;;
k) KEEP=$OPTARG ;;
*) exit 2 ;;
esac
done
if [ -z "$NEBULA" ] || [ -z "$CONFIG" ] || [ -z "$PEERS" ]; then
sed -n '4,5p' "$0"
exit 2
fi
HERE=$(cd "$(dirname "$0")" && pwd)
VPNEMU=${VPNEMU:-$HERE/vpnemu}
LOG=$(mktemp -t nebula-repro)
report() {
for p in $PEERS; do
case $p in
*:*) fam=-inet6 ping=ping6 ;;
*) fam=-inet ping="ping -t 3" ;;
esac
egress=$(route -n get $fam "$p" 2>/dev/null | awk '/interface:/{print $2}')
if $ping -c 2 "$p" >/dev/null 2>&1; then ok=ok; else ok=FAIL; fi
printf '%-8s %-24s egress=%-8s ping=%s\n' "$1" "$p" "${egress:-none}" "$ok"
done
}
cleanup() {
[ -n "${EMU:-}" ] && kill "$EMU" 2>/dev/null
[ -n "${NEB:-}" ] && kill "$NEB" 2>/dev/null
wait 2>/dev/null
}
trap cleanup EXIT INT TERM
"$NEBULA" -config "$CONFIG" >"$LOG" 2>&1 &
NEB=$!
sleep 3
if ! kill -0 $NEB 2>/dev/null; then
echo "nebula exited:"
tail -5 "$LOG"
exit 1
fi
echo "nebula on $(grep -o 'interface=utun[0-9]*' "$LOG" | head -1 | cut -d= -f2), log in $LOG"
report before
"$VPNEMU" -mode "$MODE" -keep "$KEEP" &
EMU=$!
sleep 3
report during
kill $EMU
wait $EMU 2>/dev/null
EMU=
sleep 2
report after
grep -E 'level=(WARN|ERROR).*(route|Route)' "$LOG"
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package router
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package router
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
-1
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
+8 -7
View File
@@ -135,12 +135,12 @@ listen:
# superpacket, so the receive scratch is batch * 64KiB per listening socket (~4MiB per routine at the
# default of 64). Lower this to trade peak per-syscall throughput for memory on constrained hosts.
#batch: 64
# Configure socket buffers for the udp side (outside), leave unset to use the system defaults. Values will be doubled by the kernel
# Linux only. Configure socket buffers for the udp side (outside), leave unset to use the system defaults. Values will be doubled by the kernel
# Default is net.core.rmem_default and net.core.wmem_default (/proc/sys/net/core/rmem_default and /proc/sys/net/core/rmem_default)
# Maximum is limited by memory in the system, SO_RCVBUFFORCE and SO_SNDBUFFORCE is used to avoid having to raise the system wide
# max, net.core.rmem_max and net.core.wmem_max
#read_buffer: 10485760
#write_buffer: 10485760
read_buffer: 3412512
write_buffer: 3412512
# On Windows only
# When true, Nebula installs a WFP (Windows Filtering Platform) PERMIT filter scoped to UDP at the listener port.
@@ -262,9 +262,10 @@ tun:
drop_local_broadcast: false
# Toggles forwarding of multicast packets
drop_multicast: false
# Sets the transmit queue length, if you notice lots of transmit drops on the tun it may help to raise this number. Default is 500
tx_queue: 500
# Sets the transmit queue length, if you notice lots of transmit drops on the tun it may help to raise this number. Default is 5000
tx_queue: 5000
# Default MTU for every packet, safe setting is (and the default) 1300 for internet based traffic
# Anything above 8937, here or on a route below, is capped to 8937 with a warning
mtu: 1300
# the use_offloads setting controls if Nebula will attempt to enable GSO and GRO for the tun device. Linux only, not reloadable.
@@ -410,8 +411,8 @@ logging:
#try_interval: 100ms
#retries: 20
# query_buffer is the size of the buffer channel for querying lighthouses
#query_buffer: 64
# query_buffer is the size of the buffer channel for querying lighthouses, drops are recorded in the lighthouse.query.dropped metric.
#query_buffer: 1024
# trigger_buffer is the size of the buffer channel for quickly sending handshakes
# after receiving the response for lighthouse queries
+1 -1
View File
@@ -22,7 +22,7 @@ func newFixedTicker(t *testing.T, l *slog.Logger, cacheLen int) *ConntrackCacheT
l: l,
cache: make(ConntrackCache, cacheLen),
}
for i := 0; i < cacheLen; i++ {
for i := range cacheLen {
c.cache[Packet{LocalPort: uint16(i) + 1}] = struct{}{}
}
c.cacheTick.Store(1) // cacheV starts at 0, so Get() takes the reset path
+10 -12
View File
@@ -5,33 +5,33 @@ go 1.26.0
require (
dario.cat/mergo v1.0.2
filippo.io/bigmod v0.1.0
github.com/DefinedNet/netlink v1.3.2-dn.1
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
github.com/armon/go-radix v1.0.0
github.com/flynn/noise v1.1.0
github.com/gaissmai/bart v0.28.0
github.com/gaissmai/bart v0.29.0
github.com/gogo/protobuf v1.3.2
github.com/google/gopacket v1.1.19
github.com/kardianos/service v1.3.0
github.com/miekg/dns v1.1.72
github.com/miekg/dns v1.1.73
github.com/miekg/pkcs11 v1.1.2
github.com/nbrownus/go-metrics-prometheus v0.0.0-20210712211119-974a6260965f
github.com/prometheus/client_golang v1.24.1
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6
github.com/stretchr/testify v1.12.0
github.com/vishvananda/netlink v1.3.1
github.com/stretchr/testify v1.12.1
go.uber.org/goleak v1.3.0
go.yaml.in/yaml/v3 v3.0.5
golang.org/x/crypto v0.54.0
golang.org/x/net v0.57.0
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0
golang.org/x/term v0.45.0
golang.org/x/crypto v0.57.0
golang.org/x/net v0.59.0
golang.org/x/sync v0.23.0
golang.org/x/sys v0.48.0
golang.org/x/term v0.46.0
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2
golang.zx2c4.com/wireguard v0.0.0-20230325221338-052af4a8072b
golang.zx2c4.com/wireguard/windows v1.0.1
google.golang.org/protobuf v1.36.11
google.golang.org/protobuf v1.36.12
gopkg.in/yaml.v3 v3.0.1
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe
)
@@ -45,7 +45,5 @@ require (
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/vishvananda/netns v0.0.5 // indirect
golang.org/x/mod v0.36.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.45.0 // indirect
)
+20 -26
View File
@@ -3,6 +3,8 @@ dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
filippo.io/bigmod v0.1.0 h1:UNzDk7y9ADKST+axd9skUpBQeW7fG2KrTZyOE4uGQy8=
filippo.io/bigmod v0.1.0/go.mod h1:OjOXDNlClLblvXdwgFFOQFJEocLhhtai8vGLy0JCZlI=
github.com/DefinedNet/netlink v1.3.2-dn.1 h1:Md2PsUE1Xuy1Zxs/Bm6356n1kt5bq2HspXlPJ6pEspg=
github.com/DefinedNet/netlink v1.3.2-dn.1/go.mod h1:o4YSn8Jw7zoDcYXDmV77TRd4eS5sSLuE+vu2zAN67C8=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
@@ -23,8 +25,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/flynn/noise v1.1.0 h1:KjPQoQCEFdZDiP03phOvGi11+SVVhBG2wOWAorLsstg=
github.com/flynn/noise v1.1.0/go.mod h1:xbMo+0i6+IGbYdJhF31t2eR1BIU0CYc12+BNAKwUTag=
github.com/gaissmai/bart v0.28.0 h1:89yZLo8NmyqD0RYgJ3QO9HhqqGGw+oWhf90cZm69Lko=
github.com/gaissmai/bart v0.28.0/go.mod h1:GREWQfTLRWz/c5FTOsIw+KkscuFkIV5t8Rp7Nd1Td5c=
github.com/gaissmai/bart v0.29.0 h1:wO6HGE8g9YE0Wm0bCpYxwRzfQ4+fbJKOhL64e5ACGCI=
github.com/gaissmai/bart v0.29.0/go.mod h1:GREWQfTLRWz/c5FTOsIw+KkscuFkIV5t8Rp7Nd1Td5c=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/kit v0.9.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-kit/log v0.1.0/go.mod h1:zbhenjAZHb184qTLMA9ZjW7ThYL0H2mk7Q6pNt4vbaY=
@@ -81,8 +83,8 @@ github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/miekg/dns v1.1.72 h1:vhmr+TF2A3tuoGNkLDFK9zi36F2LS+hKTRW0Uf8kbzI=
github.com/miekg/dns v1.1.72/go.mod h1:+EuEPhdHOsfk6Wk5TT2CzssZdqkmFhf8r+aVyDEToIs=
github.com/miekg/dns v1.1.73 h1:uhT8nJxmTrPJYClxVxTCX+CVn6qnzSiybRk72Z6DgrE=
github.com/miekg/dns v1.1.73/go.mod h1:RW2Obtfd5NZHvOFe3zYG0W8koWOQtAzyHaLo8vASBuQ=
github.com/miekg/pkcs11 v1.1.2 h1:/VxmeAX5qU6Q3EwafypogwWbYryHFmF2RpkJmw3m4MQ=
github.com/miekg/pkcs11 v1.1.2/go.mod h1:XsNlhZGX73bx86s2hdc/FuaLm2CPZJemRLMA+WTFxgs=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -136,10 +138,8 @@ github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.12.0 h1:K6Mr6jO9JICuend/5xzTM03ydSV3vdNRYAdPSukj8uI=
github.com/stretchr/testify v1.12.0/go.mod h1:bOYBZb5qJ00vPzWfIqBUZPaxK8jWiXc6d3ErP4Ca9Gw=
github.com/vishvananda/netlink v1.3.1 h1:3AEMt62VKqz90r0tmNhog0r/PpWKmrEShJU0wJW6bV0=
github.com/vishvananda/netlink v1.3.1/go.mod h1:ARtKouGSTGchR8aMwmkzC0qiNPrrWO5JS/XMVl45+b4=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/vishvananda/netns v0.0.5 h1:DfiHV+j8bA32MFM7bfEunvT8IAqQ/NzSJHtcmW5zdEY=
github.com/vishvananda/netns v0.0.5/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -155,14 +155,12 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -173,8 +171,8 @@ golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -182,8 +180,8 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -197,13 +195,11 @@ golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@@ -214,8 +210,6 @@ golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtn
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -234,8 +228,8 @@ google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miE
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+18 -5
View File
@@ -31,6 +31,7 @@ package nebula
import (
"bufio"
"context"
"fmt"
"net"
"strconv"
@@ -51,17 +52,29 @@ type graphiteConfigExport struct {
Percentiles []float64 // Percentiles to export from timers and histograms
}
// graphiteTimeout bounds one submission, the connect and the write together. An export is a few KB, a host that can't
// take that in this long isn't going to.
const graphiteTimeout = 30 * time.Second
// graphiteOnce performs a single submission to Graphite, returning a non-nil
// error on failed connections.
func graphiteOnce(c graphiteConfigExport) error {
// error if it could not connect or write all of it within timeout. It gives up
// as soon as ctx is done.
func graphiteOnce(ctx context.Context, c graphiteConfigExport, timeout time.Duration) error {
now := time.Now().Unix()
du := float64(c.DurationUnit)
flushSeconds := float64(c.FlushInterval) / float64(time.Second)
conn, err := net.DialTCP("tcp", nil, c.Addr)
deadline := time.Now().Add(timeout)
d := net.Dialer{Deadline: deadline}
conn, err := d.DialContext(ctx, "tcp", c.Addr.String())
if err != nil {
return err
}
defer conn.Close()
if err := conn.SetDeadline(deadline); err != nil {
return err
}
stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
defer stop()
w := bufio.NewWriter(conn)
c.Registry.Each(func(name string, i any) {
switch metric := i.(type) {
@@ -111,7 +124,7 @@ func graphiteOnce(c graphiteConfigExport) error {
fmt.Fprintf(w, "%s.%s.fifteen-minute %.2f %d\n", c.Prefix, name, t.Rate15(), now)
fmt.Fprintf(w, "%s.%s.mean-rate %.2f %d\n", c.Prefix, name, t.RateMean(), now)
}
w.Flush()
})
return nil
// A failed write sticks, so this reports the first one
return w.Flush()
}
+315
View File
@@ -0,0 +1,315 @@
package nebula
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net"
"strings"
"sync"
"testing"
"time"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// graphiteHost accepts connections and hands each one to the test
type graphiteHost struct {
ln net.Listener
conns chan net.Conn
mu sync.Mutex
closed bool
open []net.Conn
}
// newGraphiteHost accepts connections for the test. A non-zero window shrinks each connection's receive buffer, so a
// sender that is not read from blocks after a little whatever the host's TCP tuning. Only for tests that need a stall,
// a small window makes reading very slow on Linux.
func newGraphiteHost(t *testing.T, window int) *graphiteHost {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
h := &graphiteHost{ln: ln, conns: make(chan net.Conn, 8)}
done := make(chan struct{})
t.Cleanup(func() {
_ = ln.Close()
h.mu.Lock()
h.closed = true
for _, c := range h.open {
_ = c.Close()
}
h.mu.Unlock()
close(done)
})
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
h.mu.Lock()
if h.closed {
h.mu.Unlock()
_ = c.Close()
return
}
h.open = append(h.open, c)
h.mu.Unlock()
if tc, ok := c.(*net.TCPConn); ok && window > 0 {
_ = tc.SetReadBuffer(window)
}
select {
case h.conns <- c:
case <-done:
return
}
}
}()
return h
}
func (h *graphiteHost) addr(t *testing.T) *net.TCPAddr {
a, err := net.ResolveTCPAddr("tcp", h.ln.Addr().String())
require.NoError(t, err)
return a
}
// readExport reads what one send wrote, bounded so a send that never finishes fails the test rather than hanging it
func readExport(t *testing.T, c net.Conn) []byte {
t.Helper()
require.NoError(t, c.SetReadDeadline(time.Now().Add(10*time.Second)))
b, err := io.ReadAll(c)
require.NoError(t, err)
return b
}
func (h *graphiteHost) next(t *testing.T, wait time.Duration) net.Conn {
t.Helper()
select {
case c := <-h.conns:
return c
case <-time.After(wait):
t.Fatal("no connection")
return nil
}
}
// lockedBuffer is a log destination a test can read while the code under test writes to it from another goroutine
type lockedBuffer struct {
mu sync.Mutex
buf bytes.Buffer
}
func (b *lockedBuffer) Write(p []byte) (int, error) {
b.mu.Lock()
defer b.mu.Unlock()
return b.buf.Write(p)
}
func (b *lockedBuffer) String() string {
b.mu.Lock()
defer b.mu.Unlock()
return b.buf.String()
}
func testGraphiteConfig(addr *net.TCPAddr, r metrics.Registry) graphiteConfigExport {
return graphiteConfigExport{Addr: addr, Registry: r, FlushInterval: time.Second, DurationUnit: time.Nanosecond, Prefix: "nebula"}
}
// Enough metrics that one export overflows the socket buffers on any platform, so a send to a host that stops
// reading blocks
func bigGraphiteRegistry(r metrics.Registry, t *testing.T) {
for i := range 200000 {
name := fmt.Sprintf("big.gauge.%d", i)
metrics.GetOrRegisterGauge(name, r).Update(int64(i))
t.Cleanup(func() { r.Unregister(name) })
}
}
func TestGraphiteOnce_sends(t *testing.T) {
r := metrics.NewRegistry()
metrics.GetOrRegisterGauge("g", r).Update(7)
metrics.GetOrRegisterCounter("c", r).Inc(3)
h := newGraphiteHost(t, 0)
require.NoError(t, graphiteOnce(t.Context(), testGraphiteConfig(h.addr(t), r), time.Minute))
b := readExport(t, h.next(t, 5*time.Second))
assert.Contains(t, string(b), "nebula.g.value 7 ")
assert.Contains(t, string(b), "nebula.c.count 3 ")
assert.Contains(t, string(b), "nebula.c.count_ps 3.00 ")
}
// A host that accepts and never reads is given up on, and the failed write is reported
func TestGraphiteOnce_stalledHostTimesOut(t *testing.T) {
r := metrics.NewRegistry()
bigGraphiteRegistry(r, t)
h := newGraphiteHost(t, 4096)
done := make(chan error, 1)
cfg := testGraphiteConfig(h.addr(t), r)
go func() { done <- graphiteOnce(t.Context(), cfg, 300*time.Millisecond) }()
h.next(t, 5*time.Second) // accepted, never read
select {
case err := <-done:
require.Error(t, err)
case <-time.After(10 * time.Second):
t.Fatal("the send did not give up on a host that stopped reading")
}
}
// The timeout covers the whole send, a host that keeps reading but too slowly is still given up on
func TestGraphiteOnce_timeoutCoversTheWholeSend(t *testing.T) {
r := metrics.NewRegistry()
bigGraphiteRegistry(r, t)
h := newGraphiteHost(t, 0)
done := make(chan error, 1)
start := time.Now()
cfg := testGraphiteConfig(h.addr(t), r)
go func() { done <- graphiteOnce(t.Context(), cfg, 500*time.Millisecond) }()
c := h.next(t, 5*time.Second)
stopReading := make(chan struct{})
defer close(stopReading)
// 32KB every 10ms keeps every write well inside the timeout, the whole export takes seconds
go func() {
buf := make([]byte, 32<<10)
for {
select {
case <-stopReading:
return
case <-time.After(10 * time.Millisecond):
}
if _, err := c.Read(buf); err != nil {
return
}
}
}()
select {
case err := <-done:
require.Error(t, err)
assert.Less(t, time.Since(start), 5*time.Second)
case <-time.After(10 * time.Second):
t.Fatal("a host that kept reading slowly held the send past its timeout")
}
}
func TestGraphiteOnce_cancelledCtxDoesNotConnect(t *testing.T) {
h := newGraphiteHost(t, 0)
ctx, cancel := context.WithCancel(t.Context())
cancel()
require.Error(t, graphiteOnce(ctx, testGraphiteConfig(h.addr(t), metrics.NewRegistry()), time.Minute))
select {
case <-h.conns:
t.Fatal("connected on a cancelled ctx")
case <-time.After(200 * time.Millisecond):
}
}
// A stop or reload abandons a stuck send rather than waiting out its timeout
func TestGraphiteOnce_cancelAbandonsSend(t *testing.T) {
r := metrics.NewRegistry()
bigGraphiteRegistry(r, t)
h := newGraphiteHost(t, 4096)
ctx, cancel := context.WithCancel(t.Context())
done := make(chan error, 1)
cfg := testGraphiteConfig(h.addr(t), r)
go func() { done <- graphiteOnce(ctx, cfg, time.Minute) }()
h.next(t, 5*time.Second)
select {
case err := <-done:
t.Fatalf("the send should be stuck on a host that is not reading, it ended with %v", err)
case <-time.After(200 * time.Millisecond):
}
cancel()
select {
case err := <-done:
require.Error(t, err)
case <-time.After(5 * time.Second):
t.Fatal("the send is still stuck after its ctx was cancelled")
}
}
// buildRuntime hands graphite the configured prefix and interval
func TestStatsServer_graphiteExportUsesConfig(t *testing.T) {
const name = "graphite.wiring.test"
metrics.GetOrRegisterCounter(name, nil).Inc(8)
t.Cleanup(func() { metrics.DefaultRegistry.Unregister(name) })
h := newGraphiteHost(t, 0)
s, c := newTestStatsServer(t)
setStatsConfig(c, map[string]any{"type": "graphite", "interval": "4s", "prefix": "pfx", "host": h.ln.Addr().String()})
cfg, err := loadStatsConfig(c)
require.NoError(t, err)
s.runMu.Lock()
fns, _ := s.buildRuntime(t.Context(), cfg)
s.runMu.Unlock()
for _, fn := range fns {
fn()
}
b := readExport(t, h.next(t, 5*time.Second))
assert.Contains(t, string(b), "pfx."+name+".count 8 ")
assert.Contains(t, string(b), "pfx."+name+".count_ps 2.00 ")
}
func TestStatsServer_graphiteFailureIsLogged(t *testing.T) {
logs := &lockedBuffer{}
s, c := newTestStatsServer(t)
s.l = test.NewLoggerWithOutput(logs)
setStatsConfig(c, map[string]any{"type": "graphite", "interval": "1h", "host": "127.0.0.1:" + freeTCPPort(t)})
cfg, err := loadStatsConfig(c)
require.NoError(t, err)
s.runMu.Lock()
fns, _ := s.buildRuntime(t.Context(), cfg)
s.runMu.Unlock()
for _, fn := range fns {
fn()
}
assert.Contains(t, logs.String(), "Graphite export failed")
}
// Stop abandons the running export, so it goes through Start rather than handing buildRuntime a ctx, and a stop is
// not logged as a failure
func TestStatsServer_stopAbandonsGraphiteSend(t *testing.T) {
bigGraphiteRegistry(metrics.DefaultRegistry, t)
h := newGraphiteHost(t, 4096)
logs := &lockedBuffer{}
s, c := newTestStatsServer(t)
s.l = test.NewLoggerWithOutput(logs)
setStatsConfig(c, map[string]any{"type": "graphite", "interval": "50ms", "host": h.ln.Addr().String()})
require.NoError(t, s.reload(c, true))
started := make(chan struct{})
go func() {
s.Start()
close(started)
}()
stuck := h.next(t, 30*time.Second)
s.Stop()
select {
case <-started:
case <-time.After(5 * time.Second):
t.Fatal("Start did not return after Stop")
}
// A closed sender resets the connection once it sees this, one still sending just buffers it. Either way the
// read below doesn't have to drain the export through the small window to find out
_, err := stuck.Write([]byte("x"))
require.NoError(t, err)
require.NoError(t, stuck.SetReadDeadline(time.Now().Add(10*time.Second)))
n, err := io.Copy(io.Discard, stuck)
var ne net.Error
if errors.As(err, &ne) && ne.Timeout() {
t.Fatalf("the send kept the connection after Stop, read %d bytes", n)
}
assert.Less(t, n, int64(1<<20), "the send kept going after Stop")
// The capture loop logs on its own goroutine after Start has returned, give it the chance
assert.Never(t, func() bool { return strings.Contains(logs.String(), "Graphite export failed") }, 500*time.Millisecond, 10*time.Millisecond, "a stop is not a failure")
}
+11 -14
View File
@@ -54,7 +54,8 @@ type HandshakeConfig struct {
}
type HandshakeManager struct {
// Mutex for interacting with the vpnIps and indexes maps
// Mutex for interacting with the vpnIps and indexes maps.
// Lock order is hostmap, then this lock, then the lighthouse, then a RemoteList.
sync.RWMutex
vpnIps map[netip.Addr]*HandshakeHostInfo
@@ -259,14 +260,6 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
"style": header.SubTypeName(header.Handshake, hh.machine.Subtype()),
}
// Get a remotes object if we don't already have one.
// This is mainly to protect us as this should never be the case
// NB ^ This comment doesn't jive. It's how the thing gets initialized.
// It's the common path. Should it update every time, in case a future LH query/queries give us more info?
if hostinfo.remotes == nil {
hostinfo.remotes = hm.lightHouse.QueryCache([]netip.Addr{vpnIp})
}
remotes := hostinfo.remotes.CopyAddrs(hm.mainHostMap.GetPreferredRanges())
remotesHaveChanged := !slices.Equal(remotes, hh.lastRemotes)
@@ -371,6 +364,9 @@ func (hm *HandshakeManager) StartHandshake(vpnAddr netip.Addr, cacheCb func(*Han
hostinfo := &HostInfo{
vpnAddrs: []netip.Addr{vpnAddr},
HandshakePacket: make(map[uint8][]byte, 0),
// Attached here so a pending hostinfo never exists without one: SetRemote dereferences
// it, and create-tunnel -address can reach SetRemote before the handshake loop runs.
remotes: hm.lightHouse.QueryCache([]netip.Addr{vpnAddr}),
relayState: RelayState{
relays: nil,
relayForByAddr: map[netip.Addr]*Relay{},
@@ -807,7 +803,7 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
}
hm.sendHandshakeResponse(via, response, hostinfo, false)
hostinfo.remotes.RefreshFromHandshake(vpnAddrs, remoteCert.Certificate.Version())
hostinfo.remotes.RefreshFromHandshake(vpnAddrs)
// Don't wait for UpdateWorker
if f.lightHouse.IsAnyLighthouseAddr(vpnAddrs) {
@@ -946,13 +942,14 @@ func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostIn
hm.DeleteHostInfo(hostinfo)
hm.StartHandshake(hostinfo.vpnAddrs[0], func(newHH *HandshakeHostInfo) {
newHH.hostinfo.remotes = hostinfo.remotes
newHH.hostinfo.remotes.BlockRemote(via)
newHH.packetStore = hh.packetStore
hh.packetStore = []*cachedPacket{}
hostinfo.vpnAddrs = vpnAddrs
f.sendCloseTunnel(hostinfo)
})
// sendCloseTunnel may take the hostmap lock, so it must not run under the handshake
// manager lock that StartHandshake holds around the callback.
hostinfo.vpnAddrs = vpnAddrs
f.sendCloseTunnel(hostinfo)
return
}
@@ -995,7 +992,7 @@ func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostIn
f.cachedPacketMetrics.sent.Inc(int64(len(hh.packetStore)))
}
hostinfo.remotes.RefreshFromHandshake(vpnAddrs, remoteCert.Certificate.Version())
hostinfo.remotes.RefreshFromHandshake(vpnAddrs)
f.metricHandshakes.Update(duration)
// Don't wait for UpdateWorker
+68 -2
View File
@@ -11,6 +11,7 @@ import (
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func Test_NewHandshakeManagerVpnIp(t *testing.T) {
@@ -42,8 +43,6 @@ func Test_NewHandshakeManagerVpnIp(t *testing.T) {
i2 := blah.StartHandshake(ip, nil)
assert.Same(t, i, i2)
i.remotes = NewRemoteList([]netip.Addr{}, nil)
// Adding something to pending should not affect the main hostmap
assert.Empty(t, mainHM.Hosts)
@@ -235,3 +234,70 @@ func TestHandleIncomingDispatch(t *testing.T) {
assert.Empty(t, hm.indexes, "orphan stage-2 must not create state")
})
}
// Test_PendingHandshakeHasRemotes covers create-tunnel -address: a pending hostinfo must already
// carry its RemoteList when StartHandshake runs its callback, so SetRemote there, before the
// handshake loop has run, is safe and the address becomes one the handshake is sent to.
func Test_PendingHandshakeHasRemotes(t *testing.T) {
l := test.NewLogger()
mainHM := newHostMap(l)
preferredRanges := []netip.Prefix{}
mainHM.preferredRanges.Store(&preferredRanges)
hm := NewHandshakeManager(l, mainHM, newTestLighthouse(), &udp.NoopConn{}, defaultHandshakeConfig)
vpnIp := netip.MustParseAddr("172.1.1.2")
remote := netip.MustParseAddrPort("192.168.1.9:4242")
hi := hm.StartHandshake(vpnIp, func(hh *HandshakeHostInfo) {
require.NotNil(t, hh.hostinfo.remotes, "a pending handshake must never be without its RemoteList")
hh.hostinfo.SetRemote(remote)
})
assert.Contains(t, hi.remotes.CopyAddrs(preferredRanges), remote, "the handshake must be sent to the supplied remote")
assert.Equal(t, remote, hi.GetRemote())
// A RemoteList keeps one learned address per host, so a second remote replaces the
// first — the same behaviour change-remote has on an established tunnel.
other := netip.MustParseAddrPort("192.168.1.10:4242")
hi.SetRemote(other)
assert.Equal(t, []netip.AddrPort{other}, hi.remotes.CopyAddrs(preferredRanges))
assert.Equal(t, other, hi.GetRemote())
}
// newRemotesRaceFixture builds a handshake manager with an established tunnel to vpnAddr whose
// RemoteList is the lighthouse's cached list, the state every teardown-during-handshake race starts from.
func newRemotesRaceFixture(t *testing.T, vpnAddr netip.Addr) (*HandshakeManager, *Interface, *HostInfo) {
t.Helper()
l := test.NewLogger()
mainHM := newHostMap(l)
preferredRanges := []netip.Prefix{}
mainHM.preferredRanges.Store(&preferredRanges)
lh := newTestLighthouse()
hm := NewHandshakeManager(l, mainHM, lh, &udp.NoopConn{}, defaultHandshakeConfig)
f := &Interface{hostMap: mainHM, lightHouse: lh, handshakeManager: hm, l: l}
hm.f = f
established := &HostInfo{
vpnAddrs: []netip.Addr{vpnAddr},
localIndexId: 1,
remoteIndexId: 101,
remotes: lh.QueryCache([]netip.Addr{vpnAddr}),
HandshakePacket: map[uint8][]byte{},
}
mainHM.Lock()
mainHM.unlockedAddHostInfo(established, f)
mainHM.Unlock()
return hm, f, established
}
// With no handshake pending, tearing down the last tunnel still evicts the list.
func Test_CloseTunnelEvictsWithoutPendingHandshake(t *testing.T) {
vpnAddr := netip.MustParseAddr("172.1.1.2")
_, f, established := newRemotesRaceFixture(t, vpnAddr)
f.closeTunnel(established)
assert.NotContains(t, f.hostMap.Hosts, vpnAddr)
assert.NotContains(t, f.lightHouse.addrMap, vpnAddr)
}
+5
View File
@@ -24,6 +24,11 @@ type m = map[string]any
const (
Version uint8 = 1
Len = 16
// MaxOverhead is the most a payload grows on its way to the underlay.
// A relayed packet has two headers and two AEAD tags, one for the inner tunnel and another for the relay tunnel.
// Every supported cipher (today) has a 16 byte tag.
MaxOverhead = 2 * (Len + 16)
)
type MessageType uint8
-58
View File
@@ -6,7 +6,6 @@ import (
"errors"
"fmt"
"log/slog"
"net"
"net/netip"
"slices"
"sync"
@@ -18,7 +17,6 @@ import (
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/logging"
)
const defaultPromoteEvery = 1000 // Count of packets sent before we try moving a tunnel to a preferred underlay ip address
@@ -930,59 +928,3 @@ func (i *HostInfo) logger(l *slog.Logger) *slog.Logger {
return li
}
// Utility functions
func localAddrs(l *slog.Logger, allowList *LocalAllowList) []netip.Addr {
//FIXME: This function is pretty garbage
var finalAddrs []netip.Addr
ifaces, _ := net.Interfaces()
for _, i := range ifaces {
allow := allowList.AllowName(i.Name)
if l.Enabled(context.Background(), logging.LevelTrace) {
l.Log(context.Background(), logging.LevelTrace, "localAllowList.AllowName",
"interfaceName", i.Name,
"allow", allow,
)
}
if !allow {
continue
}
addrs, _ := i.Addrs()
for _, rawAddr := range addrs {
var addr netip.Addr
switch v := rawAddr.(type) {
case *net.IPNet:
//continue
addr, _ = netip.AddrFromSlice(v.IP)
case *net.IPAddr:
addr, _ = netip.AddrFromSlice(v.IP)
}
if !addr.IsValid() {
if l.Enabled(context.Background(), slog.LevelDebug) {
l.Debug("addr was invalid", "localAddr", rawAddr)
}
continue
}
addr = addr.Unmap()
if addr.IsLoopback() == false && addr.IsLinkLocalUnicast() == false {
isAllowed := allowList.Allow(addr)
if l.Enabled(context.Background(), logging.LevelTrace) {
l.Log(context.Background(), logging.LevelTrace, "localAllowList.Allow",
"localAddr", addr,
"allowed", isAllowed,
)
}
if !isAllowed {
continue
}
finalAddrs = append(finalAddrs, addr)
}
}
}
return finalAddrs
}
+34 -17
View File
@@ -10,6 +10,7 @@ import (
"github.com/slackhq/nebula/firewall"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/iputil"
"github.com/slackhq/nebula/logging"
"github.com/slackhq/nebula/noiseutil"
"github.com/slackhq/nebula/overlay/batch"
"github.com/slackhq/nebula/overlay/tio"
@@ -198,8 +199,8 @@ func (f *Interface) sendInsideMessage(hostinfo *HostInfo, pkt tio.Packet, nb []b
}
err = tio.SegmentSuperpacket(pkt, func(seg []byte) error {
//relay header + header + plaintext + AEAD tag (16 bytes for both AES-GCM and ChaCha20-Poly1305) + relay tag
scratch := sendBatch.Reserve(header.Len + header.Len + len(seg) + 16 + 16)
//relay header + header + plaintext + AEAD tag + relay tag
scratch := sendBatch.Reserve(len(seg) + header.MaxOverhead)
innerPacket := f.sendInsideEncrypt(hostinfo, ci, seg, scratch[header.Len:], nb)
if innerPacket == nil {
@@ -517,7 +518,9 @@ func (f *Interface) SendVia(via *HostInfo, relay *Relay, ad, nb, out []byte, noc
err = f.writers[q].WriteTo(toSend, via.GetRemote())
if err != nil {
via.logger(f.l).Info("Failed to WriteTo in sendVia", "error", err)
if l, ok := logging.RateLimited(f.l, f.udpRelayTxErrLimiter); ok {
via.logger(l).Info("Failed to WriteTo in sendVia", "error", err)
}
}
}
@@ -526,16 +529,26 @@ func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType
return
}
useRelay := !remote.IsValid() && !hostinfo.GetRemote().IsValid()
fullOut := out
// The encrypt appends to out in place and a relayed packet is then wrapped in place,
// so out has to fit the whole packet up front or the encrypt reallocates out from under SendVia.
// We cap the MTU of tun devices and routes we create ourselves, so only a host that set a bigger MTU than we allow gets here.
need := header.Len + len(p) + ci.eKey.Overhead()
if useRelay {
if len(out) < header.Len {
// out always has a capacity of mtu, but not always a length greater than the header.Len.
// Grow it to make sure the next operation works.
out = out[:header.Len]
// The relay tunnel runs the same cipher, and adds its own header and tag
need += header.Len + ci.eKey.Overhead()
}
if cap(out) < need {
if l, ok := logging.RateLimited(f.l, f.txOversizeLimiter); ok {
hostinfo.logger(l).Error("Dropping outbound packet, out buffer not large enough", "outCap", cap(out), "payloadLen", len(p), "relayed", useRelay)
}
return
}
fullOut := out
if useRelay {
// Save a header's worth of data at the front of the 'out' buffer.
out = out[header.Len:]
out = out[header.Len:header.Len]
}
if noiseutil.EncryptLockNeeded {
@@ -582,18 +595,22 @@ func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType
if remote.IsValid() {
err = f.writers[q].WriteTo(out, remote)
if err != nil {
hostinfo.logger(f.l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", remote,
)
if l, ok := logging.RateLimited(f.l, f.udpTxErrLimiter); ok {
hostinfo.logger(l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", remote,
)
}
}
} else if hr := hostinfo.GetRemote(); hr.IsValid() {
err = f.writers[q].WriteTo(out, hr)
if err != nil {
hostinfo.logger(f.l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", hr,
)
if l, ok := logging.RateLimited(f.l, f.udpTxErrLimiter); ok {
hostinfo.logger(l).Error("Failed to write outgoing packet",
"error", err,
"udpAddr", hr,
)
}
}
} else {
// Try to send via a relay
+24 -18
View File
@@ -24,6 +24,7 @@ import (
"github.com/slackhq/nebula/overlay"
"github.com/slackhq/nebula/overlay/batch"
"github.com/slackhq/nebula/overlay/tio"
"github.com/slackhq/nebula/ratelimit"
"github.com/slackhq/nebula/udp"
)
@@ -124,6 +125,10 @@ type Interface struct {
batchers []*batch.MultiCoalescer
wg sync.WaitGroup
udpTxErrLimiter *ratelimit.Limiter // udp direct transmit error log limiter
udpRelayTxErrLimiter *ratelimit.Limiter // udp relay transmit error log limiter
txOversizeLimiter *ratelimit.Limiter // oversized outbound packet drop log limiter
// fatalErr holds the first unexpected reader error that caused shutdown.
// nil means "no fatal error" (yet)
fatalErr atomic.Pointer[error]
@@ -198,6 +203,19 @@ func NewInterface(ctx context.Context, c *InterfaceConfig) (*Interface, error) {
return nil, errors.New("no connection manager")
}
udpTxErrLimiter, err := ratelimit.New(1, time.Second)
if err != nil {
return nil, err
}
udpRelayTxErrLimiter, err := ratelimit.New(1, time.Second)
if err != nil {
return nil, err
}
txOversizeLimiter, err := ratelimit.New(1, time.Second)
if err != nil {
return nil, err
}
if c.routines <= 1 {
c.PinThreads = false //pinning is not useful unless there's more than one tun reader
}
@@ -230,6 +248,9 @@ func NewInterface(ctx context.Context, c *InterfaceConfig) (*Interface, error) {
conntrackCacheTimeout: c.ConntrackCacheTimeout,
cpuAffinity: c.CpuAffinity,
pinThreads: c.PinThreads,
udpTxErrLimiter: udpTxErrLimiter,
udpRelayTxErrLimiter: udpRelayTxErrLimiter,
txOversizeLimiter: txOversizeLimiter,
metricHandshakes: metrics.GetOrRegisterHistogram("handshakes", nil, metrics.NewExpDecaySample(1028, 0.015)),
metricTxDropped: metrics.GetOrRegisterCounter("udp.tx.dropped", nil),
@@ -624,17 +645,15 @@ func (f *Interface) reloadMisc(c *config.C) {
}
}
func (f *Interface) emitStats(ctx context.Context, i time.Duration) {
ticker := time.NewTicker(i)
defer ticker.Stop()
// statsEmitter updates the interface's gauges, the stats server runs it on its capture loop.
func (f *Interface) statsEmitter() func() {
udpStats := udp.NewUDPStatsEmitter(f.writers)
certExpirationGauge := metrics.GetOrRegisterGauge("certificate.ttl_seconds", nil)
certInitiatingVersion := metrics.GetOrRegisterGauge("certificate.initiating_version", nil)
certMaxVersion := metrics.GetOrRegisterGauge("certificate.max_version", nil)
emit := func() {
return func() {
f.firewall.EmitStats()
f.handshakeManager.EmitStats()
udpStats()
@@ -651,19 +670,6 @@ func (f *Interface) emitStats(ctx context.Context, i time.Duration) {
certMaxVersion.Update(int64(certState.v1Cert.Version()))
}
}
// Prime gauges so a Prometheus scrape that lands before the first tick
// sees real values instead of the zero defaults (issue #907).
emit()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
emit()
}
}
}
func (f *Interface) GetHostInfo(vpnIp netip.Addr) *HostInfo {
+5 -14
View File
@@ -3,7 +3,6 @@
package nebula
import (
"context"
"net/netip"
"testing"
"time"
@@ -18,11 +17,8 @@ import (
"github.com/stretchr/testify/require"
)
// Test_emitStats_primesGauges covers issue #907: a Prometheus scrape that
// landed before the first ticker fire used to read 0 for the cert gauges.
// emitStats now primes the gauges before entering the ticker loop. We assert
// the gauge is zero before the first call and non-zero after.
func Test_emitStats_primesGauges(t *testing.T) {
// Priming before the first tick (issue #907) is Start's job, see TestStatsServer_Start_primes.
func Test_statsEmitter_setsGauges(t *testing.T) {
defer metrics.DefaultRegistry.UnregisterAll()
l := test.NewLogger()
@@ -56,17 +52,12 @@ func Test_emitStats_primesGauges(t *testing.T) {
ifce.pki.cs.Store(cs)
ttlGauge := metrics.GetOrRegisterGauge("certificate.ttl_seconds", nil)
require.Zero(t, ttlGauge.Value(), "gauge should be zero before emitStats runs")
require.Zero(t, ttlGauge.Value(), "gauge should be zero before the emitter runs")
// Pre-cancel the context so emitStats returns after priming the gauges
// without ever reading from ticker.C. The one hour interval is just a
// belt-and-suspenders, the test does not expect the ticker to fire.
ctx, cancel := context.WithCancel(context.Background())
cancel()
ifce.emitStats(ctx, time.Hour)
ifce.statsEmitter()()
ttl := ttlGauge.Value()
assert.Positive(t, ttl, "ttl gauge should be primed by emitStats before its first tick")
assert.Positive(t, ttl, "ttl gauge should be set by one run of the emitter")
assert.LessOrEqual(t, ttl, int64(3600))
assert.Equal(t, int64(cert.Version1), metrics.GetOrRegisterGauge("certificate.initiating_version", nil).Value())
assert.Equal(t, int64(cert.Version1), metrics.GetOrRegisterGauge("certificate.max_version", nil).Value())
+3 -13
View File
@@ -45,8 +45,7 @@ func setTransportChecksum4(packet []byte) {
if ihl < ipv4.HeaderLen || end < ihl || end > len(packet) {
return
}
// The checksum covers the whole datagram, which a fragment (MF set or a
// non-zero offset) does not carry.
// do not attempt to set checksums for fragments
if binary.BigEndian.Uint16(packet[6:8])&0x3fff != 0 {
return
}
@@ -68,10 +67,7 @@ func setTransportChecksum6(packet []byte) {
return
}
// The checksum covers the whole datagram, which a fragment does not carry.
// An unknown extension header hides where the transport header starts. A
// chain longer than the walk's budget ends it early, at an offset that was
// never checked against the packet.
// we must find the transport header in order to write a checksum
proto, offset, _, anyFragment, err := IPv6FindUpperProtocol(packet[:end])
if err != nil || anyFragment || offset >= end {
return
@@ -109,8 +105,7 @@ func transportExtent(transport []byte, proto uint8) ([]byte, bool) {
// writeTransportChecksum stores the checksum of transport, taken over the
// pseudo-header sum csum, in the header's checksum field. A UDP checksum that
// computes to zero goes on the wire as 0xffff: zero means no checksum was
// computed (RFC 768), and over IPv6 the checksum is mandatory (RFC 8200
// section 8.1).
// computed (RFC 768), and over IPv6 the checksum is mandatory (RFC 8200 section 8.1).
func writeTransportChecksum(transport []byte, proto uint8, csum uint32) {
var at, minLen int
switch proto {
@@ -133,11 +128,6 @@ func writeTransportChecksum(transport []byte, proto uint8, csum uint32) {
binary.BigEndian.PutUint16(transport[at:], sum)
}
// fold reduces a pseudo-header sum to the 16 bit seed Checksum takes. Carrying
// the high half back into the low half is what keeps the reduction lossless, so
// the seed sums exactly as the wider value would; 0xffff is its fixed point.
// Every term of that sum comes from a 16 bit field, so it stays far below the
// width at which the accumulator would wrap.
func fold(csum uint32) uint16 {
for csum > 0xffff {
csum = (csum >> 16) + (csum & 0xffff)
+13 -16
View File
@@ -345,14 +345,11 @@ func ipv6CreateRejectTCPPacket(packet []byte, out []byte, offset int) []byte {
return out
}
// IPv6FindUpperProtocol walks the ipv6 extension header chain and returns the upper layer protocol, the
// offset it begins at, and whether the packet is a non-first fragment. Only the RFC 8200 and IANA extension
// headers below are walked. Everything else, including Mobility (135), HIP (139), Shim6 (140), experimental
// 253/254, and real upper layer protocols like SCTP or GRE, is terminal. Walking those as extension headers
// is a firewall bypass, so they fail closed. For a non-first fragment the returned protocol is the fragmented
// protocol and offset points at the fragment header, there is no transport header to locate. Returns
// ErrIPv6CouldNotFindPayload if packet is smaller than an ipv6 header or the chain is truncated before a
// terminal protocol is reached.
// IPv6FindUpperProtocol walks the ipv6 extension header chain and returns the upper layer protocol, the offset it begins at, and whether the packet is a non-first fragment.
// Only the RFC 8200 and IANA extension headers below are walked.
// Everything else, including Mobility (135), HIP (139), Shim6 (140), experimental 253/254, and real upper layer protocols like SCTP or GRE, is terminal.
// For a non-first fragment the returned protocol is the fragmented protocol and offset points at the fragment header, there is no transport header to locate.
// Returns ErrIPv6CouldNotFindPayload if packet is smaller than an ipv6 header, or the chain is truncated, or if more than 8 extension headers are present.
func IPv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragment bool, anyFragment bool, err error) {
const maxIPv6ExtHeaders = 8
if len(packet) < ipv6.HeaderLen {
@@ -365,14 +362,14 @@ func IPv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragm
switch nextHeader {
case 0, 43, 60: // Hop-by-Hop, Routing, Destination
if len(packet) < offset+2 {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
nextHeader = packet[offset]
offset += (int(packet[offset+1]) + 1) << 3
case 44: // Fragment
if len(packet) < offset+8 {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
anyFragment = true
// Non-first fragments carry no transport header, report the fragmented protocol and stop
@@ -384,21 +381,21 @@ func IPv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragm
case 51: // AH
if len(packet) < offset+2 {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
nextHeader = packet[offset]
offset += (int(packet[offset+1]) + 2) << 2
default:
// A prior extension header can declare a length that advances offset past the packet. The terminal
// protocol's header isn't actually here, so treat the chain as truncated rather than classifying it.
if offset > len(packet) {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
// A prior extension header can declare a length that advances offset past the packet, reject this.
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
return nextHeader, offset, isFragment, anyFragment, nil
return nextHeader, offset, isFragment, anyFragment, nil //success!
}
}
return nextHeader, offset, isFragment, anyFragment, nil
// Fail closed
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
func CreateICMPEchoResponse(packet, out []byte) []byte {
+17 -5
View File
@@ -529,6 +529,17 @@ func Test_IPv6FindUpperProtocol(t *testing.T) {
firstFragToUDP := []byte{17, 0, 0, 1, 0, 0, 0, 1} // frag offset 0, M=1, next = UDP
nonFirstFrag := []byte{17, 0, 0, 9, 0, 0, 0, 1} // frag offset non-zero, next = UDP
transport := []byte{0, 80, 1, 187, 0, 0, 0, 0} // stand-in bytes, IPv6FindUpperProtocol never reads ports
extToHbH := []byte{0, 0, 0, 0, 0, 0, 0, 0} // len 0 -> 8 bytes, next = Hop-by-Hop
// 8 walked headers spend the budget before the 9th slot's terminal is ever classified. This is the
// report 4040546 shape: without failing closed the walk hands back offset 104, and a caller reads ports
// from the attacker's bytes there.
var budgetChain []byte
for range 7 {
budgetChain = append(budgetChain, extToHbH...)
}
budgetChain = append(budgetChain, extToTCP...)
budgetChain = append(budgetChain, transport...)
tests := []struct {
name string
@@ -549,9 +560,10 @@ func Test_IPv6FindUpperProtocol(t *testing.T) {
{"first fragment walks to transport", 44, append(firstFragToUDP, transport...), 17, ipv6.HeaderLen + 8, false, true, nil},
{"non-first fragment stops", 44, append(nonFirstFrag, transport...), 17, ipv6.HeaderLen, true, true, nil},
{"unknown protocol is terminal", 132, transport, 132, ipv6.HeaderLen, false, false, nil}, // SCTP
{"truncated extension header", 0, nil, 0, ipv6.HeaderLen, false, false, ErrIPv6CouldNotFindPayload},
{"truncated extension header", 0, nil, 0, 0, false, false, ErrIPv6CouldNotFindPayload},
// Destination Options with a declared length (255+1)*8 = 2048 that runs past the 48 byte buffer, next = SCTP
{"extension length past buffer", 60, []byte{132, 255, 0, 0, 0, 0, 0, 0}, 132, ipv6.HeaderLen + 2048, false, false, ErrIPv6CouldNotFindPayload},
{"extension length past buffer", 0, []byte{132, 255, 0, 0, 0, 0, 0, 0}, 0, 0, false, false, ErrIPv6CouldNotFindPayload},
{"budget exhausted fails closed", 0, budgetChain, 0, 0, false, false, ErrIPv6CouldNotFindPayload},
}
for _, tt := range tests {
@@ -559,10 +571,10 @@ func Test_IPv6FindUpperProtocol(t *testing.T) {
packet := makeIPv6Packet(src, dst, tt.nextHeader, tt.payload)
proto, offset, isFragment, anyFragment, err := IPv6FindUpperProtocol(packet)
if tt.wantErr != nil {
assert.ErrorIs(t, err, tt.wantErr)
return
require.ErrorIs(t, err, tt.wantErr)
} else {
require.NoError(t, err)
}
require.NoError(t, err)
assert.Equal(t, tt.wantProto, proto)
assert.Equal(t, tt.wantOffset, offset)
assert.Equal(t, tt.wantFragment, isFragment)
+88 -57
View File
@@ -15,10 +15,12 @@ import (
"time"
"github.com/gaissmai/bart"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/logging"
"github.com/slackhq/nebula/nicfinder"
"github.com/slackhq/nebula/udp"
"github.com/slackhq/nebula/util"
)
@@ -34,12 +36,18 @@ type LightHouse struct {
myVpnNetworks []netip.Prefix
myVpnNetworksTable *bart.Lite
punchy *Punchy
// myVpnAddrsTable contains our overlay host addrs, as opposed to the overlay networks
myVpnAddrsTable *bart.Lite
punchy *Punchy
// localAddrsFn enumerates the underlay addresses we advertise. It is a field so tests can supply simulated
// addresses rather than whatever this machine's NICs happen to be. Set it before Start.
localAddrsFn func(*LocalAllowList) []netip.Addr
// lastLocalAddrsErr is the previous localAddrsFn failure. Enumeration runs on every update, so an
// unchanged failure is demoted to Debug rather than warning every lighthouse.interval forever.
lastLocalAddrsErr atomic.Pointer[string]
// Local cache of answers from light houses
// map of vpn addr to answers
addrMap map[netip.Addr]*RemoteList
@@ -74,6 +82,8 @@ type LightHouse struct {
updateTrigger chan struct{}
queryChan chan netip.Addr
// queryDropped counts queries QueryServer dropped because queryChan was full.
queryDropped metrics.Counter
calculatedRemotes atomic.Pointer[bart.Table[[]*calculatedRemote]] // Maps VpnAddr to []*calculatedRemote
@@ -104,15 +114,19 @@ func NewLightHouseFromConfig(ctx context.Context, l *slog.Logger, c *config.C, c
amLighthouse: amLighthouse,
myVpnNetworks: cs.myVpnNetworks,
myVpnNetworksTable: cs.myVpnNetworksTable,
myVpnAddrsTable: cs.myVpnAddrsTable,
addrMap: make(map[netip.Addr]*RemoteList),
nebulaPort: nebulaPort,
punchy: p,
updateTrigger: make(chan struct{}, 1),
queryChan: make(chan netip.Addr, c.GetUint32("handshakes.query_buffer", 64)),
queryChan: make(chan netip.Addr, c.GetUint32("handshakes.query_buffer", 1024)),
queryDropped: metrics.GetOrRegisterCounter("lighthouse.query.dropped", nil),
l: l,
}
h.localAddrsFn = func(al *LocalAllowList) []netip.Addr {
return localAddrs(h.l, al)
addrs, err := nicfinder.Find(ctx, h.l, al)
h.logLocalAddrsErr(err)
return addrs
}
lighthouses := make([]netip.Addr, 0)
@@ -345,7 +359,13 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
}
}
if initial || c.HasChanged("relay.relays") {
// am_relay decides whether relay.relays applies to us at all
if initial || c.HasChanged("relay.relays") || c.HasChanged("relay.am_relay") {
var before []netip.Addr
if !initial {
before = lh.GetRelaysForMe()
}
switch c.GetBool("relay.am_relay", false) {
case true:
// Relays aren't allowed to specify other relays
@@ -370,6 +390,11 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
}
lh.relaysForMe.Store(&relaysForMe)
}
// Tell the lighthouses now rather than at the next interval, but only when what we advertise changed
if !initial && !slices.Equal(before, lh.GetRelaysForMe()) {
lh.TriggerUpdate()
}
}
return nil
@@ -502,26 +527,33 @@ func (lh *LightHouse) Query(vpnAddr netip.Addr) *RemoteList {
return nil
}
// QueryServer is asynchronous so no reply should be expected
// QueryServer is asynchronous and non-blocking, no reply should be expected.
// If the query channel is full, the query is dropped.
func (lh *LightHouse) QueryServer(vpnAddr netip.Addr) {
// Don't put lighthouse addrs in the query channel because we can't query lighthouses about lighthouses
if lh.amLighthouse || lh.IsLighthouseAddr(vpnAddr) {
return
}
lh.queryChan <- vpnAddr
select {
case lh.queryChan <- vpnAddr:
default:
lh.queryDropped.Inc(1)
}
}
func (lh *LightHouse) QueryCache(vpnAddrs []netip.Addr) *RemoteList {
rl, ok := lh.findRemoteList(vpnAddrs)
if ok {
return rl
lh.RLock()
if v, ok := lh.addrMap[vpnAddrs[0]]; ok {
lh.RUnlock()
return v
}
lh.RUnlock()
lh.Lock()
defer lh.Unlock()
// Add an entry if we don't already have one
return lh.unlockedGetRemoteList(vpnAddrs) //todo this re-calls unlockedFindRemoteList
return lh.unlockedGetRemoteList(vpnAddrs) //todo CERT-V2 this contains addrmap lookups we could potentially skip
}
// queryAndPrepMessage is a lock helper on RemoteList, assisting the caller to build a lighthouse message containing
@@ -667,58 +699,23 @@ func (lh *LightHouse) addCalculatedRemotes(vpnAddr netip.Addr) bool {
return len(calculatedV4) > 0 || len(calculatedV6) > 0
}
func (lh *LightHouse) findRemoteList(vpnAddrs []netip.Addr) (*RemoteList, bool) {
lh.RLock()
defer lh.RUnlock()
return lh.unlockedFindRemoteList(vpnAddrs)
}
// unlockedFindRemoteList checks addrMap for each of vpnAddrs. It returns the first RemoteList found,
// and true if that RemoteList is present for all vpnAddrs.
// If false, it means the addrMap, and possibly the RemoteList, need to be corrected.
func (lh *LightHouse) unlockedFindRemoteList(vpnAddrs []netip.Addr) (*RemoteList, bool) {
var am *RemoteList
//todo: if a host with addresses A and B is "split", so it only has address A and a new host has only address B
//todo: we don't handly that correctly, I'm pretty sure.
missingOrDifferent := false
for _, addr := range vpnAddrs {
found, ok := lh.addrMap[addr]
if !ok {
missingOrDifferent = true
} else if am == nil {
am = found //the first list we find wins
} else if am != found {
missingOrDifferent = true
}
}
return am, !missingOrDifferent
}
// unlockedGetRemoteList assumes you have the lh lock
func (lh *LightHouse) unlockedGetRemoteList(allAddrs []netip.Addr) *RemoteList {
// before we go and make a new remotelist, we need to make sure we don't have one for any of this set of vpnaddrs yet
am, ok := lh.unlockedFindRemoteList(allAddrs)
// we failed to find any RemoteLists: make a new one, fill out the addrMap
if am == nil {
am = NewRemoteList(allAddrs, lh.shouldAdd)
for _, addr := range allAddrs {
lh.addrMap[addr] = am
}
return am
}
// we found one! Do we need to fix it?
if !ok {
am.Lock()
am.vpnAddrs = make([]netip.Addr, len(allAddrs))
copy(am.vpnAddrs, allAddrs)
am.Unlock()
for _, addr := range allAddrs {
lh.addrMap[addr] = am
for i, addr := range allAddrs {
am, ok := lh.addrMap[addr]
if ok {
if i != 0 {
lh.addrMap[allAddrs[0]] = am
}
return am
}
}
am := NewRemoteList(allAddrs, lh.shouldAdd)
for _, addr := range allAddrs {
lh.addrMap[addr] = am
}
return am
}
@@ -946,6 +943,26 @@ func (lh *LightHouse) TriggerUpdate() {
}
}
// logLocalAddrsErr reports a localAddrs failure at Warn the first time it is seen and at Debug while
// it persists unchanged, so a permanent failure does not warn on every update forever.
func (lh *LightHouse) logLocalAddrsErr(err error) {
if err == nil {
lh.lastLocalAddrsErr.Store(nil)
return
}
msg := err.Error()
prev := lh.lastLocalAddrsErr.Swap(&msg)
if prev != nil && *prev == msg {
if lh.l.Enabled(context.Background(), slog.LevelDebug) {
lh.l.Debug("Failed to collect local addresses to advertise", "error", err)
}
return
}
lh.l.Warn("Failed to collect local addresses to advertise", "error", err)
}
func (lh *LightHouse) SendUpdate() {
var v4 []*V4AddrPort
var v6 []*V6AddrPort
@@ -1191,6 +1208,17 @@ func (lhh *LightHouseHandler) handleHostQuery(n *NebulaMeta, fromVpnAddrs []neti
return
}
// Don't respond to requests for us.
if lhh.lh.myVpnAddrsTable.Contains(queryVpnAddr) {
if lhh.l.Enabled(context.Background(), slog.LevelDebug) {
lhh.l.Debug("Ignoring HostQuery for one of my own addresses",
"fromVpnAddrs", fromVpnAddrs,
"queryVpnAddr", queryVpnAddr,
)
}
return
}
found, ln, err := lhh.lh.queryAndPrepMessage(queryVpnAddr, func(c *cache) (int, error) {
n = lhh.resetMeta()
n.Type = NebulaMeta_HostQueryReply
@@ -1227,10 +1255,13 @@ func (lhh *LightHouseHandler) handleHostQuery(n *NebulaMeta, fromVpnAddrs []neti
// sendHostPunchNotification signals the other side to punch some zero byte udp packets
func (lhh *LightHouseHandler) sendHostPunchNotification(n *NebulaMeta, fromVpnAddrs []netip.Addr, punchNotifDest netip.Addr, w EncWriter) {
whereToPunch := fromVpnAddrs[0]
// Resolve the target before taking the querier's RemoteList lock below. GetHostInfo takes the
// hostmap lock, and the hostmap is taken before the lighthouse, which is taken before a
// RemoteList, so looking it up inside the callback would deadlock.
targetHI := lhh.lh.ifce.GetHostInfo(punchNotifDest)
found, ln, err := lhh.lh.queryAndPrepMessage(whereToPunch, func(c *cache) (int, error) {
n = lhh.resetMeta()
n.Type = NebulaMeta_HostPunchNotification
targetHI := lhh.lh.ifce.GetHostInfo(punchNotifDest)
var useVersion cert.Version
if targetHI == nil {
useVersion = lhh.lh.ifce.GetCertState().initiatingVersion
+285 -146
View File
@@ -1,16 +1,21 @@
package nebula
import (
"bytes"
"encoding/binary"
"errors"
"fmt"
"net/netip"
"sync"
"testing"
"time"
"github.com/gaissmai/bart"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.yaml.in/yaml/v3"
@@ -27,15 +32,27 @@ func TestOldIPv4Only(t *testing.T) {
assert.Equal(t, binary.BigEndian.Uint32(bp[:]), m.GetAddr())
}
func testCertState(networks ...netip.Prefix) *CertState {
cs := &CertState{
myVpnNetworks: networks,
myVpnNetworksTable: new(bart.Lite),
myVpnAddrs: make([]netip.Addr, 0, len(networks)),
myVpnAddrsTable: new(bart.Lite),
}
for _, n := range networks {
cs.myVpnNetworksTable.Insert(n)
cs.myVpnAddrs = append(cs.myVpnAddrs, n.Addr())
cs.myVpnAddrsTable.Insert(netip.PrefixFrom(n.Addr(), n.Addr().BitLen()))
}
return cs
}
func Test_lhStaticMapping(t *testing.T) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/16")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh1 := "10.128.0.2"
c := config.NewC(l)
@@ -55,12 +72,7 @@ func Test_lhStaticMapping(t *testing.T) {
func TestReloadLighthouseInterval(t *testing.T) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/16")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh1 := "10.128.0.2"
c := config.NewC(l)
@@ -90,12 +102,7 @@ func TestReloadLighthouseInterval(t *testing.T) {
func BenchmarkLighthouseHandleRequest(b *testing.B) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/0")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
c := config.NewC(l)
lh, err := NewLightHouseFromConfig(b.Context(), l, c, cs, nil, nil)
@@ -195,12 +202,7 @@ func TestLighthouse_Memory(t *testing.T) {
c.Settings["listen"] = map[string]any{"port": 4242}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
lh.ifce = &mockEncWriter{}
require.NoError(t, err)
@@ -280,12 +282,7 @@ func TestLighthouse_reload(t *testing.T) {
c.Settings["listen"] = map[string]any{"port": 4242}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
@@ -315,12 +312,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
@@ -429,7 +421,9 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
assert.Equal(t, []netip.AddrPort{netip.MustParseAddrPort("3.3.3.3:4242")}, rl.CopyAddrs([]netip.Prefix{}))
}
func newLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, lhh *LightHouseHandler) testLhReply {
// sendLHHostRequest delivers a HostQuery to lhh and hands back the writer that
// captured what it emitted. Pass a nil filter to see every message.
func sendLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, lhh *LightHouseHandler, filter *NebulaMeta_MessageType) *testEncWriter {
req := &NebulaMeta{
Type: NebulaMeta_HostQuery,
Details: &NebulaMetaDetails{},
@@ -447,12 +441,59 @@ func newLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, l
panic(err)
}
filter := NebulaMeta_HostQueryReply
w := &testEncWriter{
metaFilter: &filter,
}
w := &testEncWriter{metaFilter: filter}
lhh.HandleRequest(fromAddr, []netip.Addr{myVpnIp}, b, w)
return w.lastReply
return w
}
func newLHHostRequest(fromAddr netip.AddrPort, myVpnIp, queryVpnIp netip.Addr, lhh *LightHouseHandler) testLhReply {
filter := NebulaMeta_HostQueryReply
return sendLHHostRequest(fromAddr, myVpnIp, queryVpnIp, lhh, &filter).lastReply
}
func TestLighthouse_IgnoresHostQueryForItself(t *testing.T) {
// Validate that we don't answer host queries for our own address.
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
myVpnIp := myVpnNet.Addr()
c := config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"am_lighthouse": true}
c.Settings["listen"] = map[string]any{"port": 4242}
// Add a static_host_map entry for ourselves, so our address
// is in the addrMap.
c.Settings["static_host_map"] = map[string]any{
myVpnIp.String(): []any{"192.168.100.1:4242"},
}
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(myVpnNet), nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
lhh := lh.NewRequestHandler()
peerVpnIp := netip.MustParseAddr("10.128.0.2")
peerUdpAddr := netip.MustParseAddrPort("10.0.0.2:4242")
otherVpnIp := netip.MustParseAddr("10.128.0.3")
otherUdpAddr := netip.MustParseAddrPort("10.0.0.3:4242")
newLHHostUpdate(peerUdpAddr, peerVpnIp, []netip.AddrPort{peerUdpAddr}, lhh)
newLHHostUpdate(otherUdpAddr, otherVpnIp, []netip.AddrPort{otherUdpAddr}, lhh)
// Control: a query about a real peer is still answered, and still ends with
// the punch notification aimed at the host that was asked about.
w := sendLHHostRequest(peerUdpAddr, peerVpnIp, otherVpnIp, lhh, nil)
require.NotNil(t, w.lastReply.msg)
assert.Equal(t, NebulaMeta_HostPunchNotification, w.lastReply.msg.Type)
assert.Equal(t, otherVpnIp, w.lastReply.vpnIp)
// Now validate that we don't send to ourselves.
found, _, err := lh.queryAndPrepMessage(myVpnIp, func(*cache) (int, error) { return 0, nil })
require.NoError(t, err)
require.True(t, found, "the lighthouse should hold a cache entry for its own address")
w = sendLHHostRequest(peerUdpAddr, peerVpnIp, myVpnIp, lhh, nil)
assert.Nil(t, w.lastReply.msg, "a query about our own address must produce no reply and no punch notification")
}
func newLHHostUpdate(fromAddr netip.AddrPort, vpnIp netip.Addr, addrs []netip.AddrPort, lhh *LightHouseHandler) {
@@ -642,12 +683,7 @@ func TestLighthouse_Dont_Delete_Static_Hosts(t *testing.T) {
}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
@@ -708,12 +744,7 @@ func TestLighthouse_DeletesWork(t *testing.T) {
}
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
nt := new(bart.Lite)
nt.Insert(myVpnNet)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
cs := testCertState(myVpnNet)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
@@ -739,122 +770,230 @@ func TestLighthouse_DeletesWork(t *testing.T) {
assert.Nil(t, out)
}
// newLHHostUpdateV2 sends a v2-style HostUpdateNotification where the sending tunnel carries
// multiple vpn addrs (a dual-stack v2 cert). Details.VpnAddr is left blank like SendUpdate does.
func newLHHostUpdateV2(fromAddr netip.AddrPort, vpnAddrs []netip.Addr, addrs []netip.AddrPort, lhh *LightHouseHandler) {
req := &NebulaMeta{
Type: NebulaMeta_HostUpdateNotification,
Details: &NebulaMetaDetails{},
}
for _, v := range addrs {
if v.Addr().Is4() {
req.Details.V4AddrPorts = append(req.Details.V4AddrPorts, netAddrToProtoV4AddrPort(v.Addr(), v.Port()))
} else {
req.Details.V6AddrPorts = append(req.Details.V6AddrPorts, netAddrToProtoV6AddrPort(v.Addr(), v.Port()))
}
}
b, err := req.Marshal()
if err != nil {
panic(err)
}
lhh.HandleRequest(fromAddr, vpnAddrs, b, &testEncWriter{})
func TestLightHouse_logLocalAddrsErr(t *testing.T) {
out := &bytes.Buffer{}
lh := &LightHouse{l: test.NewLoggerWithOutput(out)}
// The first sighting of a failure warns.
lh.logLocalAddrsErr(errors.New("permission denied"))
assert.Contains(t, out.String(), "level=WARN")
assert.Contains(t, out.String(), "permission denied")
// Repeating unchanged does not warn again, which is what keeps a permanent failure from warning
// on every lighthouse.interval for the life of the process.
out.Reset()
lh.logLocalAddrsErr(errors.New("permission denied"))
assert.NotContains(t, out.String(), "level=WARN")
// A different failure is a new event and warns.
out.Reset()
lh.logLocalAddrsErr(errors.New("something else"))
assert.Contains(t, out.String(), "level=WARN")
assert.Contains(t, out.String(), "something else")
// Recovering resets, so the same failure returning later warns again.
out.Reset()
lh.logLocalAddrsErr(nil)
assert.Empty(t, out.String())
lh.logLocalAddrsErr(errors.New("something else"))
assert.Contains(t, out.String(), "level=WARN")
}
func newIssue1868Lighthouse(t *testing.T) (*LightHouse, *LightHouseHandler) {
// heldEncWriter holds the query worker in its first GetHostInfo until release closes.
type heldEncWriter struct {
mockEncWriter
once sync.Once
held chan struct{}
release chan struct{}
}
func (w *heldEncWriter) GetHostInfo(netip.Addr) *HostInfo {
w.once.Do(func() { close(w.held) })
<-w.release
return nil
}
// QueryServer never waits on the query worker, since the packet readers call it and nothing drains the channel at
// shutdown; a query that finds the channel full is dropped and counted. The worker is held on its first query so the
// channel fills.
func TestLighthouse_QueryServerDoesNotBlock(t *testing.T) {
l := test.NewLogger()
cs := testCertState(netip.MustParsePrefix("10.128.0.1/16"))
lh1 := "10.128.0.2"
c := config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"hosts": []any{lh1}}
c.Settings["static_host_map"] = map[string]any{lh1: []any{"1.1.1.1:4242"}}
c.Settings["handshakes"] = map[string]any{"query_buffer": 1}
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
w := &heldEncWriter{held: make(chan struct{}), release: make(chan struct{})}
lh.ifce = w
t.Cleanup(func() { close(w.release) })
lh.QueryServer(netip.MustParseAddr("10.128.0.10"))
select {
case <-w.held:
case <-time.After(5 * time.Second):
t.Fatal("the query worker never took the first query")
}
dropped := lh.queryDropped.Count()
done := make(chan struct{})
go func() {
lh.QueryServer(netip.MustParseAddr("10.128.0.11"))
lh.QueryServer(netip.MustParseAddr("10.128.0.12"))
close(done)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("QueryServer blocked on a full query channel")
}
assert.EqualValues(t, 1, lh.queryDropped.Count()-dropped)
}
// hostmapEncWriter answers GetHostInfo from a real hostmap, the way Interface does.
type hostmapEncWriter struct {
mockEncWriter
hm *HostMap
}
func (w *hostmapEncWriter) GetHostInfo(a netip.Addr) *HostInfo { return w.hm.QueryVpnAddr(a) }
func (w *hostmapEncWriter) GetCertState() *CertState {
return &CertState{initiatingVersion: cert.Version1}
}
// A lighthouse runs four paths that each hold one lock while waiting for the next: CheckAndComplete
// holds the hostmap and takes the handshake manager, StartHandshake holds the handshake manager and
// takes the lighthouse, a host update holds the lighthouse and takes the sender's RemoteList, and a
// punch notification holds the querier's RemoteList while it resolves the target. If that last step
// reaches the hostmap under the RemoteList lock, the four deadlock. Drive them all at once.
func TestLighthouse_PunchNotificationDoesNotCloseLockCycle(t *testing.T) {
l := test.NewLogger()
myVpnNet := netip.MustParsePrefix("10.128.0.1/24")
c := config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"am_lighthouse": true}
c.Settings["listen"] = map[string]any{"port": 4242}
myVpnNet4 := netip.MustParsePrefix("10.128.0.1/24")
myVpnNet6 := netip.MustParsePrefix("fd00::1/64")
nt := new(bart.Lite)
nt.Insert(myVpnNet4)
nt.Insert(myVpnNet6)
cs := &CertState{
myVpnNetworks: []netip.Prefix{myVpnNet4, myVpnNet6},
myVpnNetworksTable: nt,
}
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(myVpnNet), nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
return lh, lh.NewRequestHandler()
}
// Scenario A: host registers via v2 (both addrs), then a v1 handshake with the same host completes on
// the lighthouse (rehandshake after cert renewal, relay-initiated handshake, traffic to the LH's v4
// addr...). handshake_manager does QueryCache(vpnAddrs) + RefreshFromHandshake(vpnAddrs) with the v1
// cert's single address, which truncates RemoteList.vpnAddrs.
func TestLighthouse_Issue1868_V1HandshakeTruncatesVpnAddrs(t *testing.T) {
lh, lhh := newIssue1868Lighthouse(t)
mainHM := newHostMap(l)
pr := []netip.Prefix{}
mainHM.preferredRanges.Store(&pr)
lh.ifce = &hostmapEncWriter{hm: mainHM}
hostV4 := netip.MustParseAddr("10.128.0.3")
hostV6 := netip.MustParseAddr("fd00::3")
hostUdp := netip.MustParseAddrPort("192.0.2.3:4242")
hostLan := netip.MustParseAddrPort("10.0.0.3:4242")
hm := NewHandshakeManager(l, mainHM, lh, &udp.NoopConn{}, defaultHandshakeConfig)
f := &Interface{hostMap: mainHM, lightHouse: lh, handshakeManager: hm, l: l}
hm.f = f
askerV4 := netip.MustParseAddr("10.128.0.2")
askerUdp := netip.MustParseAddrPort("192.0.2.2:4242")
peer := netip.MustParseAddr("10.128.0.2")
peerUdp := netip.MustParseAddrPort("10.0.0.2:4242")
target := netip.MustParseAddr("10.128.0.3")
targetUdp := netip.MustParseAddrPort("10.0.0.3:4242")
outbound := netip.MustParseAddr("10.128.0.4")
inbound := netip.MustParseAddr("10.128.0.5")
// Boot: host handshakes with the LH using its v2 cert and sends an update
newLHHostUpdateV2(hostUdp, []netip.Addr{hostV4, hostV6}, []netip.AddrPort{hostLan}, lhh)
seed := lh.NewRequestHandler()
newLHHostUpdate(peerUdp, peer, []netip.AddrPort{peerUdp}, seed)
newLHHostUpdate(targetUdp, target, []netip.AddrPort{targetUdp}, seed)
// Both addresses resolve
r := newLHHostRequest(askerUdp, askerV4, hostV4, lhh)
require.NotNil(t, r.msg, "v4 query should be answered")
assertIp4InArray(t, r.msg.Details.V4AddrPorts, hostLan)
stop := make(chan struct{})
var wg sync.WaitGroup
loop := func(body func()) {
defer wg.Done()
for {
select {
case <-stop:
return
default:
body()
}
}
}
wg.Add(4)
r = newLHHostRequest(askerUdp, askerV4, hostV6, lhh)
require.NotNil(t, r.msg, "v6 query should be answered before the v1 handshake")
assertIp4InArray(t, r.msg.Details.V4AddrPorts, hostLan)
// Outbound handshakes keep starting and being dropped.
go loop(func() {
hm.DeleteHostInfo(hm.StartHandshake(outbound, nil))
})
// Later: a v1 handshake with the same host completes on the LH. This is exactly what
// handshake_manager.go does on completion, with the v1 cert's single vpn addr.
rl := lh.QueryCache([]netip.Addr{hostV4})
rl.RefreshFromHandshake([]netip.Addr{hostV4}, cert.Version1)
// Inbound handshakes keep completing and being torn down.
var idx uint32
go loop(func() {
idx++
hi := &HostInfo{
vpnAddrs: []netip.Addr{inbound},
localIndexId: idx,
remoteIndexId: idx,
remotes: lh.QueryCache([]netip.Addr{inbound}),
HandshakePacket: map[uint8][]byte{handshakePacketStage0: {byte(idx)}},
}
if _, err := hm.CheckAndComplete(hi, handshakePacketStage0, f); err == nil {
mainHM.DeleteHostInfo(hi)
}
})
// The host keeps sending v2 updates over its v2 tunnel too
newLHHostUpdateV2(hostUdp, []netip.Addr{hostV4, hostV6}, []netip.AddrPort{hostLan}, lhh)
// The peer keeps sending host updates.
updates := lh.NewRequestHandler()
go loop(func() {
newLHHostUpdate(peerUdp, peer, []netip.AddrPort{peerUdp}, updates)
})
r = newLHHostRequest(askerUdp, askerV4, hostV4, lhh)
require.NotNil(t, r.msg, "v4 query should still be answered")
assertIp4InArray(t, r.msg.Details.V4AddrPorts, hostLan)
// The same peer keeps querying for the target, which ends in a punch notification.
queries := lh.NewRequestHandler()
go loop(func() {
sendLHHostRequest(peerUdp, peer, target, queries, nil)
})
r = newLHHostRequest(askerUdp, askerV4, hostV6, lhh)
if assert.NotNil(t, r.msg, "BUG: v6 query is silently dropped after a v1 handshake truncated RemoteList.vpnAddrs") {
assertIp4InArray(t, r.msg.Details.V4AddrPorts, hostLan)
time.Sleep(time.Second)
close(stop)
done := make(chan struct{})
go func() { wg.Wait(); close(done) }()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("deadlock: the lighthouse paths never finished after stop was signalled")
}
}
// Scenario B: the LH first creates a RemoteList for the host keyed only by its v4 addr (a pending
// LH-initiated v1 handshake does QueryCache([v4]) in handleOutbound), then the host arrives with v2.
// unlockedGetRemoteList/QueryCache hit on allAddrs[0] and never add the v6 key to addrMap.
func TestLighthouse_Issue1868_V4OnlyListNeverGainsV6Key(t *testing.T) {
lh, lhh := newIssue1868Lighthouse(t)
func TestLighthouse_reloadRelaysTriggersUpdate(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
base := "lighthouse:\n am_lighthouse: true\nlisten:\n port: 4242\n"
require.NoError(t, c.LoadString(base+"relay:\n relays: [10.128.0.5]\n"))
hostV4 := netip.MustParseAddr("10.128.0.3")
hostV6 := netip.MustParseAddr("fd00::3")
hostUdp := netip.MustParseAddrPort("192.0.2.3:4242")
hostLan := netip.MustParseAddrPort("10.0.0.3:4242")
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
assert.Empty(t, lh.updateTrigger)
askerV4 := netip.MustParseAddr("10.128.0.2")
askerUdp := netip.MustParseAddrPort("192.0.2.2:4242")
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6]\n"))
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.6")}, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
<-lh.updateTrigger
// LH is a relay and someone asked it to relay to hostV4 while the host was offline:
// StartHandshake(hostV4) -> handleOutbound -> QueryCache([hostV4]) creates a v4-only list.
_ = lh.QueryCache([]netip.Addr{hostV4})
// The config changed but what we advertise did not, a bad entry is skipped
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6, nope]\n"))
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.6")}, lh.GetRelaysForMe())
assert.Empty(t, lh.updateTrigger)
}
// Host boots and handshakes v2 with the LH (responder path does QueryCache + RefreshFromHandshake)
rl := lh.QueryCache([]netip.Addr{hostV4, hostV6})
rl.RefreshFromHandshake([]netip.Addr{hostV4, hostV6}, cert.Version2)
newLHHostUpdateV2(hostUdp, []netip.Addr{hostV4, hostV6}, []netip.AddrPort{hostLan}, lhh)
func TestLighthouse_reloadAmRelayAlone(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
base := "lighthouse:\n am_lighthouse: true\nlisten:\n port: 4242\nrelay:\n relays: [10.128.0.5]\n"
require.NoError(t, c.LoadString(base+" am_relay: false\n"))
r := newLHHostRequest(askerUdp, askerV4, hostV4, lhh)
require.NotNil(t, r.msg, "v4 query should be answered")
assertIp4InArray(t, r.msg.Details.V4AddrPorts, hostLan)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.5")}, lh.GetRelaysForMe())
r = newLHHostRequest(askerUdp, askerV4, hostV6, lhh)
if assert.NotNil(t, r.msg, "BUG: v6 query is silently dropped, addrMap never got the v6 key") {
assertIp4InArray(t, r.msg.Details.V4AddrPorts, hostLan)
}
// Relays aren't allowed to have relays, flipping am_relay alone must drop them and tell the lighthouses
require.NoError(t, c.ReloadConfigString(base+" am_relay: true\n"))
assert.Empty(t, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
<-lh.updateTrigger
require.NoError(t, c.ReloadConfigString(base+" am_relay: false\n"))
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.5")}, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
}
+31
View File
@@ -0,0 +1,31 @@
package logging
import "log/slog"
// RateLimiter is what RateLimited needs from a rate limiter, such as a *ratelimit.Limiter.
// Allow reports whether this call may proceed, and how many calls were denied since the last one that did.
type RateLimiter interface {
Allow() (denied uint64, ok bool)
}
// RateLimited checks lim before anything is built.
// When allowed it returns l, with a "suppressed" count attached if lim denied calls since its last allowed one.
// A nil lim allows everything.
//
// if l, ok := logging.RateLimited(f.l, f.udpTxErrLimiter); ok {
// hostinfo.logger(l).Error("Failed to write outgoing packet", "error", err)
// }
func RateLimited(l *slog.Logger, lim RateLimiter) (*slog.Logger, bool) {
if lim == nil {
return l, true
}
denied, ok := lim.Allow()
if !ok {
return nil, false
}
if denied > 0 {
return l.With("suppressed", denied), true
}
return l, true
}
+106
View File
@@ -0,0 +1,106 @@
package logging
import (
"bytes"
"errors"
"io"
"log/slog"
"net/netip"
"strings"
"testing"
"time"
"github.com/slackhq/nebula/ratelimit"
)
type fakeLimiter struct {
denied uint64
ok bool
}
func (f fakeLimiter) Allow() (uint64, bool) { return f.denied, f.ok }
func TestRateLimited(t *testing.T) {
var buf bytes.Buffer
base := NewLogger(&buf)
if l, ok := RateLimited(base, fakeLimiter{ok: false}); ok || l != nil {
t.Fatalf("denied call: ok=%v logger=%v", ok, l)
}
if l, ok := RateLimited(base, fakeLimiter{ok: true}); !ok || l != base {
t.Fatal("nothing denied should hand back the same logger")
}
if l, ok := RateLimited(base, nil); !ok || l != base {
t.Fatal("a nil limiter should allow with the same logger")
}
l, ok := RateLimited(base, fakeLimiter{denied: 7, ok: true})
if !ok {
t.Fatal("allowed call denied")
}
l.Info("hello", "k", "v")
if !strings.Contains(buf.String(), "suppressed=7") {
t.Fatalf("want suppressed=7, got %q", buf.String())
}
}
// The benchmarks compare a log line through slog directly against the same line behind RateLimited: logged with
// nothing denied, denied, and logged after denials.
var (
benchErr = errors.New("sendto: no buffer space available")
benchAddr = netip.MustParseAddrPort("192.0.2.1:4242")
)
func benchLogger() *slog.Logger {
return NewLogger(io.Discard).With("localIndex", 1234, "remoteIndex", 5678)
}
func BenchmarkRateLimitedBaseline(b *testing.B) {
lg := benchLogger()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
lg.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
func BenchmarkRateLimitedAllowed(b *testing.B) {
lg, lim := benchLogger(), mustLimiter(b, 1, 0)
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if l, ok := RateLimited(lg, lim); ok {
l.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
}
func BenchmarkRateLimitedDenied(b *testing.B) {
lg, lim := benchLogger(), mustLimiter(b, 1, time.Hour)
lim.Allow()
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if l, ok := RateLimited(lg, lim); ok {
l.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
}
func BenchmarkRateLimitedAfterDenied(b *testing.B) {
lg, lim := benchLogger(), fakeLimiter{denied: 5, ok: true}
b.ReportAllocs()
for i := 0; i < b.N; i++ {
if l, ok := RateLimited(lg, lim); ok {
l.Error("Failed to write outgoing packet", "error", benchErr, "udpAddr", benchAddr)
}
}
}
func mustLimiter(b *testing.B, burst int, every time.Duration) *ratelimit.Limiter {
l, err := ratelimit.New(burst, every)
if err != nil {
b.Fatal(err)
}
return l
}
+10 -5
View File
@@ -32,9 +32,12 @@ const _ = noiseutil.RejectHeadroom - 4*maxRoutines
func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, deviceFactory overlay.DeviceFactory) (retcon *Control, reterr error) {
ctx, cancel := context.WithCancel(context.Background())
// Automatically cancel the context if Main returns an error, to signal all created goroutines to quit.
// The goroutines started below stop only when this context does, and only a caller holding the
// Control can arrange that. Cancel whenever we are not handing one back, which covers an error
// and a config test alike: a config test used to leave the lighthouse query worker, and a
// hostname resolver per dns named static host, running for the life of the process.
defer func() {
if reterr != nil {
if retcon == nil {
cancel()
}
}()
@@ -297,6 +300,7 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
}
var ifce *Interface
var emitStats func()
if !configTest {
ifce, err = NewInterface(ctx, ifConfig)
if err != nil {
@@ -315,9 +319,12 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
go handshakeManager.Run(ctx)
punchy.Start(ctx, ifce, hostMap, lightHouse)
// On the stats server's loop, so the gauges follow the interval it accepted
emitStats = ifce.statsEmitter()
}
stats, err := newStatsServerFromConfig(ctx, l, c, buildVersion, configTest)
stats, err := newStatsServerFromConfig(ctx, l, c, buildVersion, configTest, emitStats)
if err != nil {
return nil, util.ContextualizeIfNeeded("Failed to start stats emitter", err)
}
@@ -326,8 +333,6 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
return nil, nil
}
go ifce.emitStats(ctx, c.GetDuration("stats.interval", time.Second*10))
attachCommands(l, c, ssh, ifce)
networkChanges := udp.NewNetworkChangeMonitor(ctx, l, c)
+74
View File
@@ -1,12 +1,21 @@
package nebula
import (
"fmt"
"net/netip"
"os"
"path/filepath"
"testing"
"time"
"github.com/slackhq/nebula/cert"
cert_test "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/util"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/goleak"
)
func TestParseCpuAffinity(t *testing.T) {
@@ -49,3 +58,68 @@ func TestParseCpuAffinity(t *testing.T) {
assert.Nil(t, parseCpuAffinity(newConfig([]any{1 << 20}), l, 1))
}
}
// TestMain_ConfigTestReleasesItsGoroutines pins the rule that Main only leaves goroutines running
// when it hands back a Control to stop them with.
//
// A config test gets no Control, so anything it started had nothing to stop it: the lighthouse
// query worker, and a hostname resolver per dns named static host, ran for the life of the
// process. That matters to every embedder that validates a config in process rather than by
// exec'ing, dnclient and the apple clients included, because they do it on each config load and
// the leak accumulates.
func TestMain_ConfigTestReleasesItsGoroutines(t *testing.T) {
defer goleak.VerifyNone(t, goleak.IgnoreCurrent())
l := test.NewLogger()
dir := t.TempDir()
before := time.Now().Add(-time.Hour)
after := time.Now().Add(time.Hour)
ca, _, caKey, caPEM := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, before, after, nil, nil, nil)
networks := []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")}
_, _, keyPEM, certPEM := cert_test.NewTestCert(
cert.Version2, cert.Curve_CURVE25519, ca, caKey, "config-test", before, after, networks, nil, nil)
caPath := filepath.Join(dir, "ca.pem")
certPath := filepath.Join(dir, "cert.pem")
keyPath := filepath.Join(dir, "key.pem")
require.NoError(t, os.WriteFile(caPath, caPEM, 0o600))
require.NoError(t, os.WriteFile(certPath, certPEM, 0o600))
require.NoError(t, os.WriteFile(keyPath, keyPEM, 0o600))
// A static host by address, not by name: the query worker is the goroutine under test and a
// hostname would drag a real dns lookup into a unit test.
configBody := fmt.Sprintf(`
pki:
ca: %s
cert: %s
key: %s
static_host_map:
"10.0.0.2": ["192.0.2.1:4242"]
lighthouse:
hosts:
- "10.0.0.2"
listen:
host: 127.0.0.1
port: 0
tun:
disabled: true
firewall:
outbound:
- port: any
proto: any
host: any
inbound:
- port: any
proto: any
host: any
`, caPath, certPath, keyPath)
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.yml"), []byte(configBody), 0o600))
c := config.NewC(l)
require.NoError(t, c.Load(dir))
ctrl, err := Main(c, true, "config-test", l, nil)
require.NoError(t, err)
require.Nil(t, ctrl, "a config test hands back nothing to stop, so it must stop itself")
}
+39
View File
@@ -0,0 +1,39 @@
//go:build !android
package nicfinder
import (
"context"
"errors"
"fmt"
"log/slog"
"net"
"net/netip"
)
func localAddrs(ctx context.Context, l *slog.Logger, filter Filter) ([]netip.Addr, error) {
ifaces, err := net.Interfaces()
if err != nil {
return nil, fmt.Errorf("failed to enumerate local interfaces: %w", err)
}
var out []netip.Addr
var errs []error
for _, i := range ifaces {
if !allowName(ctx, l, filter, i.Name) {
continue
}
addrs, err := i.Addrs()
if err != nil {
errs = append(errs, fmt.Errorf("failed to get addresses for %s: %w", i.Name, err))
continue
}
for _, rawAddr := range addrs {
if addr, ok := allowedAddr(ctx, l, filter, rawAddr); ok {
out = append(out, addr)
}
}
}
return out, errors.Join(errs...)
}
+104
View File
@@ -0,0 +1,104 @@
//go:build android
package nicfinder
import (
"context"
"errors"
"fmt"
"log/slog"
"net/netip"
"os"
"github.com/DefinedNet/netlink"
"github.com/DefinedNet/netlink/nl"
"golang.org/x/sys/unix"
)
// Android denies bind() on netlink route sockets and RTM_GETLINK, so net.Interfaces fails with EACCES.
// Enumerate with an RTM_GETADDR dump on an unbound socket instead.
// localInterfaces returns every interface that has an address, with its addresses. It lists
// addresses over a netlink socket that is never bound and names each index with SIOCGIFNAME, the
// same operations Bionic's getifaddrs uses for app UIDs, which the sandbox permits.
func localAddrs(ctx context.Context, l *slog.Logger, filter Filter) ([]netip.Addr, error) {
var errs []error
addrs, err := netlinkAddrList()
if err != nil {
if !errors.Is(err, netlink.ErrDumpInterrupted) {
return nil, fmt.Errorf("failed to enumerate local interfaces: %w", err)
}
// The address table kept changing under the dump and netlink gave up retrying. What it
// returned is still worth advertising.
errs = append(errs, fmt.Errorf("local addresses may be incomplete: %w", err))
}
// Any socket will do for SIOCGIFNAME.
sfd, err := unix.Socket(unix.AF_INET, unix.SOCK_DGRAM|unix.SOCK_CLOEXEC, 0)
if err != nil {
return nil, os.NewSyscallError("socket", err)
}
defer unix.Close(sfd)
var out []netip.Addr
nameAllowed := make(map[int]bool) // by interface index, once named
for _, a := range addrs {
allow, named := nameAllowed[a.LinkIndex]
if (named && !allow) || a.IPNet == nil {
continue
}
addr, ok := allowedAddr(ctx, l, filter, a.IPNet)
if !ok {
continue
}
if !named {
name, err := interfaceNameByIndex(sfd, a.LinkIndex)
// ENODEV means the interface went away after the dump. Any other failure is reported
// and the interface skipped, so the rest are still returned.
if err != nil && !errors.Is(err, unix.ENODEV) {
errs = append(errs, fmt.Errorf("failed to resolve name of interface %d: %w", a.LinkIndex, err))
}
allow = err == nil && allowName(ctx, l, filter, name)
nameAllowed[a.LinkIndex] = allow
}
if allow {
out = append(out, addr)
}
}
return out, errors.Join(errs...)
}
// netlinkAddrList is netlink.AddrList over a socket that is never bound. The kernel autobinds it
// on the first send, which the sandbox permits.
func netlinkAddrList() ([]netlink.Addr, error) {
fd, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW|unix.SOCK_CLOEXEC, unix.NETLINK_ROUTE)
if err != nil {
return nil, os.NewSyscallError("socket", err)
}
s, err := nl.NewNetlinkSocketFromFd(fd)
if err != nil {
unix.Close(fd)
return nil, err
}
h := netlink.NewHandleFromSockets(
map[int]*nl.SocketHandle{unix.NETLINK_ROUTE: {Socket: s}},
netlink.HandleOptions{RetryInterrupted: true},
)
defer h.Close()
if err := h.SetSocketTimeout(netlink.GetSocketTimeout()); err != nil {
return nil, err
}
return h.AddrList(nil, netlink.FAMILY_ALL)
}
// interfaceNameByIndex resolves an interface index with SIOCGIFNAME.
func interfaceNameByIndex(fd int, index int) (string, error) {
var ifr unix.Ifreq
ifr.SetUint32(uint32(index)) // ifr_ifindex is in the ifr_ifru union
if err := unix.IoctlIfreq(fd, unix.SIOCGIFNAME, &ifr); err != nil {
return "", err
}
return ifr.Name(), nil
}
+59
View File
@@ -0,0 +1,59 @@
package nicfinder
import (
"context"
"log/slog"
"net"
"net/netip"
"github.com/slackhq/nebula/logging"
)
// Filter decides which interfaces, by name, and which of their addresses Find returns.
type Filter interface {
AllowName(string) bool
Allow(netip.Addr) bool
}
// Find returns the addresses of this host's network interfaces that allowList accepts, with
// loopback and link-local addresses left out. When only some interfaces could be read it returns
// their addresses alongside an error for the rest. allowList must not be nil; it is called as
// given, so a nil pointer of a type whose methods accept a nil receiver is fine.
func Find(ctx context.Context, l *slog.Logger, allowList Filter) ([]netip.Addr, error) {
return localAddrs(ctx, l, allowList)
}
// allowName asks filter about an interface name and traces the answer.
func allowName(ctx context.Context, l *slog.Logger, filter Filter, name string) bool {
allow := filter.AllowName(name)
if l.Enabled(ctx, logging.LevelTrace) {
l.Log(ctx, logging.LevelTrace, "localAllowList.AllowName", "interfaceName", name, "allow", allow)
}
return allow
}
// allowedAddr converts one address a platform reports and says whether to advertise it: it must
// be a *net.IPNet or *net.IPAddr carrying an IP, not be loopback or link-local, and be accepted by
// filter, whose answer is traced. IPv4 comes back unmapped.
func allowedAddr(ctx context.Context, l *slog.Logger, filter Filter, rawAddr net.Addr) (netip.Addr, bool) {
var ip net.IP
switch v := rawAddr.(type) {
case *net.IPNet:
ip = v.IP
case *net.IPAddr:
ip = v.IP
}
addr, ok := netip.AddrFromSlice(ip)
if !ok {
return netip.Addr{}, false
}
addr = addr.Unmap()
if addr.IsLoopback() || addr.IsLinkLocalUnicast() {
return netip.Addr{}, false
}
isAllowed := filter.Allow(addr)
if l.Enabled(ctx, logging.LevelTrace) {
l.Log(ctx, logging.LevelTrace, "localAllowList.Allow", "localAddr", addr, "allowed", isAllowed)
}
return addr, isAllowed
}
+117
View File
@@ -0,0 +1,117 @@
package nicfinder
import (
"context"
"net"
"net/netip"
"regexp"
"testing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
)
type allowAll struct{}
func (allowAll) Allow(netip.Addr) bool { return true }
func (allowAll) AllowName(string) bool { return true }
type testAllowList struct {
bannedName *regexp.Regexp
bannedAddr netip.Addr
}
func (l *testAllowList) Allow(addr netip.Addr) bool {
return addr != l.bannedAddr
}
func (l *testAllowList) AllowName(n string) bool {
return !l.bannedName.MatchString(n)
}
func newTestAllowList(bannedName string) *testAllowList {
return &testAllowList{
bannedName: regexp.MustCompile(bannedName),
}
}
func TestAllowedAddr(t *testing.T) {
ctx, l := context.Background(), test.NewLogger()
allowed := func(filter Filter, addrs []net.Addr) []netip.Addr {
var out []netip.Addr
for _, rawAddr := range addrs {
if addr, ok := allowedAddr(ctx, l, filter, rawAddr); ok {
out = append(out, addr)
}
}
return out
}
addrs := []net.Addr{
&net.IPNet{IP: net.ParseIP("127.0.0.1"), Mask: net.CIDRMask(8, 32)},
&net.IPNet{IP: net.ParseIP("::1"), Mask: net.CIDRMask(128, 128)},
&net.IPNet{IP: net.ParseIP("10.0.0.5"), Mask: net.CIDRMask(24, 32)},
&net.IPNet{IP: net.ParseIP("169.254.10.10"), Mask: net.CIDRMask(16, 32)},
&net.IPNet{IP: net.ParseIP("fe80::1"), Mask: net.CIDRMask(64, 128)},
&net.IPAddr{IP: net.ParseIP("fd00::5")},
&net.IPNet{},
otherAddr{},
nil,
}
// Loopback, link local and anything without an IP are dropped, and IPv4 comes back unmapped
// even though net.ParseIP hands over a 16 byte slice.
assert.Equal(t, []netip.Addr{
netip.MustParseAddr("10.0.0.5"),
netip.MustParseAddr("fd00::5"),
}, allowed(allowAll{}, addrs))
// The filter sees the unmapped form and can drop a single address.
al := newTestAllowList("none")
al.bannedAddr = netip.MustParseAddr("10.0.0.5")
assert.Equal(t, []netip.Addr{netip.MustParseAddr("fd00::5")}, allowed(al, addrs))
}
// otherAddr is a net.Addr that is neither *net.IPNet nor *net.IPAddr. netlink.Addr is one such
// type: it satisfies net.Addr through an embedded *net.IPNet, and passing it directly once
// silently produced no addresses.
type otherAddr struct{}
func (otherAddr) Network() string { return "other" }
func (otherAddr) String() string { return "other" }
// hostFilter counts AllowName calls and can reject every name or every address.
type hostFilter struct {
rejectNames bool
rejectAddrs bool
asked map[string]int
}
func (f *hostFilter) Allow(netip.Addr) bool { return !f.rejectAddrs }
func (f *hostFilter) AllowName(name string) bool {
f.asked[name]++
return !f.rejectNames
}
// Runs this platform's localAddrs against the host and checks it honors the whole filter.
func TestLocalAddrsAppliesFilter(t *testing.T) {
ctx, l := context.Background(), test.NewLogger()
f := &hostFilter{asked: map[string]int{}}
all, err := localAddrs(ctx, l, f)
if len(f.asked) == 0 {
t.Skipf("no interfaces to enumerate here: %v", err)
}
for name, n := range f.asked {
assert.Equal(t, 1, n, "AllowName asked %d times for %s", n, name)
}
// Loopback and link-local never come back, whatever the filter says.
for _, addr := range all {
assert.False(t, addr.IsLoopback() || addr.IsLinkLocalUnicast(), "%s", addr)
}
none, _ := localAddrs(ctx, l, &hostFilter{rejectNames: true, asked: map[string]int{}})
assert.Empty(t, none, "addresses got past a filter that rejects every interface name")
none, _ = localAddrs(ctx, l, &hostFilter{rejectAddrs: true, asked: map[string]int{}})
assert.Empty(t, none, "addresses got past a filter that rejects every address")
}
-1
View File
@@ -1,5 +1,4 @@
//go:build boringcrypto
// +build boringcrypto
package noiseutil
-1
View File
@@ -1,5 +1,4 @@
//go:build boringcrypto
// +build boringcrypto
package noiseutil
+11 -8
View File
@@ -160,9 +160,7 @@ func (f *Interface) readOutsidePackets(via ViaSender, packet []byte, rxc *rxCont
case header.TestReply:
// No-op, useful for the Roaming and connectionManager side-effects above
case header.TestRequest:
const maxCipherOverhead = 16 //todo we use this too often, needs a real importable const
const maxOverhead = header.Len + header.Len + maxCipherOverhead + maxCipherOverhead
if maxOverhead+len(out) > len(rxc.scratch) {
if header.MaxOverhead+len(out) > len(rxc.scratch) {
// A reply that cannot fit in scratch is dropped no matter the log level.
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("dropping oversized test request", "payloadLen", len(out), "from", via)
@@ -263,11 +261,16 @@ func (f *Interface) handleOutsideRelayPacket(hostinfo *HostInfo, via ViaSender,
// closeTunnel closes a tunnel locally, it does not send a closeTunnel packet to the remote
func (f *Interface) closeTunnel(hostInfo *HostInfo) {
final := f.hostMap.DeleteHostInfo(hostInfo)
if final {
// We no longer have any tunnels with this vpn addr, clear learned lighthouse state to lower memory usage
f.lightHouse.DeleteVpnAddrs(hostInfo.vpnAddrs)
// Hold the hostmap lock across both the delete and the lighthouse eviction,
// to avoid a race with handshake completion.
f.hostMap.Lock()
defer f.hostMap.Unlock()
if !f.hostMap.unlockedDeleteHostInfo(hostInfo) {
return
}
// We no longer have any tunnels with this vpn addr, clear learned lighthouse state to lower memory usage
f.lightHouse.DeleteVpnAddrs(hostInfo.vpnAddrs)
}
// sendCloseTunnel is a helper function to send a proper close tunnel packet to a remote
@@ -354,7 +357,7 @@ func parseV6(data []byte, incoming bool, fp *firewall.ParsedPacket) error {
// and cannot drift into misreading an unknown protocol (SCTP, GRE, etc.) as a forged transport.
proto, offset, isFragment, anyFragment, err := iputil.IPv6FindUpperProtocol(data)
if err != nil {
return ErrIPv6PacketTooShort
return err
}
fp.Protocol = proto
+7 -7
View File
@@ -117,7 +117,7 @@ func Test_newPacket_v6(t *testing.T) {
require.NoError(t, err)
err = newPacket(buffer.Bytes(), true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
// A v6 packet with a hop-by-hop extension
// ICMPv6 Payload (Echo Request)
@@ -151,12 +151,12 @@ func Test_newPacket_v6(t *testing.T) {
// A full IPv6 header and 1 byte in the first extension, but missing
// the length byte.
err = newPacket(buffer.Bytes()[:41], true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
// A full IPv6 header plus 1 full extension, but only 1 byte of the
// next layer, missing length byte
err = newPacket(buffer.Bytes()[:49], true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
err = nil
// A good ICMP packet
@@ -354,12 +354,12 @@ func Test_newPacket_v6(t *testing.T) {
// Ensure buffer bounds checking during processing, a truncated AH header can't reach the payload
err = newPacket(b[:41], true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
// Invalid AH header
b = buffer.Bytes()
err = newPacket(b, true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
}
func Test_newPacket_ipv6Fragment(t *testing.T) {
@@ -466,7 +466,7 @@ func Test_newPacket_ipv6Fragment(t *testing.T) {
// Too short of a fragment packet
err = newPacket(secondFrag[:len(secondFrag)-10], false, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
require.ErrorIs(t, err, iputil.ErrIPv6CouldNotFindPayload)
}
func BenchmarkParseV6(b *testing.B) {
@@ -707,7 +707,7 @@ func Test_newPacket_v6ExtHeaderPastBuffer(t *testing.T) {
pkt[40] = byte(layers.IPProtocolSCTP) // Dest Options next header = SCTP
pkt[41] = 255 // declared length (255+1)*8 = 2048, past the 48 byte buffer
require.ErrorIs(t, newPacket(pkt, true, p), ErrIPv6PacketTooShort)
require.ErrorIs(t, newPacket(pkt, true, p), iputil.ErrIPv6CouldNotFindPayload)
}
// Test_newPacket_v6ExtHeaderConfusion is a regression test for parseV6 walking any unrecognized
+2 -2
View File
@@ -127,7 +127,7 @@ func TestPseudoSumIPv6MatchesReference(t *testing.T) {
func TestIPv4HdrChecksumMatchesReference(t *testing.T) {
rng := rand.New(rand.NewSource(0x1791))
for _, hdrLen := range []int{20, 24, 40, 60} {
for trial := 0; trial < 200; trial++ {
for trial := range 200 {
hdr := make([]byte, hdrLen)
rng.Read(hdr)
hdr[0] = 0x40 | byte(hdrLen/4)
@@ -156,7 +156,7 @@ func TestIPv4HdrChecksumMatchesReference(t *testing.T) {
// the way a receiver does (pseudo-header + L4 must sum to all-ones).
func TestChecksumSeedReceiverAcceptance(t *testing.T) {
rng := rand.New(rand.NewSource(0x1826))
for trial := 0; trial < 200; trial++ {
for trial := range 200 {
src := [4]byte{byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256))}
dst := [4]byte{byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256)), byte(rng.Intn(256))}
payLen := rng.Intn(1500)
+1 -1
View File
@@ -540,7 +540,7 @@ func TestCoalescerCapBySegments(t *testing.T) {
c := newTestTCPCoalescer(t, w)
pay := make([]byte, 512)
seq := uint32(1000)
for i := 0; i < tcpCoalesceMaxSegs+5; i++ {
for range tcpCoalesceMaxSegs + 5 {
if err := c.Commit(buildTCPv4(seq, tcpAck, pay)); err != nil {
t.Fatal(err)
}
+2 -2
View File
@@ -28,7 +28,7 @@ func TestSendBatchReserveCommitFlush(t *testing.T) {
b := NewSendBatch(fw, 4, 32)
ap := netip.MustParseAddrPort("10.0.0.1:4242")
for i := 0; i < 4; i++ {
for i := range 4 {
slot := b.Reserve(32)
if cap(slot) != 32 {
t.Fatalf("slot %d: cap=%d want 32", i, cap(slot))
@@ -72,7 +72,7 @@ func TestSendBatchSlotsDoNotOverlap(t *testing.T) {
b := NewSendBatch(fw, 3, 8)
ap := netip.MustParseAddrPort("10.0.0.1:80")
for i := 0; i < 3; i++ {
for i := range 3 {
s := b.Reserve(8)
pkt := append(s[:0], byte(0xA0+i), byte(0xB0+i))
b.Commit(pkt, ap)
+3 -3
View File
@@ -129,7 +129,7 @@ func TestUDPCoalescerCoalescesEqualSized(t *testing.T) {
w := &fakeTunWriter{gsoEnabled: true}
c := newTestUDPCoalescer(t, w)
pay := make([]byte, 1200)
for i := 0; i < 3; i++ {
for range 3 {
if err := c.Commit(buildUDPv4(1000, 53, pay)); err != nil {
t.Fatal(err)
}
@@ -259,7 +259,7 @@ func TestUDPCoalescerCapsAtMaxSegs(t *testing.T) {
w := &fakeTunWriter{gsoEnabled: true}
c := newTestUDPCoalescer(t, w)
pay := make([]byte, 100)
for i := 0; i < udpCoalesceMaxSegs+5; i++ {
for range udpCoalesceMaxSegs + 5 {
if err := c.Commit(buildUDPv4(1000, 53, pay)); err != nil {
t.Fatal(err)
}
@@ -317,7 +317,7 @@ func TestUDPCoalescerIPv6Coalesces(t *testing.T) {
w := &fakeTunWriter{gsoEnabled: true}
c := newTestUDPCoalescer(t, w)
pay := make([]byte, 1200)
for i := 0; i < 3; i++ {
for range 3 {
if err := c.Commit(buildUDPv6(1000, 53, pay)); err != nil {
t.Fatal(err)
}
+1 -1
View File
@@ -151,7 +151,7 @@ func TestChecksumTailPaths(t *testing.T) {
offsets := []int{0, 1, 3, 7, 15} // mix of aligned and odd starts
for k := 0; k <= maxK; k++ {
for tail := 0; tail < 64; tail++ {
for tail := range 64 {
length := 64*k + tail
for _, seed := range seeds {
for _, off := range offsets {
+57
View File
@@ -0,0 +1,57 @@
//go:build !e2e_testing
package overlay
import (
"fmt"
"log/slog"
"golang.org/x/sys/windows"
"golang.org/x/sys/windows/registry"
)
var tcpipInterfaceKeys = []string{
`SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces`,
`SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\Interfaces`,
}
// disableDNSRegistration prevents publishing overlay addresses in the host's
// DNS records, where clients outside the overlay cannot reach them.
// Failures are non-fatal so registry access cannot prevent tunnel creation.
func disableDNSRegistration(l *slog.Logger, guid windows.GUID) {
id := guid.String()
for _, base := range tcpipInterfaceKeys {
path := base + `\` + id
if err := setDNSRegistrationValues(path); err != nil {
l.Warn("Failed to disable DNS registration on the tun adapter", "error", err, "key", path)
continue
}
l.Debug("Disabled DNS registration on the tun adapter", "key", path)
}
}
func setDNSRegistrationValues(path string) error {
// The stack may not have created the adapter's interface key yet.
k, _, err := registry.CreateKey(registry.LOCAL_MACHINE, path, registry.SET_VALUE)
if err != nil {
return fmt.Errorf("open HKLM\\%s: %w", path, err)
}
defer k.Close()
// Neither RegistrationEnabled nor DisableDynamicUpdate alone reliably
// prevents registration, so all three settings are required.
values := []struct {
name string
data uint32
}{
{"RegistrationEnabled", 0},
{"DisableDynamicUpdate", 1},
{"MaxNumberOfAddressesToRegister", 0},
}
for _, v := range values {
if err := k.SetDWordValue(v.name, v.data); err != nil {
return fmt.Errorf("set %s: %w", v.name, err)
}
}
return nil
}
-1
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
-1
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
+28 -4
View File
@@ -50,7 +50,13 @@ func (r Route) String() string {
func makeRouteTree(l *slog.Logger, routes []Route, allowMTU bool) (*bart.Table[routing.Gateways], error) {
routeTree := new(bart.Table[routing.Gateways])
seen := make(map[netip.Prefix]bool, len(routes))
for _, r := range routes {
if seen[r.Cidr] {
l.Warn("route is listed more than once, only the last entry's via is used", "route", r)
}
seen[r.Cidr] = true
if !allowMTU && r.MTU > 0 {
l.Warn("route MTU is not supported on this platform",
"goos", runtime.GOOS,
@@ -98,7 +104,11 @@ func parseRoutes(c *config.C, networks []netip.Prefix) ([]Route, error) {
mtu, ok := rMtu.(int)
if !ok {
mtu, err = strconv.Atoi(rMtu.(string))
s, ok := rMtu.(string)
if !ok {
return nil, fmt.Errorf("entry %v.mtu in tun.routes is not an integer: %v", i+1, rMtu)
}
mtu, err = strconv.Atoi(s)
if err != nil {
return nil, fmt.Errorf("entry %v.mtu in tun.routes is not an integer: %v", i+1, err)
}
@@ -174,7 +184,11 @@ func parseUnsafeRoutes(c *config.C, networks []netip.Prefix) ([]Route, error) {
if rMtu, ok := m["mtu"]; ok {
mtu, ok = rMtu.(int)
if !ok {
mtu, err = strconv.Atoi(rMtu.(string))
s, ok := rMtu.(string)
if !ok {
return nil, fmt.Errorf("entry %v.mtu in tun.unsafe_routes is not an integer: %v", i+1, rMtu)
}
mtu, err = strconv.Atoi(s)
if err != nil {
return nil, fmt.Errorf("entry %v.mtu in tun.unsafe_routes is not an integer: %v", i+1, err)
}
@@ -192,10 +206,15 @@ func parseUnsafeRoutes(c *config.C, networks []netip.Prefix) ([]Route, error) {
metric, ok := rMetric.(int)
if !ok {
_, err = strconv.ParseInt(rMetric.(string), 10, 32)
s, ok := rMetric.(string)
if !ok {
return nil, fmt.Errorf("entry %v.metric in tun.unsafe_routes is not an integer: %v", i+1, rMetric)
}
n, err := strconv.ParseInt(s, 10, 32)
if err != nil {
return nil, fmt.Errorf("entry %v.metric in tun.unsafe_routes is not an integer: %v", i+1, err)
}
metric = int(n)
}
if metric < 0 || metric > math.MaxInt32 {
@@ -248,10 +267,15 @@ func parseUnsafeRoutes(c *config.C, networks []netip.Prefix) ([]Route, error) {
gatewayWeight, ok := rGatewayWeight.(int)
if !ok {
_, err = strconv.ParseInt(rGatewayWeight.(string), 10, 32)
s, ok := rGatewayWeight.(string)
if !ok {
return nil, fmt.Errorf("entry .weight in tun.unsafe_routes[%v].via[%v] is not an integer", i+1, ig+1)
}
w, err := strconv.ParseInt(s, 10, 32)
if err != nil {
return nil, fmt.Errorf("entry .weight in tun.unsafe_routes[%v].via[%v] is not an integer", i+1, ig+1)
}
gatewayWeight = int(w)
}
if gatewayWeight < 1 || gatewayWeight > math.MaxInt32 {
+116
View File
@@ -1,13 +1,17 @@
package overlay
import (
"bytes"
"fmt"
"net/netip"
"strings"
"testing"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -402,3 +406,115 @@ func Test_makeMultipathUnsafeRouteTree(t *testing.T) {
routing.CalculateBucketsForGateways(expectedGateways)
assert.ElementsMatch(t, expectedGateways, r)
}
func Test_getMTU(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
// unset
assert.Equal(t, DefaultMTU, getMTU(l, c))
// the largest a relayed packet can carry in one underlay buffer
require.Equal(t, udp.MTU, MaxMTU+header.MaxOverhead)
c.Settings["tun"] = map[string]any{"mtu": MaxMTU}
assert.Equal(t, MaxMTU, getMTU(l, c))
// too big is capped
c.Settings["tun"] = map[string]any{"mtu": 9000}
assert.Equal(t, MaxMTU, getMTU(l, c))
}
func Test_getAllRoutesFromConfigCapsMTU(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
c.Settings["tun"] = map[string]any{
"routes": []any{
map[string]any{"mtu": "9000", "route": "10.0.0.0/29"},
map[string]any{"mtu": "1300", "route": "10.0.0.8/29"},
},
"unsafe_routes": []any{
map[string]any{"via": "127.0.0.1", "mtu": "9000", "route": "1.0.0.0/29"},
},
}
changed, routes, err := getAllRoutesFromConfig(l, c, []netip.Prefix{netip.MustParsePrefix("10.0.0.0/24")}, true)
require.NoError(t, err)
assert.True(t, changed)
mtus := map[string]int{}
for _, r := range routes {
mtus[r.Cidr.String()] = r.MTU
}
assert.Equal(t, map[string]int{"10.0.0.0/29": MaxMTU, "10.0.0.8/29": 1300, "1.0.0.0/29": MaxMTU}, mtus)
}
// The route tree keeps one entry per CIDR, so a CIDR listed twice only routes by the last one. Say so instead of
// letting the earlier via, or on Windows the earlier metric, quietly lose
func Test_makeRouteTree_WarnsOnDuplicate(t *testing.T) {
buf := &bytes.Buffer{}
l := test.NewLoggerWithOutput(buf)
via := func(s string) routing.Gateways {
return routing.Gateways{routing.NewGateway(netip.MustParseAddr(s), 1)}
}
cidr := netip.MustParsePrefix("10.251.1.0/24")
_, err := makeRouteTree(l, []Route{
{Cidr: cidr, Via: via("10.250.0.2"), Install: true, Metric: 100},
{Cidr: netip.MustParsePrefix("10.251.2.0/24"), Via: via("10.250.0.2"), Install: true},
}, false)
require.NoError(t, err)
assert.Empty(t, buf.String())
tree, err := makeRouteTree(l, []Route{
{Cidr: cidr, Via: via("10.250.0.2"), Install: true, Metric: 100},
{Cidr: cidr, Via: via("10.250.0.3"), Install: true, Metric: 200},
}, false)
require.NoError(t, err)
assert.Equal(t, 1, strings.Count(buf.String(), "route is listed more than once, only the last entry's via is used"))
gw, ok := tree.Lookup(netip.MustParseAddr("10.251.1.1"))
require.True(t, ok)
assert.Equal(t, netip.MustParseAddr("10.250.0.3"), gw[0].Addr(), "the last entry wins")
}
// A quoted metric or weight is a number written as a string, it means that number. Anything else is an error, not a
// panic
func Test_parseUnsafeRoutes_QuotedNumbers(t *testing.T) {
c := config.NewC(test.NewLogger())
n := netip.MustParsePrefix("10.0.0.1/24")
route := func(metric, weight any) {
c.Settings["tun"] = map[string]any{"unsafe_routes": []any{map[string]any{
"route": "1.0.0.0/8",
"metric": metric,
"via": []any{map[string]any{"gateway": "10.0.0.2", "weight": weight}},
}}}
}
route("100", "5")
routes, err := parseUnsafeRoutes(c, []netip.Prefix{n})
require.NoError(t, err)
require.Len(t, routes, 1)
assert.Equal(t, 100, routes[0].Metric)
assert.Equal(t, "{addr: 10.0.0.2, weight: 5}", routes[0].Via[0].String())
route(1.5, 1)
_, err = parseUnsafeRoutes(c, []netip.Prefix{n})
require.EqualError(t, err, "entry 1.metric in tun.unsafe_routes is not an integer: 1.5")
route(1, true)
_, err = parseUnsafeRoutes(c, []netip.Prefix{n})
require.EqualError(t, err, "entry .weight in tun.unsafe_routes[1].via[1] is not an integer")
}
// An mtu that is neither a number nor a quoted one is an error, not a panic
func Test_parseRoutes_MTUNotANumber(t *testing.T) {
c := config.NewC(test.NewLogger())
n := netip.MustParsePrefix("10.0.0.1/24")
c.Settings["tun"] = map[string]any{"routes": []any{map[string]any{"route": "1.0.0.0/8", "mtu": 1500.5}}}
_, err := parseRoutes(c, []netip.Prefix{n})
require.EqualError(t, err, "entry 1.mtu in tun.routes is not an integer: 1500.5")
c.Settings["tun"] = map[string]any{"unsafe_routes": []any{map[string]any{"route": "1.0.0.0/8", "via": "10.0.0.2", "mtu": 1500.5}}}
_, err = parseUnsafeRoutes(c, []netip.Prefix{n})
require.EqualError(t, err, "entry 1.mtu in tun.unsafe_routes is not an integer: 1500.5")
}
-1
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package tio
-1
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package tio
-1
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package tio
-1
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package tio
-1
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package tio
+4 -7
View File
@@ -1,5 +1,4 @@
//go:build linux && !android && !e2e_testing
// +build linux,!android,!e2e_testing
package tio
@@ -117,17 +116,15 @@ func TestPoll_ConcurrentWrite_NoRace(t *testing.T) {
}()
var wg sync.WaitGroup
for w := 0; w < writers; w++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < perWriter; i++ {
for range writers {
wg.Go(func() {
for range perWriter {
if _, werr := p.Write(payload); werr != nil {
t.Errorf("write: %v", werr)
return
}
}
}()
})
}
wg.Wait()
-1
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package tio
+5 -6
View File
@@ -1,5 +1,4 @@
//go:build linux && !android && !e2e_testing
// +build linux,!android,!e2e_testing
package tio
@@ -137,7 +136,7 @@ func buildTSOv4(t *testing.T, payLen, mss int) ([]byte, virtio.Hdr) {
binary.BigEndian.PutUint16(pkt[34:36], 65535) // window
// payload
for i := 0; i < payLen; i++ {
for i := range payLen {
pkt[ipLen+tcpLen+i] = byte(i & 0xff)
}
return pkt, virtio.NewHeader(
@@ -257,7 +256,7 @@ func TestSegmentTCPv6(t *testing.T) {
pkt[53] = 0x19 // FIN | ACK | PSH — exercise FIN clearing too
binary.BigEndian.PutUint16(pkt[54:56], 65535)
for i := 0; i < payLen; i++ {
for i := range payLen {
pkt[ipLen+tcpLen+i] = byte(i)
}
@@ -361,7 +360,7 @@ func buildUSOv4(t *testing.T, payLen, gsoSize int) ([]byte, virtio.Hdr) {
binary.BigEndian.PutUint16(pkt[22:24], 53) // dport
binary.BigEndian.PutUint16(pkt[24:26], uint16(udpLen+payLen)) // superpacket length
for i := 0; i < payLen; i++ {
for i := range payLen {
pkt[ipLen+udpLen+i] = byte(i & 0xff)
}
@@ -473,7 +472,7 @@ func TestSegmentUDPv6(t *testing.T) {
// Superpacket-wide length, as the kernel supplies it; see buildUSOv4.
binary.BigEndian.PutUint16(pkt[44:46], uint16(udpLen+payLen))
for i := 0; i < payLen; i++ {
for i := range payLen {
pkt[ipLen+udpLen+i] = byte(i)
}
@@ -772,7 +771,7 @@ func buildTSOv6(payLen, gso int) []byte {
pkt[53] = 0x10 // ACK only
binary.BigEndian.PutUint16(pkt[54:56], 65535)
for i := 0; i < payLen; i++ {
for i := range payLen {
pkt[ipLen+tcpLen+i] = byte(i)
}
return pkt
-1
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package virtio
+4 -11
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
// Package virtio implements the pure validation, header-correction, and
// per-segment slicing logic for kernel-supplied TSO/USO superpackets on
@@ -254,12 +253,9 @@ func SegmentTCP(pkt []byte, hdrLenU, csumStartU, gsoSizeU uint16, yield func(seg
var savedHdr [maxSegHdrLen]byte
copy(savedHdr[:headerLen], pkt[:headerLen])
for i := 0; i < numSeg; i++ {
for i := range numSeg {
segStart := i * gsoSize
segEnd := segStart + gsoSize
if segEnd > payLen {
segEnd = payLen
}
segEnd := min(segStart+gsoSize, payLen)
segPayLen := segEnd - segStart
segLen := headerLen + segPayLen
headerOff := i * gsoSize
@@ -359,12 +355,9 @@ func SegmentUDP(pkt []byte, hdrLenU, csumStartU, gsoSizeU uint16, yield func(seg
var savedHdr [maxSegHdrLen]byte
copy(savedHdr[:headerLen], pkt[:headerLen])
for i := 0; i < numSeg; i++ {
for i := range numSeg {
segStart := i * gsoSize
segEnd := segStart + gsoSize
if segEnd > payLen {
segEnd = payLen
}
segEnd := min(segStart+gsoSize, payLen)
segPayLen := segEnd - segStart
segLen := headerLen + segPayLen
headerOff := i * gsoSize
+7 -8
View File
@@ -1,5 +1,4 @@
//go:build linux && !android
// +build linux,!android
package virtio
@@ -58,7 +57,7 @@ func buildTCPv4Super(payLen int) (pkt []byte, hdrLen, csumStart uint16) {
pkt[33] = 0x18 // ACK | PSH
binary.BigEndian.PutUint16(pkt[34:36], 65535) // window
for i := 0; i < payLen; i++ {
for i := range payLen {
pkt[ipLen+tcpLen+i] = byte(i & 0xff)
}
return pkt, ipLen + tcpLen, ipLen
@@ -82,7 +81,7 @@ func buildUDPv4Super(payLen int) (pkt []byte, hdrLen, csumStart uint16) {
binary.BigEndian.PutUint16(pkt[20:22], 12345) // sport
binary.BigEndian.PutUint16(pkt[22:24], 53) // dport
for i := 0; i < payLen; i++ {
for i := range payLen {
pkt[ipLen+udpLen+i] = byte(i & 0xff)
}
return pkt, ipLen + udpLen, ipLen
@@ -188,7 +187,7 @@ func TestSegmentTCPHeaderNotCorrupted(t *testing.T) {
// Payload bytes must be the original contiguous slice.
segPayLen := len(seg) - int(hdrLen)
wantPay := make([]byte, segPayLen)
for k := 0; k < segPayLen; k++ {
for k := range segPayLen {
wantPay[k] = byte((off + k) & 0xff)
}
if !bytes.Equal(seg[hdrLen:], wantPay) {
@@ -317,7 +316,7 @@ func TestSegmentUDPHeaderNotCorrupted(t *testing.T) {
}
wantPay := make([]byte, segPayLen)
for k := 0; k < segPayLen; k++ {
for k := range segPayLen {
wantPay[k] = byte((off + k) & 0xff)
}
if !bytes.Equal(seg[hdrLen:], wantPay) {
@@ -365,7 +364,7 @@ func buildUDPv4Single(payload []byte) (pkt []byte, hdr Hdr) {
// 0xffff, because all-zero is the reserved "no checksum" encoding that IPv6 rejects outright.
func TestFinishChecksumUDPZeroStoresAllOnes(t *testing.T) {
var payload []byte
for i := 0; i < 0x10000; i++ {
for i := range 0x10000 {
p := []byte{byte(i >> 8), byte(i)}
pkt, hdr := buildUDPv4Single(p)
cs, co := int(hdr.CsumStart), int(hdr.CsumOffset)
@@ -544,7 +543,7 @@ func TestBaseSumsMatchZeroingReference(t *testing.T) {
t.Run("ipv4", func(t *testing.T) {
for ihl := ipv4HeaderMinLen; ihl <= ipv4HeaderMaxLen; ihl += 4 {
for iter := 0; iter < 5000; iter++ {
for range 5000 {
pkt := make([]byte, ihl)
for i := range pkt {
pkt[i] = randByte(&state)
@@ -574,7 +573,7 @@ func TestBaseSumsMatchZeroingReference(t *testing.T) {
for dataOff := 5; dataOff <= 15; dataOff++ {
tcpLen := dataOff * 4
headerLen := csumStart + tcpLen
for iter := 0; iter < 5000; iter++ {
for range 5000 {
pkt := make([]byte, headerLen+64)
for i := range pkt {
pkt[i] = randByte(&state)
+26 -4
View File
@@ -7,11 +7,27 @@ import (
"net/netip"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/udp"
"github.com/slackhq/nebula/util"
)
const DefaultMTU = 1300
// MaxMTU is the largest tun.mtu, or route mtu, the underlay can carry.
// Every underlay buffer is udp.MTU bytes, and a relayed packet adds header.MaxOverhead to its plaintext.
const MaxMTU = udp.MTU - header.MaxOverhead
// getMTU reads tun.mtu, capping a value too big for the underlay to carry.
func getMTU(l *slog.Logger, c *config.C) int {
mtu := c.GetInt("tun.mtu", DefaultMTU)
if mtu > MaxMTU {
l.Warn("tun.mtu is too big for the underlay to carry, capping it", "mtu", mtu, "maxMTU", MaxMTU)
return MaxMTU
}
return mtu
}
type NameError struct {
Name string
Underlying error
@@ -27,7 +43,7 @@ type DeviceFactory func(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix,
func NewDeviceFromConfig(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, routines int) (Device, error) {
switch {
case c.GetBool("tun.disabled", false):
tun := newDisabledTun(vpnNetworks, c.GetInt("tun.tx_queue", 500), c.GetBool("stats.message_metrics", false), l)
tun := newDisabledTun(vpnNetworks, c.GetInt("tun.tx_queue", 5000), c.GetBool("stats.message_metrics", false), l)
return tun, nil
default:
@@ -41,7 +57,7 @@ func NewFdDeviceFromConfig(fd *int) DeviceFactory {
}
}
func getAllRoutesFromConfig(c *config.C, vpnNetworks []netip.Prefix, initial bool) (bool, []Route, error) {
func getAllRoutesFromConfig(l *slog.Logger, c *config.C, vpnNetworks []netip.Prefix, initial bool) (bool, []Route, error) {
if !initial && !c.HasChanged("tun.routes") && !c.HasChanged("tun.unsafe_routes") {
return false, nil, nil
}
@@ -57,6 +73,12 @@ func getAllRoutesFromConfig(c *config.C, vpnNetworks []netip.Prefix, initial boo
}
routes = append(routes, unsafeRoutes...)
for i, r := range routes {
if r.MTU > MaxMTU {
l.Warn("route mtu is too big for the underlay to carry, capping it", "route", r.Cidr, "mtu", r.MTU, "maxMTU", MaxMTU)
routes[i].MTU = MaxMTU
}
}
return true, routes, nil
}
@@ -93,14 +115,14 @@ func prefixToMask(prefix netip.Prefix) netip.Addr {
}
func flipBytes(b []byte) []byte {
for i := 0; i < len(b); i++ {
for i := range b {
b[i] ^= 0xFF
}
return b
}
func orBytes(a []byte, b []byte) []byte {
ret := make([]byte, len(a))
for i := 0; i < len(a); i++ {
for i := range a {
ret[i] = a[i] | b[i]
}
return ret
+1 -2
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
@@ -69,7 +68,7 @@ func (t *tun) Activate() error {
}
func (t *tun) reload(c *config.C, initial bool) error {
change, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
-1
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
+63 -6
View File
@@ -1,5 +1,4 @@
//go:build !ios && !e2e_testing
// +build !ios,!e2e_testing
package overlay
@@ -30,7 +29,10 @@ type tun struct {
Routes atomic.Pointer[[]Route]
routeTree atomic.Pointer[bart.Table[routing.Gateways]]
linkAddr *netroute.LinkAddr
l *slog.Logger
// hostOwned means the fd arrived from the OS, which has already configured addressing, mtu
// and routes for it. NEPacketTunnelProvider on darwin does this.
hostOwned bool
l *slog.Logger
}
type ifReq struct {
@@ -124,7 +126,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
f: os.NewFile(uintptr(fd), ""),
Device: name,
vpnNetworks: vpnNetworks,
DefaultMTU: c.GetInt("tun.mtu", DefaultMTU),
DefaultMTU: getMTU(l, c),
l: l,
}
@@ -150,8 +152,48 @@ func (t *tun) deviceBytes() (o [16]byte) {
return
}
func newTunFromFd(_ *config.C, _ *slog.Logger, _ int, _ []netip.Prefix) (*tun, error) {
return nil, fmt.Errorf("newTunFromFd not supported in Darwin")
// newTunFromFd adopts a utun the host already created and configured, which is how a darwin
// network extension is handed its device. Everything about moving packets is shared with newTun,
// only the setup differs: the host owns addressing and routing here.
func newTunFromFd(c *config.C, l *slog.Logger, deviceFd int, vpnNetworks []netip.Prefix) (*tun, error) {
if err := unix.SetNonblock(deviceFd, true); err != nil {
// We own the fd from the moment it is handed to us
_ = unix.Close(deviceFd)
return nil, fmt.Errorf("failed to set the tun fd to non-blocking mode: %w", err)
}
file := os.NewFile(uintptr(deviceFd), "/dev/tun")
t := &tun{
f: file,
Device: utunNameFromFd(deviceFd),
vpnNetworks: vpnNetworks,
DefaultMTU: getMTU(l, c),
hostOwned: true,
l: l,
}
if err := t.reload(c, true); err != nil {
_ = file.Close()
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
if err := t.reload(c, false); err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
}
})
return t, nil
}
// utunNameFromFd asks the socket what interface it is, for logs. A blank name is not worth
// failing a tunnel over, so an error just leaves it empty.
func utunNameFromFd(fd int) string {
name, err := unix.GetsockoptString(fd, unix.AF_SYS_CONTROL, _UTUN_OPT_IFNAME)
if err != nil {
return ""
}
return name
}
func (t *tun) Close() error {
@@ -162,6 +204,12 @@ func (t *tun) Close() error {
}
func (t *tun) Activate() error {
// The host handed us a configured device. Its addresses, mtu and routes come from the network
// settings it applied, and a sandboxed extension cannot change them anyway.
if t.hostOwned {
return nil
}
devName := t.deviceBytes()
s, err := unix.Socket(
@@ -302,7 +350,7 @@ func (t *tun) activate6(network netip.Prefix) error {
}
func (t *tun) reload(c *config.C, initial bool) error {
change, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
@@ -375,6 +423,11 @@ func getLinkAddr(name string) (*netroute.LinkAddr, error) {
}
func (t *tun) addRoutes(logErrors bool) error {
// The route tree is still ours, the system routing table is not
if t.hostOwned {
return nil
}
routes := *t.Routes.Load()
for _, r := range routes {
@@ -404,6 +457,10 @@ func (t *tun) addRoutes(logErrors bool) error {
}
func (t *tun) removeRoutes(routes []Route) error {
if t.hostOwned {
return nil
}
for _, r := range routes {
if !r.Install {
continue
+2 -3
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
@@ -371,7 +370,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
t := &tun{
Device: deviceName,
vpnNetworks: vpnNetworks,
MTU: c.GetInt("tun.mtu", DefaultMTU),
MTU: getMTU(l, c),
l: l,
fd: fd,
shutdownR: shutdownR,
@@ -511,7 +510,7 @@ func (t *tun) setMTU() error {
}
func (t *tun) reload(c *config.C, initial bool) error {
change, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
+1 -2
View File
@@ -1,5 +1,4 @@
//go:build ios && !e2e_testing
// +build ios,!e2e_testing
package overlay
@@ -69,7 +68,7 @@ func (t *tun) Activate() error {
}
func (t *tun) reload(c *config.C, initial bool) error {
change, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
+15 -7
View File
@@ -1,5 +1,4 @@
//go:build !android && !e2e_testing
// +build !android,!e2e_testing
package overlay
@@ -16,12 +15,12 @@ import (
"time"
"unsafe"
"github.com/DefinedNet/netlink"
"github.com/gaissmai/bart"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/overlay/tio"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/util"
"github.com/vishvananda/netlink"
"golang.org/x/sys/unix"
)
@@ -222,7 +221,7 @@ func newTunGeneric(c *config.C, l *slog.Logger, fd int, vnetHdr bool, offloadFla
vnetHdr: vnetHdr,
offloadFlags: offloadFlags,
vpnNetworks: vpnNetworks,
TXQueueLen: c.GetInt("tun.tx_queue", 500),
TXQueueLen: c.GetInt("tun.tx_queue", 5000),
useSystemRoutes: c.GetBool("tun.use_system_route_table", false),
useSystemRoutesBufferSize: c.GetInt("tun.use_system_route_table_buffer_size", 0),
routesFromSystem: map[netip.Prefix]routing.Gateways{},
@@ -245,7 +244,7 @@ func newTunGeneric(c *config.C, l *slog.Logger, fd int, vnetHdr bool, offloadFla
}
func (t *tun) reload(c *config.C, initial bool) error {
routeChange, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
routeChange, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
@@ -254,6 +253,15 @@ func (t *tun) reload(c *config.C, initial bool) error {
return nil
}
// Only the mtu changed. The routes are the same, parse them again so the ones without their own mtu pick up the
// new default. Without this an mtu change would remove them all
if !routeChange {
_, routes, err = getAllRoutesFromConfig(t.l, c, t.vpnNetworks, true)
if err != nil {
return err
}
}
routeTree, err := makeRouteTree(t.l, routes, true)
if err != nil {
return err
@@ -268,14 +276,14 @@ func (t *tun) reload(c *config.C, initial bool) error {
oldDefaultMTU := t.DefaultMTU
oldMaxMTU := t.MaxMTU
newDefaultMTU := c.GetInt("tun.mtu", DefaultMTU)
newDefaultMTU := getMTU(t.l, c)
newMaxMTU := newDefaultMTU
for i, r := range routes {
if r.MTU == 0 {
routes[i].MTU = newDefaultMTU
}
if r.MTU > t.MaxMTU {
if r.MTU > newMaxMTU {
newMaxMTU = r.MTU
}
}
@@ -548,7 +556,7 @@ func (t *tun) setDefaultRoute(cidr netip.Prefix) error {
if err != nil {
t.l.Warn("Failed to set default route MTU, retrying", "error", err, "cidr", cidr)
//retry twice more -- on some systems there appears to be a race condition where if we set routes too soon, netlink says `invalid argument`
for i := 0; i < 2; i++ {
for range 2 {
time.Sleep(100 * time.Millisecond)
err = netlink.RouteReplace(&nr)
if err == nil {
-1
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
+2 -3
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
@@ -100,7 +99,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
fd: fd,
Device: deviceName,
vpnNetworks: vpnNetworks,
MTU: c.GetInt("tun.mtu", DefaultMTU),
MTU: getMTU(l, c),
l: l,
}
@@ -341,7 +340,7 @@ func (t *tun) doIoctlByName(ctl uintptr, value uint32) error {
}
func (t *tun) reload(c *config.C, initial bool) error {
change, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
-1
View File
@@ -1,5 +1,4 @@
//go:build !windows
// +build !windows
package overlay
+2 -3
View File
@@ -1,5 +1,4 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
@@ -91,7 +90,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
fd: fd,
Device: deviceName,
vpnNetworks: vpnNetworks,
MTU: c.GetInt("tun.mtu", DefaultMTU),
MTU: getMTU(l, c),
l: l,
}
@@ -320,7 +319,7 @@ func (t *tun) doIoctlByName(ctl uintptr, value uint32) error {
}
func (t *tun) reload(c *config.C, initial bool) error {
change, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
+56
View File
@@ -0,0 +1,56 @@
//go:build !e2e_testing
package overlay
import (
"net/netip"
"testing"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Only tun.mtu changes. The system calls fail without a real device, what matters is the routes nebula keeps
func TestReloadMTUOnlyKeepsRoutes(t *testing.T) {
routes := " unsafe_routes:\n - route: 192.168.100.0/24\n via: 10.1.0.1\n"
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString("tun:\n mtu: 1300\n"+routes))
tn := &tun{
l: test.NewLogger(),
vpnNetworks: []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")},
ioctlFd: ^uintptr(0),
routesFromSystem: map[netip.Prefix]routing.Gateways{},
}
require.NoError(t, tn.reload(c, true))
require.NoError(t, c.ReloadConfigString("tun:\n mtu: 1400\n"+routes))
_ = tn.reload(c, false)
got := *tn.Routes.Load()
require.Len(t, got, 1)
assert.Equal(t, netip.MustParsePrefix("192.168.100.0/24"), got[0].Cidr)
assert.Equal(t, 1400, got[0].MTU)
assert.Equal(t, 1400, tn.DefaultMTU)
}
// The device mtu has to fit the largest route mtu, wherever it sits in the list
func TestReloadMaxMTUIsTheLargestRoute(t *testing.T) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString("tun:\n mtu: 1300\n unsafe_routes:\n"+
" - route: 192.168.100.0/24\n via: 10.1.0.1\n mtu: 1500\n"+
" - route: 192.168.101.0/24\n via: 10.1.0.1\n mtu: 1400\n"))
tn := &tun{
l: test.NewLogger(),
vpnNetworks: []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")},
ioctlFd: ^uintptr(0),
routesFromSystem: map[netip.Prefix]routing.Gateways{},
}
require.NoError(t, tn.reload(c, true))
assert.Equal(t, 1500, tn.MaxMTU)
assert.Equal(t, 1300, tn.DefaultMTU)
}
+1 -2
View File
@@ -1,5 +1,4 @@
//go:build e2e_testing
// +build e2e_testing
package overlay
@@ -32,7 +31,7 @@ type TestTun struct {
}
func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*TestTun, error) {
_, routes, err := getAllRoutesFromConfig(c, vpnNetworks, true)
_, routes, err := getAllRoutesFromConfig(l, c, vpnNetworks, true)
if err != nil {
return nil, err
}
+109 -7
View File
@@ -1,16 +1,17 @@
//go:build !e2e_testing
// +build !e2e_testing
package overlay
import (
"crypto"
"errors"
"fmt"
"log/slog"
"net/netip"
"os"
"path/filepath"
"runtime"
"slices"
"sync/atomic"
"syscall"
"unsafe"
@@ -75,7 +76,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*w
t := &winTun{
Device: deviceName,
vpnNetworks: vpnNetworks,
MTU: c.GetInt("tun.mtu", DefaultMTU),
MTU: getMTU(l, c),
guid: *guid,
networkCategory: cat,
setCategory: setCat,
@@ -104,6 +105,9 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*w
}
t.tun = tunDevice.(*wintun.NativeTun)
// Disable registration before Activate assigns addresses and triggers it.
disableDNSRegistration(l, *guid)
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
if err != nil {
@@ -115,7 +119,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*w
}
func (t *winTun) reload(c *config.C, initial bool) error {
change, routes, err := getAllRoutesFromConfig(c, t.vpnNetworks, initial)
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
}
@@ -135,7 +139,7 @@ func (t *winTun) reload(c *config.C, initial bool) error {
if !initial {
// Remove first, if the system removes a wanted route hopefully it will be re-added next
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes))
err := t.removeRoutes(findRemovedSystemRoutes(routes, *oldRoutes))
if err != nil {
util.LogWithContextIfNeeded("Failed to remove routes", err, t.l)
}
@@ -182,15 +186,29 @@ func (t *winTun) addRoutes(logErrors bool) error {
luid := winipcfg.LUID(t.tun.LUID())
routes := *t.Routes.Load()
foundDefault4 := false
carriesV6 := slices.ContainsFunc(t.vpnNetworks, func(p netip.Prefix) bool { return p.Addr().Is6() })
for _, r := range routes {
// A CIDR listed twice routes by its last entry, install only that one or each reload would flip between them. The
// same rule as systemRoutes
last := make(map[netip.Prefix]int, len(routes))
for i, r := range routes {
last[r.Cidr] = i
}
for i, r := range routes {
if len(r.Via) == 0 || !r.Install {
// We don't allow route MTUs so only install routes with a via
continue
}
if last[r.Cidr] != i {
continue
}
// A v6 unsafe_route is legal under a v4-only cert; uninstalled ones put nothing on the adapter.
carriesV6 = carriesV6 || r.Cidr.Addr().Is6()
// Add our unsafe route as an on-link route to the nebula tun device.
err := luid.AddRoute(r.Cidr, unspecifiedNextHop(r.Cidr), uint32(r.Metric))
result, err := addRoute(luid, r)
if err != nil {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors {
@@ -200,7 +218,12 @@ func (t *winTun) addRoutes(logErrors bool) error {
return retErr
}
} else {
t.l.Info("Added route", "route", r)
switch result {
case routeAdded:
t.l.Info("Added route", "route", r)
case routeMetricUpdated:
t.l.Info("Updated route metric", "route", r)
}
}
if !foundDefault4 {
@@ -210,6 +233,11 @@ func (t *winTun) addRoutes(logErrors bool) error {
}
}
return t.setMTU(luid, foundDefault4, carriesV6)
}
// setMTU applies tun.mtu per address family. The default route metric rides along on the v4 handle.
func (t *winTun) setMTU(luid winipcfg.LUID, foundDefault4, carriesV6 bool) error {
ipif, err := luid.IPInterface(windows.AF_INET)
if err != nil {
return fmt.Errorf("failed to get ip interface: %w", err)
@@ -224,6 +252,25 @@ func (t *winTun) addRoutes(logErrors bool) error {
if err := ipif.Set(); err != nil {
return fmt.Errorf("failed to set ip interface: %w", err)
}
// Windows tracks NLMTU per family and wintun sets neither, so v6 keeps the adapter default of 65535.
// Gated so a v4-only overlay under 1280 boots; a v6 one deliberately does not, as linux also refuses.
if !carriesV6 {
return nil
}
ipif6, err := luid.IPInterface(windows.AF_INET6)
if err != nil {
// No v6 on the adapter means there is no NLMTU to get wrong. A failed Set below is not the same thing.
t.l.Info("Skipping ipv6 MTU, no ipv6 interface on this adapter", "error", err)
return nil
}
ipif6.NLMTU = uint32(t.MTU)
if err := ipif6.Set(); err != nil {
return fmt.Errorf("failed to set ipv6 interface: %w", err)
}
return nil
}
@@ -288,6 +335,61 @@ func (t *winTun) Close() error {
return t.tun.Close()
}
// systemRoutes maps each CIDR to the route that puts it in the system table. A CIDR's last entry decides, including
// one that says not to install it
func systemRoutes(routes []Route) map[netip.Prefix]Route {
out := make(map[netip.Prefix]Route, len(routes))
for _, r := range routes {
if len(r.Via) > 0 && r.Install {
out[r.Cidr] = r
} else {
delete(out, r.Cidr)
}
}
return out
}
// findRemovedSystemRoutes is the routes to delete from the system table. Windows deletes by destination and next hop,
// so only a CIDR nothing installs anymore goes, a changed metric is set on the route in place by addRoute
func findRemovedSystemRoutes(newRoutes, oldRoutes []Route) []Route {
keep := systemRoutes(newRoutes)
var removed []Route
for _, r := range systemRoutes(oldRoutes) {
if _, ok := keep[r.Cidr]; !ok {
removed = append(removed, r)
}
}
return removed
}
type routeResult int
const (
routeAdded routeResult = iota
routeMetricUpdated
routeAlreadyInstalled
)
// addRoute installs r on the interface. A reload re-adds every route to catch any the system dropped, and Windows
// refuses one it has with the same destination and next hop whatever its metric. One that is there with another
// metric, ours from before a metric change or someone else's, is brought to ours in place
func addRoute(luid winipcfg.LUID, r Route) (routeResult, error) {
nextHop := unspecifiedNextHop(r.Cidr)
err := luid.AddRoute(r.Cidr, nextHop, uint32(r.Metric))
if !errors.Is(err, windows.ERROR_OBJECT_ALREADY_EXISTS) {
return routeAdded, err
}
row, err := luid.Route(r.Cidr, nextHop)
if err != nil {
return routeAlreadyInstalled, err
}
if row.Metric != uint32(r.Metric) {
row.Metric = uint32(r.Metric)
return routeMetricUpdated, row.Set()
}
return routeAlreadyInstalled, nil
}
func unspecifiedNextHop(p netip.Prefix) netip.Addr {
if p.Addr().Is4() {
return netip.IPv4Unspecified()
+168
View File
@@ -0,0 +1,168 @@
//go:build windows && !e2e_testing
package overlay
import (
"bytes"
"fmt"
"net/netip"
"strings"
"testing"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/sys/windows"
"golang.zx2c4.com/wireguard/windows/tunnel/winipcfg"
)
type routeTest struct {
c *config.C
tun *winTun
luid winipcfg.LUID
log *bytes.Buffer
}
// newRouteTest makes a wintun device of its own, before installing tun.unsafe_routes from cfg
func newRouteTest(t *testing.T, cfg string) *routeTest {
t.Helper()
if !windows.GetCurrentProcessToken().IsElevated() {
t.Skip("creating a wintun device needs an elevated process")
}
if err := checkWinTunExists(); err != nil {
t.Skipf("wintun.dll is not where nebula looks for it: %v", err)
}
rt := &routeTest{log: &bytes.Buffer{}}
l := test.NewLoggerWithOutput(rt.log)
rt.c = config.NewC(l)
require.NoError(t, rt.c.LoadString(cfg))
var err error
rt.tun, err = newTun(rt.c, l, []netip.Prefix{netip.MustParsePrefix("10.250.0.1/24")}, false)
require.NoError(t, err)
t.Cleanup(func() { _ = rt.tun.Close() })
rt.luid = winipcfg.LUID(rt.tun.tun.LUID())
return rt
}
func (rt *routeTest) metric(t *testing.T, cidr string) uint32 {
t.Helper()
p := netip.MustParsePrefix(cidr)
row, err := rt.luid.Route(p, unspecifiedNextHop(p))
require.NoError(t, err)
return row.Metric
}
func unsafeRoutes(routes ...string) string {
var b strings.Builder
b.WriteString("tun:\n dev: nebroutetest\n unsafe_routes:\n")
// cidr[@metric][!], the ! for install: false
for _, r := range routes {
r, noInstall := strings.CutSuffix(r, "!")
cidr, metric, _ := strings.Cut(r, "@")
fmt.Fprintf(&b, " - route: %s\n via: 10.250.0.2\n", cidr)
if metric != "" {
fmt.Fprintf(&b, " metric: %s\n", metric)
}
if noInstall {
b.WriteString(" install: false\n")
}
}
return b.String()
}
// A reload that adds a route re-adds the ones already installed, Windows refusing those as already there isn't a failure
func TestWinTun_ReloadKeepsInstalledRoutes(t *testing.T) {
forFamilies(t, func(t *testing.T, a, b string) {
rt := newRouteTest(t, unsafeRoutes(a))
require.NoError(t, rt.tun.Activate())
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a, b)))
assert.NotContains(t, rt.log.String(), "Failed to add route")
rt.metric(t, a)
rt.metric(t, b)
})
}
// Windows calls a route the same by destination and next hop, so a changed metric is set on the installed route in
// place rather than deleted and added, which would leave a moment with no route
func TestWinTun_ReloadMetricChangeApplies(t *testing.T) {
forFamilies(t, func(t *testing.T, a, b string) {
rt := newRouteTest(t, unsafeRoutes(a+"@100"))
require.NoError(t, rt.tun.Activate())
require.Equal(t, uint32(100), rt.metric(t, a))
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a+"@200")))
assert.NotContains(t, rt.log.String(), "Failed to add route")
assert.NotContains(t, rt.log.String(), "Removed route")
assert.Equal(t, uint32(200), rt.metric(t, a))
})
}
// A route someone else put on the device with another metric is brought to ours, at startup too
func TestWinTun_ForeignRouteGetsOurMetric(t *testing.T) {
forFamilies(t, func(t *testing.T, a, b string) {
rt := newRouteTest(t, unsafeRoutes(a+"@100"))
p := netip.MustParsePrefix(a)
require.NoError(t, rt.luid.AddRoute(p, unspecifiedNextHop(p), 50))
require.NoError(t, rt.tun.Activate())
assert.Equal(t, uint32(100), rt.metric(t, a))
assert.Contains(t, rt.log.String(), "level=INFO msg=\"Updated route metric\"")
})
}
// A CIDR listed twice is installed once, by its last entry, and reloads leave it there instead of flipping between the
// two metrics
func TestWinTun_DuplicateCidrInstallsTheLastEntry(t *testing.T) {
forFamilies(t, func(t *testing.T, a, b string) {
rt := newRouteTest(t, unsafeRoutes(a+"@100", a+"@200"))
require.NoError(t, rt.tun.Activate())
assert.Equal(t, uint32(200), rt.metric(t, a))
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a+"@100", a+"@200", b)))
assert.Equal(t, uint32(200), rt.metric(t, a))
assert.NotContains(t, rt.log.String(), "Updated route metric")
assert.NotContains(t, rt.log.String(), "Failed to add route")
})
}
// Dropping the duplicate that wasn't installed leaves the installed route alone. Windows deletes by destination and
// next hop, deleting the dropped one would take the live route with it
func TestWinTun_DroppingAnUninstalledDuplicateKeepsTheRoute(t *testing.T) {
forFamilies(t, func(t *testing.T, a, b string) {
rt := newRouteTest(t, unsafeRoutes(a+"@100", a+"@200"))
require.NoError(t, rt.tun.Activate())
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a+"@200")))
assert.NotContains(t, rt.log.String(), "Removed route")
assert.Equal(t, uint32(200), rt.metric(t, a))
})
}
// A reload ends where a fresh start with the same config would, a CIDR whose last entry says not to install it is
// removed
func TestWinTun_ReloadMatchesAFreshStart(t *testing.T) {
forFamilies(t, func(t *testing.T, a, b string) {
cidr := netip.MustParsePrefix(a)
routes := unsafeRoutes(a+"@100", a+"@100!")
fresh := newRouteTest(t, routes)
require.NoError(t, fresh.tun.Activate())
_, err := fresh.luid.Route(cidr, unspecifiedNextHop(cidr))
require.Error(t, err, "a fresh start installed it")
require.NoError(t, fresh.tun.Close())
rt := newRouteTest(t, unsafeRoutes(a+"@100"))
require.NoError(t, rt.tun.Activate())
require.NoError(t, rt.c.ReloadConfigString(routes))
_, err = rt.luid.Route(cidr, unspecifiedNextHop(cidr))
assert.Error(t, err, "the reload left it installed")
})
}
// forFamilies runs f with two unsafe_routes CIDRs of each address family, the tun's own network stays IPv4
func forFamilies(t *testing.T, f func(t *testing.T, a, b string)) {
t.Run("ipv4", func(t *testing.T) { f(t, "10.251.1.0/24", "10.251.2.0/24") })
t.Run("ipv6", func(t *testing.T) { f(t, "fd00:251:1::/64", "fd00:251:2::/64") })
}
Loaded 100 of 126 files, more files were not shown because too many files have changed in this diff. Show more