Compare commits

..
Author SHA1 Message Date
Caleb Jasik 4dbeff889c v1.11.2 changelog (#1888) 2026-09-21 18:19:35 -05:00
Wade SimmonsandJack Doan 673f59b2b2 smoke: Prove windows honors NlMtu with a 9000 byte ping (#1873) (#1884)
(cherry picked from commit 019cc46a7f)

Co-authored-by: Jack Doan <me@jackdoan.com>
2026-09-21 18:01:28 -04:00
Jack Doan 3871faf26e Advertise underlay addresses on Android (#1881) (#1887)
(cherry picked from commit 3a45c01656)
2026-09-21 16:29:12 -05:00
Wade SimmonsandCaleb Jasik b66c9bb362 backport v1.11: Recompute the transport checksum on self-forwarded packets (#1880)
* Recompute the transport checksum on self-forwarded packets (#1862)

(cherry picked from commit ec3304e3a9)

* try to backport

* fix miss backport

* backport anyFragment logic

* restore test

---------

Co-authored-by: Caleb Jasik <calebjasik@jasik.xyz>
2026-09-15 18:27:43 -04:00
Wade SimmonsandNate Brown 501fda7a40 windows: Fix NLMTU for ipv6 (#1871) (#1878)
(cherry picked from commit 89178f45ba)

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-09-14 16:17:20 -04:00
Wade Simmons 992e36b678 v1.11.1 (#1854)
Prepare the v1.11.1 CHANGELOG updates
2026-08-21 15:23:25 -04:00
Nate Brown 5d7c8c905b release: retry the STS assume until the secret key survives Windows (#1856) 2026-08-21 11:03:53 -05:00
Wade Simmonsandsn0w c2ad75e459 cmd/ca: prevent out-of-memory on 32bit systems (#1834) (#1852)
(cherry picked from commit 6fcb926334)

Co-authored-by: sn0w <me@sn0w.re>
2026-08-20 16:52:50 -04:00
Wade SimmonsandNate Brown 0a916b1d4b parseV6: only walk real ipv6 extension headers, fail closed on unknown protocols (#1840) (#1853)
(cherry picked from commit 49e35d1283)

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-08-20 16:52:36 -04:00
Wade SimmonsandNate Brown 33abaf95cd Enforce outbound message counter limits to prevent nonce reuse (#1841) (#1850)
(cherry picked from commit f15d10fc54)

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-08-20 14:24:11 -04:00
20c68ef3c4 Drop dependency on github.com/cyberdelia/go-metrics-graphite (#1832) (#1849)
The go-metrics-graphite package has been unmaintained for 10+ years,
which is a packaging and supply-chain concern for downstream
distributors (see #1831). Nebula only used its Config struct and Once()
entrypoint, so inline just those (~65 lines) into a local graphite.go,
preserving the upstream BSD-2-Clause copyright notice, and remove the
dependency.

Fixes #1831


(cherry picked from commit 28d82f7b8a)

Co-authored-by: Jay R. Wren <jay.wren@slack-corp.com>
Co-authored-by: Claude <svc-devxp-claude@slack-corp.com>
2026-08-20 14:23:45 -04:00
Wade SimmonsandNate Brown f46abae87e Take Apple signing credentials from Secrets Manager (#1833) (#1848)
(cherry picked from commit e7b4e094b9)

Co-authored-by: Nate Brown <nbrown.us@gmail.com>
2026-08-20 07:52:12 -04:00
Wade SimmonsandJohn Maguire ae688d038d Tolerate ErrDumpInterrupted when listing tun addresses (#1842)
Backport from #1835 to release 1.11

    ## What

    Since the bump to netlink v1.3.0, `netlink.AddrList` returns `ErrDumpInterrupted` ("results may be incomplete or inconsistent") when the kernel sets `NLM_F_DUMP_INTR` on an address dump. `addIPs` treats any error from `AddrList` as fatal, so a transient interrupted dump aborts startup:

    ```
    failed to start nebula: failed to get tun address list: results may be incomplete or inconsistent
    ```

    Before v1.3.0, the pinned netlink release had no `NLM_F_DUMP_INTR` handling at all and silently returned partial results, so this condition was previously invisible.

    ## Root cause

    The dump is racing nebula's own setup. `RTM_GETADDR` dumps addresses for the whole system, and the kernel flags the dump as interrupted if any address changes between recvmsg batches. When the cert contains an IPv6 network, the `AddrReplace` a few lines above returns while the new address is still tentative; the kernel's DAD worker flips it to preferred asynchronously ~100-400us later, and that flip lands inside the immediately-following dump. The race is entirely self-contained - no concurrent restart or interface churn is required, and the hit rate scales with how many addresses are on the host (more dump batches = wider window).

    Reproduced with a standalone tool replaying this exact netlink sequence on a host with ~80 addresses: an IPv6 /80 alone interrupts 10/200 cycles and v4+v6 together 16/200, while IPv4-only runs 0/800. `ip -ts monitor addr` shows the tentative add and the async flag-clear as two events during the dump window. Hosts with DAD disabled (`net.ipv6.conf.all.accept_dad=0`, e.g. LXC containers) never reproduce, confirming the flip is the trigger. Observed in the wild as dnclient package upgrades reliably failing their post-install restart (systemd start timeout, package left half-configured).

    ## How

    Treat `ErrDumpInterrupted` as success with a warning. netlink still returns the partial result set alongside the error, and this call site only uses the list to prune addresses that are not in the certs; a missed stale address survives until the next reload, which is strictly better than failing startup. All cert addresses were already applied via `AddrReplace` above.

Co-authored-by: John Maguire <john@defined.net>
2026-08-19 11:35:06 -04:00
47 changed files with 2105 additions and 526 deletions

No files matched your search

+9 -2
View File
@@ -43,8 +43,15 @@ runs:
with:
role-to-assume: ${{ inputs.role }}
aws-region: ${{ inputs.region }}
# Default is 12 retries to ride out IAM trust-policy propagation; once
# the role is stable we want a real misconfiguration to fail fast.
# An STS secret key with special characters does not survive the
# pwsh -> make -> MSYS sh -> aws.exe chain, and SigV4 then signs with a
# key that no longer matches, so the first S3 upload fails with
# SignatureDoesNotMatch. Retries the assume until it comes back clean.
# Same fix as DefinedNet/dnclient#867.
special-characters-workaround: true
# Overridden by the workaround above and kept for whenever that goes:
# the default 12 rides out IAM trust-policy propagation, and once the
# role is stable a real misconfiguration should fail fast.
retry-max-attempts: 5
- name: Sign .exe files
+82 -11
View File
@@ -73,8 +73,11 @@ jobs:
build-darwin:
name: Build Universal Darwin
env:
HAS_SIGNING_CREDS: ${{ secrets.AC_USERNAME != '' }}
HAS_SIGNING_CREDS: ${{ secrets.APPLE_SIGNING_ROLE_ARN != '' }}
runs-on: macos-latest
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v7
@@ -83,17 +86,68 @@ jobs:
go-version: '1.26'
check-latest: true
# GitHub holds ARNs, not credentials, and ARNs outlive a rotation
- name: Configure AWS credentials
if: env.HAS_SIGNING_CREDS == 'true'
uses: aws-actions/configure-aws-credentials@v6
with:
role-to-assume: ${{ secrets.APPLE_SIGNING_ROLE_ARN }}
aws-region: us-east-2
# parse-json-secrets unpacks into SIGNING_* and ASC_*, masked on the way in
- name: Fetch signing credentials
if: env.HAS_SIGNING_CREDS == 'true'
uses: aws-actions/aws-secretsmanager-get-secrets@v3
with:
parse-json-secrets: true
secret-ids: |
SIGNING,${{ secrets.APPLE_SIGNING_DEVELOPER_ID_ARN }}
ASC,${{ secrets.APPLE_NOTARY_KEY_ARN }}
- name: Import certificates
if: env.HAS_SIGNING_CREDS == 'true'
uses: Apple-Actions/import-codesign-certs@v7
with:
p12-file-base64: ${{ secrets.APPLE_DEVELOPER_CERTIFICATE_P12_BASE64 }}
p12-password: ${{ secrets.APPLE_DEVELOPER_CERTIFICATE_PASSWORD }}
p12-file-base64: ${{ env.SIGNING_P12_BASE64 }}
p12-password: ${{ env.SIGNING_PASSWORD }}
# The action imports but does not check the chain validates, which is how a p12
# missing its intermediate reaches a failing codesign
- name: Check the identity is usable
if: env.HAS_SIGNING_CREDS == 'true'
run: |
: "${SIGNING_IDENTITY_SHA1:?empty, so the secret has no identity_sha1}"
identities=$(security find-identity -v -p codesigning signing_temp.keychain)
case "$identities" in
*"$SIGNING_IDENTITY_SHA1"*) ;;
*) printf '%s\n' "$identities" >&2; exit 1 ;;
esac
# notarytool wants the key as a file
- name: Write the App Store Connect key
if: env.HAS_SIGNING_CREDS == 'true'
run: |
mkdir -p ~/private_keys
chmod 700 ~/private_keys
key_path="$HOME/private_keys/AuthKey_${ASC_KEY_ID}.p8"
(umask 077; printf '%s\n' "$ASC_PRIVATE_KEY" > "$key_path")
echo "ASC_P8=$key_path" >> "$GITHUB_ENV"
- name: Drop the credentials from the environment
if: env.HAS_SIGNING_CREDS == 'true'
run: |
# The action's own inventory, so a new field in a secret is covered
python3 -c '
import json, os
raw = os.environ.get("SECRETS_LIST_CLEAN_UP")
if raw is None and os.environ.get("SIGNING_P12_BASE64"):
raise SystemExit("SECRETS_LIST_CLEAN_UP is gone, fetched secrets are not being scrubbed")
keep = {"SIGNING_IDENTITY_SHA1", "ASC_KEY_ID", "ASC_ISSUER_ID"}
names = [n for n in json.loads(raw or "[]") if n not in keep]
print("\n".join(f"{n}=" for n in dict.fromkeys(names)))
' >> "$GITHUB_ENV"
- name: Build, sign, and notarize
env:
AC_USERNAME: ${{ secrets.AC_USERNAME }}
AC_PASSWORD: ${{ secrets.AC_PASSWORD }}
run: |
rm -rf release
mkdir release
@@ -102,17 +156,34 @@ jobs:
lipo -create -output ./release/nebula ./build/darwin-amd64/nebula ./build/darwin-arm64/nebula
lipo -create -output ./release/nebula-cert ./build/darwin-amd64/nebula-cert ./build/darwin-arm64/nebula-cert
if [ -n "$AC_USERNAME" ]; then
codesign -s "10BC1FDDEB6CE753550156C0669109FAC49E4D1E" -f -v --timestamp --options=runtime -i "net.defined.nebula" ./release/nebula
codesign -s "10BC1FDDEB6CE753550156C0669109FAC49E4D1E" -f -v --timestamp --options=runtime -i "net.defined.nebula-cert" ./release/nebula-cert
# Unset in a fork, which has no credentials to sign with
if [ -n "$SIGNING_IDENTITY_SHA1" ]; then
codesign -s "$SIGNING_IDENTITY_SHA1" -f -v --timestamp --options=runtime -i "net.defined.nebula" ./release/nebula
codesign -s "$SIGNING_IDENTITY_SHA1" -f -v --timestamp --options=runtime -i "net.defined.nebula-cert" ./release/nebula-cert
fi
zip -j release/nebula-darwin.zip release/nebula-cert release/nebula
if [ -n "$AC_USERNAME" ]; then
xcrun notarytool submit ./release/nebula-darwin.zip --team-id "576H3XS7FP" --apple-id "$AC_USERNAME" --password "$AC_PASSWORD" --wait
if [ -n "$ASC_P8" ]; then
xcrun notarytool submit ./release/nebula-darwin.zip --key "$ASC_P8" --key-id "$ASC_KEY_ID" --issuer "$ASC_ISSUER_ID" --wait
fi
- name: Drop the signing key
if: always() && env.HAS_SIGNING_CREDS == 'true'
run: |
# Locked, not deleted: import-codesign-certs deletes it in its own post
# step and fails the job if it is already gone. Locked is unusable.
security lock-keychain signing_temp.keychain || true
rm -f "$ASC_P8"
# Nothing later in this job needs AWS
python3 -c '
import json, os
names = json.loads(os.environ.get("SECRETS_LIST_CLEAN_UP") or "[]")
names += ["ASC_P8", "SIGNING_IDENTITY_SHA1", "ASC_KEY_ID", "ASC_ISSUER_ID",
"AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN"]
print("\n".join(f"{n}=" for n in dict.fromkeys(names)))
' >> "$GITHUB_ENV"
- name: Upload artifacts
uses: actions/upload-artifact@v7
with:
+21
View File
@@ -61,3 +61,24 @@ jobs:
run: NAME="smoke-p256" ./smoke.sh
timeout-minutes: 10
smoke-self:
name: Run self traffic smoke test on macOS
runs-on: macos-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-go@v7
with:
go-version: '1.26'
check-latest: true
- name: build
run: make bin
- name: run smoke-self
working-directory: ./.github/workflows/smoke
run: ./smoke-self.sh
timeout-minutes: 10
+130
View File
@@ -0,0 +1,130 @@
#!/bin/bash
# A host must be able to reach its own overlay address. Where the kernel sends
# that traffic through the tun rather than over loopback, nebula sees it and
# hands it straight back (immediatelyForwardToSelf), and whether the kernel
# accepts what comes back is only answerable against a real kernel. Runs one
# nebula on this machine as root and aims every probe at its own address.
set -e -x
set -o pipefail
V4=192.0.2.1
V6=2001:db8::1
case "$(uname -s)" in
Darwin) TUN_DEV=utun ;;
*) TUN_DEV=tun0 ;;
esac
ROOT="$(cd ../../.. && pwd)"
rm -rf build/self
mkdir -p build/self
cd build/self
cleanup() {
echo
echo " *** cleanup"
echo
set +e
if [ -n "$NEBULA_PID" ]
then
sudo kill "$NEBULA_PID"
fi
{ kill $(jobs -p); wait; } 2>/dev/null
sed 's/^/ [self] /' nebula.log
}
trap cleanup EXIT
# perl is on every platform this runs on; timeout(1) is not.
alarm() {
perl -e 'alarm shift; exec @ARGV' "$@"
}
RESULTS=""
FAILED=""
probe() {
local name="$1"
shift
if "$@"
then
RESULTS="$RESULTS $name=ok"
else
RESULTS="$RESULTS $name=FAIL"
FAILED="$FAILED $name"
fi
}
# Send one datagram, then wait for the listener to have written it out.
udp_probe() {
echo self | alarm 5 nc -u -w1 "$1" 3000 || true
set +x
for _ in $(seq 1 20)
do
if grep -q self "$2"
then
set -x
return 0
fi
sleep 0.25
done
set -x
return 1
}
"$ROOT/nebula-cert" ca -name "Smoke Test"
"$ROOT/nebula-cert" sign -name self -networks "$V4/24,$V6/64"
HOST=self AM_LIGHTHOUSE=true TUN_DEV="$TUN_DEV" ../../genconfig.sh >self.yml
"$ROOT/nebula" -config self.yml -test
sudo -v
sudo "$ROOT/nebula" -config self.yml >nebula.log 2>&1 &
NEBULA_PID=$!
for _ in $(seq 1 40)
do
ifconfig | grep "inet6 $V6 " >/dev/null && break
sleep 0.25
done
ifconfig | grep "inet $V4 "
ifconfig | grep "inet6 $V6 "
nc -l "$V4" 2000 >/dev/null &
nc -l "$V6" 2000 >/dev/null &
nc -u -l "$V4" 3000 >udp4.txt &
nc -u -l "$V6" 3000 >udp6.txt &
sleep 1
set +x
echo
echo " *** Testing self traffic from $V4"
echo
set -x
probe icmp4 alarm 5 ping -c1 "$V4"
probe tcp4 alarm 5 nc -z "$V4" 2000
probe udp4 udp_probe "$V4" udp4.txt
set +x
echo
echo " *** Testing self traffic from $V6"
echo
set -x
probe icmp6 alarm 5 ping6 -c1 "$V6"
probe tcp6 alarm 5 nc -z "$V6" 2000
probe udp6 udp_probe "$V6" udp6.txt
set +x
echo
echo " *** self traffic:$RESULTS"
echo
if [ -n "$FAILED" ]
then
echo "self traffic failed:$FAILED" >&2
exit 1
fi
+48 -5
View File
@@ -51,15 +51,19 @@ wsl -d $Distro -- bash -c "rm -rf $WslDir && mkdir -p $WslDir" | Out-Null
$DevName = 'nebula-smoke'
$Ip1 = '192.168.241.1'
$Ip2 = '192.168.241.2'
# Dual stack on purpose: a v4-only overlay never exercises the v6 side of tun.mtu.
$Ip6_1 = 'fd42:4242:241::1'
$Ip6_2 = 'fd42:4242:241::2'
$Mtu = 1300
$Port = 4242
& $NebulaCert ca -name 'smoke-ca' -out-crt "$WorkDir\ca.crt" -out-key "$WorkDir\ca.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert ca failed (exit $LASTEXITCODE)" }
& $NebulaCert sign -name 'lighthouse' -networks "$Ip1/24" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\lighthouse.crt" -out-key "$WorkDir\lighthouse.key"
& $NebulaCert sign -name 'lighthouse' -networks "$Ip1/24,$Ip6_1/64" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\lighthouse.crt" -out-key "$WorkDir\lighthouse.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert sign lighthouse failed (exit $LASTEXITCODE)" }
& $NebulaCert sign -name 'peer' -networks "$Ip2/24" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\peer.crt" -out-key "$WorkDir\peer.key"
& $NebulaCert sign -name 'peer' -networks "$Ip2/24,$Ip6_2/64" -ca-crt "$WorkDir\ca.crt" -ca-key "$WorkDir\ca.key" -out-crt "$WorkDir\peer.crt" -out-key "$WorkDir\peer.key"
if ($LASTEXITCODE -ne 0) { throw "nebula-cert sign peer failed (exit $LASTEXITCODE)" }
# Windows lighthouse config.
@@ -82,7 +86,7 @@ tun:
drop_local_broadcast: false
drop_multicast: false
tx_queue: 500
mtu: 1300
mtu: $Mtu
network_category: private
logging:
level: info
@@ -126,7 +130,7 @@ tun:
drop_local_broadcast: false
drop_multicast: false
tx_queue: 500
mtu: 1300
mtu: $Mtu
logging:
level: info
format: text
@@ -169,7 +173,7 @@ Write-Host '=== WSL diagnostic ==='
wsl --version 2>&1 | Out-Host
wsl --list --verbose 2>&1 | Out-Host
wsl -d $Distro -u root -- uname -a | Out-Host
wsl -d $Distro -u root -- bash -c "modprobe tun 2>&1 || true; mkdir -p /dev/net; [ -c /dev/net/tun ] || mknod /dev/net/tun c 10 200; chmod 600 /dev/net/tun; ls -l /dev/net/tun"
wsl -d $Distro -u root -- bash -c "modprobe tun 2>&1 || true; mkdir -p /dev/net; [ -c /dev/net/tun ] || mknod /dev/net/tun c 10 200; chmod 600 /dev/net/tun; { echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6; echo 0 > /proc/sys/net/ipv6/conf/default/disable_ipv6; } 2>/dev/null || true; ls -l /dev/net/tun"
if ($LASTEXITCODE -ne 0) { throw "failed to prepare /dev/net/tun in WSL (TUN support missing?)" }
# Deliberately no New-NetFirewallRule calls here -- nebula's windows_bypass_wdf
@@ -214,6 +218,16 @@ try {
}
Write-Host "OK: $DevName NetworkCategory=Private"
# v6 silently kept the adapter default of 65535 while v4 was correct.
foreach ($family in @('IPv4', 'IPv6')) {
Wait-Until -TimeoutSec 30 -What "$DevName $family NlMtu=$Mtu" -Predicate {
if ($lhProc.HasExited) { throw "lighthouse exited (code $($lhProc.ExitCode)) before $family mtu was set" }
$rows = @(Get-NetIPInterface -InterfaceAlias $DevName -AddressFamily $family -ErrorAction SilentlyContinue)
$rows.Count -gt 0 -and -not ($rows | Where-Object { $_.NlMtu -ne $Mtu })
}
Write-Host "OK: $DevName $family NlMtu=$Mtu"
}
Wait-Until -TimeoutSec 30 -What "WSL nebula1 with $Ip2" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before tun was ready" }
$r = wsl -d $Distro -u root -- bash -c "ip -o addr show nebula1 2>/dev/null | grep -q 'inet $Ip2' && echo yes"
@@ -221,6 +235,13 @@ try {
}
Write-Host "OK: WSL nebula1 has $Ip2"
Wait-Until -TimeoutSec 30 -What "WSL nebula1 with $Ip6_2" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the v6 address was up" }
$r = wsl -d $Distro -u root -- bash -c "ip -o addr show nebula1 2>/dev/null | grep -q 'inet6 $Ip6_2' && echo yes"
("$r").Trim() -eq 'yes'
}
Write-Host "OK: WSL nebula1 has $Ip6_2"
Wait-Until -TimeoutSec 30 -What "ping from WSL peer to windows lighthouse ($Ip1)" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before ping succeeded" }
$r = wsl -d $Distro -u root -- bash -c "ping -c1 -W1 $Ip1 >/dev/null 2>&1 && echo OK"
@@ -234,6 +255,28 @@ try {
}
Write-Host "OK: windows lighthouse -> WSL peer"
# Otherwise the v6 networks only prove the interface exists, not that it forwards.
Wait-Until -TimeoutSec 30 -What "v6 ping from WSL peer to windows lighthouse ($Ip6_1)" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the v6 ping succeeded" }
$r = wsl -d $Distro -u root -- bash -c "ping -6 -c1 -W1 $Ip6_1 >/dev/null 2>&1 && echo OK"
("$r").Trim() -eq 'OK'
}
Write-Host "OK: WSL peer -> windows lighthouse over v6"
# The NlMtu check above proves the value was written, not that windows honors it. A payload this size only
# crosses the tunnel as fragments cut at NlMtu on the way out of the adapter. Left at the adapter default it
# reaches nebula whole, overflows the udp.MTU (9001) write buffer and is dropped, so the ping never answers.
# Anything smaller would ride out as one oversized datagram and survive on IP fragmentation of the underlay.
$BigPayload = 9000
foreach ($target in @(@{ Family = 'v4'; Ip = $Ip2 }, @{ Family = 'v6'; Ip = $Ip6_2 })) {
Wait-Until -TimeoutSec 30 -What "$($target.Family) ping with a $BigPayload byte payload from windows lighthouse to WSL peer ($($target.Ip))" -Predicate {
if ($peerProc.HasExited) { throw "peer exited (code $($peerProc.ExitCode)) before the large $($target.Family) ping succeeded" }
$null = & ping.exe -n 1 -w 1000 -l $BigPayload $target.Ip
$LASTEXITCODE -eq 0
}
Write-Host "OK: windows lighthouse -> WSL peer, $BigPayload byte $($target.Family) payload"
}
Write-Host ''
Write-Host 'All smoke checks passed.'
}
+46 -1
View File
@@ -7,6 +7,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [1.11.2] - 2026-09-21
See the [v1.11.2](https://github.com/slackhq/nebula/milestone/31?closed=1) milestone for a complete list of changes.
### Fixed
- Windows tracks NLMTU per address family and Nebula only ever set it for `AF_INET`, so the IPv6 MTU on every
Windows adapter sat at the adapter default of 65535 regardless of `tun.mtu` and the stack could hand Nebula
packets far larger than configured. An IPv6 enabled overlay with `tun.mtu` under 1280 now refuses to start,
matching the other platforms. (#1871)
- Android 11+ denies `bind()` on netlink route sockets inside the app sandbox, so local address discovery
failed and an Android node advertised no underlay addresses, leaving peers on the same LAN to reach it only
at the address a lighthouse observed. (#1881)
- Recompute the transport checksum on packets a host sends to its own overlay address. On macOS the kernel
leaves the checksum unfinished for hardware offload, which does not survive the trip through userspace, so
TCP and UDP to a host's own IPv6 overlay address were dropped as corrupt. (#1862)
## [1.11.1] - 2026-08-21
See the [v1.11.1](https://github.com/slackhq/nebula/milestone/30?closed=1) milestone for a complete list of changes.
### Changed
- IPv6 packets whose next header is a protocol Nebula does not parse (SCTP, GRE, IP-in-IP, etc.) are now
classified as that protocol with no ports, closing a firewall bypass where a crafted payload could steer
the classifier into reading one as TCP/UDP and matching a TCP/UDP rule. These packets are now matched as
their true protocol, so only a `proto: any` rule allows them. If you carry one of these protocols over the
overlay, confirm a `proto: any` rule covers it before upgrading, it may have been passing only through this
bypass. (#1840)
- Drop the dependency on `github.com/cyberdelia/go-metrics-graphite`, which has been unmaintained for over ten
years, by inlining the small amount of code Nebula used. (#1832)
### Fixed
- The ICMPv6 type was read from the wrong byte when classifying IPv6 packets, so the echo identifier used
for conntrack was never picked up. (#1840)
- Enforce outbound message counter limits so a tunnel is rehandshaked before the counter can wrap, preventing
nonce reuse. This is unreachable in practice, but is enforced as a defense-in-depth measure. (#1841)
- Prevent `nebula-cert ca` from running out of memory on 32bit systems when generating encrypted private keys. (#1834)
- Tolerate `ErrDumpInterrupted` when listing tun addresses on Linux, so a transient interrupted netlink dump
no longer aborts startup. (#1835)
## [1.11.0] - 2026-07-23
See the [v1.11.0](https://github.com/slackhq/nebula/milestone/25?closed=1) milestone for a complete list of changes.
@@ -870,7 +912,10 @@ created.)
- Initial public release.
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.10.3...HEAD
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.11.2...HEAD
[1.11.2]: https://github.com/slackhq/nebula/releases/tag/v1.11.2
[1.11.1]: https://github.com/slackhq/nebula/releases/tag/v1.11.1
[1.11.0]: https://github.com/slackhq/nebula/releases/tag/v1.11.0
[1.10.3]: https://github.com/slackhq/nebula/releases/tag/v1.10.3
[1.10.2]: https://github.com/slackhq/nebula/releases/tag/v1.10.2
[1.10.1]: https://github.com/slackhq/nebula/releases/tag/v1.10.1
+7 -1
View File
@@ -222,11 +222,14 @@ test-cov-html:
go test -coverprofile=coverage.out
go tool cover -html=coverage.out
# The package builds only compile. The final line links an android binary so a linker-only failure
# cannot pass CI.
build-test-mobile:
GOARCH=amd64 GOOS=ios go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=ios go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=amd64 GOOS=android go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=android go build $(shell go list ./... | grep -v '/cmd/\|/examples/')
GOARCH=arm64 GOOS=android go build -o /dev/null ${NEBULA_CMD_PATH}
bench:
go test -bench=.
@@ -275,10 +278,13 @@ smoke-docker-race: BUILD_ARGS = -race
smoke-docker-race: CGO_ENABLED = 1
smoke-docker-race: smoke-docker
smoke-self: bin
cd .github/workflows/smoke/ && ./smoke-self.sh
smoke-vagrant/%: bin-docker build/%/nebula
cd .github/workflows/smoke/ && ./build.sh $*
cd .github/workflows/smoke/ && ./smoke-vagrant.sh $*
.FORCE:
.PHONY: all all-linux all-freebsd all-openbsd all-netbsd all-darwin all-windows all-cross-linux all-cross-linux-arm all-cross-linux-mips all-cross-linux-other all-cross-darwin all-cross-windows bench bench-cpu bench-cpu-long bin build-test-mobile e2e e2ev e2evv e2evvv e2evvvv proto release service smoke-docker smoke-docker-race test test-cov-html smoke-vagrant/%
.PHONY: all all-linux all-freebsd all-openbsd all-netbsd all-darwin all-windows all-cross-linux all-cross-linux-arm all-cross-linux-mips all-cross-linux-other all-cross-darwin all-cross-windows bench bench-cpu bench-cpu-long bin build-test-mobile e2e e2ev e2evv e2evvv e2evvvv proto release service smoke-docker smoke-docker-race smoke-self test test-cov-html smoke-vagrant/%
.DEFAULT_GOAL := bin
+17 -2
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"io"
"math"
"math/bits"
"net/netip"
"os"
"strings"
@@ -44,6 +45,20 @@ type caFlags struct {
}
func newCaFlags() *caFlags {
// prevent running out of memory on 32-bit systems by defaulting to
// RFC9106's recommendation for memory-constrained environments
var (
defaultArgonMemory uint
defaultArgonIterations uint
)
if bits.UintSize == 32 {
defaultArgonMemory = 64 * 1024
defaultArgonIterations = 3
} else {
defaultArgonMemory = 2 * 1024 * 1024
defaultArgonIterations = 1
}
cf := caFlags{set: flag.NewFlagSet("ca", flag.ContinueOnError)}
cf.set.Usage = func() {}
cf.name = cf.set.String("name", "", "Required: name of the certificate authority")
@@ -55,9 +70,9 @@ func newCaFlags() *caFlags {
cf.groups = cf.set.String("groups", "", "Optional: comma separated list of groups. This will limit which groups subordinate certs can use")
cf.networks = cf.set.String("networks", "", "Optional: comma separated list of ip address and network in CIDR notation. This will limit which ip addresses and networks subordinate certs can use in networks")
cf.unsafeNetworks = cf.set.String("unsafe-networks", "", "Optional: comma separated list of ip address and network in CIDR notation. This will limit which ip addresses and networks subordinate certs can use in unsafe networks")
cf.argonMemory = cf.set.Uint("argon-memory", 2*1024*1024, "Optional: Argon2 memory parameter (in KiB) used for encrypted private key passphrase")
cf.argonMemory = cf.set.Uint("argon-memory", defaultArgonMemory, "Optional: Argon2 memory parameter (in KiB) used for encrypted private key passphrase")
cf.argonParallelism = cf.set.Uint("argon-parallelism", 4, "Optional: Argon2 parallelism parameter used for encrypted private key passphrase")
cf.argonIterations = cf.set.Uint("argon-iterations", 1, "Optional: Argon2 iterations parameter used for encrypted private key passphrase")
cf.argonIterations = cf.set.Uint("argon-iterations", defaultArgonIterations, "Optional: Argon2 iterations parameter used for encrypted private key passphrase")
cf.encryption = cf.set.Bool("encrypt", false, "Optional: prompt for passphrase and write out-key in an encrypted format")
cf.curve = cf.set.String("curve", "25519", "EdDSA/ECDSA Curve (25519, P256)")
cf.p11url = p11Flag(cf.set)
+25 -5
View File
@@ -7,7 +7,9 @@ import (
"bytes"
"encoding/pem"
"errors"
"math/bits"
"os"
"strconv"
"strings"
"testing"
"time"
@@ -22,6 +24,18 @@ func Test_caSummary(t *testing.T) {
}
func Test_caHelp(t *testing.T) {
var (
defaultArgonMemory string
defaultArgonIterations string
)
if bits.UintSize == 32 {
defaultArgonMemory = strconv.Itoa(64 * 1024)
defaultArgonIterations = strconv.Itoa(3)
} else {
defaultArgonMemory = strconv.Itoa(2 * 1024 * 1024)
defaultArgonIterations = strconv.Itoa(1)
}
ob := &bytes.Buffer{}
caHelp(ob)
assert.Equal(
@@ -29,9 +43,9 @@ func Test_caHelp(t *testing.T) {
"Usage of "+os.Args[0]+" ca <flags>: create a self signed certificate authority\n"+
" Pass \"-\" to any path flag to read from stdin or write to stdout.\n"+
" -argon-iterations uint\n"+
" \tOptional: Argon2 iterations parameter used for encrypted private key passphrase (default 1)\n"+
" \tOptional: Argon2 iterations parameter used for encrypted private key passphrase (default "+defaultArgonIterations+")\n"+
" -argon-memory uint\n"+
" \tOptional: Argon2 memory parameter (in KiB) used for encrypted private key passphrase (default 2097152)\n"+
" \tOptional: Argon2 memory parameter (in KiB) used for encrypted private key passphrase (default "+defaultArgonMemory+")\n"+
" -argon-parallelism uint\n"+
" \tOptional: Argon2 parallelism parameter used for encrypted private key passphrase (default 4)\n"+
" -curve string\n"+
@@ -188,10 +202,16 @@ func Test_ca(t *testing.T) {
k, _ := pem.Decode(rb)
ned, err := cert.UnmarshalNebulaEncryptedData(k.Bytes)
require.NoError(t, err)
// we won't know salt in advance, so just check start of string
assert.Equal(t, uint32(2*1024*1024), ned.EncryptionMetadata.Argon2Parameters.Memory)
if bits.UintSize == 32 {
assert.Equal(t, uint32(64*1024), ned.EncryptionMetadata.Argon2Parameters.Memory)
assert.Equal(t, uint32(3), ned.EncryptionMetadata.Argon2Parameters.Iterations)
} else {
assert.Equal(t, uint32(2*1024*1024), ned.EncryptionMetadata.Argon2Parameters.Memory)
assert.Equal(t, uint32(1), ned.EncryptionMetadata.Argon2Parameters.Iterations)
}
assert.Equal(t, uint8(4), ned.EncryptionMetadata.Argon2Parameters.Parallelism)
assert.Equal(t, uint32(1), ned.EncryptionMetadata.Argon2Parameters.Iterations)
// verify the key is valid and decrypt-able
var curve cert.Curve
+20
View File
@@ -323,6 +323,12 @@ func (cm *connectionManager) makeTrafficDecision(localIndex uint32, now time.Tim
return closeTunnel, hostinfo, nil
}
if hostinfo.ConnectionState != nil && hostinfo.ConnectionState.messageCounter.Load() >= RejectAfterMessages {
// Send path can't encrypt a CloseTunnel notify, so just delete locally; the peer recovers via recv_error.
hostinfo.logger(cm.l).Error("Dropping tunnel, message counter is exhausted")
return deleteTunnel, hostinfo, nil
}
primary := cm.hostMap.Hosts[hostinfo.vpnAddrs[0]]
mainHostInfo := true
if primary != nil && primary != hostinfo {
@@ -448,6 +454,11 @@ func (cm *connectionManager) shouldSwapPrimary(current *HostInfo) bool {
return false
}
if current.ConnectionState.messageCounter.Load() >= RehandshakeAfterMessages {
// This tunnel is being rolled for counter exhaustion, never swap back onto its spent key.
return false
}
crt := cm.intf.pki.getCertState().getCertificate(current.ConnectionState.myCert.Version())
if crt == nil {
//my cert was reloaded away. We should definitely swap from this tunnel
@@ -544,6 +555,15 @@ func (cm *connectionManager) tryRehandshake(hostinfo *HostInfo) {
"reason", "current cert version < pki.initiatingVersion",
)
cm.intf.handshakeManager.StartHandshake(hostinfo.vpnAddrs[0], nil)
return
}
if hostinfo.ConnectionState.messageCounter.Load() >= RehandshakeAfterMessages {
cm.l.Info("Re-handshaking with remote",
"vpnAddrs", hostinfo.vpnAddrs,
"reason", "message counter rehandshake threshold reached",
)
cm.intf.handshakeManager.StartHandshake(hostinfo.vpnAddrs[0], nil)
return
}
+73
View File
@@ -199,6 +199,79 @@ func Test_NewConnectionManagerTest2(t *testing.T) {
assert.Contains(t, nc.hostMap.Hosts, hostinfo.vpnAddrs[0])
}
func Test_NewConnectionManager_CounterLimits(t *testing.T) {
l := test.NewLogger()
localrange := netip.MustParsePrefix("10.1.1.1/24")
vpnIp := netip.MustParseAddr("172.1.1.2")
preferredRanges := []netip.Prefix{localrange}
// Very incomplete mock objects
hostMap := newHostMap(l)
hostMap.preferredRanges.Store(&preferredRanges)
cs := &CertState{
initiatingVersion: cert.Version1,
privateKey: []byte{},
v1Cert: &dummyCert{version: cert.Version1},
v1Credential: nil,
}
lh := newTestLighthouse()
ifce := &Interface{
hostMap: hostMap,
inside: &overlaytest.NoopTun{},
outside: &udp.NoopConn{},
firewall: &Firewall{},
lightHouse: lh,
pki: &PKI{},
myVpnAddrs: []netip.Addr{netip.MustParseAddr("172.1.1.1")}, // sorts below vpnIp so shouldSwapPrimary can proceed
handshakeManager: NewHandshakeManager(l, hostMap, lh, &udp.NoopConn{}, defaultHandshakeConfig),
l: l,
}
ifce.pki.cs.Store(cs)
conf := config.NewC(test.NewLogger())
punchy := NewPunchyFromConfig(test.NewLogger(), conf, nil)
nc := newConnectionManagerFromConfig(test.NewLogger(), conf, hostMap, punchy)
nc.intf = ifce
hostinfo := &HostInfo{
vpnAddrs: []netip.Addr{vpnIp},
localIndexId: 1099,
remoteIndexId: 9901,
}
hostinfo.ConnectionState = &ConnectionState{
myCert: &dummyCert{version: cert.Version1},
}
nc.hostMap.unlockedAddHostInfo(hostinfo, ifce)
// Below the rehandshake threshold, no handshake is started
hostinfo.ConnectionState.messageCounter.Store(RehandshakeAfterMessages - 1)
nc.tryRehandshake(hostinfo)
assert.Nil(t, ifce.handshakeManager.QueryVpnAddr(vpnIp))
// A tunnel on its current cert would normally swap to primary
assert.True(t, nc.shouldSwapPrimary(hostinfo))
// At the rehandshake threshold, a new handshake is started
hostinfo.ConnectionState.messageCounter.Store(RehandshakeAfterMessages)
nc.tryRehandshake(hostinfo)
assert.NotNil(t, ifce.handshakeManager.QueryVpnAddr(vpnIp))
// An exhausted tunnel being rolled must never swap back to primary onto its spent key
assert.False(t, nc.shouldSwapPrimary(hostinfo))
// Still below the reject limit, the tunnel stays up
nc.In(hostinfo)
decision, _, _ := nc.makeTrafficDecision(hostinfo.localIndexId, time.Now())
assert.Equal(t, tryRehandshake, decision)
// At the reject limit, the tunnel is deleted locally without a doomed CloseTunnel notify
hostinfo.ConnectionState.messageCounter.Store(RejectAfterMessages)
decision, _, _ = nc.makeTrafficDecision(hostinfo.localIndexId, time.Now())
assert.Equal(t, deleteTunnel, decision)
}
func Test_NewConnectionManager_DisconnectInactive(t *testing.T) {
l := test.NewLogger()
localrange := netip.MustParsePrefix("10.1.1.1/24")
+30 -3
View File
@@ -2,6 +2,7 @@ package nebula
import (
"encoding/json"
"fmt"
"log/slog"
"sync"
"sync/atomic"
@@ -12,7 +13,18 @@ import (
"github.com/slackhq/nebula/noiseutil"
)
const ReplayWindow = 1024
const (
ReplayWindow = 1024
// RehandshakeAfterMessages rolls keys inside the AES-GCM data-volume margin (~2^-36 advantage at 64KB frames).
RehandshakeAfterMessages = uint64(1) << 34
// RejectAfterMessages is the nonce ceiling enforced by noiseutil; a tunnel here is deleted locally, not notified.
RejectAfterMessages = noiseutil.RejectAfterMessages
)
// RehandshakeAfterMessages must stay below RejectAfterMessages so tunnels roll before the hard send stop.
const _ = RejectAfterMessages - RehandshakeAfterMessages
type ConnectionState struct {
eKey noiseutil.CipherState
@@ -30,7 +42,12 @@ type ConnectionState struct {
// completed handshake.Result. It seeds messageCounter and the replay window so
// that the post-handshake message indices already used on the wire don't count
// as missed traffic in the data plane.
func newConnectionStateFromResult(r *handshake.Result) *ConnectionState {
func newConnectionStateFromResult(r *handshake.Result) (*ConnectionState, error) {
// Refuse a MessageIndex too big for the replay window: it can only be a bug, and would spin the seed loop below.
if r.MessageIndex >= ReplayWindow {
return nil, fmt.Errorf("handshake message index %d exceeds replay window", r.MessageIndex)
}
ci := &ConnectionState{
myCert: r.MyCert,
initiator: r.Initiator,
@@ -43,7 +60,7 @@ func newConnectionStateFromResult(r *handshake.Result) *ConnectionState {
for i := uint64(1); i <= r.MessageIndex; i++ {
ci.window.Update(nil, i)
}
return ci
return ci, nil
}
func (cs *ConnectionState) MarshalJSON() ([]byte, error) {
@@ -54,6 +71,16 @@ func (cs *ConnectionState) MarshalJSON() ([]byte, error) {
})
}
// NextMessageCounter reserves the next 1-based counter; RejectAfterMessages is the first we refuse, pinned to not wrap.
func (cs *ConnectionState) NextMessageCounter() (uint64, bool) {
c := cs.messageCounter.Add(1)
if c >= RejectAfterMessages {
cs.messageCounter.Store(RejectAfterMessages)
return c, false
}
return c, true
}
func (cs *ConnectionState) Curve() cert.Curve {
return cs.myCert.Curve()
}
+53 -2
View File
@@ -6,10 +6,12 @@ import (
"time"
"github.com/flynn/noise"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert"
ct "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -79,11 +81,51 @@ func runTestHandshake(t *testing.T) (initR, respR *handshake.Result) {
return initR, respR
}
func TestConnectionState_NextMessageCounter(t *testing.T) {
cs := &ConnectionState{}
cs.messageCounter.Store(RejectAfterMessages - 2)
c, ok := cs.NextMessageCounter()
assert.True(t, ok)
assert.Equal(t, RejectAfterMessages-1, c)
// Hitting the limit refuses and pins the counter there
c, ok = cs.NextMessageCounter()
assert.False(t, ok)
assert.Equal(t, RejectAfterMessages, c)
assert.Equal(t, RejectAfterMessages, cs.messageCounter.Load())
// Continued send attempts stay refused and the counter never wraps
for i := 0; i < 10; i++ {
_, ok = cs.NextMessageCounter()
assert.False(t, ok)
}
assert.Equal(t, RejectAfterMessages, cs.messageCounter.Load())
}
// TestSendNoMetricsDropsExhausted drives the send path to the exhausted drop; metric and out flag prove it.
func TestSendNoMetricsDropsExhausted(t *testing.T) {
initR, _ := runTestHandshake(t)
ci, err := newConnectionStateFromResult(initR)
require.NoError(t, err)
ci.messageCounter.Store(RejectAfterMessages - 1)
f := &Interface{l: test.NewLogger(), messageMetrics: &MessageMetrics{txExhausted: metrics.NewCounter()}}
hostinfo := &HostInfo{vpnAddrs: []netip.Addr{netip.MustParseAddr("10.0.0.1")}, ConnectionState: ci}
f.sendNoMetrics(header.Message, 0, ci, hostinfo, netip.AddrPort{}, []byte{}, make([]byte, 12), make([]byte, mtu), 0)
// The crossing send is refused: it records an exhaustion drop and never reaches connectionManager.Out.
assert.Equal(t, int64(1), f.messageMetrics.txExhausted.Count())
assert.False(t, hostinfo.out.Load())
}
func TestNewConnectionStateFromResult(t *testing.T) {
initR, respR := runTestHandshake(t)
t.Run("initiator", func(t *testing.T) {
ci := newConnectionStateFromResult(initR)
ci, err := newConnectionStateFromResult(initR)
require.NoError(t, err)
assert.True(t, ci.initiator)
assert.Equal(t, initR.MyCert, ci.myCert)
assert.Equal(t, initR.RemoteCert, ci.peerCert)
@@ -102,8 +144,17 @@ func TestNewConnectionStateFromResult(t *testing.T) {
assert.True(t, ci.window.Check(nil, 3), "counter 3 must not be pre-seeded")
})
t.Run("message index too large is refused", func(t *testing.T) {
bad := *initR
bad.MessageIndex = ReplayWindow
ci, err := newConnectionStateFromResult(&bad)
require.Error(t, err)
assert.Nil(t, ci)
})
t.Run("responder", func(t *testing.T) {
ci := newConnectionStateFromResult(respR)
ci, err := newConnectionStateFromResult(respR)
require.NoError(t, err)
assert.False(t, ci.initiator)
assert.Equal(t, respR.MyCert, ci.myCert)
assert.Equal(t, respR.RemoteCert, ci.peerCert)
-136
View File
@@ -1,136 +0,0 @@
//go:build e2e_testing
// +build e2e_testing
package e2e
import (
"testing"
"time"
"github.com/slackhq/nebula"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/e2e/router"
"github.com/slackhq/nebula/udp"
)
// TestRecoveryTiming measures how long a tunnel takes to come back after the peer stops accepting our traffic,
// which is what a laptop waking on a new network looks like from the peer's side: its NAT has no state for where
// we are now, so everything we send disappears.
//
// It is a measurement, not a pass/fail assertion. Recovery is timed to the moment the peer punches back at us,
// since that is when its NAT opens and the tunnel is usable again.
//
// go test -tags e2e_testing -v -run TestRecoveryTiming ./e2e/
func TestRecoveryTiming(t *testing.T) {
for _, tc := range []struct {
name string
rebind bool
}{
{"no trigger", false},
{"rebind counter", true},
} {
t.Run(tc.name, func(t *testing.T) {
d, lost := measureRecovery(t, tc.rebind)
t.Logf("RESULT %-16s recovered in %-9v (%d packets lost)", tc.name, d.Round(time.Millisecond), lost)
})
}
}
// measureRecovery returns how long until the peer punched back, and how many of our packets died meanwhile. When
// rebind is true we call RebindUDPServer once the tunnel goes dark, which is what the darwin network change
// monitor does and what iOS has always done. When false, nothing tells nebula anything is wrong.
func measureRecovery(t *testing.T, rebind bool) (time.Duration, int) {
t.Helper()
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
lhControl, lhVpnIpNet, lhUdpAddr, _ := newSimpleServer(cert.Version2, ca, caKey, "lh", "10.128.0.1/24", m{
"lighthouse": m{"am_lighthouse": true},
})
peerCfg := m{
"lighthouse": m{
"hosts": []any{lhVpnIpNet[0].Addr().String()},
"interval": 600,
"local_allow_list": m{
"10.0.0.0/24": true,
"::/0": false,
},
},
"static_host_map": m{
lhVpnIpNet[0].Addr().String(): []any{lhUdpAddr.String()},
},
}
myControl, myVpnIpNet, myUdpAddr, _ := newSimpleServer(cert.Version2, ca, caKey, "me", "10.128.0.2/24", peerCfg)
theirControl, theirVpnIpNet, theirUdpAddr, _ := newSimpleServer(cert.Version2, ca, caKey, "them", "10.128.0.3/24", peerCfg)
r := router.NewR(t, lhControl, myControl, theirControl)
defer r.RenderFlow()
defer func() {
lhControl.Stop()
myControl.Stop()
theirControl.Stop()
}()
lhControl.Start()
myControl.Start()
theirControl.Start()
r.RouteFor(time.Millisecond * 500)
myControl.InjectLightHouseAddr(theirVpnIpNet[0].Addr(), theirUdpAddr)
theirControl.InjectLightHouseAddr(myVpnIpNet[0].Addr(), myUdpAddr)
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpnIpNet[0].Addr(), 80, myVpnIpNet[0].Addr(), 80, []byte("establish")))
r.RouteFor(time.Second)
if myControl.GetHostInfoByVpnAddr(theirVpnIpNet[0].Addr(), false) == nil {
t.Fatal("failed to establish the tunnel we are measuring")
}
r.RouteFor(time.Millisecond * 500)
// From here the peer's NAT has no state for us, everything we send it disappears
start := time.Now()
blackholed := 0
var recovered time.Duration
if rebind {
myControl.RebindUDPServer()
}
// Keep the tun busy the way someone retrying a stalled connection would
stop := make(chan struct{})
defer close(stop)
go func() {
tick := time.NewTicker(time.Millisecond * 200)
defer tick.Stop()
for {
select {
case <-stop:
return
case <-tick.C:
myControl.InjectTunPacket(BuildTunUDPPacket(
theirVpnIpNet[0].Addr(), 80, myVpnIpNet[0].Addr(), 80, []byte("retry")))
}
}
}()
r.RouteForAllExitFuncOrTimeout(time.Second*30, func(p *udp.Packet, c *nebula.Control) router.ExitType {
if c == theirControl && p.From == myControl.GetUDPAddr() {
blackholed++
return router.Drop
}
// The peer reaching us directly is the moment its NAT opened, whether that is a punch or a handshake
if c == myControl && p.From == theirUdpAddr {
recovered = time.Since(start)
return router.RouteAndExit
}
return router.KeepRouting
})
if recovered == 0 {
t.Fatalf("no recovery within 30s (%d packets blackholed)", blackholed)
}
return recovered, blackholed
}
+2 -19
View File
@@ -153,9 +153,6 @@ const (
ExitNow ExitType = 1
// RouteAndExit routes this packet and exits immediately afterwards
RouteAndExit ExitType = 2
// Drop discards this packet without delivering it and keeps routing. Use it to simulate a blackhole, such as
// a restrictive NAT refusing traffic from an address it has not seen.
Drop ExitType = 3
)
type ExitFunc func(packet *udp.Packet, receiver *nebula.Control) ExitType
@@ -166,9 +163,7 @@ type ExitFunc func(packet *udp.Packet, receiver *nebula.Control) ExitType
func NewR(t testing.TB, controls ...*nebula.Control) *R {
ctx, cancel := context.WithCancel(context.Background())
// t.Name() contains a slash for subtests, so the flow log can land in a nested directory
fn := filepath.Join("mermaid", fmt.Sprintf("%s.md", t.Name()))
if err := os.MkdirAll(filepath.Dir(fn), 0755); err != nil {
if err := os.MkdirAll("mermaid", 0755); err != nil {
panic(err)
}
@@ -179,7 +174,7 @@ func NewR(t testing.TB, controls ...*nebula.Control) *R {
outNat: make(map[outNatKey]netip.AddrPort),
flow: []flowEntry{},
ignoreFlows: []ignoreFlow{},
fn: fn,
fn: filepath.Join("mermaid", fmt.Sprintf("%s.md", t.Name())),
t: t,
cancelRender: cancel,
}
@@ -692,10 +687,6 @@ func (r *R) RouteExitFunc(sender *nebula.Control, whatDo ExitFunc) {
p.Release()
return
case Drop:
// Record it so the flow log shows the attempt, but never hand it to the receiver
r.unlockedInjectFlow(sender, receiver, p, false)
case KeepRouting:
fp := r.unlockedInjectFlow(sender, receiver, p, false)
receiver.InjectUDPPacket(p)
@@ -788,10 +779,6 @@ func (r *R) RouteForAllExitFuncOrTimeout(timeout time.Duration, whatDo ExitFunc)
p.Release()
return true
case Drop:
// Record it so the flow log shows the attempt, but never hand it to the receiver
r.unlockedInjectFlow(cm[x], receiver, p, false)
case KeepRouting:
fp := r.unlockedInjectFlow(cm[x], receiver, p, false)
receiver.InjectUDPPacket(p)
@@ -897,10 +884,6 @@ func (r *R) RouteForAllExitFunc(whatDo ExitFunc) {
p.Release()
return
case Drop:
// Record it so the flow log shows the attempt, but never hand it to the receiver
r.unlockedInjectFlow(cm[x], receiver, p, false)
case KeepRouting:
fp := r.unlockedInjectFlow(cm[x], receiver, p, false)
receiver.InjectUDPPacket(p)
+1 -2
View File
@@ -5,9 +5,9 @@ go 1.26.0
require (
dario.cat/mergo v1.0.2
filippo.io/bigmod v0.1.0
github.com/DefinedNet/netlink v1.3.2-dn.1
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
github.com/armon/go-radix v1.0.0
github.com/cyberdelia/go-metrics-graphite v0.0.0-20161219230853-39f87cc3b432
github.com/flynn/noise v1.1.0
github.com/gaissmai/bart v0.28.0
github.com/gogo/protobuf v1.3.2
@@ -21,7 +21,6 @@ require (
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
github.com/stefanberger/go-pkcs11uri v0.0.0-20230803200340-78284954bff6
github.com/stretchr/testify v1.11.1
github.com/vishvananda/netlink v1.3.1
go.uber.org/goleak v1.3.0
go.yaml.in/yaml/v3 v3.0.4
golang.org/x/crypto v0.54.0
+2 -6
View File
@@ -3,6 +3,8 @@ dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8=
dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA=
filippo.io/bigmod v0.1.0 h1:UNzDk7y9ADKST+axd9skUpBQeW7fG2KrTZyOE4uGQy8=
filippo.io/bigmod v0.1.0/go.mod h1:OjOXDNlClLblvXdwgFFOQFJEocLhhtai8vGLy0JCZlI=
github.com/DefinedNet/netlink v1.3.2-dn.1 h1:Md2PsUE1Xuy1Zxs/Bm6356n1kt5bq2HspXlPJ6pEspg=
github.com/DefinedNet/netlink v1.3.2-dn.1/go.mod h1:o4YSn8Jw7zoDcYXDmV77TRd4eS5sSLuE+vu2zAN67C8=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
@@ -19,8 +21,6 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cyberdelia/go-metrics-graphite v0.0.0-20161219230853-39f87cc3b432 h1:M5QgkYacWj0Xs8MhpIK/5uwU02icXpEoSo9sM2aRCps=
github.com/cyberdelia/go-metrics-graphite v0.0.0-20161219230853-39f87cc3b432/go.mod h1:xwIwAxMvYnVrGJPe2FKx5prTrnAjGOD8zvDOnxnrrkM=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -145,8 +145,6 @@ github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81P
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/vishvananda/netlink v1.3.1 h1:3AEMt62VKqz90r0tmNhog0r/PpWKmrEShJU0wJW6bV0=
github.com/vishvananda/netlink v1.3.1/go.mod h1:ARtKouGSTGchR8aMwmkzC0qiNPrrWO5JS/XMVl45+b4=
github.com/vishvananda/netns v0.0.5 h1:DfiHV+j8bA32MFM7bfEunvT8IAqQ/NzSJHtcmW5zdEY=
github.com/vishvananda/netns v0.0.5/go.mod h1:SpkAiCQRtJ6TvvxPnOSyH3BMl6unz3xZlaprSwhNNJM=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -206,8 +204,6 @@ golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+117
View File
@@ -0,0 +1,117 @@
package nebula
// This file is a trimmed, inlined copy of the graphite exporter from
// github.com/cyberdelia/go-metrics-graphite, retaining only the Config type and
// the Once entrypoint that Nebula uses. The upstream package has been
// unmaintained for 10+ years, so it was vendored here to drop the dependency.
// See https://github.com/slackhq/nebula/issues/1831.
//
// Copyright 2015 Timothée Peignier. All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
//
// 1. Redistributions of source code must retain the above copyright notice,
// this list of conditions and the following disclaimer.
//
// 2. Redistributions in binary form must reproduce the above copyright notice,
// this list of conditions and the following disclaimer in the documentation
// and/or other materials provided with the distribution.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
// DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
// FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
// CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
import (
"bufio"
"fmt"
"net"
"strconv"
"strings"
"time"
"github.com/rcrowley/go-metrics"
)
// graphiteConfigExport provides a container with configuration parameters for
// the Graphite exporter.
type graphiteConfigExport struct {
Addr *net.TCPAddr // Network address to connect to
Registry metrics.Registry // Registry to be exported
FlushInterval time.Duration // Flush interval
DurationUnit time.Duration // Time conversion unit for durations
Prefix string // Prefix to be prepended to metric names
Percentiles []float64 // Percentiles to export from timers and histograms
}
// graphiteOnce performs a single submission to Graphite, returning a non-nil
// error on failed connections.
func graphiteOnce(c graphiteConfigExport) error {
now := time.Now().Unix()
du := float64(c.DurationUnit)
flushSeconds := float64(c.FlushInterval) / float64(time.Second)
conn, err := net.DialTCP("tcp", nil, c.Addr)
if err != nil {
return err
}
defer conn.Close()
w := bufio.NewWriter(conn)
c.Registry.Each(func(name string, i any) {
switch metric := i.(type) {
case metrics.Counter:
count := metric.Count()
fmt.Fprintf(w, "%s.%s.count %d %d\n", c.Prefix, name, count, now)
fmt.Fprintf(w, "%s.%s.count_ps %.2f %d\n", c.Prefix, name, float64(count)/flushSeconds, now)
case metrics.Gauge:
fmt.Fprintf(w, "%s.%s.value %d %d\n", c.Prefix, name, metric.Value(), now)
case metrics.GaugeFloat64:
fmt.Fprintf(w, "%s.%s.value %f %d\n", c.Prefix, name, metric.Value(), now)
case metrics.Histogram:
h := metric.Snapshot()
ps := h.Percentiles(c.Percentiles)
fmt.Fprintf(w, "%s.%s.count %d %d\n", c.Prefix, name, h.Count(), now)
fmt.Fprintf(w, "%s.%s.min %d %d\n", c.Prefix, name, h.Min(), now)
fmt.Fprintf(w, "%s.%s.max %d %d\n", c.Prefix, name, h.Max(), now)
fmt.Fprintf(w, "%s.%s.mean %.2f %d\n", c.Prefix, name, h.Mean(), now)
fmt.Fprintf(w, "%s.%s.std-dev %.2f %d\n", c.Prefix, name, h.StdDev(), now)
for psIdx, psKey := range c.Percentiles {
key := strings.Replace(strconv.FormatFloat(psKey*100.0, 'f', -1, 64), ".", "", 1)
fmt.Fprintf(w, "%s.%s.%s-percentile %.2f %d\n", c.Prefix, name, key, ps[psIdx], now)
}
case metrics.Meter:
m := metric.Snapshot()
fmt.Fprintf(w, "%s.%s.count %d %d\n", c.Prefix, name, m.Count(), now)
fmt.Fprintf(w, "%s.%s.one-minute %.2f %d\n", c.Prefix, name, m.Rate1(), now)
fmt.Fprintf(w, "%s.%s.five-minute %.2f %d\n", c.Prefix, name, m.Rate5(), now)
fmt.Fprintf(w, "%s.%s.fifteen-minute %.2f %d\n", c.Prefix, name, m.Rate15(), now)
fmt.Fprintf(w, "%s.%s.mean %.2f %d\n", c.Prefix, name, m.RateMean(), now)
case metrics.Timer:
t := metric.Snapshot()
ps := t.Percentiles(c.Percentiles)
count := t.Count()
fmt.Fprintf(w, "%s.%s.count %d %d\n", c.Prefix, name, count, now)
fmt.Fprintf(w, "%s.%s.count_ps %.2f %d\n", c.Prefix, name, float64(count)/flushSeconds, now)
fmt.Fprintf(w, "%s.%s.min %d %d\n", c.Prefix, name, t.Min()/int64(du), now)
fmt.Fprintf(w, "%s.%s.max %d %d\n", c.Prefix, name, t.Max()/int64(du), now)
fmt.Fprintf(w, "%s.%s.mean %.2f %d\n", c.Prefix, name, t.Mean()/du, now)
fmt.Fprintf(w, "%s.%s.std-dev %.2f %d\n", c.Prefix, name, t.StdDev()/du, now)
for psIdx, psKey := range c.Percentiles {
key := strings.Replace(strconv.FormatFloat(psKey*100.0, 'f', -1, 64), ".", "", 1)
fmt.Fprintf(w, "%s.%s.%s-percentile %.2f %d\n", c.Prefix, name, key, ps[psIdx]/du, now)
}
fmt.Fprintf(w, "%s.%s.one-minute %.2f %d\n", c.Prefix, name, t.Rate1(), now)
fmt.Fprintf(w, "%s.%s.five-minute %.2f %d\n", c.Prefix, name, t.Rate5(), now)
fmt.Fprintf(w, "%s.%s.fifteen-minute %.2f %d\n", c.Prefix, name, t.Rate15(), now)
fmt.Fprintf(w, "%s.%s.mean-rate %.2f %d\n", c.Prefix, name, t.RateMean(), now)
}
w.Flush()
})
return nil
}
+14 -2
View File
@@ -749,8 +749,14 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
return
}
connState, err := newConnectionStateFromResult(result)
if err != nil {
f.l.Error("Discarding handshake with an invalid message index", "error", err, "vpnAddrs", vpnAddrs)
return
}
hostinfo := &HostInfo{
ConnectionState: newConnectionStateFromResult(result),
ConnectionState: connState,
localIndexId: result.LocalIndex,
remoteIndexId: result.RemoteIndex,
vpnAddrs: vpnAddrs,
@@ -868,7 +874,13 @@ func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostIn
}
// Handshake complete; build the ConnectionState now that we have keys and a verified peer cert.
hostinfo.ConnectionState = newConnectionStateFromResult(result)
cs, err := newConnectionStateFromResult(result)
if err != nil {
f.l.Error("Discarding handshake with an invalid message index", "error", err, "vpnAddrs", hostinfo.vpnAddrs)
hm.DeleteHostInfo(hostinfo)
return
}
hostinfo.ConnectionState = cs
remoteCert := result.RemoteCert
if remoteCert == nil {
-58
View File
@@ -6,7 +6,6 @@ import (
"errors"
"fmt"
"log/slog"
"net"
"net/netip"
"slices"
"sync"
@@ -18,7 +17,6 @@ import (
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/logging"
)
const defaultPromoteEvery = 1000 // Count of packets sent before we try moving a tunnel to a preferred underlay ip address
@@ -865,59 +863,3 @@ func (i *HostInfo) logger(l *slog.Logger) *slog.Logger {
return li
}
// Utility functions
func localAddrs(l *slog.Logger, allowList *LocalAllowList) []netip.Addr {
//FIXME: This function is pretty garbage
var finalAddrs []netip.Addr
ifaces, _ := net.Interfaces()
for _, i := range ifaces {
allow := allowList.AllowName(i.Name)
if l.Enabled(context.Background(), logging.LevelTrace) {
l.Log(context.Background(), logging.LevelTrace, "localAllowList.AllowName",
"interfaceName", i.Name,
"allow", allow,
)
}
if !allow {
continue
}
addrs, _ := i.Addrs()
for _, rawAddr := range addrs {
var addr netip.Addr
switch v := rawAddr.(type) {
case *net.IPNet:
//continue
addr, _ = netip.AddrFromSlice(v.IP)
case *net.IPAddr:
addr, _ = netip.AddrFromSlice(v.IP)
}
if !addr.IsValid() {
if l.Enabled(context.Background(), slog.LevelDebug) {
l.Debug("addr was invalid", "localAddr", rawAddr)
}
continue
}
addr = addr.Unmap()
if addr.IsLoopback() == false && addr.IsLinkLocalUnicast() == false {
isAllowed := allowList.Allow(addr)
if l.Enabled(context.Background(), logging.LevelTrace) {
l.Log(context.Background(), logging.LevelTrace, "localAllowList.Allow",
"localAddr", addr,
"allowed", isAllowed,
)
}
if !isAllowed {
continue
}
finalAddrs = append(finalAddrs, addr)
}
}
}
return finalAddrs
}
+25 -2
View File
@@ -37,6 +37,7 @@ func (f *Interface) consumeInsidePacket(packet []byte, fwPacket *firewall.Packet
// routes packets from the Nebula addr to the Nebula addr through the Nebula
// TUN device.
if immediatelyForwardToSelf {
iputil.SetTransportChecksum(packet)
_, err := f.readers[q].Write(packet)
if err != nil {
f.l.Error("Failed to forward to tun", "error", err)
@@ -275,6 +276,14 @@ func (f *Interface) sendTo(t header.MessageType, st header.MessageSubType, ci *C
f.sendNoMetrics(t, st, ci, hostinfo, remote, p, nb, out, 0)
}
// dropExhausted records an exhaustion drop and logs once, on the crossing send, for a spent tunnel.
func (f *Interface) dropExhausted(hostinfo *HostInfo, c uint64, msg string) {
f.messageMetrics.TxExhausted(1)
if c == RejectAfterMessages {
hostinfo.logger(f.l).Error(msg)
}
}
// SendVia sends a payload through a Relay tunnel. No authentication or encryption is done
// to the payload for the ultimate target host, making this a useful method for sending
// handshake messages to peers through relay tunnels.
@@ -294,7 +303,14 @@ func (f *Interface) SendVia(via *HostInfo,
// NOTE: for goboring AESGCMTLS we need to lock because of the nonce check
via.ConnectionState.writeLock.Lock()
}
c := via.ConnectionState.messageCounter.Add(1)
c, ok := via.ConnectionState.NextMessageCounter()
if !ok {
if noiseutil.EncryptLockNeeded {
via.ConnectionState.writeLock.Unlock()
}
f.dropExhausted(via, c, "Dropping outbound relay packets, tunnel message counter is exhausted")
return
}
out = header.Encode(out, header.Version, header.Message, header.MessageRelay, relay.RemoteIndex, c)
f.connectionManager.Out(via)
@@ -361,7 +377,14 @@ func (f *Interface) sendNoMetrics(t header.MessageType, st header.MessageSubType
// NOTE: for goboring AESGCMTLS we need to lock because of the nonce check
ci.writeLock.Lock()
}
c := ci.messageCounter.Add(1)
c, ok := ci.NextMessageCounter()
if !ok {
if noiseutil.EncryptLockNeeded {
ci.writeLock.Unlock()
}
f.dropExhausted(hostinfo, c, "Dropping outbound packets, tunnel message counter is exhausted")
return
}
//l.WithField("trace", string(debug.Stack())).Error("out Header ", &Header{Version, t, st, 0, hostinfo.remoteIndexId, c}, p)
out = header.Encode(out, header.Version, t, st, hostinfo.remoteIndexId, c)
+263
View File
@@ -0,0 +1,263 @@
package nebula
import (
"encoding/binary"
"io"
"net/netip"
"testing"
"github.com/gaissmai/bart"
"github.com/slackhq/nebula/firewall"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const (
ipv4HeaderLen = 20
ipv6HeaderLen = 40
)
// capturingTun is a tio.Queue that records what is written to it. A queue that
// discards writes is indistinguishable from a packet that was never forwarded.
type capturingTun struct {
writes [][]byte
}
func (c *capturingTun) Read([]byte) (int, error) { return 0, io.EOF }
func (c *capturingTun) Close() error { return nil }
func (c *capturingTun) Write(b []byte) (int, error) {
c.writes = append(c.writes, append([]byte(nil), b...))
return len(b), nil
}
func newSelfForwardInterface(myAddrs ...netip.Addr) (*Interface, *capturingTun) {
vpnAddrs := &bart.Lite{}
for _, a := range myAddrs {
vpnAddrs.Insert(netip.PrefixFrom(a, a.BitLen()))
}
tun := &capturingTun{}
return &Interface{
l: test.NewLogger(),
myVpnAddrsTable: vpnAddrs,
myBroadcastAddrsTable: &bart.Lite{},
readers: []io.ReadWriteCloser{tun},
}, tun
}
func consumeInside(f *Interface, packet []byte) {
f.consumeInsidePacket(packet, &firewall.Packet{}, make([]byte, 12), make([]byte, mtu), 0, nil)
}
// l4Proto describes one upper-layer header for these tests: its IP next-header
// value, where its checksum field sits within the header, and how to build a
// minimal instance of it.
type l4Proto struct {
name string
nextHdr uint8
cksumAt int
build func() []byte
}
var (
tcpSyn = l4Proto{"tcp", firewall.ProtoTCP, 16, func() []byte {
h := make([]byte, 20)
binary.BigEndian.PutUint16(h[0:2], 49152)
binary.BigEndian.PutUint16(h[2:4], 443)
binary.BigEndian.PutUint32(h[4:8], 0x11223344) // sequence
h[12] = 5 << 4 // data offset, no options
h[13] = 0x02 // SYN
binary.BigEndian.PutUint16(h[14:16], 65535) // window
return h
}}
udpDatagram = l4Proto{"udp", firewall.ProtoUDP, 6, func() []byte {
h := make([]byte, 8+4)
binary.BigEndian.PutUint16(h[0:2], 49152)
binary.BigEndian.PutUint16(h[2:4], 53)
binary.BigEndian.PutUint16(h[4:6], uint16(len(h)))
copy(h[8:], "ping")
return h
}}
icmpEcho = l4Proto{"icmp", firewall.ProtoICMP, 2, func() []byte { return echoRequest(8) }}
icmpv6Echo = l4Proto{"icmpv6", firewall.ProtoICMPv6, 2, func() []byte { return echoRequest(128) }}
)
// echoRequest builds an echo request body. The type differs between ICMP and
// ICMPv6, the rest of the header does not.
func echoRequest(typ uint8) []byte {
h := make([]byte, 8)
h[0] = typ
binary.BigEndian.PutUint16(h[4:6], 0xbeef) // identifier
binary.BigEndian.PutUint16(h[6:8], 1) // sequence
return h
}
func buildIPv6(src, dst netip.Addr, p l4Proto) []byte {
l4 := p.build()
pkt := make([]byte, ipv6HeaderLen+len(l4))
pkt[0] = 0x60
binary.BigEndian.PutUint16(pkt[4:6], uint16(len(l4)))
pkt[6] = p.nextHdr
pkt[7] = 64
copy(pkt[8:24], src.AsSlice())
copy(pkt[24:40], dst.AsSlice())
copy(pkt[ipv6HeaderLen:], l4)
if l4 := pkt[ipv6HeaderLen:]; p.nextHdr == firewall.ProtoTCP || p.nextHdr == firewall.ProtoUDP {
sum := ipv6PseudoheaderSum(src, dst, uint32(p.nextHdr), uint32(len(l4)))
binary.BigEndian.PutUint16(l4[p.cksumAt:], ^fold(sumBytes(l4, sum)))
}
return pkt
}
func buildIPv4(src, dst netip.Addr, p l4Proto) []byte {
l4 := p.build()
pkt := make([]byte, ipv4HeaderLen+len(l4))
pkt[0] = 0x45
binary.BigEndian.PutUint16(pkt[2:4], uint16(len(pkt)))
pkt[8] = 64
pkt[9] = p.nextHdr
copy(pkt[12:16], src.AsSlice())
copy(pkt[16:20], dst.AsSlice())
copy(pkt[ipv4HeaderLen:], l4)
if l4 := pkt[ipv4HeaderLen:]; p.nextHdr == firewall.ProtoTCP || p.nextHdr == firewall.ProtoUDP {
sum := sumBytes(pkt[12:20], uint32(p.nextHdr)+uint32(len(l4)))
binary.BigEndian.PutUint16(l4[p.cksumAt:], ^fold(sumBytes(l4, sum)))
}
return pkt
}
// ipv6PseudoheaderSum is the RFC 2460 section 8.1 pseudo-header sum: source,
// destination, a 32 bit upper-layer packet length and a 32 bit zero-padded next
// header. Kept local to the test so these assertions do not check nebula's
// checksum code against itself.
func ipv6PseudoheaderSum(src, dst netip.Addr, nextHeader, length uint32) uint32 {
var csum uint32
s, d := src.AsSlice(), dst.AsSlice()
for i := 0; i < 16; i += 2 {
csum += uint32(s[i])<<8 | uint32(s[i+1])
csum += uint32(d[i])<<8 | uint32(d[i+1])
}
return csum + length + nextHeader
}
func sumBytes(b []byte, csum uint32) uint32 {
for i := 0; i+1 < len(b); i += 2 {
csum += uint32(b[i])<<8 | uint32(b[i+1])
}
if len(b)%2 == 1 {
csum += uint32(b[len(b)-1]) << 8
}
return csum
}
func fold(csum uint32) uint16 {
for csum > 0xffff {
csum = (csum >> 16) + (csum & 0xffff)
}
return uint16(csum)
}
// l4ChecksumValid6 verifies an IPv6 upper-layer checksum the way a receiver
// does: the pseudo-header plus the whole upper-layer segment, checksum field
// included, folds to 0xffff. The next header field is the upper-layer protocol
// only while there are no extension headers, which is all this file builds.
func l4ChecksumValid6(pkt []byte) bool {
src, _ := netip.AddrFromSlice(pkt[8:24])
dst, _ := netip.AddrFromSlice(pkt[24:40])
l4 := pkt[ipv6HeaderLen:]
return fold(sumBytes(l4, ipv6PseudoheaderSum(src, dst, uint32(pkt[6]), uint32(len(l4))))) == 0xffff
}
// l4ChecksumValid4 is the IPv4 counterpart: the RFC 793/768 pseudo-header is
// source, destination, a zero byte, the protocol and the upper-layer length.
func l4ChecksumValid4(pkt []byte) bool {
ihl := int(pkt[0]&0x0f) << 2
l4 := pkt[ihl:]
return fold(sumBytes(l4, sumBytes(pkt[12:20], uint32(pkt[9])+uint32(len(l4))))) == 0xffff
}
// TestConsumeInsidePacketSelfTraffic covers the self-addressed branch of
// consumeInsidePacket, taken where immediatelyForwardToSelf is set (see
// inside_bsd.go): the packet goes straight back to the tun, ahead of the
// firewall and the handshake.
func TestConsumeInsidePacketSelfTraffic(t *testing.T) {
v4 := netip.MustParseAddr("100.100.1.42")
v6 := netip.MustParseAddr("fd00::42")
tests := []struct {
name string
addr netip.Addr
pkt []byte
}{
{"ipv4/tcp", v4, buildIPv4(v4, v4, tcpSyn)},
{"ipv4/udp", v4, buildIPv4(v4, v4, udpDatagram)},
{"ipv4/icmp", v4, buildIPv4(v4, v4, icmpEcho)},
{"ipv6/tcp", v6, buildIPv6(v6, v6, tcpSyn)},
{"ipv6/udp", v6, buildIPv6(v6, v6, udpDatagram)},
{"ipv6/icmpv6", v6, buildIPv6(v6, v6, icmpv6Echo)},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
f, tun := newSelfForwardInterface(tt.addr)
// consumeInsidePacket writes through the slice it is handed, so a
// packet that arrived with a valid checksum must come back out of
// bytes taken before the call, unchanged.
want := append([]byte(nil), tt.pkt...)
consumeInside(f, tt.pkt)
if immediatelyForwardToSelf {
require.Len(t, tun.writes, 1)
assert.Equal(t, want, tun.writes[0])
} else {
assert.Empty(t, tun.writes, "self traffic reaches the tun over loopback here and must be dropped")
}
})
}
}
// TestConsumeInsidePacketSelfTrafficChecksum shows that the self-forward
// returns the bytes it was handed, so a packet that arrived with a wrong
// upper-layer checksum is written back with that same wrong checksum and the
// kernel drops it on re-entry.
//
// This is how a macOS host loses TCP and UDP to its own IPv6 overlay address:
// the kernel writes only the pseudo-header sum into the checksum field and
// defers completion to hardware offload, state that does not survive the
// crossing into userspace. Which kernels do this, for which protocols and IP
// versions, is a property of the kernel and belongs to a test against a live
// one; here the checksum is simply wrong, and the forward must make it right.
func TestConsumeInsidePacketSelfTrafficChecksum(t *testing.T) {
if !immediatelyForwardToSelf {
t.Skip("self traffic never reaches the tun on this platform")
}
versions := []struct {
name string
addr netip.Addr
build func(src, dst netip.Addr, p l4Proto) []byte
l4At int
valid func(pkt []byte) bool
}{
{"v4", netip.MustParseAddr("100.100.1.42"), buildIPv4, ipv4HeaderLen, l4ChecksumValid4},
{"v6", netip.MustParseAddr("fd00::42"), buildIPv6, ipv6HeaderLen, l4ChecksumValid6},
}
for _, v := range versions {
for _, p := range []l4Proto{tcpSyn, udpDatagram} {
t.Run(v.name+"/"+p.name, func(t *testing.T) {
pkt := v.build(v.addr, v.addr, p)
binary.BigEndian.PutUint16(pkt[v.l4At+p.cksumAt:], 0x1234)
require.False(t, v.valid(pkt), "the packet under test must start with a wrong checksum")
f, tun := newSelfForwardInterface(v.addr)
consumeInside(f, pkt)
require.Len(t, tun.writes, 1)
assert.True(t, v.valid(tun.writes[0]),
"a forwarded %s packet must carry a valid checksum, got 0x%04x",
p.name, binary.BigEndian.Uint16(tun.writes[0][v.l4At+p.cksumAt:]))
})
}
}
}
+147
View File
@@ -0,0 +1,147 @@
package iputil
import (
"encoding/binary"
"github.com/google/gopacket/layers"
"golang.org/x/net/ipv4"
"golang.org/x/net/ipv6"
"gvisor.dev/gvisor/pkg/tcpip/checksum"
)
const udpHeaderLen = 8
// SetTransportChecksum recomputes the TCP or UDP checksum of an IPv4 or IPv6
// packet in place.
//
// A kernel that offloads checksums to the NIC hands a packet to a tun with the
// transport checksum unfinished: only the pseudo-header sum is in the field and
// the rest is left for hardware that a tun does not have. A packet written
// straight back to that tun is dropped on re-entry unless the checksum is
// completed first. ICMP is left alone; it arrived complete on the kernels this
// was measured against.
//
// So is any packet whose transport header cannot be located: fragments, unknown
// extension headers and truncated packets. An IPv6 fragment header is declined
// even when it carries the whole datagram (RFC 6946 atomic fragment), because
// the walk reports only that a fragment header was present.
func SetTransportChecksum(packet []byte) {
if len(packet) < 1 {
return
}
switch int(packet[0] >> 4) {
case ipv4.Version:
setTransportChecksum4(packet)
case ipv6.Version:
setTransportChecksum6(packet)
}
}
func setTransportChecksum4(packet []byte) {
if len(packet) < ipv4.HeaderLen {
return
}
ihl := int(packet[0]&0x0f) << 2
end := int(binary.BigEndian.Uint16(packet[2:4]))
if ihl < ipv4.HeaderLen || end < ihl || end > len(packet) {
return
}
// The checksum covers the whole datagram, which a fragment (MF set or a
// non-zero offset) does not carry.
if binary.BigEndian.Uint16(packet[6:8])&0x3fff != 0 {
return
}
transport, ok := transportExtent(packet[ihl:end], packet[9])
if !ok {
return
}
csum := ipv4PseudoheaderChecksum(packet[12:16], packet[16:20], uint32(packet[9]), uint32(len(transport)))
writeTransportChecksum(transport, layers.IPProtocol(packet[9]), csum)
}
func setTransportChecksum6(packet []byte) {
if len(packet) < ipv6.HeaderLen {
return
}
end := ipv6.HeaderLen + int(binary.BigEndian.Uint16(packet[4:6]))
if end > len(packet) {
return
}
// The checksum covers the whole datagram, which a fragment does not carry.
// An unknown extension header hides where the transport header starts. A
// chain longer than the walk's budget ends it early, at an offset that was
// never checked against the packet.
proto, offset, _, anyFragment, err := IPv6FindUpperProtocol(packet[:end])
if err != nil || anyFragment || offset >= end {
return
}
transport, ok := transportExtent(packet[offset:end], proto)
if !ok {
return
}
csum := ipv6PseudoheaderChecksum(packet[8:24], packet[24:40], uint32(proto), uint32(len(transport)))
writeTransportChecksum(transport, layers.IPProtocol(proto), csum)
}
// transportExtent narrows a segment to the length its own header declares. UDP
// carries a Length field, and RFC 768 and RFC 8200 section 8.1 both make that
// field, not the IP payload extent, the length the pseudo-header counts and the
// checksum covers; a datagram padded out to a link's minimum frame is the usual
// way the two differ. TCP has no such field, so its segment runs to the end of
// the IP payload. A Length that overruns the bytes IP delivered describes a
// datagram that is not there.
func transportExtent(transport []byte, proto uint8) ([]byte, bool) {
if layers.IPProtocol(proto) != layers.IPProtocolUDP {
return transport, true
}
if len(transport) < udpHeaderLen {
return nil, false
}
ulen := int(binary.BigEndian.Uint16(transport[4:6]))
if ulen < udpHeaderLen || ulen > len(transport) {
return nil, false
}
return transport[:ulen], true
}
// writeTransportChecksum stores the checksum of transport, taken over the
// pseudo-header sum csum, in the header's checksum field. A UDP checksum that
// computes to zero goes on the wire as 0xffff: zero means no checksum was
// computed (RFC 768), and over IPv6 the checksum is mandatory (RFC 8200
// section 8.1).
func writeTransportChecksum(transport []byte, proto layers.IPProtocol, csum uint32) {
var at, minLen int
switch proto {
case layers.IPProtocolTCP:
at, minLen = 16, 20
case layers.IPProtocolUDP:
at, minLen = 6, udpHeaderLen
default:
return
}
if len(transport) < minLen {
return
}
transport[at], transport[at+1] = 0, 0
sum := ^checksum.Checksum(transport, fold(csum))
if sum == 0 && proto == layers.IPProtocolUDP {
sum = 0xffff
}
binary.BigEndian.PutUint16(transport[at:], sum)
}
// fold reduces a pseudo-header sum to the 16 bit seed Checksum takes. Carrying
// the high half back into the low half is what keeps the reduction lossless, so
// the seed sums exactly as the wider value would; 0xffff is its fixed point.
// Every term of that sum comes from a 16 bit field, so it stays far below the
// width at which the accumulator would wrap.
func fold(csum uint32) uint16 {
for csum > 0xffff {
csum = (csum >> 16) + (csum & 0xffff)
}
return uint16(csum)
}
+242
View File
@@ -0,0 +1,242 @@
package iputil
import (
"encoding/binary"
"net"
"testing"
"github.com/google/gopacket"
"github.com/google/gopacket/layers"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/net/ipv6"
)
// serialize builds a packet with gopacket, whose checksums are computed
// independently of this package.
func serialize(t *testing.T, ls ...gopacket.SerializableLayer) []byte {
buf := gopacket.NewSerializeBuffer()
require.NoError(t, gopacket.SerializeLayers(buf, gopacket.SerializeOptions{FixLengths: true, ComputeChecksums: true}, ls...))
return append([]byte(nil), buf.Bytes()...)
}
// withExtensionHeader inserts an 8 byte IPv6 extension header of the given
// type between the IPv6 header and its payload. The transport checksum does not
// change: the pseudo-header counts only upper-layer bytes.
func withExtensionHeader(pkt []byte, typ layers.IPProtocol, hdr [8]byte) []byte {
hdr[0] = pkt[6]
out := make([]byte, 0, len(pkt)+8)
out = append(out, pkt[:40]...)
out = append(out, hdr[:]...)
out = append(out, pkt[40:]...)
out[6] = byte(typ)
binary.BigEndian.PutUint16(out[4:6], binary.BigEndian.Uint16(pkt[4:6])+8)
return out
}
// truncate copies the first n bytes into a buffer of exactly that capacity, so
// a read past the length panics instead of quietly succeeding.
func truncate(pkt []byte, n int) []byte {
out := make([]byte, n)
copy(out, pkt)
return out
}
// extChain builds an IPv6 packet fronted by n Destination Options headers. Each
// points at another one, so the walk spends its whole budget without reaching a
// transport header. lastExtLen inflates the final header's declared length,
// which is how the walk ends up past the end of the packet.
func extChain(n int, lastExtLen byte) []byte {
pkt := make([]byte, ipv6.HeaderLen)
pkt[0], pkt[6], pkt[7] = 0x60, 60, 64
for i := range n {
h := make([]byte, 8)
h[0] = 60
if i == n-1 {
h[1] = lastExtLen
}
pkt = append(pkt, h...)
}
pkt = append(pkt, make([]byte, 20)...)
binary.BigEndian.PutUint16(pkt[4:6], uint16(len(pkt)-ipv6.HeaderLen))
return pkt
}
func TestSetTransportChecksum(t *testing.T) {
// Source and destination differ so that a pseudo-header built from the wrong
// one, or from the two swapped, does not land on the same checksum anyway.
v4 := func(proto layers.IPProtocol) *layers.IPv4 {
return &layers.IPv4{Version: 4, TTL: 64, Id: 0x1234, Protocol: proto, SrcIP: net.IPv4(192, 0, 2, 1).To4(), DstIP: net.IPv4(198, 51, 100, 2).To4()}
}
v6 := func(proto layers.IPProtocol) *layers.IPv6 {
return &layers.IPv6{Version: 6, HopLimit: 64, NextHeader: proto, SrcIP: net.ParseIP("2001:db8::1"), DstIP: net.ParseIP("2001:db8:1::2")}
}
tcp := func(ip gopacket.NetworkLayer) *layers.TCP {
l := &layers.TCP{SrcPort: 49152, DstPort: 443, SYN: true, Window: 65535}
require.NoError(t, l.SetNetworkLayerForChecksum(ip))
return l
}
udp := func(ip gopacket.NetworkLayer) *layers.UDP {
l := &layers.UDP{SrcPort: 49152, DstPort: 53}
require.NoError(t, l.SetNetworkLayerForChecksum(ip))
return l
}
payload := gopacket.Payload("self")
nop := layers.IPv4Option{OptionType: 1, OptionLength: 1}
ip4tcp := v4(layers.IPProtocolTCP)
ip4opts := v4(layers.IPProtocolTCP)
ip4opts.Options = []layers.IPv4Option{nop, nop, nop, nop}
ip4udp := v4(layers.IPProtocolUDP)
ip6tcp := v6(layers.IPProtocolTCP)
ip6udp := v6(layers.IPProtocolUDP)
hopByHop := [8]byte{0, 0, 1, 4} // next header, length 0, PadN of 4
// Bytes past the length the IP header declares are not part of the
// datagram and must not be summed.
trailing4 := append(serialize(t, ip4tcp, tcp(ip4tcp), payload), []byte("trailing")...)
trailing6 := append(serialize(t, ip6tcp, tcp(ip6tcp), payload), []byte("trailing")...)
// A datagram padded out past the length UDP declares: the pseudo-header
// counts the UDP Length field, so the checksum is the unpadded one.
padded4 := append(serialize(t, ip4udp, udp(ip4udp), payload), []byte("pad!")...)
binary.BigEndian.PutUint16(padded4[2:4], uint16(len(padded4)))
padded6 := append(serialize(t, ip6udp, udp(ip6udp), payload), []byte("pad!")...)
binary.BigEndian.PutUint16(padded6[4:6], uint16(len(padded6)-ipv6.HeaderLen))
// Corrupting the checksum and asking for it back must yield gopacket's
// packet, byte for byte.
recomputed := []struct {
name string
pkt []byte
cksum int
}{
{"v4 tcp", serialize(t, ip4tcp, tcp(ip4tcp), payload), 20 + 16},
{"v4 tcp with ip options", serialize(t, ip4opts, tcp(ip4opts), payload), 24 + 16},
{"v4 udp", serialize(t, ip4udp, udp(ip4udp), payload), 20 + 6},
{"v4 tcp header only", serialize(t, ip4tcp, tcp(ip4tcp)), 20 + 16},
{"v4 udp header only", serialize(t, ip4udp, udp(ip4udp)), 20 + 6},
{"v6 tcp", serialize(t, ip6tcp, tcp(ip6tcp), payload), 40 + 16},
{"v6 udp", serialize(t, ip6udp, udp(ip6udp), payload), 40 + 6},
{"v6 udp header only", serialize(t, ip6udp, udp(ip6udp)), 40 + 6},
{"v6 tcp behind hop-by-hop", withExtensionHeader(serialize(t, ip6tcp, tcp(ip6tcp), payload), layers.IPProtocolIPv6HopByHop, hopByHop), 48 + 16},
{"v4 tcp with bytes past the total length", trailing4, 20 + 16},
{"v6 tcp with bytes past the payload length", trailing6, 40 + 16},
{"v4 udp padded past its declared length", padded4, 20 + 6},
{"v6 udp padded past its declared length", padded6, 40 + 6},
}
for _, tt := range recomputed {
t.Run(tt.name, func(t *testing.T) {
got := append([]byte(nil), tt.pkt...)
binary.BigEndian.PutUint16(got[tt.cksum:], 0x1234)
require.NotEqual(t, tt.pkt, got)
SetTransportChecksum(got)
assert.Equal(t, tt.pkt, got)
})
}
ip4frag := v4(layers.IPProtocolTCP)
ip4frag.Flags = layers.IPv4MoreFragments
ip4later := v4(layers.IPProtocolTCP)
ip4later.FragOffset = 1
ip4icmp := v4(layers.IPProtocolICMPv4)
badIHL := serialize(t, ip4tcp, tcp(ip4tcp), payload)
badIHL[0] = 0x44 // header length 16, shorter than an ipv4 header
shortTotalLen := serialize(t, ip4tcp, tcp(ip4tcp), payload)
binary.BigEndian.PutUint16(shortTotalLen[2:4], 10) // shorter than the header it introduces
cutTCP := serialize(t, ip4tcp, tcp(ip4tcp), payload)
binary.BigEndian.PutUint16(cutTCP[2:4], 20+19) // one byte short of a tcp header
cutTCP = truncate(cutTCP, 20+19)
cutUDP := serialize(t, ip4udp, udp(ip4udp), payload)
binary.BigEndian.PutUint16(cutUDP[2:4], 20+7) // one byte short of a udp header
cutUDP = truncate(cutUDP, 20+7)
// Two bytes short, so a transport header survives whole and the minimum
// length check cannot stand in for the bounds check.
cutV6 := truncate(serialize(t, ip6tcp, tcp(ip6tcp), payload), 62)
fragment := [8]byte{0, 0, 0, 1, 0, 0, 0, 1} // next header, reserved, offset 0 with M set, id
overrun4 := serialize(t, ip4udp, udp(ip4udp), payload)
binary.BigEndian.PutUint16(overrun4[24:26], uint16(len(overrun4)-20+1)) // one byte past what ip delivered
overrun6 := serialize(t, ip6udp, udp(ip6udp), payload)
binary.BigEndian.PutUint16(overrun6[44:46], uint16(len(overrun6)-ipv6.HeaderLen+1))
shortUDPLen := serialize(t, ip4udp, udp(ip4udp), payload)
binary.BigEndian.PutUint16(shortUDPLen[24:26], 7) // shorter than the header it counts
// Where the checksum cannot be completed the packet is left as it came.
untouched := []struct {
name string
pkt []byte
cksum int
}{
{"v4 first fragment", serialize(t, ip4frag, tcp(ip4frag), payload), 20 + 16},
{"v4 later fragment", serialize(t, ip4later, tcp(ip4later), payload), 20 + 16},
{"v4 icmp", serialize(t, ip4icmp, &layers.ICMPv4{TypeCode: layers.CreateICMPv4TypeCode(8, 0), Id: 1, Seq: 1}, payload), 20 + 2},
{"v4 header length below the minimum", badIHL, 20 + 16},
{"v4 total length below the header length", shortTotalLen, 20 + 16},
{"v4 truncated below its total length", truncate(serialize(t, ip4tcp, tcp(ip4tcp), payload), 30), -1},
{"v4 tcp header cut short", cutTCP, 20 + 16},
{"v4 udp header cut short", cutUDP, -1},
{"v6 fragment", withExtensionHeader(serialize(t, ip6tcp, tcp(ip6tcp), payload), layers.IPProtocolIPv6Fragment, fragment), 48 + 16},
{"v6 truncated below its payload length", truncate(serialize(t, ip6tcp, tcp(ip6tcp), payload), 50), -1},
{"v6 truncated with a whole transport header still present", cutV6, 40 + 16},
{"v6 extension header chain longer than the walk", extChain(9, 0), 112 + 16},
{"v6 extension header chain running past the packet", extChain(8, 255), 104 + 16},
{"v4 udp length past the end of the datagram", overrun4, 20 + 6},
{"v6 udp length past the end of the datagram", overrun6, 40 + 6},
{"v4 udp length below a udp header", shortUDPLen, 20 + 6},
}
for _, tt := range untouched {
t.Run(tt.name, func(t *testing.T) {
if tt.cksum >= 0 {
binary.BigEndian.PutUint16(tt.pkt[tt.cksum:], 0x1234)
}
want := append([]byte(nil), tt.pkt...)
SetTransportChecksum(tt.pkt)
assert.Equal(t, want, tt.pkt)
})
}
t.Run("too short to carry a header", func(t *testing.T) {
for _, pkt := range [][]byte{nil, {}, {0x45}, {0x60}} {
assert.NotPanics(t, func() { SetTransportChecksum(pkt) })
}
})
t.Run("tcp checksum of zero goes out as zero", func(t *testing.T) {
pkt := serialize(t, ip4tcp, tcp(ip4tcp), gopacket.Payload{0, 0})
c := binary.BigEndian.Uint16(pkt[36:38])
require.NotZero(t, c)
// Only udp reserves zero to mean "not computed", so tcp keeps it.
binary.BigEndian.PutUint16(pkt[40:42], c)
SetTransportChecksum(pkt)
assert.Zero(t, binary.BigEndian.Uint16(pkt[36:38]))
})
t.Run("udp checksum of zero goes out as 0xffff", func(t *testing.T) {
pkt := serialize(t, ip4udp, udp(ip4udp), gopacket.Payload{0, 0})
c := binary.BigEndian.Uint16(pkt[26:28])
require.NotZero(t, c)
// The one's complement sum is now 0xffff - c; adding c to the payload
// makes it 0xffff, whose complement is zero.
binary.BigEndian.PutUint16(pkt[28:30], c)
SetTransportChecksum(pkt)
assert.Equal(t, uint16(0xffff), binary.BigEndian.Uint16(pkt[26:28]))
})
}
func TestFold(t *testing.T) {
// 0xffff is the fold's fixed point, so a loop bound one notch tight never
// terminates on it.
for _, tt := range []struct {
in uint32
want uint16
}{
{0, 0},
{0xffff, 0xffff},
{0x10000, 1},
{0x1fffe, 0xffff},
{0xffffffff, 0xffff},
} {
assert.Equal(t, tt.want, fold(tt.in))
}
}
+31 -8
View File
@@ -2,11 +2,16 @@ package iputil
import (
"encoding/binary"
"errors"
"golang.org/x/net/ipv4"
"golang.org/x/net/ipv6"
)
// ErrIPv6CouldNotFindPayload is returned when the ipv6 extension header chain is truncated before a terminal
// upper layer protocol is reached.
var ErrIPv6CouldNotFindPayload = errors.New("could not find payload in ipv6 packet")
const (
// MaxIPv4RejectPacketSize is the largest IPv4 reject packet:
// - 20 byte ipv4 header
@@ -199,8 +204,8 @@ func ipv4CreateRejectTCPPacket(packet []byte, out []byte) []byte {
}
func ipv6CreateRejectPacket(packet []byte, out []byte) []byte {
proto, offset, isFragment := ipv6FindUpperProtocol(packet)
if isFragment {
proto, offset, isFragment, _, err := IPv6FindUpperProtocol(packet)
if err != nil || isFragment {
return nil
}
switch proto {
@@ -333,7 +338,18 @@ func ipv6CreateRejectTCPPacket(packet []byte, out []byte, offset int) []byte {
return out
}
func ipv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragment bool) {
// IPv6FindUpperProtocol walks the ipv6 extension header chain and returns the upper layer protocol, the
// offset it begins at, and whether the packet is a non-first fragment. Only the RFC 8200 and IANA extension
// headers below are walked. Everything else, including Mobility (135), HIP (139), Shim6 (140), experimental
// 253/254, and real upper layer protocols like SCTP or GRE, is terminal. Walking those as extension headers
// is a firewall bypass, so they fail closed. For a non-first fragment the returned protocol is the fragmented
// protocol and offset points at the fragment header, there is no transport header to locate. Returns
// ErrIPv6CouldNotFindPayload if packet is smaller than an ipv6 header or the chain is truncated before a
// terminal protocol is reached.
func IPv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragment, anyFragment bool, err error) {
if len(packet) < ipv6.HeaderLen {
return 0, 0, false, false, ErrIPv6CouldNotFindPayload
}
nextHeader = packet[6]
offset = ipv6.HeaderLen
@@ -341,30 +357,37 @@ func ipv6FindUpperProtocol(packet []byte) (nextHeader uint8, offset int, isFragm
switch nextHeader {
case 0, 43, 60: // Hop-by-Hop, Routing, Destination
if len(packet) < offset+2 {
return nextHeader, offset, isFragment
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
}
nextHeader = packet[offset]
offset += (int(packet[offset+1]) + 1) << 3
case 44: // Fragment
if len(packet) < offset+8 {
return nextHeader, offset, isFragment
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
}
anyFragment = true
// Non-first fragments carry no transport header, report the fragmented protocol and stop
if packet[offset+2] != 0 || packet[offset+3]&0xf8 != 0 {
isFragment = true
return packet[offset], offset, true, anyFragment, nil
}
nextHeader = packet[offset]
offset += 8
case 51: // AH
if len(packet) < offset+2 {
return nextHeader, offset, isFragment
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
}
nextHeader = packet[offset]
offset += (int(packet[offset+1]) + 2) << 2
default:
return nextHeader, offset, isFragment
// A prior extension header can declare a length that advances offset past the packet. The terminal
// protocol's header isn't actually here, so treat the chain as truncated rather than classifying it.
if offset > len(packet) {
return nextHeader, offset, isFragment, anyFragment, ErrIPv6CouldNotFindPayload
}
return nextHeader, offset, isFragment, anyFragment, nil
}
}
}
+61
View File
@@ -6,6 +6,7 @@ import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/net/ipv4"
"golang.org/x/net/ipv6"
)
@@ -474,3 +475,63 @@ func TestCreateICMPEchoResponse_IPv6_NotICMPv6(t *testing.T) {
result := CreateICMPEchoResponse(packet, out)
assert.Nil(t, result)
}
func Test_IPv6FindUpperProtocol(t *testing.T) {
src := net.ParseIP("fd00::1")
dst := net.ParseIP("fd00::2")
// 8 byte extension/transport stand-ins, first byte is the next header, second is the length field
extToTCP := []byte{6, 0, 0, 0, 0, 0, 0, 0} // len 0 -> 8 bytes, next = TCP
extToUDP := []byte{17, 0, 0, 0, 0, 0, 0, 0} // len 0 -> 8 bytes, next = UDP
extToRouting := []byte{43, 0, 0, 0, 0, 0, 0, 0} // len 0 -> 8 bytes, next = Routing
ahToUDP := []byte{17, 0, 0, 0, 0, 0, 0, 0} // AH len 0 -> (0+2)<<2 = 8 bytes, next = UDP
firstFragToUDP := []byte{17, 0, 0, 1, 0, 0, 0, 1} // frag offset 0, M=1, next = UDP
nonFirstFrag := []byte{17, 0, 0, 9, 0, 0, 0, 1} // frag offset non-zero, next = UDP
transport := []byte{0, 80, 1, 187, 0, 0, 0, 0} // stand-in bytes, IPv6FindUpperProtocol never reads ports
tests := []struct {
name string
nextHeader uint8
payload []byte
wantProto uint8
wantOffset int
wantFragment bool
wantAnyFrag bool
wantErr error
}{
{"plain udp", 17, transport, 17, ipv6.HeaderLen, false, false, nil},
{"hop-by-hop then tcp", 0, append(extToTCP, transport...), 6, ipv6.HeaderLen + 8, false, false, nil},
{"routing then tcp", 43, append(extToTCP, transport...), 6, ipv6.HeaderLen + 8, false, false, nil},
{"destination then udp", 60, append(extToUDP, transport...), 17, ipv6.HeaderLen + 8, false, false, nil},
{"hop-by-hop, routing, then tcp", 0, append(append(extToRouting, extToTCP...), transport...), 6, ipv6.HeaderLen + 16, false, false, nil},
{"ah then udp", 51, append(ahToUDP, transport...), 17, ipv6.HeaderLen + 8, false, false, nil},
{"first fragment walks to transport", 44, append(firstFragToUDP, transport...), 17, ipv6.HeaderLen + 8, false, true, nil},
{"non-first fragment stops", 44, append(nonFirstFrag, transport...), 17, ipv6.HeaderLen, true, true, nil},
{"unknown protocol is terminal", 132, transport, 132, ipv6.HeaderLen, false, false, nil}, // SCTP
{"truncated extension header", 0, nil, 0, ipv6.HeaderLen, false, false, ErrIPv6CouldNotFindPayload},
// Destination Options with a declared length (255+1)*8 = 2048 that runs past the 48 byte buffer, next = SCTP
{"extension length past buffer", 60, []byte{132, 255, 0, 0, 0, 0, 0, 0}, 132, ipv6.HeaderLen + 2048, false, false, ErrIPv6CouldNotFindPayload},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
packet := makeIPv6Packet(src, dst, tt.nextHeader, tt.payload)
proto, offset, isFragment, anyFragment, err := IPv6FindUpperProtocol(packet)
if tt.wantErr != nil {
assert.ErrorIs(t, err, tt.wantErr)
return
}
require.NoError(t, err)
assert.Equal(t, tt.wantProto, proto)
assert.Equal(t, tt.wantOffset, offset)
assert.Equal(t, tt.wantFragment, isFragment)
assert.Equal(t, tt.wantAnyFrag, anyFragment)
})
}
// A packet smaller than an ipv6 header must error rather than panic reading byte 6
t.Run("shorter than ipv6 header", func(t *testing.T) {
_, _, _, _, err := IPv6FindUpperProtocol(make([]byte, 6))
assert.ErrorIs(t, err, ErrIPv6CouldNotFindPayload)
})
}
+28 -1
View File
@@ -19,6 +19,7 @@ import (
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/logging"
"github.com/slackhq/nebula/nicfinder"
"github.com/slackhq/nebula/udp"
"github.com/slackhq/nebula/util"
)
@@ -40,6 +41,10 @@ type LightHouse struct {
// addresses rather than whatever this machine's NICs happen to be. Set it before Start.
localAddrsFn func(*LocalAllowList) []netip.Addr
// lastLocalAddrsErr is the previous localAddrsFn failure. Enumeration runs on every update, so an
// unchanged failure is demoted to Debug rather than warning every lighthouse.interval forever.
lastLocalAddrsErr atomic.Pointer[string]
// Local cache of answers from light houses
// map of vpn addr to answers
addrMap map[netip.Addr]*RemoteList
@@ -112,7 +117,9 @@ func NewLightHouseFromConfig(ctx context.Context, l *slog.Logger, c *config.C, c
l: l,
}
h.localAddrsFn = func(al *LocalAllowList) []netip.Addr {
return localAddrs(h.l, al)
addrs, err := nicfinder.Find(ctx, h.l, al)
h.logLocalAddrsErr(err)
return addrs
}
lighthouses := make([]netip.Addr, 0)
@@ -913,6 +920,26 @@ func (lh *LightHouse) TriggerUpdate() {
}
}
// logLocalAddrsErr reports a localAddrs failure at Warn the first time it is seen and at Debug while
// it persists unchanged, so a permanent failure does not warn on every update forever.
func (lh *LightHouse) logLocalAddrsErr(err error) {
if err == nil {
lh.lastLocalAddrsErr.Store(nil)
return
}
msg := err.Error()
prev := lh.lastLocalAddrsErr.Swap(&msg)
if prev != nil && *prev == msg {
if lh.l.Enabled(context.Background(), slog.LevelDebug) {
lh.l.Debug("Failed to collect local addresses to advertise", "error", err)
}
return
}
lh.l.Warn("Failed to collect local addresses to advertise", "error", err)
}
func (lh *LightHouse) SendUpdate() {
var v4 []*V4AddrPort
var v6 []*V6AddrPort
+31
View File
@@ -1,7 +1,9 @@
package nebula
import (
"bytes"
"encoding/binary"
"errors"
"fmt"
"net/netip"
"testing"
@@ -738,3 +740,32 @@ func TestLighthouse_DeletesWork(t *testing.T) {
out = lh.Query(testHost)
assert.Nil(t, out)
}
func TestLightHouse_logLocalAddrsErr(t *testing.T) {
out := &bytes.Buffer{}
lh := &LightHouse{l: test.NewLoggerWithOutput(out)}
// The first sighting of a failure warns.
lh.logLocalAddrsErr(errors.New("permission denied"))
assert.Contains(t, out.String(), "level=WARN")
assert.Contains(t, out.String(), "permission denied")
// Repeating unchanged does not warn again, which is what keeps a permanent failure from warning
// on every lighthouse.interval for the life of the process.
out.Reset()
lh.logLocalAddrsErr(errors.New("permission denied"))
assert.NotContains(t, out.String(), "level=WARN")
// A different failure is a new event and warns.
out.Reset()
lh.logLocalAddrsErr(errors.New("something else"))
assert.Contains(t, out.String(), "level=WARN")
assert.Contains(t, out.String(), "something else")
// Recovering resets, so the same failure returning later warns again.
out.Reset()
lh.logLocalAddrsErr(nil)
assert.Empty(t, out.String())
lh.logLocalAddrsErr(errors.New("something else"))
assert.Contains(t, out.String(), "level=WARN")
}
+15 -8
View File
@@ -11,6 +11,7 @@ import (
"time"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/noiseutil"
"github.com/slackhq/nebula/overlay"
"github.com/slackhq/nebula/sshd"
"github.com/slackhq/nebula/udp"
@@ -20,14 +21,17 @@ import (
type m = map[string]any
// maxRoutines caps routines below the RejectHeadroom nonce gap so concurrent senders can't race the counter past wrap.
const maxRoutines = 1 << 16
// The reject headroom must exceed every sender that can be mid-reservation at once, about two per routine.
const _ = noiseutil.RejectHeadroom - 4*maxRoutines
func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, deviceFactory overlay.DeviceFactory) (retcon *Control, reterr error) {
ctx, cancel := context.WithCancel(context.Background())
// The goroutines started below stop only when this context does, and only a caller holding the
// Control can arrange that. Cancel whenever we are not handing one back, which covers an error
// and a config test alike: a config test used to leave the lighthouse query worker, and a
// hostname resolver per dns named static host, running for the life of the process.
// Automatically cancel the context if Main returns an error, to signal all created goroutines to quit.
defer func() {
if retcon == nil {
if reterr != nil {
cancel()
}
}()
@@ -84,9 +88,6 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
if routines < 1 {
routines = 1
}
if routines > 1 {
l.Info("Using multiple routines", "routines", routines)
}
} else {
// deprecated and undocumented
tunQueues := c.GetInt("tun.routines", 1)
@@ -96,6 +97,12 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
l.Warn("Setting tun.routines and listen.routines is deprecated. Use `routines` instead", "routines", routines)
}
}
if routines > maxRoutines {
l.Warn("Using multiple routines", "routines", maxRoutines, "clamped", true, "requestedRoutines", routines)
routines = maxRoutines
} else if routines > 1 {
l.Info("Using multiple routines", "routines", routines)
}
// EXPERIMENTAL
// Intentionally not documented yet while we do more testing and determine
-82
View File
@@ -1,82 +0,0 @@
package nebula
import (
"fmt"
"net/netip"
"os"
"path/filepath"
"testing"
"time"
"github.com/slackhq/nebula/cert"
cert_test "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/require"
"go.uber.org/goleak"
)
// TestMain_ConfigTestReleasesItsGoroutines pins the rule that Main only leaves goroutines running
// when it hands back a Control to stop them with.
//
// A config test gets no Control, so anything it started had nothing to stop it: the lighthouse
// query worker, and a hostname resolver per dns named static host, ran for the life of the
// process. That matters to every embedder that validates a config in process rather than by
// exec'ing, dnclient and the apple clients included, because they do it on each config load and
// the leak accumulates.
func TestMain_ConfigTestReleasesItsGoroutines(t *testing.T) {
defer goleak.VerifyNone(t, goleak.IgnoreCurrent())
l := test.NewLogger()
dir := t.TempDir()
before := time.Now().Add(-time.Hour)
after := time.Now().Add(time.Hour)
ca, _, caKey, caPEM := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, before, after, nil, nil, nil)
networks := []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")}
_, _, keyPEM, certPEM := cert_test.NewTestCert(
cert.Version2, cert.Curve_CURVE25519, ca, caKey, "config-test", before, after, networks, nil, nil)
caPath := filepath.Join(dir, "ca.pem")
certPath := filepath.Join(dir, "cert.pem")
keyPath := filepath.Join(dir, "key.pem")
require.NoError(t, os.WriteFile(caPath, caPEM, 0o600))
require.NoError(t, os.WriteFile(certPath, certPEM, 0o600))
require.NoError(t, os.WriteFile(keyPath, keyPEM, 0o600))
// A static host by address, not by name: the query worker is the goroutine under test and a
// hostname would drag a real dns lookup into a unit test.
configBody := fmt.Sprintf(`
pki:
ca: %s
cert: %s
key: %s
static_host_map:
"10.0.0.2": ["192.0.2.1:4242"]
lighthouse:
hosts:
- "10.0.0.2"
listen:
host: 127.0.0.1
port: 0
tun:
disabled: true
firewall:
outbound:
- port: any
proto: any
host: any
inbound:
- port: any
proto: any
host: any
`, caPath, certPath, keyPath)
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.yml"), []byte(configBody), 0o600))
c := config.NewC(l)
require.NoError(t, c.Load(dir))
ctrl, err := Main(c, true, "config-test", l, nil)
require.NoError(t, err)
require.Nil(t, ctrl, "a config test hands back nothing to stop, so it must stop itself")
}
+13 -4
View File
@@ -14,7 +14,8 @@ type MessageMetrics struct {
rxUnknown metrics.Counter
txUnknown metrics.Counter
rxInvalid metrics.Counter
rxInvalid metrics.Counter
txExhausted metrics.Counter
}
func (m *MessageMetrics) Rx(t header.MessageType, s header.MessageSubType, i int64) {
@@ -41,6 +42,13 @@ func (m *MessageMetrics) RxInvalid(i int64) {
}
}
// TxExhausted counts outbound packets dropped because the tunnel's message counter is spent.
func (m *MessageMetrics) TxExhausted(i int64) {
if m != nil && m.txExhausted != nil {
m.txExhausted.Inc(i)
}
}
func newMessageMetrics() *MessageMetrics {
gen := func(t string) [][]metrics.Counter {
return [][]metrics.Counter{
@@ -61,9 +69,10 @@ func newMessageMetrics() *MessageMetrics {
rx: gen("rx"),
tx: gen("tx"),
rxUnknown: metrics.GetOrRegisterCounter("messages.rx.other", nil),
txUnknown: metrics.GetOrRegisterCounter("messages.tx.other", nil),
rxInvalid: metrics.GetOrRegisterCounter("messages.rx.invalid", nil),
rxUnknown: metrics.GetOrRegisterCounter("messages.rx.other", nil),
txUnknown: metrics.GetOrRegisterCounter("messages.tx.other", nil),
rxInvalid: metrics.GetOrRegisterCounter("messages.rx.invalid", nil),
txExhausted: metrics.GetOrRegisterCounter("messages.tx.exhausted", nil),
}
}
+39
View File
@@ -0,0 +1,39 @@
//go:build !android
package nicfinder
import (
"context"
"errors"
"fmt"
"log/slog"
"net"
"net/netip"
)
func localAddrs(ctx context.Context, l *slog.Logger, filter Filter) ([]netip.Addr, error) {
ifaces, err := net.Interfaces()
if err != nil {
return nil, fmt.Errorf("failed to enumerate local interfaces: %w", err)
}
var out []netip.Addr
var errs []error
for _, i := range ifaces {
if !allowName(ctx, l, filter, i.Name) {
continue
}
addrs, err := i.Addrs()
if err != nil {
errs = append(errs, fmt.Errorf("failed to get addresses for %s: %w", i.Name, err))
continue
}
for _, rawAddr := range addrs {
if addr, ok := allowedAddr(ctx, l, filter, rawAddr); ok {
out = append(out, addr)
}
}
}
return out, errors.Join(errs...)
}
+104
View File
@@ -0,0 +1,104 @@
//go:build android
package nicfinder
import (
"context"
"errors"
"fmt"
"log/slog"
"net/netip"
"os"
"github.com/DefinedNet/netlink"
"github.com/DefinedNet/netlink/nl"
"golang.org/x/sys/unix"
)
// Android denies bind() on netlink route sockets and RTM_GETLINK, so net.Interfaces fails with EACCES.
// Enumerate with an RTM_GETADDR dump on an unbound socket instead.
// localInterfaces returns every interface that has an address, with its addresses. It lists
// addresses over a netlink socket that is never bound and names each index with SIOCGIFNAME, the
// same operations Bionic's getifaddrs uses for app UIDs, which the sandbox permits.
func localAddrs(ctx context.Context, l *slog.Logger, filter Filter) ([]netip.Addr, error) {
var errs []error
addrs, err := netlinkAddrList()
if err != nil {
if !errors.Is(err, netlink.ErrDumpInterrupted) {
return nil, fmt.Errorf("failed to enumerate local interfaces: %w", err)
}
// The address table kept changing under the dump and netlink gave up retrying. What it
// returned is still worth advertising.
errs = append(errs, fmt.Errorf("local addresses may be incomplete: %w", err))
}
// Any socket will do for SIOCGIFNAME.
sfd, err := unix.Socket(unix.AF_INET, unix.SOCK_DGRAM|unix.SOCK_CLOEXEC, 0)
if err != nil {
return nil, os.NewSyscallError("socket", err)
}
defer unix.Close(sfd)
var out []netip.Addr
nameAllowed := make(map[int]bool) // by interface index, once named
for _, a := range addrs {
allow, named := nameAllowed[a.LinkIndex]
if (named && !allow) || a.IPNet == nil {
continue
}
addr, ok := allowedAddr(ctx, l, filter, a.IPNet)
if !ok {
continue
}
if !named {
name, err := interfaceNameByIndex(sfd, a.LinkIndex)
// ENODEV means the interface went away after the dump. Any other failure is reported
// and the interface skipped, so the rest are still returned.
if err != nil && !errors.Is(err, unix.ENODEV) {
errs = append(errs, fmt.Errorf("failed to resolve name of interface %d: %w", a.LinkIndex, err))
}
allow = err == nil && allowName(ctx, l, filter, name)
nameAllowed[a.LinkIndex] = allow
}
if allow {
out = append(out, addr)
}
}
return out, errors.Join(errs...)
}
// netlinkAddrList is netlink.AddrList over a socket that is never bound. The kernel autobinds it
// on the first send, which the sandbox permits.
func netlinkAddrList() ([]netlink.Addr, error) {
fd, err := unix.Socket(unix.AF_NETLINK, unix.SOCK_RAW|unix.SOCK_CLOEXEC, unix.NETLINK_ROUTE)
if err != nil {
return nil, os.NewSyscallError("socket", err)
}
s, err := nl.NewNetlinkSocketFromFd(fd)
if err != nil {
unix.Close(fd)
return nil, err
}
h := netlink.NewHandleFromSockets(
map[int]*nl.SocketHandle{unix.NETLINK_ROUTE: {Socket: s}},
netlink.HandleOptions{RetryInterrupted: true},
)
defer h.Close()
if err := h.SetSocketTimeout(netlink.GetSocketTimeout()); err != nil {
return nil, err
}
return h.AddrList(nil, netlink.FAMILY_ALL)
}
// interfaceNameByIndex resolves an interface index with SIOCGIFNAME.
func interfaceNameByIndex(fd int, index int) (string, error) {
var ifr unix.Ifreq
ifr.SetUint32(uint32(index)) // ifr_ifindex is in the ifr_ifru union
if err := unix.IoctlIfreq(fd, unix.SIOCGIFNAME, &ifr); err != nil {
return "", err
}
return ifr.Name(), nil
}
+59
View File
@@ -0,0 +1,59 @@
package nicfinder
import (
"context"
"log/slog"
"net"
"net/netip"
"github.com/slackhq/nebula/logging"
)
// Filter decides which interfaces, by name, and which of their addresses Find returns.
type Filter interface {
AllowName(string) bool
Allow(netip.Addr) bool
}
// Find returns the addresses of this host's network interfaces that allowList accepts, with
// loopback and link-local addresses left out. When only some interfaces could be read it returns
// their addresses alongside an error for the rest. allowList must not be nil; it is called as
// given, so a nil pointer of a type whose methods accept a nil receiver is fine.
func Find(ctx context.Context, l *slog.Logger, allowList Filter) ([]netip.Addr, error) {
return localAddrs(ctx, l, allowList)
}
// allowName asks filter about an interface name and traces the answer.
func allowName(ctx context.Context, l *slog.Logger, filter Filter, name string) bool {
allow := filter.AllowName(name)
if l.Enabled(ctx, logging.LevelTrace) {
l.Log(ctx, logging.LevelTrace, "localAllowList.AllowName", "interfaceName", name, "allow", allow)
}
return allow
}
// allowedAddr converts one address a platform reports and says whether to advertise it: it must
// be a *net.IPNet or *net.IPAddr carrying an IP, not be loopback or link-local, and be accepted by
// filter, whose answer is traced. IPv4 comes back unmapped.
func allowedAddr(ctx context.Context, l *slog.Logger, filter Filter, rawAddr net.Addr) (netip.Addr, bool) {
var ip net.IP
switch v := rawAddr.(type) {
case *net.IPNet:
ip = v.IP
case *net.IPAddr:
ip = v.IP
}
addr, ok := netip.AddrFromSlice(ip)
if !ok {
return netip.Addr{}, false
}
addr = addr.Unmap()
if addr.IsLoopback() || addr.IsLinkLocalUnicast() {
return netip.Addr{}, false
}
isAllowed := filter.Allow(addr)
if l.Enabled(ctx, logging.LevelTrace) {
l.Log(ctx, logging.LevelTrace, "localAllowList.Allow", "localAddr", addr, "allowed", isAllowed)
}
return addr, isAllowed
}
+117
View File
@@ -0,0 +1,117 @@
package nicfinder
import (
"context"
"net"
"net/netip"
"regexp"
"testing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
)
type allowAll struct{}
func (allowAll) Allow(netip.Addr) bool { return true }
func (allowAll) AllowName(string) bool { return true }
type testAllowList struct {
bannedName *regexp.Regexp
bannedAddr netip.Addr
}
func (l *testAllowList) Allow(addr netip.Addr) bool {
return addr != l.bannedAddr
}
func (l *testAllowList) AllowName(n string) bool {
return !l.bannedName.MatchString(n)
}
func newTestAllowList(bannedName string) *testAllowList {
return &testAllowList{
bannedName: regexp.MustCompile(bannedName),
}
}
func TestAllowedAddr(t *testing.T) {
ctx, l := context.Background(), test.NewLogger()
allowed := func(filter Filter, addrs []net.Addr) []netip.Addr {
var out []netip.Addr
for _, rawAddr := range addrs {
if addr, ok := allowedAddr(ctx, l, filter, rawAddr); ok {
out = append(out, addr)
}
}
return out
}
addrs := []net.Addr{
&net.IPNet{IP: net.ParseIP("127.0.0.1"), Mask: net.CIDRMask(8, 32)},
&net.IPNet{IP: net.ParseIP("::1"), Mask: net.CIDRMask(128, 128)},
&net.IPNet{IP: net.ParseIP("10.0.0.5"), Mask: net.CIDRMask(24, 32)},
&net.IPNet{IP: net.ParseIP("169.254.10.10"), Mask: net.CIDRMask(16, 32)},
&net.IPNet{IP: net.ParseIP("fe80::1"), Mask: net.CIDRMask(64, 128)},
&net.IPAddr{IP: net.ParseIP("fd00::5")},
&net.IPNet{},
otherAddr{},
nil,
}
// Loopback, link local and anything without an IP are dropped, and IPv4 comes back unmapped
// even though net.ParseIP hands over a 16 byte slice.
assert.Equal(t, []netip.Addr{
netip.MustParseAddr("10.0.0.5"),
netip.MustParseAddr("fd00::5"),
}, allowed(allowAll{}, addrs))
// The filter sees the unmapped form and can drop a single address.
al := newTestAllowList("none")
al.bannedAddr = netip.MustParseAddr("10.0.0.5")
assert.Equal(t, []netip.Addr{netip.MustParseAddr("fd00::5")}, allowed(al, addrs))
}
// otherAddr is a net.Addr that is neither *net.IPNet nor *net.IPAddr. netlink.Addr is one such
// type: it satisfies net.Addr through an embedded *net.IPNet, and passing it directly once
// silently produced no addresses.
type otherAddr struct{}
func (otherAddr) Network() string { return "other" }
func (otherAddr) String() string { return "other" }
// hostFilter counts AllowName calls and can reject every name or every address.
type hostFilter struct {
rejectNames bool
rejectAddrs bool
asked map[string]int
}
func (f *hostFilter) Allow(netip.Addr) bool { return !f.rejectAddrs }
func (f *hostFilter) AllowName(name string) bool {
f.asked[name]++
return !f.rejectNames
}
// Runs this platform's localAddrs against the host and checks it honors the whole filter.
func TestLocalAddrsAppliesFilter(t *testing.T) {
ctx, l := context.Background(), test.NewLogger()
f := &hostFilter{asked: map[string]int{}}
all, err := localAddrs(ctx, l, f)
if len(f.asked) == 0 {
t.Skipf("no interfaces to enumerate here: %v", err)
}
for name, n := range f.asked {
assert.Equal(t, 1, n, "AllowName asked %d times for %s", n, name)
}
// Loopback and link-local never come back, whatever the filter says.
for _, addr := range all {
assert.False(t, addr.IsLoopback() || addr.IsLinkLocalUnicast(), "%s", addr)
}
none, _ := localAddrs(ctx, l, &hostFilter{rejectNames: true, asked: map[string]int{}})
assert.Empty(t, none, "addresses got past a filter that rejects every interface name")
none, _ = localAddrs(ctx, l, &hostFilter{rejectAddrs: true, asked: map[string]int{}})
assert.Empty(t, none, "addresses got past a filter that rejects every address")
}
+3
View File
@@ -25,6 +25,9 @@ func (s *CipherStateAESGCM) EncryptDanger(out, ad, plaintext []byte, n uint64, n
if s == nil {
return nil, errors.New("no cipher state available to encrypt")
}
if n >= RejectAfterMessages {
return nil, ErrMessageCounterExhausted
}
nb[0] = 0
nb[1] = 0
nb[2] = 0
+3
View File
@@ -24,6 +24,9 @@ func (s *CipherStateChaChaPoly) EncryptDanger(out, ad, plaintext []byte, n uint6
if s == nil {
return nil, errors.New("no cipher state available to encrypt")
}
if n >= RejectAfterMessages {
return nil, ErrMessageCounterExhausted
}
nb[0] = 0
nb[1] = 0
nb[2] = 0
+11
View File
@@ -1,11 +1,22 @@
package noiseutil
import (
"errors"
"fmt"
"math"
"github.com/flynn/noise"
)
// RejectHeadroom is the wrap gap for senders racing the counter, sized large enough for any routine count.
const RejectHeadroom = uint64(1) << 40
// RejectAfterMessages is the nonce ceiling: encrypting stops RejectHeadroom short of the wrap.
const RejectAfterMessages = math.MaxUint64 - RejectHeadroom
// ErrMessageCounterExhausted is returned by EncryptDanger once the nonce reaches RejectAfterMessages.
var ErrMessageCounterExhausted = errors.New("message counter exhausted")
// CipherState is the post-handshake AEAD cipher used for the data plane.
// Each supported cipher has its own concrete implementation in this package with the nonce endianness hardcoded,
// so the encrypt/decrypt fast path avoids interface dispatch on the byte order.
+19
View File
@@ -1,6 +1,7 @@
package noiseutil
import (
"math"
"testing"
"github.com/flynn/noise"
@@ -89,6 +90,24 @@ func roundtrip(t *testing.T, enc, dec CipherState) {
assert.Equal(t, 16, enc.Overhead())
}
func TestEncryptRejectsExhaustedCounter(t *testing.T) {
// Pin the headroom below the uint64 wrap so a typo can't silently move the ceiling.
require.Equal(t, uint64(1)<<40, RejectHeadroom)
require.Equal(t, math.MaxUint64-RejectHeadroom, RejectAfterMessages)
encA, _ := buildCipherStates(t, CipherAESGCM)
encC, _ := buildCipherStates(t, noise.CipherChaChaPoly)
nb := make([]byte, 12)
for _, cs := range []CipherState{NewCipherStateAESGCM(encA), NewCipherStateChaChaPoly(encC)} {
_, err := cs.EncryptDanger(nil, nil, []byte("x"), RejectAfterMessages-1, nb)
require.NoError(t, err)
_, err = cs.EncryptDanger(nil, nil, []byte("x"), RejectAfterMessages, nb)
require.ErrorIs(t, err, ErrMessageCounterExhausted)
}
}
func BenchmarkCipherStateEncryptAESGCM(b *testing.B) {
enc, _ := buildCipherStatesB(b, CipherAESGCM)
benchEncryptCipherState(b, NewCipherState(enc, CipherAESGCM))
+43 -85
View File
@@ -13,6 +13,7 @@ import (
"github.com/slackhq/nebula/firewall"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/iputil"
"golang.org/x/net/ipv4"
)
@@ -299,7 +300,6 @@ var (
ErrIPv4InvalidHeaderLength = errors.New("invalid ipv4 header length")
ErrIPv4PacketTooShort = errors.New("ipv4 packet is too short")
ErrIPv6PacketTooShort = errors.New("ipv6 packet is too short")
ErrIPv6CouldNotFindPayload = errors.New("could not find payload in ipv6 packet")
)
// newPacket validates and parses the interesting bits for the firewall out of the ip and sub protocol headers
@@ -332,101 +332,59 @@ func parseV6(data []byte, incoming bool, fp *firewall.Packet) error {
fp.RemoteAddr, _ = netip.AddrFromSlice(data[24:40])
}
protoAt := 6 // NextHeader is at 6 bytes into the ipv6 header
offset := ipv6.HeaderLen // Start at the end of the ipv6 header
next := 0
for {
if protoAt >= dataLen {
break
// Walk the extension header chain to the upper layer protocol. iputil.IPv6FindUpperProtocol is the single
// source of truth for which headers are extension headers, so this stays in lockstep with the reject path
// and cannot drift into misreading an unknown protocol (SCTP, GRE, etc.) as a forged transport.
proto, offset, isFragment, _, err := iputil.IPv6FindUpperProtocol(data)
if err != nil {
return ErrIPv6PacketTooShort
}
fp.Protocol = proto
fp.Fragment = isFragment
if isFragment {
// Non-first fragments carry no transport header, so we have no ports to read
fp.RemotePort = 0
fp.LocalPort = 0
return nil
}
switch layers.IPProtocol(proto) {
case layers.IPProtocolICMPv6:
// An ICMPv6 message is at least type, code and checksum, 4 bytes. Only echo carries more than we read.
if dataLen < offset+4 {
return ErrIPv6PacketTooShort
}
proto := layers.IPProtocol(data[protoAt])
switch proto {
case layers.IPProtocolESP, layers.IPProtocolNoNextHeader:
fp.Protocol = uint8(proto)
fp.RemotePort = 0
fp.LocalPort = 0
fp.Fragment = false
return nil
case layers.IPProtocolICMPv6:
fp.LocalPort = 0 //incoming vs outgoing doesn't matter for icmpv6
switch data[offset] { //icmp type
case layers.ICMPv6TypeEchoRequest, layers.ICMPv6TypeEchoReply:
if dataLen < offset+6 {
return ErrIPv6PacketTooShort
}
fp.Protocol = uint8(proto)
fp.LocalPort = 0 //incoming vs outgoing doesn't matter for icmpv6
icmptype := data[offset+1]
switch icmptype {
case layers.ICMPv6TypeEchoRequest, layers.ICMPv6TypeEchoReply:
fp.RemotePort = binary.BigEndian.Uint16(data[offset+4 : offset+6]) //identifier
default:
fp.RemotePort = 0
}
fp.Fragment = false
return nil
case layers.IPProtocolTCP, layers.IPProtocolUDP:
if dataLen < offset+4 {
return ErrIPv6PacketTooShort
}
fp.Protocol = uint8(proto)
if incoming {
fp.RemotePort = binary.BigEndian.Uint16(data[offset : offset+2])
fp.LocalPort = binary.BigEndian.Uint16(data[offset+2 : offset+4])
} else {
fp.LocalPort = binary.BigEndian.Uint16(data[offset : offset+2])
fp.RemotePort = binary.BigEndian.Uint16(data[offset+2 : offset+4])
}
fp.Fragment = false
return nil
case layers.IPProtocolIPv6Fragment:
// Fragment header is 8 bytes, need at least offset+4 to read the offset field
if dataLen < offset+8 {
return ErrIPv6PacketTooShort
}
// Check if this is the first fragment
fragmentOffset := binary.BigEndian.Uint16(data[offset+2:offset+4]) &^ uint16(0x7) // Remove the reserved and M flag bits
if fragmentOffset != 0 {
// Non-first fragment, use what we have now and stop processing
fp.Protocol = data[offset]
fp.Fragment = true
fp.RemotePort = 0
fp.LocalPort = 0
return nil
}
// The next loop should be the transport layer since we are the first fragment
next = 8 // Fragment headers are always 8 bytes
case layers.IPProtocolAH:
// Auth headers, used by IPSec, have a different meaning for header length
if dataLen <= offset+1 {
break
}
next = (int(data[offset+1]) + 2) << 2
fp.RemotePort = binary.BigEndian.Uint16(data[offset+4 : offset+6]) //identifier
default:
// Normal ipv6 header length processing
if dataLen <= offset+1 {
break
}
next = (int(data[offset+1]) + 1) << 3
fp.RemotePort = 0
}
if next <= 0 {
// Safety check, each ipv6 header has to be at least 8 bytes
next = 8
case layers.IPProtocolTCP, layers.IPProtocolUDP:
if dataLen < offset+4 {
return ErrIPv6PacketTooShort
}
if incoming {
fp.RemotePort = binary.BigEndian.Uint16(data[offset : offset+2])
fp.LocalPort = binary.BigEndian.Uint16(data[offset+2 : offset+4])
} else {
fp.LocalPort = binary.BigEndian.Uint16(data[offset : offset+2])
fp.RemotePort = binary.BigEndian.Uint16(data[offset+2 : offset+4])
}
protoAt = offset
offset = offset + next
default:
// don't set ports for protocols Nebula doesn't inspect
fp.RemotePort = 0
fp.LocalPort = 0
}
return ErrIPv6CouldNotFindPayload
return nil
}
func parseV4(data []byte, incoming bool, fp *firewall.Packet) error {
+89 -9
View File
@@ -14,6 +14,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/net/ipv4"
"golang.org/x/net/ipv6"
)
func Test_newPacket(t *testing.T) {
@@ -115,12 +116,12 @@ func Test_newPacket_v6(t *testing.T) {
require.NoError(t, err)
err = newPacket(buffer.Bytes(), true, p)
require.ErrorIs(t, err, ErrIPv6CouldNotFindPayload)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
// A v6 packet with a hop-by-hop extension
// ICMPv6 Payload (Echo Request)
icmpLayer := layers.ICMPv6{
TypeCode: layers.ICMPv6TypeEchoRequest,
TypeCode: layers.CreateICMPv6TypeCode(layers.ICMPv6TypeEchoRequest, 0),
}
// Hop-by-Hop Extension Header
hopOption := layers.IPv6HopByHopOption{}
@@ -149,12 +150,12 @@ func Test_newPacket_v6(t *testing.T) {
// A full IPv6 header and 1 byte in the first extension, but missing
// the length byte.
err = newPacket(buffer.Bytes()[:41], true, p)
require.ErrorIs(t, err, ErrIPv6CouldNotFindPayload)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
// A full IPv6 header plus 1 full extension, but only 1 byte of the
// next layer, missing length byte
err = newPacket(buffer.Bytes()[:49], true, p)
require.ErrorIs(t, err, ErrIPv6CouldNotFindPayload)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
err = nil
// A good ICMP packet
@@ -167,7 +168,7 @@ func Test_newPacket_v6(t *testing.T) {
}
icmp := layers.ICMPv6{
TypeCode: layers.ICMPv6TypeEchoRequest,
TypeCode: layers.CreateICMPv6TypeCode(layers.ICMPv6TypeEchoRequest, 0),
Checksum: 0x1234,
}
@@ -189,6 +190,18 @@ func Test_newPacket_v6(t *testing.T) {
assert.Equal(t, uint16(0), p.LocalPort)
assert.False(t, p.Fragment)
// A minimal 4 byte non-echo ICMPv6 message (type, code, checksum), no identifier to read
icmpMin := make([]byte, ipv6.HeaderLen+4)
copy(icmpMin, buffer.Bytes()[:ipv6.HeaderLen])
icmpMin[6] = byte(layers.IPProtocolICMPv6)
icmpMin[ipv6.HeaderLen] = 1 // type 1, destination unreachable, not echo
err = newPacket(icmpMin, true, p)
require.NoError(t, err)
assert.Equal(t, uint8(layers.IPProtocolICMPv6), p.Protocol)
assert.Equal(t, uint16(0), p.RemotePort)
assert.Equal(t, uint16(0), p.LocalPort)
assert.False(t, p.Fragment)
// A good ESP packet
b := buffer.Bytes()
b[6] = byte(layers.IPProtocolESP)
@@ -213,11 +226,15 @@ func Test_newPacket_v6(t *testing.T) {
assert.Equal(t, uint16(0), p.LocalPort)
assert.False(t, p.Fragment)
// An unknown protocol packet
// An unknown protocol packet, we don't dissect it so we fail closed on its true protocol with no ports
b = buffer.Bytes()
b[6] = 255 // 255 is a reserved protocol number
err = newPacket(b, true, p)
require.ErrorIs(t, err, ErrIPv6CouldNotFindPayload)
require.NoError(t, err)
assert.Equal(t, uint8(255), p.Protocol)
assert.Equal(t, uint16(0), p.RemotePort)
assert.Equal(t, uint16(0), p.LocalPort)
assert.False(t, p.Fragment)
// A good UDP packet
ip = layers.IPv6{
@@ -334,14 +351,14 @@ func Test_newPacket_v6(t *testing.T) {
assert.Equal(t, uint16(22), p.LocalPort)
assert.False(t, p.Fragment)
// Ensure buffer bounds checking during processing
// Ensure buffer bounds checking during processing, a truncated AH header can't reach the payload
err = newPacket(b[:41], true, p)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
// Invalid AH header
b = buffer.Bytes()
err = newPacket(b, true, p)
require.ErrorIs(t, err, ErrIPv6CouldNotFindPayload)
require.ErrorIs(t, err, ErrIPv6PacketTooShort)
}
func Test_newPacket_ipv6Fragment(t *testing.T) {
@@ -675,3 +692,66 @@ func Test_newPacket_v6ExtHeaderOverflow(t *testing.T) {
// the host delivers to, not the forged 443 at the overflowed offset.
assert.Equal(t, uint16(22), p.LocalPort, "firewall must parse the real transport header, not the overflowed offset")
}
// Test_newPacket_v6ExtHeaderPastBuffer is a regression test for an extension header whose declared length
// advances the walk past the end of the packet. The upper layer protocol's header isn't actually present,
// so parseV6 must drop the packet rather than classify it as the terminal protocol with no ports.
func Test_newPacket_v6ExtHeaderPastBuffer(t *testing.T) {
p := &firewall.Packet{}
pkt := make([]byte, 48)
pkt[0] = 0x60
pkt[6] = byte(layers.IPProtocolIPv6Destination) // Destination Options
pkt[7] = 64 // hop limit
pkt[40] = byte(layers.IPProtocolSCTP) // Dest Options next header = SCTP
pkt[41] = 255 // declared length (255+1)*8 = 2048, past the 48 byte buffer
require.ErrorIs(t, newPacket(pkt, true, p), ErrIPv6PacketTooShort)
}
// Test_newPacket_v6ExtHeaderConfusion is a regression test for parseV6 walking any unrecognized
// Next Header as if it were an ipv6 extension header. A real upper layer protocol Nebula doesn't
// dissect (SCTP here) is not walkable, so applying the (len+1)*8 formula marched into the SCTP
// payload and landed on a byte that looked like UDP, forging a protocol/port pair the firewall
// would trust while the host delivered the real SCTP datagram. The fix fails closed: the packet
// is classified as its true protocol with no ports, so it only matches an `any` rule.
func Test_newPacket_v6ExtHeaderConfusion(t *testing.T) {
p := &firewall.Packet{}
pkt := make([]byte, 52)
pkt[0] = 0x60 // version 6
pkt[6] = byte(layers.IPProtocolSCTP) // NextHeader = SCTP, a real protocol, not an extension header
pkt[7] = 64 // hop limit
// Real SCTP header at offset 40. Pre-fix parseV6 walked SCTP as an extension header: byte 41 (0x00, the
// low byte of the src port below) was read as the header length, giving next=(0+1)*8=8, which landed the
// walk on byte 40 (0x11), misread as NextHeader=UDP, then bytes 48-51 as ports.
binary.BigEndian.PutUint16(pkt[40:42], 0x1100) // SCTP src port; byte 40=0x11, byte 41=0x00
binary.BigEndian.PutUint16(pkt[42:44], 445) // SCTP dst port, never read by parseV6
binary.BigEndian.PutUint16(pkt[48:50], 53) // SCTP checksum bytes, pre-fix forged RemotePort
binary.BigEndian.PutUint16(pkt[50:52], 53) // pre-fix forged LocalPort
require.NoError(t, newPacket(pkt, true, p))
assert.Equal(t, uint8(layers.IPProtocolSCTP), p.Protocol, "must classify as the true protocol, not the forged UDP")
assert.Equal(t, uint16(0), p.RemotePort)
assert.Equal(t, uint16(0), p.LocalPort)
assert.False(t, p.Fragment)
// Same confusion, but the unknown protocol sits after a real extension header. The HopByHop is walked
// correctly, then SCTP must still fail closed instead of being walked into its own payload. Protocol is
// the only assertion that discriminates the fix here, a regression that walked SCTP would misclassify it.
chained := make([]byte, 60)
chained[0] = 0x60 // version 6
chained[6] = byte(layers.IPProtocolIPv6HopByHop) // NextHeader = HopByHop extension
chained[7] = 64 // hop limit
chained[40] = byte(layers.IPProtocolSCTP) // HopByHop NextHeader = SCTP
chained[41] = 0 // HopByHop length 0 -> 8 bytes, SCTP begins at offset 48
binary.BigEndian.PutUint16(chained[48:50], 0x1100) // SCTP src port, pre-fix forged NextHeader/length bait
binary.BigEndian.PutUint16(chained[50:52], 445) // SCTP dst port, never read by parseV6
require.NoError(t, newPacket(chained, true, p))
assert.Equal(t, uint8(layers.IPProtocolSCTP), p.Protocol, "must fail closed on the unknown protocol after the extension header")
assert.Equal(t, uint16(0), p.RemotePort)
assert.Equal(t, uint16(0), p.LocalPort)
assert.False(t, p.Fragment)
}
+3 -61
View File
@@ -30,10 +30,7 @@ type tun struct {
Routes atomic.Pointer[[]Route]
routeTree atomic.Pointer[bart.Table[routing.Gateways]]
linkAddr *netroute.LinkAddr
// hostOwned means the fd arrived from the OS, which has already configured addressing, mtu
// and routes for it. NEPacketTunnelProvider on darwin does this.
hostOwned bool
l *slog.Logger
l *slog.Logger
}
type ifReq struct {
@@ -153,48 +150,8 @@ func (t *tun) deviceBytes() (o [16]byte) {
return
}
// newTunFromFd adopts a utun the host already created and configured, which is how a darwin
// network extension is handed its device. Everything about moving packets is shared with newTun,
// only the setup differs: the host owns addressing and routing here.
func newTunFromFd(c *config.C, l *slog.Logger, deviceFd int, vpnNetworks []netip.Prefix) (*tun, error) {
if err := unix.SetNonblock(deviceFd, true); err != nil {
// We own the fd from the moment it is handed to us
_ = unix.Close(deviceFd)
return nil, fmt.Errorf("failed to set the tun fd to non-blocking mode: %w", err)
}
file := os.NewFile(uintptr(deviceFd), "/dev/tun")
t := &tun{
f: file,
Device: utunNameFromFd(deviceFd),
vpnNetworks: vpnNetworks,
DefaultMTU: c.GetInt("tun.mtu", DefaultMTU),
hostOwned: true,
l: l,
}
if err := t.reload(c, true); err != nil {
_ = file.Close()
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
if err := t.reload(c, false); err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
}
})
return t, nil
}
// utunNameFromFd asks the socket what interface it is, for logs. A blank name is not worth
// failing a tunnel over, so an error just leaves it empty.
func utunNameFromFd(fd int) string {
name, err := unix.GetsockoptString(fd, unix.AF_SYS_CONTROL, _UTUN_OPT_IFNAME)
if err != nil {
return ""
}
return name
func newTunFromFd(_ *config.C, _ *slog.Logger, _ int, _ []netip.Prefix) (*tun, error) {
return nil, fmt.Errorf("newTunFromFd not supported in Darwin")
}
func (t *tun) Close() error {
@@ -205,12 +162,6 @@ func (t *tun) Close() error {
}
func (t *tun) Activate() error {
// The host handed us a configured device. Its addresses, mtu and routes come from the network
// settings it applied, and a sandboxed extension cannot change them anyway.
if t.hostOwned {
return nil
}
devName := t.deviceBytes()
s, err := unix.Socket(
@@ -424,11 +375,6 @@ func getLinkAddr(name string) (*netroute.LinkAddr, error) {
}
func (t *tun) addRoutes(logErrors bool) error {
// The route tree is still ours, the system routing table is not
if t.hostOwned {
return nil
}
routes := *t.Routes.Load()
for _, r := range routes {
@@ -458,10 +404,6 @@ func (t *tun) addRoutes(logErrors bool) error {
}
func (t *tun) removeRoutes(routes []Route) error {
if t.hostOwned {
return nil
}
for _, r := range routes {
if !r.Install {
continue
+12 -2
View File
@@ -5,6 +5,7 @@ package overlay
import (
"encoding/binary"
"errors"
"fmt"
"io"
"log/slog"
@@ -17,11 +18,11 @@ import (
"time"
"unsafe"
"github.com/DefinedNet/netlink"
"github.com/gaissmai/bart"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/util"
"github.com/vishvananda/netlink"
"golang.org/x/sys/unix"
)
@@ -483,7 +484,16 @@ func (t *tun) addIPs(link netlink.Link) error {
//iterate over remainder, remove whoever shouldn't be there
al, err := netlink.AddrList(link, netlink.FAMILY_ALL)
if err != nil {
return fmt.Errorf("failed to get tun address list: %s", err)
//RTM_GETADDR dumps the whole system, so any concurrent address change
//interrupts it - including the kernel's async tentative->preferred
//flip of an IPv6 address the AddrReplace calls above just added,
//which makes this a race against our own setup. Partial results are
//still returned; the worst case is a stale address surviving until
//the next config reload, which beats failing startup over it.
if !errors.Is(err, netlink.ErrDumpInterrupted) {
return fmt.Errorf("failed to get tun address list: %s", err)
}
t.l.Warn("tun address list dump was interrupted, stale addresses may remain")
}
for i := range al {
+29
View File
@@ -12,6 +12,7 @@ import (
"os"
"path/filepath"
"runtime"
"slices"
"sync/atomic"
"syscall"
"unsafe"
@@ -178,6 +179,7 @@ func (t *winTun) addRoutes(logErrors bool) error {
luid := winipcfg.LUID(t.tun.LUID())
routes := *t.Routes.Load()
foundDefault4 := false
carriesV6 := slices.ContainsFunc(t.vpnNetworks, func(p netip.Prefix) bool { return p.Addr().Is6() })
for _, r := range routes {
if len(r.Via) == 0 || !r.Install {
@@ -185,6 +187,9 @@ func (t *winTun) addRoutes(logErrors bool) error {
continue
}
// A v6 unsafe_route is legal under a v4-only cert; uninstalled ones put nothing on the adapter.
carriesV6 = carriesV6 || r.Cidr.Addr().Is6()
// Add our unsafe route as an on-link route to the nebula tun device.
err := luid.AddRoute(r.Cidr, unspecifiedNextHop(r.Cidr), uint32(r.Metric))
if err != nil {
@@ -206,6 +211,11 @@ func (t *winTun) addRoutes(logErrors bool) error {
}
}
return t.setMTU(luid, foundDefault4, carriesV6)
}
// setMTU applies tun.mtu per address family. The default route metric rides along on the v4 handle.
func (t *winTun) setMTU(luid winipcfg.LUID, foundDefault4, carriesV6 bool) error {
ipif, err := luid.IPInterface(windows.AF_INET)
if err != nil {
return fmt.Errorf("failed to get ip interface: %w", err)
@@ -220,6 +230,25 @@ func (t *winTun) addRoutes(logErrors bool) error {
if err := ipif.Set(); err != nil {
return fmt.Errorf("failed to set ip interface: %w", err)
}
// Windows tracks NLMTU per family and wintun sets neither, so v6 keeps the adapter default of 65535.
// Gated so a v4-only overlay under 1280 boots; a v6 one deliberately does not, as linux also refuses.
if !carriesV6 {
return nil
}
ipif6, err := luid.IPInterface(windows.AF_INET6)
if err != nil {
// No v6 on the adapter means there is no NLMTU to get wrong. A failed Set below is not the same thing.
t.l.Info("Skipping ipv6 MTU, no ipv6 interface on this adapter", "error", err)
return nil
}
ipif6.NLMTU = uint32(t.MTU)
if err := ipif6.Set(); err != nil {
return fmt.Errorf("failed to set ipv6 interface: %w", err)
}
return nil
}
+2 -3
View File
@@ -13,7 +13,6 @@ import (
"sync/atomic"
"time"
graphite "github.com/cyberdelia/go-metrics-graphite"
mp "github.com/nbrownus/go-metrics-prometheus"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
@@ -253,7 +252,7 @@ func (s *statsServer) buildRuntime(cfg statsConfig) ([]func(), *http.Server) {
// loadStatsConfig already resolved and validated the address; re-parse
// the resolved form (no DNS lookup) to get a *net.TCPAddr.
addr, _ := net.ResolveTCPAddr(cfg.graphite.protocol, cfg.graphite.resolvedAddr)
gcfg := graphite.Config{
gcfg := graphiteConfigExport{
Addr: addr,
Registry: metrics.DefaultRegistry,
FlushInterval: cfg.interval,
@@ -262,7 +261,7 @@ func (s *statsServer) buildRuntime(cfg statsConfig) ([]func(), *http.Server) {
Percentiles: []float64{0.5, 0.75, 0.95, 0.99, 0.999},
}
captureFns = append(captureFns, func() {
if err := graphite.Once(gcfg); err != nil {
if err := graphiteOnce(gcfg); err != nil {
s.l.Error("Graphite export failed", "error", err)
}
})
+1 -1
View File
@@ -371,7 +371,7 @@ func waitForListening(t *testing.T, addr string) {
})
}
// graphiteSink is a minimal TCP accept-and-discard server so graphite.Once
// graphiteSink is a minimal TCP accept-and-discard server so graphiteOnce
// calls in tests don't spam error logs or wedge on connection refused.
type graphiteSink struct {
ln net.Listener
+18 -5
View File
@@ -31,7 +31,8 @@ func procyield(cycles uint32)
const (
packetsPerRing = 1024
bytesPerPacket = 2048 - 32
// Caps tun.mtu at MTU-32 direct, MTU-64 relayed, unenforced anywhere else. 17.6MB page locked per socket.
bytesPerPacket = MTU
receiveSpins = 15
)
@@ -69,12 +70,14 @@ func NewRIOListener(l *slog.Logger, addr netip.Addr, port int) (*RIOConn, error)
err := u.bind(l, &windows.SockaddrInet6{Addr: addr.As16(), Port: port})
if err != nil {
u.close()
return nil, fmt.Errorf("bind: %w", err)
}
for i := 0; i < packetsPerRing; i++ {
err = u.insertReceiveRequest()
if err != nil {
u.close()
return nil, fmt.Errorf("init rx ring: %w", err)
}
}
@@ -342,15 +345,25 @@ func (u *RIOConn) Close() error {
return nil
}
u.close()
return nil
}
// Also unwinds a partial build from NewRIOListener, where isOpen is false and Close would no-op.
// Socket first, unlike wireguard-go: receive() re-arms every slot, so freeing the rings under a live socket
// hands the kernel freed pages for all packetsPerRing outstanding receives.
func (u *RIOConn) close() {
// WSASocket reports failure as InvalidHandle, not zero.
if u.sock != 0 && u.sock != windows.InvalidHandle {
windows.CloseHandle(u.sock)
}
u.sock = 0
windows.PostQueuedCompletionStatus(u.rx.iocp, 0, 0, nil)
windows.PostQueuedCompletionStatus(u.tx.iocp, 0, 0, nil)
u.rx.CloseAndZero()
u.tx.CloseAndZero()
if u.sock != 0 {
windows.CloseHandle(u.sock)
}
return nil
}
func (ring *ringBuffer) Push() *ringPacket {