Compare commits

..
Author SHA1 Message Date
dependabot[bot] 70e6ca6812 Bump the golang-x-dependencies group across 1 directory with 5 updates
Bumps the golang-x-dependencies group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [golang.org/x/crypto](https://github.com/golang/crypto) | `0.57.0` | `0.58.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.59.0` | `0.60.0` |
| [golang.org/x/sync](https://github.com/golang/sync) | `0.23.0` | `0.24.0` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.48.0` | `0.49.0` |
| [golang.org/x/term](https://github.com/golang/term) | `0.46.0` | `0.47.0` |



Updates `golang.org/x/crypto` from 0.57.0 to 0.58.0
- [Commits](https://github.com/golang/crypto/compare/v0.57.0...v0.58.0)

Updates `golang.org/x/net` from 0.59.0 to 0.60.0
- [Commits](https://github.com/golang/net/compare/v0.59.0...v0.60.0)

Updates `golang.org/x/sync` from 0.23.0 to 0.24.0
- [Commits](https://github.com/golang/sync/compare/v0.23.0...v0.24.0)

Updates `golang.org/x/sys` from 0.48.0 to 0.49.0
- [Commits](https://github.com/golang/sys/compare/v0.48.0...v0.49.0)

Updates `golang.org/x/term` from 0.46.0 to 0.47.0
- [Commits](https://github.com/golang/term/compare/v0.46.0...v0.47.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.58.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/net
  dependency-version: 0.60.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/sync
  dependency-version: 0.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/sys
  dependency-version: 0.49.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
- dependency-name: golang.org/x/term
  dependency-version: 0.47.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: golang-x-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-10-19 19:53:08 +00:00
Matt RichardsonandClaude Fable 5.1 8a4bd8b10f Report ssh command failures to the surface (#1959)
smoke-extra / freebsd-amd64 (push) Failing after 14s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 13s
smoke-extra / netbsd-amd64 (push) Failing after 14s
smoke-extra / openbsd-amd64 (push) Failing after 15s
smoke-extra / linux-386 (push) Failing after 14s
smoke / Run multi node smoke test (push) Failing after 1m46s
Build and test / Static checks (push) Successful in 2m38s
Build and test / Test linux (push) Failing after 1m12s
Build and test / Test linux-pkcs11 (push) Failing after 2m6s
Build and test / Test linux-boringcrypto (push) Failing after 3m0s
Build and test / Test linux-fips140 (push) Failing after 2m56s
Build and test / Cross-build linux-arm (push) Successful in 3m10s
Build and test / Cross-build linux-mips (push) Successful in 3m55s
Build and test / Cross-build linux-other (push) Successful in 3m15s
Build and test / Cross-build windows (push) Successful in 1m5s
Build and test / Cross-build freebsd (push) Successful in 1m37s
Build and test / Cross-build netbsd (push) Successful in 1m36s
Build and test / Cross-build openbsd (push) Successful in 1m35s
Build and test / Cross-build mobile (push) Successful in 3m32s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
Report ssh command failures to the surface (#1959)

The ssh debug commands told the user when something went wrong but
never told the surface that ran them, so `ssh host cmd` always exited 0
and a few failures were not reported at all. A failed command now
returns its error, a failure the user has already heard about is a
commands.Failure, and the ssh exec path exits 1 for either. The command
bodies also lose their ssh prefixes and the duplication they carried
over from ssh.go; both goldens pin that nothing else changed.

- Dispatch returns the callback's error, or shlex's; the ssh session
  logs it at debug, and not at all for a Failure the user already saw
- commands.Fail writes a message and returns it as a Failure in one step;
  an unknown command, `help` for a command that is not there, a failed
  reload and an unbalanced quote are failures
- ssh exec exit-status is 1 for a command, flag or failure error, 0
  otherwise; the e2e golden records the exit status for every case
- Clone rebinds help to the copy, so an ssh session's help lists logout
  and `help logout` and `logout -h` answer; sshd.LogoutCommand is the
  session's logout, exported for tests
- "did not understand" names the first word only, the one looked up;
  DispatchArgs takes just the tokens
- log-format writes its refusal to the user, as log-level does
- print-relays emits the relay error as a string and omits the key when
  there is none, where it printed {} or "Error":null
- list-hostmap and list-lighthouse-addrmap -json return the encode
  error; print-cert reports a certificate that will not marshal
- a flags type that is not the one registered is an internal error, not
  a silent success
- command functions and flag types are <name>Command and <name>Flags;
  parseVpnAddr and listHostMapFlagSet replace the repeated parse block
  and flag set; close-tunnel sends through sendCloseTunnel

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-19 10:48:26 -04:00
Matt RichardsonandClaude Fable 5.1 b426843f88 Extract the ssh debug commands into a transport neutral commands package (#1958)
Migrate commands from ssh into a transport neutral package

 - add golden tests validating that output remains unchanged
 - move commands to commands.go

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-19 10:25:28 -04:00
Wade Simmons cda3710d5a fix error return in handleOutsideMessagePacket (#1970)
smoke-extra / freebsd-amd64 (push) Failing after 17s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 16s
smoke-extra / netbsd-amd64 (push) Failing after 16s
smoke-extra / openbsd-amd64 (push) Failing after 16s
smoke-extra / linux-386 (push) Failing after 17s
smoke / Run multi node smoke test (push) Failing after 1m39s
Build and test / Static checks (push) Successful in 45s
Build and test / Test linux (push) Failing after 1m18s
Build and test / Test linux-pkcs11 (push) Failing after 2m5s
Build and test / Test linux-boringcrypto (push) Failing after 3m1s
Build and test / Test linux-fips140 (push) Failing after 2m55s
Build and test / Cross-build linux-arm (push) Successful in 3m0s
Build and test / Cross-build linux-mips (push) Successful in 3m42s
Build and test / Cross-build linux-other (push) Successful in 3m5s
Build and test / Cross-build windows (push) Successful in 1m2s
Build and test / Cross-build freebsd (push) Successful in 1m32s
Build and test / Cross-build netbsd (push) Successful in 1m31s
Build and test / Cross-build openbsd (push) Successful in 1m32s
Build and test / Cross-build mobile (push) Successful in 3m21s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
Introduced by #1961, we should still return early for invalid packets
2026-10-17 11:09:25 -04:00
John Maguire c26871fe08 service: Close listeners and tear down the netstack on shutdown (#1954) 2026-10-17 11:08:29 -04:00
Caleb Jasik 7daacc6a48 overlay/darwin: test tun framing and zero-alloc Read/Write (#1906)
smoke-extra / freebsd-amd64 (push) Failing after 16s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 16s
smoke-extra / netbsd-amd64 (push) Failing after 17s
smoke-extra / openbsd-amd64 (push) Failing after 17s
smoke-extra / linux-386 (push) Failing after 16s
smoke / Run multi node smoke test (push) Failing after 1m41s
Build and test / Static checks (push) Successful in 45s
Build and test / Test linux (push) Failing after 1m47s
Build and test / Test linux-pkcs11 (push) Failing after 2m6s
Build and test / Test linux-boringcrypto (push) Failing after 2m57s
Build and test / Test linux-fips140 (push) Failing after 2m53s
Build and test / Cross-build linux-arm (push) Successful in 3m7s
Build and test / Cross-build linux-mips (push) Successful in 3m55s
Build and test / Cross-build linux-other (push) Successful in 3m13s
Build and test / Cross-build windows (push) Successful in 1m4s
Build and test / Cross-build freebsd (push) Successful in 1m35s
Build and test / Cross-build netbsd (push) Successful in 1m35s
Build and test / Cross-build openbsd (push) Successful in 1m34s
Build and test / Cross-build mobile (push) Successful in 3m26s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
2026-10-16 18:12:58 -05:00
John Maguire 428dad94b3 Reject v2 cert names and groups at signing the parser rejects (#1967)
smoke-extra / freebsd-amd64 (push) Failing after 18s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 18s
smoke-extra / netbsd-amd64 (push) Failing after 16s
smoke-extra / openbsd-amd64 (push) Failing after 17s
smoke-extra / linux-386 (push) Failing after 17s
smoke / Run multi node smoke test (push) Failing after 1m41s
Build and test / Static checks (push) Successful in 45s
Build and test / Test linux (push) Failing after 1m19s
Build and test / Test linux-pkcs11 (push) Failing after 2m5s
Build and test / Test linux-boringcrypto (push) Failing after 2m56s
Build and test / Test linux-fips140 (push) Failing after 2m51s
Build and test / Cross-build linux-arm (push) Successful in 3m0s
Build and test / Cross-build linux-mips (push) Successful in 3m43s
Build and test / Cross-build linux-other (push) Successful in 3m7s
Build and test / Cross-build windows (push) Successful in 1m2s
Build and test / Cross-build freebsd (push) Successful in 1m32s
Build and test / Cross-build netbsd (push) Successful in 1m33s
Build and test / Cross-build openbsd (push) Successful in 1m32s
Build and test / Cross-build mobile (push) Successful in 3m23s
smoke-extra / Run windows smoke test (push) Canceled after 0s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
2026-10-16 18:05:51 -04:00
John Maguire 5bf666dd90 Treat uppercase blocked fingerprints the same as lowercase (#1964)
- Error for invalid on startup 
- Still block the valid blocklist entries when a reload has a bad one
2026-10-16 16:28:37 -04:00
John Maguire 49207da077 Remove the ssh debug profiling commands (#1965)
We plan to move this functionality into the debug socket.
2026-10-16 15:47:56 -04:00
Jack Doan bbc49669b7 fix noisy log line (#1961) 2026-10-16 12:08:55 -05:00
Nate Brown 915495d5f4 Fail the reload on a startup only value that would fail a restart, and stop asking for tun.dev restarts that change nothing (#1953)
smoke-extra / freebsd-amd64 (push) Failing after 25s
smoke-extra / linux-amd64-ipv6disable (push) Failing after 13s
smoke-extra / netbsd-amd64 (push) Failing after 11s
smoke-extra / openbsd-amd64 (push) Failing after 12s
smoke-extra / linux-386 (push) Failing after 14s
smoke / Run multi node smoke test (push) Failing after 1m43s
Build and test / Static checks (push) Successful in 2m25s
Build and test / Test linux (push) Failing after 1m28s
Build and test / Test linux-pkcs11 (push) Failing after 2m5s
Build and test / Test linux-boringcrypto (push) Failing after 2m56s
Build and test / Test linux-fips140 (push) Failing after 3m3s
Build and test / Cross-build linux-arm (push) Successful in 3m14s
Build and test / Cross-build linux-mips (push) Successful in 3m48s
Build and test / Cross-build linux-other (push) Successful in 3m13s
Build and test / Cross-build windows (push) Successful in 1m3s
Build and test / Cross-build freebsd (push) Successful in 1m34s
Build and test / Cross-build netbsd (push) Successful in 1m36s
Build and test / Cross-build openbsd (push) Successful in 1m37s
Build and test / Cross-build mobile (push) Successful in 3m33s
smoke / Run self traffic smoke test on macOS (push) Canceled after 0s
Build and test / Test macos (push) Canceled after 0s
Build and test / Test windows (push) Canceled after 0s
Build and test / CI status (push) Canceled after 0s
smoke-extra / Run windows smoke test (push) Canceled after 0s
2026-10-15 23:34:09 -05:00
Nate Brown a203d044bb Fail the reload when a config entry is skipped with a warning (#1952) 2026-10-15 23:22:47 -05:00
Nate Brown 316bf87780 Don't restart sshd when only sshd.sandbox_dir changes (#1948) 2026-10-15 22:12:12 -05:00
Nate Brown 1a0ac25db0 Report how a config reload went: failures, restarts required, and the keys that need one (#1935) 2026-10-15 15:36:39 -05:00
John Maguire 42597dda7d service: Don't hold listener lock while handing off a connection (#1939) 2026-10-15 15:14:39 -04:00
John Maguire afbdc87ed9 Update gvisor and bump to go1.27 (#1940)
gvisor's arm64 race-build compare-and-swap (pkg/sync/race_arm64.s)
loaded the new value into the register meant for the old one (R1
instead of R2), so it never succeeded. Sleepers could not commit to
parking and spun: under -race on arm64 every idle service netstack
busied about 14 cores, and TestService's static host lookup of
localhost missed its 250ms timeout and stalled for the 30s static_map
cadence. Fixed in gvisor 1b2946c4fa.

Go 1.27.2 is now the minimum because gvisor's go branch requires it.
golangci-lint is bumped to v2.14, the first release that can read
go1.27.2 export data.
2026-10-14 21:30:59 -04:00
Nate Brown 94372f4f51 Don't delete routes we never installed on darwin and the BSDs (#1933) 2026-10-13 16:34:38 -05:00
John Maguire 42280f04ce darwin: Assign host address to the utun and route vpn explicitly (#1931)
Tailscale with an exit node treats every interface address prefix as an
attached LAN and either claims it or excludes it, and configd then rewrites
the matching route away from nebula's utun. Interfaces holding a single
address are skipped, so assign /32 and /128 and add the network route
ourselves for both families.
2026-10-13 13:50:37 -04:00
Nate Brown 11c1917387 Warn when an unsafe_route is already routed through another interface on darwin (#1925) 2026-10-12 16:57:14 -05:00
Nate Brown 5ba7aff4b3 ssh: Only restart sshd on reload when its config or host key changed (#1923) 2026-10-12 16:56:51 -05:00
115 changed files with 4970 additions and 5812 deletions

No files matched your search

+2
View File
@@ -0,0 +1,2 @@
# Goldens are compared byte for byte, so they must come out of git the way they went in.
*.golden -text
+3 -3
View File
@@ -14,7 +14,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build
@@ -40,7 +40,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build
@@ -83,7 +83,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
# GitHub holds ARNs, not credentials, and ARNs outlive a rotation
+3 -3
View File
@@ -34,7 +34,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: add hashicorp source
@@ -66,7 +66,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: add hashicorp source
@@ -92,7 +92,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
# WSL2 + Ubuntu so the smoke can run a real linux peer with its own
+2 -2
View File
@@ -22,7 +22,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Smoke Docker
@@ -51,7 +51,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: build
+4 -4
View File
@@ -22,7 +22,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Install goimports
@@ -42,7 +42,7 @@ jobs:
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.12
version: v2.14
test:
name: Test ${{ matrix.name }}
@@ -87,7 +87,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build
@@ -132,7 +132,7 @@ jobs:
- uses: actions/setup-go@v7
with:
go-version: '1.26'
go-version: '1.27'
check-latest: true
- name: Build ${{ matrix.name }}
+7
View File
@@ -7,8 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Removed
- The SSH debug server's `start-cpu-profile`, `stop-cpu-profile`, `save-heap-profile`, `save-mutex-profile`, and
`mutex-profile-fraction` commands, and `sshd.sandbox_dir`, which is now ignored if set.
### Fixed
- Signing a v2 certificate with an empty name, a name longer than 253 bytes, or an empty group now fails.
These certificates were signed successfully but could not be loaded.
- `create-tunnel -address <ip:port>` panicked on a nil RemoteList and, over the ssh debug server, took nebula
down with it. A hostinfo returned by StartHandshake had no RemoteList until the handshake loop's first
pass attached one, and `create-tunnel -address` set the remote before that. The list is now attached when
+10 -1
View File
@@ -389,7 +389,16 @@ func (c *certificateV2) fromTBSCertificate(t *TBSCertificate) error {
}
func (c *certificateV2) validate() error {
// Empty names are allowed
// Mirror the name and group rules in unmarshalDetails so signing fails instead of producing an unparseable certificate
if c.details.name == "" || len(c.details.name) > MaxNameLength {
return NewErrInvalidCertificateProperties("name must be between 1 and %d bytes", MaxNameLength)
}
for _, group := range c.details.groups {
if group == "" {
return NewErrInvalidCertificateProperties("groups may not be empty")
}
}
if len(c.publicKey) == 0 {
return ErrInvalidPublicKey
+45
View File
@@ -6,6 +6,7 @@ import (
"crypto/elliptic"
"crypto/rand"
"net/netip"
"strings"
"testing"
"time"
@@ -135,3 +136,47 @@ func TestCertificate_SignP256_AlwaysNormalized(t *testing.T) {
assert.True(t, normie)
}
}
func TestCertificateV2_SignRejectsUnparseable(t *testing.T) {
_, priv, err := ed25519.GenerateKey(rand.Reader)
require.NoError(t, err)
var invalid *ErrInvalidCertificateProperties
newTBS := func() TBSCertificate {
return TBSCertificate{
Version: Version2,
Name: "testing",
Networks: []netip.Prefix{mustParsePrefixUnmapped("10.1.1.1/24")},
Groups: []string{"test-group1"},
NotBefore: time.Now().Add(-time.Minute),
NotAfter: time.Now().Add(time.Minute),
PublicKey: []byte("1234567890abcedfghij1234567890ab"),
Curve: Curve_CURVE25519,
IsCA: true,
}
}
tbs := newTBS()
tbs.Name = strings.Repeat("a", MaxNameLength)
c, err := tbs.Sign(nil, Curve_CURVE25519, priv)
require.NoError(t, err)
b, err := c.Marshal()
require.NoError(t, err)
_, err = unmarshalCertificateV2(b, nil, Curve_CURVE25519)
require.NoError(t, err)
tbs = newTBS()
tbs.Name = strings.Repeat("a", MaxNameLength+1)
_, err = tbs.Sign(nil, Curve_CURVE25519, priv)
require.ErrorAs(t, err, &invalid)
tbs = newTBS()
tbs.Name = ""
_, err = tbs.Sign(nil, Curve_CURVE25519, priv)
require.ErrorAs(t, err, &invalid)
tbs = newTBS()
tbs.Groups = []string{"test-group1", ""}
_, err = tbs.Sign(nil, Curve_CURVE25519, priv)
require.ErrorAs(t, err, &invalid)
}
+2 -1
View File
@@ -85,9 +85,10 @@ func main() {
fmt.Printf("failed to apply logging config: %s", err)
os.Exit(1)
}
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := logging.ApplyConfig(l, c); err != nil {
l.Error("Failed to reconfigure logger on reload", "error", err)
r.Failed()
}
})
+2 -1
View File
@@ -34,9 +34,10 @@ func (p *program) Start(s service.Service) error {
if err := logging.ApplyConfig(l, c); err != nil {
return fmt.Errorf("failed to apply logging config: %s", err)
}
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := logging.ApplyConfig(l, c); err != nil {
l.Error("Failed to reconfigure logger on reload", "error", err)
r.Failed()
}
})
+2 -1
View File
@@ -71,9 +71,10 @@ func main() {
fmt.Printf("failed to apply logging config: %s", err)
os.Exit(1)
}
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := logging.ApplyConfig(l, c); err != nil {
l.Error("Failed to reconfigure logger on reload", "error", err)
r.Failed()
}
})
+710
View File
@@ -0,0 +1,710 @@
package nebula
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"log/slog"
"maps"
"net/netip"
"sort"
"strings"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/logging"
)
// The commands nebula serves. attachCommands builds the one registry every surface
// reads. See the commands package for what a Command is.
type listHostMapFlags struct {
Json bool
Pretty bool
ByIndex bool
}
// listHostMapFlagSet is the flag set list-hostmap and list-pending-hostmap share.
func listHostMapFlagSet() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := listHostMapFlags{}
fl.BoolVar(&s.Json, "json", false, "outputs as json with more information")
fl.BoolVar(&s.Pretty, "pretty", false, "pretty prints json, assumes -json")
fl.BoolVar(&s.ByIndex, "by-index", false, "gets all hosts in the hostmap from the index table")
return fl, &s
}
type printCertFlags struct {
Json bool
Pretty bool
Raw bool
}
type printTunnelFlags struct {
Pretty bool
}
type changeRemoteFlags struct {
Address string
}
type closeTunnelFlags struct {
LocalOnly bool
}
type createTunnelFlags struct {
Address string
}
type deviceInfoFlags struct {
Json bool
Pretty bool
}
func attachCommands(l *slog.Logger, c *config.C, reg *commands.Registry, f *Interface) {
reg.Register(&commands.Command{
Name: commands.ListHostmap,
ShortDescription: "List all known previously connected hosts",
Flags: listHostMapFlagSet,
Callback: func(fs any, a []string, w commands.StringWriter) error {
return listHostMapCommand(f.hostMap, fs, w)
},
})
reg.Register(&commands.Command{
Name: commands.ListPendingHostmap,
ShortDescription: "List all handshaking hosts",
Flags: listHostMapFlagSet,
Callback: func(fs any, a []string, w commands.StringWriter) error {
return listHostMapCommand(f.handshakeManager, fs, w)
},
})
reg.Register(&commands.Command{
Name: commands.ListLighthouseAddrmap,
ShortDescription: "List all lighthouse map entries",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := listHostMapFlags{}
fl.BoolVar(&s.Json, "json", false, "outputs as json with more information")
fl.BoolVar(&s.Pretty, "pretty", false, "pretty prints json, assumes -json")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return listLighthouseAddrmapCommand(f.lightHouse, fs, w)
},
})
reg.Register(&commands.Command{
Name: commands.Reload,
ShortDescription: "Reloads configuration from disk, same as sending HUP to the process",
Callback: func(fs any, a []string, w commands.StringWriter) error {
return reloadCommand(l, c, w)
},
})
reg.Register(&commands.Command{
Name: commands.LogLevel,
ShortDescription: "Gets or sets the current log level",
Callback: func(fs any, a []string, w commands.StringWriter) error {
return logLevelCommand(l, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.LogFormat,
ShortDescription: "Gets or sets the current log format",
Callback: func(fs any, a []string, w commands.StringWriter) error {
return logFormatCommand(l, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.Version,
ShortDescription: "Prints the currently running version of nebula",
Callback: func(fs any, a []string, w commands.StringWriter) error {
return versionCommand(f, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.DeviceInfo,
ShortDescription: "Prints information about the network device.",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := deviceInfoFlags{}
fl.BoolVar(&s.Json, "json", false, "outputs as json with more information")
fl.BoolVar(&s.Pretty, "pretty", false, "pretty prints json, assumes -json")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return deviceInfoCommand(f, fs, w)
},
})
reg.Register(&commands.Command{
Name: commands.PrintCert,
ShortDescription: "Prints the current certificate being used or the certificate for the provided vpn addr",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := printCertFlags{}
fl.BoolVar(&s.Json, "json", false, "outputs as json")
fl.BoolVar(&s.Pretty, "pretty", false, "pretty prints json, assumes -json")
fl.BoolVar(&s.Raw, "raw", false, "raw prints the PEM encoded certificate, not compatible with -json or -pretty")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return printCertCommand(f, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.PrintTunnel,
ShortDescription: "Prints json details about a tunnel for the provided vpn addr",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := printTunnelFlags{}
fl.BoolVar(&s.Pretty, "pretty", false, "pretty prints json")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return printTunnelCommand(f, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.PrintRelays,
ShortDescription: "Prints json details about all relay info",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := printTunnelFlags{}
fl.BoolVar(&s.Pretty, "pretty", false, "pretty prints json")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return printRelaysCommand(f, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.ChangeRemote,
ShortDescription: "Changes the remote address used in the tunnel for the provided vpn addr",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := changeRemoteFlags{}
fl.StringVar(&s.Address, "address", "", "The new remote address, ip:port")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return changeRemoteCommand(f, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.CloseTunnel,
ShortDescription: "Closes a tunnel for the provided vpn addr",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := closeTunnelFlags{}
fl.BoolVar(&s.LocalOnly, "local-only", false, "Disables notifying the remote that the tunnel is shutting down")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return closeTunnelCommand(f, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.CreateTunnel,
ShortDescription: "Creates a tunnel for the provided vpn address",
Help: "The lighthouses will be queried for real addresses but you can provide one as well.",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
s := createTunnelFlags{}
fl.StringVar(&s.Address, "address", "", "Optionally provide a real remote address, ip:port ")
return fl, &s
},
Callback: func(fs any, a []string, w commands.StringWriter) error {
return createTunnelCommand(f, fs, a, w)
},
})
reg.Register(&commands.Command{
Name: commands.QueryLighthouse,
ShortDescription: "Query the lighthouses for the provided vpn address",
Help: "This command is asynchronous. Only currently known udp addresses will be printed.",
Callback: func(fs any, a []string, w commands.StringWriter) error {
return queryLighthouseCommand(f, fs, a, w)
},
})
}
func listHostMapCommand(hl controlHostLister, a any, w commands.StringWriter) error {
fs, ok := a.(*listHostMapFlags)
if !ok {
return fmt.Errorf("internal error: expected *listHostMapFlags but was %T", a)
}
var hm []ControlHostInfo
if fs.ByIndex {
hm = listHostMapIndexes(hl)
} else {
hm = listHostMapHosts(hl)
}
sort.Slice(hm, func(i, j int) bool {
return hm[i].VpnAddrs[0].Compare(hm[j].VpnAddrs[0]) < 0
})
if fs.Json || fs.Pretty {
js := json.NewEncoder(w.GetWriter())
if fs.Pretty {
js.SetIndent("", " ")
}
if err := js.Encode(hm); err != nil {
return err
}
} else {
for _, v := range hm {
err := w.WriteLine(fmt.Sprintf("%s: %s", v.VpnAddrs, v.RemoteAddrs))
if err != nil {
return err
}
}
}
return nil
}
func listLighthouseAddrmapCommand(lightHouse *LightHouse, a any, w commands.StringWriter) error {
fs, ok := a.(*listHostMapFlags)
if !ok {
return fmt.Errorf("internal error: expected *listHostMapFlags but was %T", a)
}
type lighthouseInfo struct {
VpnAddr string `json:"vpnAddr"`
Addrs *CacheMap `json:"addrs"`
}
lightHouse.RLock()
addrMap := make([]lighthouseInfo, len(lightHouse.addrMap))
x := 0
for k, v := range lightHouse.addrMap {
addrMap[x] = lighthouseInfo{
VpnAddr: k.String(),
Addrs: v.CopyCache(),
}
x++
}
lightHouse.RUnlock()
sort.Slice(addrMap, func(i, j int) bool {
return strings.Compare(addrMap[i].VpnAddr, addrMap[j].VpnAddr) < 0
})
if fs.Json || fs.Pretty {
js := json.NewEncoder(w.GetWriter())
if fs.Pretty {
js.SetIndent("", " ")
}
if err := js.Encode(addrMap); err != nil {
return err
}
} else {
for _, v := range addrMap {
b, err := json.Marshal(v.Addrs)
if err != nil {
return err
}
err = w.WriteLine(fmt.Sprintf("%s: %s", v.VpnAddr, string(b)))
if err != nil {
return err
}
}
}
return nil
}
// parseVpnAddr parses the vpn address a command takes as its first operand, failing to w
// when there is none or it does not parse. noun is how the command's messages name it,
// "vpn address" or "vpn addr", which the output golden pins per command.
func parseVpnAddr(a []string, noun string, w commands.StringWriter) (netip.Addr, error) {
if len(a) == 0 {
return netip.Addr{}, commands.Fail(w, "No vpn address was provided")
}
vpnAddr, err := netip.ParseAddr(a[0])
if err != nil {
return netip.Addr{}, commands.Fail(w, fmt.Sprintf("The provided %s could not be parsed: %s", noun, a[0]))
}
return vpnAddr, nil
}
func versionCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
return w.WriteLine(ifce.version)
}
func queryLighthouseCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
vpnAddr, err := parseVpnAddr(a, "vpn address", w)
if err != nil {
return err
}
var cm *CacheMap
rl := ifce.lightHouse.Query(vpnAddr)
if rl != nil {
cm = rl.CopyCache()
}
return json.NewEncoder(w.GetWriter()).Encode(cm)
}
func closeTunnelCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
flags, ok := fs.(*closeTunnelFlags)
if !ok {
return fmt.Errorf("internal error: expected *closeTunnelFlags but was %T", fs)
}
vpnAddr, err := parseVpnAddr(a, "vpn address", w)
if err != nil {
return err
}
hostInfo := ifce.hostMap.QueryVpnAddr(vpnAddr)
if hostInfo == nil {
return commands.Fail(w, fmt.Sprintf("Could not find tunnel for vpn address: %v", a[0]))
}
if !flags.LocalOnly {
ifce.sendCloseTunnel(hostInfo)
}
ifce.closeTunnel(hostInfo)
return w.WriteLine("Closed")
}
func createTunnelCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
flags, ok := fs.(*createTunnelFlags)
if !ok {
return fmt.Errorf("internal error: expected *createTunnelFlags but was %T", fs)
}
vpnAddr, err := parseVpnAddr(a, "vpn address", w)
if err != nil {
return err
}
hostInfo := ifce.hostMap.QueryVpnAddr(vpnAddr)
if hostInfo != nil {
return commands.Fail(w, "Tunnel already exists")
}
hostInfo = ifce.handshakeManager.QueryVpnAddr(vpnAddr)
if hostInfo != nil {
return commands.Fail(w, "Tunnel already handshaking")
}
var addr netip.AddrPort
if flags.Address != "" {
addr, err = netip.ParseAddrPort(flags.Address)
if err != nil {
return commands.Fail(w, "Address could not be parsed")
}
}
// Set the remote from the callback, which StartHandshake runs under the handshake manager lock
// before the hostinfo reaches the handshake loop or the static-host trigger, so the address is
// in the RemoteList before the first handshake is sent.
ifce.handshakeManager.StartHandshake(vpnAddr, func(hh *HandshakeHostInfo) {
if addr.IsValid() {
hh.hostinfo.SetRemote(addr)
}
})
return w.WriteLine("Created")
}
func changeRemoteCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
flags, ok := fs.(*changeRemoteFlags)
if !ok {
return fmt.Errorf("internal error: expected *changeRemoteFlags but was %T", fs)
}
vpnAddr, err := parseVpnAddr(a, "vpn address", w)
if err != nil {
return err
}
if flags.Address == "" {
return commands.Fail(w, "No address was provided")
}
addr, err := netip.ParseAddrPort(flags.Address)
if err != nil {
return commands.Fail(w, "Address could not be parsed")
}
hostInfo := ifce.hostMap.QueryVpnAddr(vpnAddr)
if hostInfo == nil {
return commands.Fail(w, fmt.Sprintf("Could not find tunnel for vpn address: %v", a[0]))
}
hostInfo.SetRemote(addr)
return w.WriteLine("Changed")
}
func logLevelCommand(l *slog.Logger, fs any, a []string, w commands.StringWriter) error {
ctrl, ok := l.Handler().(interface {
GetLevel() slog.Level
SetLevel(slog.Level)
})
if !ok {
return commands.Fail(w, "Log level is not reconfigurable on this logger")
}
if len(a) == 0 {
return w.WriteLine(fmt.Sprintf("Log level is: %s", logging.LevelName(ctrl.GetLevel())))
}
level, err := logging.ParseLevel(strings.ToLower(a[0]))
if err != nil {
return commands.Fail(w, fmt.Sprintf("Unknown log level %s. Possible log levels: trace, debug, info, warn, error", a))
}
ctrl.SetLevel(level)
return w.WriteLine(fmt.Sprintf("Log level is: %s", logging.LevelName(ctrl.GetLevel())))
}
func logFormatCommand(l *slog.Logger, fs any, a []string, w commands.StringWriter) error {
ctrl, ok := l.Handler().(interface {
GetFormat() string
SetFormat(string) error
})
if !ok {
return commands.Fail(w, "Log format is not reconfigurable on this logger")
}
if len(a) == 0 {
return w.WriteLine(fmt.Sprintf("Log format is: %s", ctrl.GetFormat()))
}
if err := ctrl.SetFormat(strings.ToLower(a[0])); err != nil {
return commands.Fail(w, fmt.Sprintf("Unable to set log format: %s", err))
}
return w.WriteLine(fmt.Sprintf("Log format is: %s", ctrl.GetFormat()))
}
func printCertCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
args, ok := fs.(*printCertFlags)
if !ok {
return fmt.Errorf("internal error: expected *printCertFlags but was %T", fs)
}
cert := ifce.pki.getCertState().GetDefaultCertificate()
if len(a) > 0 {
vpnAddr, err := parseVpnAddr(a, "vpn addr", w)
if err != nil {
return err
}
hostInfo := ifce.hostMap.QueryVpnAddr(vpnAddr)
if hostInfo == nil {
return commands.Fail(w, fmt.Sprintf("Could not find tunnel for vpn addr: %v", a[0]))
}
cert = hostInfo.GetCert().Certificate
}
if args.Json || args.Pretty {
b, err := cert.MarshalJSON()
if err != nil {
return commands.Fail(w, fmt.Sprintf("Unable to marshal the certificate: %s", err))
}
if args.Pretty {
buf := new(bytes.Buffer)
if err := json.Indent(buf, b, "", " "); err != nil {
return commands.Fail(w, fmt.Sprintf("Unable to marshal the certificate: %s", err))
}
b = buf.Bytes()
}
return w.WriteBytes(b)
}
if args.Raw {
b, err := cert.MarshalPEM()
if err != nil {
return commands.Fail(w, fmt.Sprintf("Unable to marshal the certificate: %s", err))
}
return w.WriteBytes(b)
}
return w.WriteLine(cert.String())
}
func printRelaysCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
args, ok := fs.(*printTunnelFlags)
if !ok {
return fmt.Errorf("internal error: expected *printTunnelFlags but was %T", fs)
}
ifce.hostMap.RLock()
relays := maps.Clone(ifce.hostMap.Relays)
ifce.hostMap.RUnlock()
type RelayFor struct {
Error string `json:",omitempty"`
Type string
State string
PeerAddr netip.Addr
LocalIndex uint32
RemoteIndex uint32
RelayedThrough []netip.Addr
}
type RelayOutput struct {
NebulaAddr netip.Addr
RelayForAddrs []RelayFor
}
type CmdOutput struct {
Relays []*RelayOutput
}
co := CmdOutput{}
enc := json.NewEncoder(w.GetWriter())
if args.Pretty {
enc.SetIndent("", " ")
}
for k, v := range relays {
ro := RelayOutput{NebulaAddr: v.vpnAddrs[0]}
co.Relays = append(co.Relays, &ro)
relayHI := ifce.hostMap.QueryVpnAddr(v.vpnAddrs[0])
if relayHI == nil {
ro.RelayForAddrs = append(ro.RelayForAddrs, RelayFor{Error: "could not find hostinfo"})
continue
}
for _, vpnAddr := range relayHI.relayState.CopyRelayForIps() {
rf := RelayFor{}
r, ok := relayHI.relayState.GetRelayForByAddr(vpnAddr)
if ok {
t := ""
switch r.Type {
case ForwardingType:
t = "forwarding"
case TerminalType:
t = "terminal"
default:
t = "unknown"
}
s := ""
switch r.State {
case Requested:
s = "requested"
case Established:
s = "established"
default:
s = "unknown"
}
rf.LocalIndex = r.LocalIndex
rf.RemoteIndex = r.RemoteIndex
rf.PeerAddr = r.PeerAddr
rf.Type = t
rf.State = s
if rf.LocalIndex != k {
rf.Error = fmt.Sprintf("hostmap LocalIndex '%v' does not match RelayState LocalIndex", k)
}
}
relayedHI := ifce.hostMap.QueryVpnAddr(vpnAddr)
if relayedHI != nil {
rf.RelayedThrough = append(rf.RelayedThrough, relayedHI.relayState.CopyRelayIps()...)
}
ro.RelayForAddrs = append(ro.RelayForAddrs, rf)
}
}
return enc.Encode(co)
}
func printTunnelCommand(ifce *Interface, fs any, a []string, w commands.StringWriter) error {
args, ok := fs.(*printTunnelFlags)
if !ok {
return fmt.Errorf("internal error: expected *printTunnelFlags but was %T", fs)
}
vpnAddr, err := parseVpnAddr(a, "vpn addr", w)
if err != nil {
return err
}
hostInfo := ifce.hostMap.QueryVpnAddr(vpnAddr)
if hostInfo == nil {
return commands.Fail(w, fmt.Sprintf("Could not find tunnel for vpn addr: %v", a[0]))
}
enc := json.NewEncoder(w.GetWriter())
if args.Pretty {
enc.SetIndent("", " ")
}
return enc.Encode(copyHostInfo(hostInfo, ifce.hostMap.GetPreferredRanges()))
}
func deviceInfoCommand(ifce *Interface, fs any, w commands.StringWriter) error {
flags, ok := fs.(*deviceInfoFlags)
if !ok {
return fmt.Errorf("internal error: expected flags to be deviceInfoFlags but was %+v", fs)
}
data := struct {
Name string `json:"name"`
Cidr []netip.Prefix `json:"cidr"`
}{
Name: ifce.inside.Name(),
Cidr: make([]netip.Prefix, len(ifce.inside.Networks())),
}
copy(data.Cidr, ifce.inside.Networks())
if flags.Json || flags.Pretty {
js := json.NewEncoder(w.GetWriter())
if flags.Pretty {
js.SetIndent("", " ")
}
return js.Encode(data)
} else {
return w.WriteLine(fmt.Sprintf("name=%v cidr=%v", data.Name, data.Cidr))
}
}
// reloadCommand reloads the config and writes the result to the client. It reloads even if the client has gone away
func reloadCommand(l *slog.Logger, c *config.C, w commands.StringWriter) error {
writeErr := w.WriteLine("Reloading config")
res := c.ReloadConfig()
res.Log(l)
if err := w.WriteLine(res.String()); writeErr == nil {
writeErr = err
}
if writeErr != nil {
return writeErr
}
if res.Err != nil || res.Failed {
return &commands.Failure{Msg: res.String()}
}
return nil
}
+83
View File
@@ -0,0 +1,83 @@
package commands
import (
"flag"
)
// CommandFlags is called before help or command execution to parse command line flags.
// It should return a flag.FlagSet instance and a pointer to the struct that will contain parsed flags
type CommandFlags func() (*flag.FlagSet, any)
// CommandCallback is called when your command should execute.
// fs will be a pointer to the struct provided by Command.Flags callback, if there was one. -h and -help are reserved
// and handled automatically for you.
// a will be any unconsumed arguments, if no Command.Flags was available this will be all the flags passed in.
// w is the writer to use when sending messages back to the client.
// An error returned by the callback is handed back to whoever dispatched the command, which reports a failure: exit
// status 1 over ssh. The error is not shown to the user. For a failure the user caused or can act on, write the
// message and return it as a *Failure, which Fail does in one step; the surface then reports the failure without
// logging what the user was already told. Any other error is logged by the surface as well.
type CommandCallback func(fs any, a []string, w StringWriter) error
type Command struct {
Name Name
ShortDescription string
Help string
Flags CommandFlags
Callback CommandCallback
}
// Failure is the error a command returns for a failure it has already explained to the user:
// the message it wrote is the whole story. A surface reports it as a failure and does not
// log it.
type Failure struct {
Msg string
}
func (f *Failure) Error() string {
return f.Msg
}
// Fail writes msg to w and returns it as a *Failure, so a command reports a failure to the
// user and to its surface in one line. If the write itself fails, that error is returned
// instead: the user was not told anything.
func Fail(w StringWriter, msg string) error {
if err := w.WriteLine(msg); err != nil {
return err
}
return &Failure{Msg: msg}
}
func execCommand(c *Command, args []string, w StringWriter) error {
var (
fl *flag.FlagSet
fs any
)
if c.Flags != nil {
fl, fs = c.Flags()
if fl != nil {
// SetOutput() here in case fl.Parse dumps usage.
fl.SetOutput(w.GetWriter())
err := fl.Parse(args)
if err != nil {
// fl.Parse has dumped error information to the user via the w writer.
return err
}
args = fl.Args()
}
}
return c.Callback(fs, args, w)
}
func checkHelpArgs(args []string) bool {
for _, a := range args {
if a == "-h" || a == "-help" {
return true
}
}
return false
}
+28
View File
@@ -0,0 +1,28 @@
package commands
// Name identifies a command. Every command nebula serves is a constant here, so a
// surface, a client and a test spell it the same way and are told at compile time when
// they do not. It is a string rather than an integer so a test can register a throwaway
// command without adding it here, and so the wire carries the name as the string it has
// always been.
type Name string
const (
ChangeRemote Name = "change-remote"
CloseTunnel Name = "close-tunnel"
CreateTunnel Name = "create-tunnel"
DeviceInfo Name = "device-info"
Help Name = "help"
ListHostmap Name = "list-hostmap"
ListLighthouseAddrmap Name = "list-lighthouse-addrmap"
ListPendingHostmap Name = "list-pending-hostmap"
LogFormat Name = "log-format"
LogLevel Name = "log-level"
Logout Name = "logout"
PrintCert Name = "print-cert"
PrintRelays Name = "print-relays"
PrintTunnel Name = "print-tunnel"
QueryLighthouse Name = "query-lighthouse"
Reload Name = "reload"
Version Name = "version"
)
+181
View File
@@ -0,0 +1,181 @@
package commands
import (
"cmp"
"fmt"
"maps"
"slices"
"strings"
"sync"
"github.com/anmitsu/go-shlex"
)
// Registry holds a set of commands. New is the only way to make one.
//
// It is safe for concurrent use. Commands may be registered while a surface is serving
// from the registry: the ssh server clones it per session, and a clone may overlap a
// Register.
type Registry struct {
mu sync.RWMutex
commands map[Name]*Command
}
// New returns an empty Registry with only `help` available.
func New() *Registry {
r := &Registry{commands: map[Name]*Command{}}
r.registerHelp()
return r
}
func (r *Registry) registerHelp() {
r.Register(&Command{
Name: Help,
ShortDescription: "prints available commands or help <command> for specific usage info",
Callback: func(a any, args []string, w StringWriter) error {
return r.help(args, w)
},
})
}
// Register adds a command that can be run by a user. A command registered under a name
// already held replaces it; see Clone for what that means for Help.
func (r *Registry) Register(c *Command) {
r.mu.Lock()
defer r.mu.Unlock()
r.commands[c.Name] = c
}
// Get returns the command registered under name, or nil if there is no such command.
func (r *Registry) Get(name Name) *Command {
r.mu.RLock()
defer r.mu.RUnlock()
return r.commands[name]
}
// Match returns the names of every command starting with prefix, sorted, as strings for
// a prompt to complete and print.
func (r *Registry) Match(prefix string) []string {
cmds := make([]string, 0)
r.mu.RLock()
for n := range r.commands {
if strings.HasPrefix(string(n), prefix) {
cmds = append(cmds, string(n))
}
}
r.mu.RUnlock()
slices.Sort(cmds)
return cmds
}
// All returns every registered command sorted by name, which is the order help lists them.
func (r *Registry) All() []*Command {
r.mu.RLock()
defer r.mu.RUnlock()
cmds := slices.Collect(maps.Values(r.commands))
slices.SortFunc(cmds, func(a, b *Command) int { return cmp.Compare(a.Name, b.Name) })
return cmds
}
// Clone returns a copy holding the same commands, with help bound to the copy: a
// command registered on the clone alone is listed and described there, and one the
// clone withholds is not. That rebinding replaces whatever was registered under Help on
// the source, built-in or not.
func (r *Registry) Clone() *Registry {
r.mu.RLock()
c := &Registry{commands: maps.Clone(r.commands)}
r.mu.RUnlock()
c.registerHelp()
return c
}
// Dispatch parses a command line and runs it, reporting problems to w. The error is for
// the caller to log, not to show; see CommandCallback.
func (r *Registry) Dispatch(line string, w StringWriter) error {
// A line shlex cannot split, an unbalanced quote.
args, err := shlex.Split(line, true)
if err != nil {
return Fail(w, "could not parse the command line: "+err.Error())
}
return r.DispatchArgs(args, w)
}
// DispatchArgs runs an already tokenized command line. The error is as for Dispatch.
func (r *Registry) DispatchArgs(args []string, w StringWriter) error {
if len(args) == 0 {
r.dumpCommands(w)
return nil
}
c := r.Get(Name(args[0]))
if c == nil {
// Only the first word was looked up, so only it is named.
msg := fmt.Sprintf("did not understand: %s", args[0])
if err := w.WriteLine(msg); err != nil {
return err
}
r.dumpCommands(w)
return &Failure{Msg: msg}
}
if checkHelpArgs(args) {
// Through the registered help command, so -h and `help <name>` are one path.
return execCommand(r.Get(Help), []string{string(c.Name)}, w)
}
return execCommand(c, args[1:], w)
}
// dumpCommands writes the list of available commands to w.
func (r *Registry) dumpCommands(w StringWriter) {
err := w.WriteLine("Available commands:")
if err != nil {
return
}
// In the order All gives, which is the display order.
cmds := make([]string, 0)
for _, l := range r.All() {
cmds = append(cmds, fmt.Sprintf("%s - %s", l.Name, l.ShortDescription))
}
_ = w.Write(strings.Join(cmds, "\n") + "\n\n")
}
// help writes the command list, or usage for the single command named in a.
func (r *Registry) help(a []string, w StringWriter) error {
// Just typed help
if len(a) == 0 {
r.dumpCommands(w)
return nil
}
// We are printing a specific commands help text
cmd := r.Get(Name(a[0]))
if cmd == nil {
return Fail(w, "Command not available "+a[0])
}
if err := w.WriteLine(fmt.Sprintf("%s - %s", cmd.Name, cmd.ShortDescription)); err != nil {
return err
}
if cmd.Help != "" {
if err := w.WriteLine(fmt.Sprintf(" %s", cmd.Help)); err != nil {
return err
}
}
if cmd.Flags != nil {
fs, _ := cmd.Flags()
if fs != nil {
fs.SetOutput(w.GetWriter())
fs.PrintDefaults()
}
}
return nil
}
+176
View File
@@ -0,0 +1,176 @@
package commands
import (
"bytes"
"errors"
"flag"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func testRegistry() *Registry {
r := New()
r.Register(&Command{
Name: "served",
ShortDescription: "a command this surface serves",
Callback: func(fs any, a []string, w StringWriter) error {
return w.WriteLine("read " + strings.Join(a, ","))
},
})
r.Register(&Command{
Name: "withheld",
ShortDescription: "a command this surface withholds",
Callback: func(fs any, a []string, w StringWriter) error {
return w.WriteLine("wrote")
},
})
return r
}
// TestFlagUsageGoesToTheWriter pins that a parse failure's usage lands on the command's
// writer.
func TestFlagUsageGoesToTheWriter(t *testing.T) {
r := New()
r.Register(&Command{
Name: "emit",
ShortDescription: "emits",
Flags: func() (*flag.FlagSet, any) {
fl := flag.NewFlagSet("", flag.ContinueOnError)
fl.Bool("json", false, "emit json")
return fl, nil
},
Callback: func(fs any, a []string, w StringWriter) error { return nil },
})
merged := &bytes.Buffer{}
require.Error(t, r.DispatchArgs([]string{"emit", "-nope"}, NewStringWriter(merged)))
assert.Contains(t, merged.String(), "flag provided but not defined")
}
// TestDispatchReportsUnparseableLine: a line shlex rejects, an unbalanced quote, is a
// failure the user is told about.
func TestDispatchReportsUnparseableLine(t *testing.T) {
out := &bytes.Buffer{}
err := testRegistry().Dispatch(`served "unterminated`, NewStringWriter(out))
var f *Failure
require.ErrorAs(t, err, &f)
assert.Equal(t, f.Msg+"\n", out.String())
assert.Contains(t, out.String(), "could not parse the command line: ")
}
// TestFailTellsTheUserAndTheSurface: Fail writes the message and returns it as a Failure;
// when the write fails, the user heard nothing, so the write error is what comes back.
func TestFailTellsTheUserAndTheSurface(t *testing.T) {
out := &bytes.Buffer{}
err := Fail(NewStringWriter(out), "no such tunnel")
var f *Failure
require.ErrorAs(t, err, &f)
assert.Equal(t, "no such tunnel", f.Msg)
assert.Equal(t, "no such tunnel\n", out.String())
err = Fail(NewStringWriter(failingWriter{}), "no such tunnel")
require.EqualError(t, err, "gone")
assert.NotErrorAs(t, err, &f)
}
type failingWriter struct{}
func (failingWriter) Write([]byte) (int, error) { return 0, errors.New("gone") }
// TestAllSortsByName: registration order does not matter.
func TestAllSortsByName(t *testing.T) {
r := New()
for _, n := range []Name{"zz", Version, "aa", ChangeRemote} {
r.Register(&Command{Name: n})
}
var got []Name
for _, c := range r.All() {
got = append(got, c.Name)
}
assert.Equal(t, []Name{"aa", ChangeRemote, Help, Version, "zz"}, got)
}
// TestHelpFlagAgreesWithHelpCommand pins that -h answers through the registered help
// command, and that help on a clone describes the clone: a command registered only there
// is known to both -h and `help <name>`, and one withheld from it is known to neither.
func TestHelpFlagAgreesWithHelpCommand(t *testing.T) {
r := testRegistry()
c := r.Clone()
c.Register(&Command{
Name: Logout,
ShortDescription: "ends the session",
Callback: func(fs any, a []string, w StringWriter) error { return nil },
})
flag := &bytes.Buffer{}
c.Dispatch("logout -h", NewStringWriter(flag))
cmd := &bytes.Buffer{}
c.Dispatch("help logout", NewStringWriter(cmd))
assert.Equal(t, cmd.String(), flag.String())
assert.Equal(t, "logout - ends the session\n", flag.String())
// The source never learns of it.
src := &bytes.Buffer{}
r.Dispatch("help logout", NewStringWriter(src))
assert.Equal(t, "Command not available logout\n", src.String())
// And a command both hold is described either way.
flag.Reset()
c.Dispatch("served -h", NewStringWriter(flag))
assert.Equal(t, "served - a command this surface serves\n", flag.String())
}
// TestCloneHelpListsTheClone: the command list help prints is the clone's, so a clone
// that withholds a command does not name it, and the source still does.
func TestCloneHelpListsTheClone(t *testing.T) {
r := testRegistry()
c := New()
c.Register(r.Get("served"))
c = c.Clone()
out := &bytes.Buffer{}
c.Dispatch("help", NewStringWriter(out))
assert.Equal(t, "Available commands:\n"+
"help - prints available commands or help <command> for specific usage info\n"+
"served - a command this surface serves\n\n", out.String())
out.Reset()
c.Dispatch("help withheld", NewStringWriter(out))
assert.Equal(t, "Command not available withheld\n", out.String())
out.Reset()
r.Dispatch("help withheld", NewStringWriter(out))
assert.Equal(t, "withheld - a command this surface withholds\n", out.String())
}
// TestDispatchReturnsTheCommandsError: what the callback returns reaches the caller, so
// a surface can log it.
func TestDispatchReturnsTheCommandsError(t *testing.T) {
r := New()
r.Register(&Command{
Name: "fails",
Callback: func(fs any, a []string, w StringWriter) error {
return errors.New("boom")
},
})
out := &bytes.Buffer{}
require.EqualError(t, r.Dispatch("fails", NewStringWriter(out)), "boom")
assert.Empty(t, out.String(), "a command's own error is not shown; the command speaks for itself")
}
// TestUnknownCommandNamesTheWord: only the first word is looked up, so only it is
// reported, followed by the command list, and the line is a failure to the surface.
func TestUnknownCommandNamesTheWord(t *testing.T) {
out := &bytes.Buffer{}
err := testRegistry().Dispatch(`nope "with args"`, NewStringWriter(out))
var f *Failure
require.ErrorAs(t, err, &f)
assert.Equal(t, "did not understand: nope", f.Msg)
assert.True(t, strings.HasPrefix(out.String(), "did not understand: nope\nAvailable commands:\n"), out.String())
}
+8 -1
View File
@@ -1,7 +1,9 @@
package sshd
package commands
import "io"
// StringWriter is how a command talks back to whoever invoked it: one stream, which the
// ssh surface points at its channel.
type StringWriter interface {
WriteLine(string) error
Write(string) error
@@ -13,6 +15,11 @@ type stringWriter struct {
w io.Writer
}
// NewStringWriter returns a StringWriter that writes to w.
func NewStringWriter(w io.Writer) StringWriter {
return &stringWriter{w: w}
}
func (w *stringWriter) WriteLine(s string) error {
return w.Write(s + "\n")
}
+67
View File
@@ -0,0 +1,67 @@
package nebula
import (
"bytes"
"io"
"net/netip"
"testing"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/logging"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestLogFormatRefusalIsShown: a format the logger does not know is reported to the user
// as a failure, rather than returned bare, which over ssh was silence.
func TestLogFormatRefusalIsShown(t *testing.T) {
l := logging.NewLogger(io.Discard)
out := &bytes.Buffer{}
w := commands.NewStringWriter(out)
var f *commands.Failure
require.ErrorAs(t, logFormatCommand(l, nil, []string{"bogus"}, w), &f)
assert.Equal(t, "Unable to set log format: unknown log format `bogus`. possible formats: [text json]\n", out.String())
out.Reset()
require.NoError(t, logFormatCommand(l, nil, []string{"JSON"}, w))
assert.Equal(t, "Log format is: json\n", out.String())
}
// TestPrintRelaysReportsErrorsAsText: the Error field was an error value, which encodes
// as {} and lost the message. It is the message now, and absent when there is none.
func TestPrintRelaysReportsErrorsAsText(t *testing.T) {
l := test.NewLogger()
hm := newHostMap(l)
relay := netip.MustParseAddr("10.0.0.1")
peer := netip.MustParseAddr("10.0.0.2")
// A relay host that is in Relays but not in Hosts, so its hostinfo cannot be found.
hm.Relays[1] = &HostInfo{vpnAddrs: []netip.Addr{relay}}
ifce := &Interface{hostMap: hm}
out := &bytes.Buffer{}
require.NoError(t, printRelaysCommand(ifce, &printTunnelFlags{}, nil, commands.NewStringWriter(out)))
assert.JSONEq(t, `{"Relays":[{"NebulaAddr":"10.0.0.1","RelayForAddrs":[{"Error":"could not find hostinfo","Type":"","State":"","PeerAddr":"","LocalIndex":0,"RemoteIndex":0,"RelayedThrough":null}]}]}`, out.String())
// The same relay host, found, relaying for a peer whose index matches: no Error key.
hi := hm.Relays[1]
r := &Relay{Type: ForwardingType, State: Established, PeerAddr: peer, LocalIndex: 1, RemoteIndex: 2}
hi.relayState = RelayState{
relayForByAddr: map[netip.Addr]*Relay{peer: r},
relayForByIdx: map[uint32]*Relay{r.LocalIndex: r},
}
hm.Hosts[relay] = hi
out.Reset()
require.NoError(t, printRelaysCommand(ifce, &printTunnelFlags{}, nil, commands.NewStringWriter(out)))
assert.JSONEq(t, `{"Relays":[{"NebulaAddr":"10.0.0.1","RelayForAddrs":[{"Type":"forwarding","State":"established","PeerAddr":"10.0.0.2","LocalIndex":1,"RemoteIndex":2,"RelayedThrough":null}]}]}`, out.String())
// And an index mismatch names it.
hm.Relays[7] = hi
delete(hm.Relays, 1)
out.Reset()
require.NoError(t, printRelaysCommand(ifce, &printTunnelFlags{}, nil, commands.NewStringWriter(out)))
assert.Contains(t, out.String(), `"Error":"hostmap LocalIndex '7' does not match RelayState LocalIndex"`)
}
+31 -22
View File
@@ -26,9 +26,12 @@ type C struct {
files []string
Settings map[string]any
oldSettings map[string]any
callbacks []func(*C)
callbacks []func(*C, *Reload)
l *slog.Logger
reloadLock sync.Mutex
// Set by the first reload that needs a restart and never cleared. Guarded by reloadLock
needsRestart bool
}
func NewC(l *slog.Logger) *C {
@@ -72,8 +75,9 @@ func (c *C) LoadString(raw string) error {
// RegisterReloadCallback stores a function to be called when a config reload is triggered. The functions registered
// here should decide if they need to make a change to the current process before making the change. HasChanged can be
// used to help decide if a change is necessary.
// These functions should return quickly or spawn their own go routine if they will take a while
func (c *C) RegisterReloadCallback(f func(*C)) {
// These functions should return quickly or spawn their own go routine if they will take a while.
// Use r to report anything that failed or needs a restart to apply.
func (c *C) RegisterReloadCallback(f func(c *C, r *Reload)) {
c.callbacks = append(c.callbacks, f)
}
@@ -144,50 +148,55 @@ func (c *C) CatchHUP(ctx context.Context) {
return
case <-ch:
c.l.Info("Caught HUP, reloading config")
c.ReloadConfig()
c.ReloadConfig().Log(c.l)
}
}
}()
}
func (c *C) ReloadConfig() {
// ReloadConfig reloads the config from the path provided to Load and runs every registered callback. A config that
// fails to load is logged here, like any other failure, and the result carries the error for the summary.
func (c *C) ReloadConfig() ReloadResult {
c.reloadLock.Lock()
defer c.reloadLock.Unlock()
c.oldSettings = make(map[string]any)
maps.Copy(c.oldSettings, c.Settings)
err := c.Load(c.path)
if err != nil {
c.l.Error("Error occurred while reloading config",
"config_path", c.path,
"error", err,
)
return
if err := c.Load(c.path); err != nil {
c.l.Error("Error occurred while reloading config", "config_path", c.path, "error", err)
return ReloadResult{Err: err, RestartRequired: c.needsRestart}
}
for _, v := range c.callbacks {
v(c)
}
return c.runCallbacks()
}
func (c *C) ReloadConfigString(raw string) error {
// ReloadConfigString is ReloadConfig for a config held in memory rather than on disk.
func (c *C) ReloadConfigString(raw string) ReloadResult {
c.reloadLock.Lock()
defer c.reloadLock.Unlock()
c.oldSettings = make(map[string]any)
maps.Copy(c.oldSettings, c.Settings)
err := c.LoadString(raw)
if err != nil {
return err
if err := c.LoadString(raw); err != nil {
c.l.Error("Error occurred while reloading config", "error", err)
return ReloadResult{Err: err, RestartRequired: c.needsRestart}
}
return c.runCallbacks()
}
func (c *C) runCallbacks() ReloadResult {
r := &Reload{c: c}
for _, v := range c.callbacks {
v(c)
v(c, r)
}
return nil
if r.restart {
c.needsRestart = true
}
return ReloadResult{Failed: r.failed, RestartRequired: c.needsRestart, RestartKeys: r.keys}
}
// GetString will get the string for k or return the default d if not found or invalid
@@ -391,7 +400,7 @@ func (c *C) parse() error {
var nm map[string]any
err = yaml.Unmarshal(b, &nm)
if err != nil {
return err
return fmt.Errorf("%s: %w", path, err)
}
// We need to use WithAppendSlice so that firewall rules in separate
+2 -2
View File
@@ -19,7 +19,7 @@ func TestConfig_Load(t *testing.T) {
// invalid yaml
c := NewC(l)
os.WriteFile(filepath.Join(dir, "01.yaml"), []byte(" invalid yaml"), 0644)
require.EqualError(t, c.Load(dir), "yaml: unmarshal errors:\n line 1: cannot unmarshal !!str `invalid...` into map[string]interface {}")
require.EqualError(t, c.Load(dir), filepath.Join(dir, "01.yaml")+": yaml: unmarshal errors:\n line 1: cannot unmarshal !!str `invalid...` into map[string]interface {}")
// simple multi config merge
c = NewC(l)
@@ -129,7 +129,7 @@ func TestConfig_ReloadConfig(t *testing.T) {
os.WriteFile(filepath.Join(dir, "01.yaml"), []byte("outer:\n inner: ho"), 0644)
c.RegisterReloadCallback(func(c *C) {
c.RegisterReloadCallback(func(c *C, _ *Reload) {
done <- true
})
+98
View File
@@ -0,0 +1,98 @@
package config
import (
"context"
"fmt"
"log/slog"
"slices"
)
// ReloadResult describes how a reload went. Problems are logged where they happen, Log writes a one line summary.
type ReloadResult struct {
// Err is set when the new config could not be read or parsed, no callbacks ran
Err error
// Failed is true when a callback could not apply part of the new config
Failed bool
// RestartRequired is true once any reload has needed a restart, and stays true until the process restarts
RestartRequired bool
// RestartKeys are the keys this reload changed that only apply on a restart
RestartKeys []string
}
// Summary returns a one line summary of the reload and the level to log it at.
func (r ReloadResult) Summary() (slog.Level, string) {
switch {
case r.Err != nil && r.RestartRequired:
return slog.LevelError, "Config reload failed, the new config did not load and a restart is still required"
case r.Err != nil:
return slog.LevelError, "Config reload failed, the new config did not load"
case r.Failed && r.RestartRequired:
return slog.LevelError, "Config reload completed with errors and requires a restart to fully apply, review the logs"
case r.Failed:
return slog.LevelError, "Config reload completed with errors, review the logs"
case r.RestartRequired:
return slog.LevelWarn, "Config reload was completed but requires a restart to fully apply"
default:
return slog.LevelInfo, "Config reload complete"
}
}
// String is the summary with Err and RestartKeys included, for output other than the log.
func (r ReloadResult) String() string {
_, msg := r.Summary()
if r.Err != nil {
msg = fmt.Sprintf("%s: %v", msg, r.Err)
}
if len(r.RestartKeys) > 0 {
msg = fmt.Sprintf("%s, keys %v", msg, r.RestartKeys)
}
return msg
}
// Log writes Summary to l.
func (r ReloadResult) Log(l *slog.Logger) {
level, msg := r.Summary()
if len(r.RestartKeys) > 0 {
l.Log(context.Background(), level, msg, "keys", r.RestartKeys)
return
}
l.Log(context.Background(), level, msg)
}
// Reload is passed to each reload callback to report how applying the new config went.
type Reload struct {
c *C
failed bool
restart bool
// Changed keys that only apply on a restart, from every callback
keys []string
}
// Failed marks the reload as failed, the caller logs why. Safe to call on a nil Reload, which is what the initial load
// passes.
func (r *Reload) Failed() {
if r != nil {
r.failed = true
}
}
// RestartRequired marks the reload as needing a restart, the caller logs why. Safe to call on a nil Reload.
func (r *Reload) RestartRequired() {
if r != nil {
r.restart = true
}
}
// RestartRequiredIfChanged marks the reload as needing a restart if any of keys changed, and adds them to RestartKeys.
// Use it for keys that are only read at startup. Safe to call on a nil Reload.
func (r *Reload) RestartRequiredIfChanged(keys ...string) {
if r == nil {
return
}
for _, k := range keys {
if r.c.HasChanged(k) && !slices.Contains(r.keys, k) {
r.keys = append(r.keys, k)
r.restart = true
}
}
}
+108
View File
@@ -0,0 +1,108 @@
package config
import (
"errors"
"log/slog"
"testing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestReloadConfigString(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1"))
fail := false
ran := 0
c.RegisterReloadCallback(func(c *C, r *Reload) {
ran++
if fail {
r.Failed()
}
})
assert.Equal(t, ReloadResult{}, c.ReloadConfigString("a: 2"))
fail = true
assert.Equal(t, ReloadResult{Failed: true}, c.ReloadConfigString("a: 3"))
assert.Equal(t, 2, ran)
// A config that fails to load runs no callbacks and returns the error
res := c.ReloadConfigString("a: [")
require.Error(t, res.Err)
assert.False(t, res.Failed)
assert.Equal(t, 2, ran)
assert.Equal(t, 3, c.GetInt("a", 0))
}
func TestReloadRestartRequiredLatches(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1\nb: 1"))
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequiredIfChanged("a", "b") })
// Nothing that needs a restart changed
assert.Equal(t, ReloadResult{}, c.ReloadConfigString("a: 1\nb: 1\nc: 1"))
assert.Equal(t, ReloadResult{RestartRequired: true, RestartKeys: []string{"a"}}, c.ReloadConfigString("a: 2\nb: 1\nc: 1"))
// The restart sticks through a reload with no changes, reverting the key and a config that fails to load. Only the
// reload that changed the key lists it
assert.Equal(t, ReloadResult{RestartRequired: true}, c.ReloadConfigString("a: 2\nb: 1\nc: 1"))
assert.Equal(t, ReloadResult{RestartRequired: true, RestartKeys: []string{"a"}}, c.ReloadConfigString("a: 1\nb: 1\nc: 1"))
res := c.ReloadConfigString("a: [")
require.Error(t, res.Err)
assert.True(t, res.RestartRequired)
}
// A key reported by more than one callback is listed once
func TestReloadRestartKeysDeduped(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1\nb: 1"))
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequiredIfChanged("a") })
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequiredIfChanged("a", "b") })
assert.Equal(t, []string{"a", "b"}, c.ReloadConfigString("a: 2\nb: 2").RestartKeys)
}
// A callback can require a restart without a key changing, a file the config points at may have
func TestReloadRestartRequiredWithoutAChange(t *testing.T) {
c := NewC(test.NewLogger())
require.NoError(t, c.LoadString("a: 1"))
c.RegisterReloadCallback(func(c *C, r *Reload) { r.RestartRequired() })
assert.Equal(t, ReloadResult{RestartRequired: true}, c.ReloadConfigString("a: 1"))
}
func TestReloadResultSummary(t *testing.T) {
cases := []struct {
res ReloadResult
level slog.Level
msg string
}{
{ReloadResult{}, slog.LevelInfo, "Config reload complete"},
{ReloadResult{RestartRequired: true}, slog.LevelWarn, "Config reload was completed but requires a restart to fully apply"},
{ReloadResult{Failed: true}, slog.LevelError, "Config reload completed with errors, review the logs"},
{ReloadResult{Failed: true, RestartRequired: true}, slog.LevelError, "Config reload completed with errors and requires a restart to fully apply, review the logs"},
{ReloadResult{Err: errors.New("bad yaml")}, slog.LevelError, "Config reload failed, the new config did not load"},
{ReloadResult{Err: errors.New("bad yaml"), RestartRequired: true}, slog.LevelError, "Config reload failed, the new config did not load and a restart is still required"},
}
for _, tc := range cases {
level, msg := tc.res.Summary()
assert.Equal(t, tc.level, level, tc.msg)
assert.Equal(t, tc.msg, msg)
}
res := ReloadResult{RestartRequired: true, RestartKeys: []string{"a", "b"}}
assert.Equal(t, "Config reload was completed but requires a restart to fully apply, keys [a b]", res.String())
// String includes the error
assert.Equal(t, "Config reload failed, the new config did not load: bad yaml", ReloadResult{Err: errors.New("bad yaml")}.String())
}
// The initial load passes a nil Reload
func TestReloadNilIsSafe(t *testing.T) {
var r *Reload
r.Failed()
r.RestartRequired()
r.RestartRequiredIfChanged("a")
}
+2 -1
View File
@@ -57,7 +57,8 @@ func newConnectionManagerFromConfig(l *slog.Logger, c *config.C, hm *HostMap, p
}
cm.reload(c, true)
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("timers.connection_alive_interval", "timers.pending_deletion_interval")
cm.reload(c, false)
})
+9
View File
@@ -11,6 +11,7 @@ import (
"syscall"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/overlay"
)
@@ -47,6 +48,7 @@ type Control struct {
f *Interface
l *slog.Logger
commands *commands.Registry
ctx context.Context
cancel context.CancelFunc
sshStart func()
@@ -372,6 +374,13 @@ func (c *Control) CloseAllTunnels(excludeLighthouses bool) (closed int) {
return
}
// Commands returns the registry the daemon's surfaces serve from. It is the live
// registry, not a copy: a command registered on it is served to every ssh session that
// starts afterwards. Clone it to run commands without that reach.
func (c *Control) Commands() *commands.Registry {
return c.commands
}
func (c *Control) Device() overlay.Device {
return c.f.inside
}
+3 -3
View File
@@ -151,9 +151,9 @@ func (c *fakeConn) WriteTo(_ []byte, _ netip.AddrPort) error { return nil }
func (c *fakeConn) WriteBatch(bufs [][]byte, _ []netip.AddrPort) (int, error) {
return len(bufs), nil
}
func (c *fakeConn) ReloadConfig(_ *config.C) {}
func (c *fakeConn) SupportsMultipleReaders() bool { return true }
func (c *fakeConn) Close() error { c.closed = true; return nil }
func (c *fakeConn) ReloadConfig(_ *config.C, _ *config.Reload) {}
func (c *fakeConn) SupportsMultipleReaders() bool { return true }
func (c *fakeConn) Close() error { c.closed = true; return nil }
type multiqueueDevice struct {
*fakeDevice
+2 -1
View File
@@ -69,9 +69,10 @@ func newDnsServerFromConfig(ctx context.Context, l *slog.Logger, pki *PKI, hostM
ds.mux = dns.NewServeMux()
ds.mux.HandleFunc(".", ds.handleDnsRequest)
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := ds.reload(c, false); err != nil {
ds.l.Error("Failed to reload DNS responder from config", "error", err)
r.Failed()
}
})
-131
View File
@@ -1,131 +0,0 @@
# Post-quantum handshake transport compared with QUIC
Status: analysis, 2026-10-14. Compares the fragment transport in
[pq-handshake.md](pq-handshake.md) with how QUIC makes its handshake reliable. TLS is out of
scope; only the transport mechanics are compared.
## Summary
QUIC's handshake reliability is closed-loop: the receiver acknowledges, and the sender
repairs exactly what is missing. Ours is open-loop: repair symbols are sent up front, and a
timer sends fresh symbols. At handshake sizes (K <= 15) a QUIC-style acknowledgement
shrinks to a 16-bit bitmap, and with an MDS code to a single count. So the real comparison
is forward error correction versus acknowledgements, not QUIC versus us.
FEC is the right core. One piece of feedback is worth adding, initiator to responder and for
msg2 only (see [Worth taking from QUIC](#worth-taking-from-quic)).
## What QUIC does
- **Messages as a stream.** Handshake bytes travel as `CRYPTO` frames (offset and length)
and are reassembled in order. Initial and Handshake packets each have their own packet
number space.
- **Immediate acknowledgements.** Receivers must acknowledge Initial and Handshake packets
without delay. ACK frames list ranges of received packet numbers.
- **Loss detection** (RFC 9002). A packet is lost once 3 later packets are acknowledged, or
after 9/8 of the round-trip time. Lost bytes are resent in new packets.
- **Probe timeout (PTO).** If a whole flight vanishes, a probe timer fires. Before any RTT
sample it starts at about 1 s (333 ms initial RTT plus 4x its variance) and doubles each
time. Both endpoints run it.
- **Amplification limit and anti-deadlock.** The server may send at most 3x what it
received until the client's address is validated. The client must keep probing so a
server stuck at that limit cannot deadlock.
- **Retry.** A stateless round trip to validate the address. `NEW_TOKEN` lets later
connections skip it.
- **Congestion control from the first packet.** The initial window is 10 packets of 1200
bytes.
## Side by side
| | QUIC | pq-handshake |
|---|---|---|
| Receiver feedback | ACK ranges, sent immediately | None; the next handshake message is the only acknowledgement |
| Partial loss | +1 to 2 RTT (ACK, detect, resend) | 0 RTT while at most m symbols are lost |
| Total loss | PTO: about 1 s, then doubling | Next PQ tick: 300 ms, then 1 s (alternating), or 100/300/600 ms (sticky) |
| Spurious retransmit | Wasted bytes | Still useful: fresh repair indexes add to what the receiver holds |
| Reassembly | Every byte offset, in order | Any K of up to 256 symbols |
| Who retransmits | Both sides on their own timers | Initiator on a timer; the responder resends msg2 only when a repeat msg1 arrives (rate limited) |
| State from the first packet | Server buffers and acknowledges partial flights | Capped pre-authentication table that never acknowledges |
| Address validation | Retry, 1 RTT | Cookie under load, 1 RTT |
From the [simulation results](pq-handshake.md#simulation-results): phase 2 at 5% random
loss with no FEC completes on the first attempt 39.8% of the time. Under
acknowledgement-based recovery roughly 60% of handshakes would pay at least one extra round
trip. With FEC, 0.4% wait for a retry. Spending bandwidth to save round trips is the stated
design preference, and open-loop recovery delivers it.
## Where the designs already agree
- **1200-byte floor and the 3x amplification budget**, both taken directly from QUIC.
- **Cookie and Retry.** Both cost one round trip and both can be forged by an on-path
attacker. QUIC's Retry integrity tag uses a fixed, published key.
- **On-path injection.** QUIC's Initial keys are derived from the client's connection ID,
which an observer can read, so QUIC accepts the same handshake-blocking injection
weakness documented for our symbols in
[Injected symbols](pq-handshake.md#injected-symbols).
- **Phase 2 msg3 confirmation.** Resending msg3 until any authenticated packet arrives,
with a `Test` from the responder on completion, is QUIC's handshake confirmation and
`HANDSHAKE_DONE`.
## Worth taking from QUIC
### 1. A msg2 progress probe from the initiator
This is the main gap. Today, recovering a partially lost msg2 means the initiator resends a
full msg1 attempt; the responder decodes it, matches it in the answered-msg1 cache, and
resends msg2. In phase 2 that is about 12 fragments (~13 KB) uploaded just to act as an
acknowledgement, and recovery waits for the 300 ms or 1 s tick.
- The initiator already knows the responder's index: msg2 fragments carry it as `FragID`
(`sendFragmentedResponse` passes `hostinfo.localIndexId`).
- A one-packet probe, "msg2: have n of K", sent to that index would recover in about one
round trip. The responder finds it the way phase 2 plans to find msg3.
- Timing comes for free: msg1 send to first msg2 fragment is an RTT sample. Probe at
roughly RTT/8 plus the repair delay after the first msg2 fragment if decoding has not
completed.
- Forging a probe needs the 32-bit responder index, so only an on-path attacker can, and
the existing 3x budget bounds the reply.
- If no msg2 fragment arrived at all, the initiator keeps today's msg1 resend. That is the
same tail-loss case QUIC's PTO covers.
The reverse direction, the responder reporting progress on msg1, should not be added. It
needs per-entry timers on the pre-authentication path, which is exactly the state QUIC
servers carry and the reason QUIC needs Retry.
### 2. Buffer early data in phase 2
QUIC stacks hold a few packets they cannot decrypt yet. The phase 2 plan drops data that
reaches the responder before msg3 decodes. Because repair fragments trail by
`handshakes.pq.repair_delay` (10 ms), losing one msg3 data fragment drops about 10 ms of
the initiator's first data while the repair is in flight. A small queue on the pending
responder entry, replayed once msg3 verifies, avoids this.
### 3. Burst size across remotes
QUIC handshakes on a single path. `handleOutbound` sends the same `now` and `later` sets to
every known remote and to relays (`StartRelays`).
- Phase 1 is fine.
- In phase 2, four remotes means about 48 packets (~54 KB) leaving one uplink at once,
every PQ tick, about 4x QUIC's initial window.
- The Gilbert-Elliott simulation treats each path independently, so it does not capture
this self-inflicted burst on a constrained uplink such as cellular.
- Options: send the full FEC set only to the most likely remote and a reduced set to the
others, or interleave packets across remotes.
## Not worth taking
- **Packet numbers, ACK ranges, an RTT estimator, loss thresholds, congestion control.**
QUIC pays for this machinery because it also serves a data transport. Nebula carries
packets and leaves reliability to the overlay's own transports. Building it only for
handshakes of at most 15 fragments buys nothing a symbol count cannot.
- **Offset-based reassembly.** It gives up the any-K-of-256 property and makes spurious
retransmits wasteful again.
- **An RTT-adaptive retry timer.** Low value. The fixed 100 ms linear schedule already fires
sooner than QUIC's 1 s first PTO, and a spurious resend costs bandwidth, not correctness.
## Conclusion
Keep FEC as the core and add the msg2 progress probe. That gets QUIC's roughly one round
trip tail recovery without pre-authentication state, and cuts phase 2's retry cost on the
uplink by about 10x. Items 2 and 3 are phase 2 concerns; item 1 also helps phase 1.
-348
View File
@@ -1,348 +0,0 @@
# Post-quantum handshakes
Status: draft. Decisions below were made on 2026-10-13; open items are listed at the end.
Phase 1 is implemented (see [Implementation](#implementation-phase-1)).
## Goals
1. **Confidentiality against harvest-now-decrypt-later.** Traffic recorded today must stay
secret after a cryptographically relevant quantum computer exists. This needs a
post-quantum key exchange now.
2. **Post-quantum authentication, later.** Needs certificates whose CA signature and host
keys are post-quantum. This requires reissuing certificates, so it ships second.
3. **Never rely on IP fragmentation.** ML-KEM and ML-DSA do not fit in one packet, so the
handshake must carry messages larger than the path MTU itself.
Non-goals: identity hiding beyond what IX offers today, and signatures at handshake time.
The only signature in the system stays the CA signature inside the certificate.
## Primitives and sizes
| Primitive | Size |
|---|---|
| ML-KEM-1024 encapsulation key | 1568 B |
| ML-KEM-1024 ciphertext | 1568 B |
| ML-DSA-87 public key / signature | 2592 B / 4627 B |
| Today's IX msg1 / msg2 (v2 cert, 151 B) | 252 B / 287 B |
`crypto/mlkem` is available in Go 1.26 (the current `go.mod`); `crypto/mldsa` arrives in
Go 1.27 and is only needed for phase 2.
## Phase 1: ephemeral ML-KEM on today's certificates
A new handshake subtype, `ix_psk2_mlkem1024` (2), IX-shaped (2 messages; the responder
completes on msg1 as today), using the flynn/noise `psk2` modifier. The Noise protocol name
becomes `Noise_IXpsk2_...`, which separates it from the classical subtype.
- **msg1**: the initiator generates a fresh ML-KEM-1024 key pair and places the
encapsulation key in `NebulaHandshakeDetails.KEMKey` (field 9). msg1's payload is mixed
into the Noise transcript, so it cannot be stripped or replaced without msg2 failing.
With the psk modifier Noise also mixes `e` into the cipher key, so msg1's static key and
payload are encrypted, though only under a key derived from the public ephemeral.
- **Responder**: after `ReadMessage(msg1)` it encapsulates to that key, obtaining
`(ct, ss)`, and calls `HandshakeState.SetPresharedKey(psk)` before writing msg2, with
`psk = HKDF-Extract(SHA-256, salt = "nebula ix_psk2_mlkem1024 v1", ikm = ss || ct)`.
The ciphertext is bound in because, unlike the encapsulation key, it is not part of the
Noise transcript.
- **msg2** on the wire is `ct || noise_msg2`. The ciphertext travels outside the Noise
message because the initiator needs `ss` before processing msg2's `psk` token. Any change
to `ct` yields a different `ss` (ML-KEM implicit rejection) and the handshake fails.
Transport keys depend on both the X25519 DH outputs and the ML-KEM shared secret.
Authentication is unchanged and classical. With a minimal v2 certificate msg1 and msg2 are
about 1.84 KB each, so each travels as 2 data and 2 repair fragments of about 936 bytes.
A KEM key on the classical subtype, or a missing or malformed one on this subtype, is a fatal
protocol error; a forged ciphertext only fails that packet.
## Phase 2: pqIX hybrid with certificate v3
Certificate v3 is hybrid: the classical static key and CA signature plus a static
ML-KEM-1024 key and an ML-DSA-87 CA signature. Host authentication is by KEM rather than by
signature, following PQNoise's pqIX, while keeping the classical DH tokens:
- **msg1** (I to R): `e, s`, ephemeral ML-KEM key, initiator cert.
- **msg2** (R to I): `e, s`, ciphertext to I's ephemeral KEM key, ciphertext to I's static
KEM key, responder cert.
- **msg3** (I to R): ciphertext to R's static KEM key, plus first data. R authenticates I
when msg3 decrypts; I authenticates R implicitly when R's first reply decrypts.
msg3 is about 1.6 KB, 2 data and 2 repair fragments, and adds no new message types:
- **Responder.** After sending msg2 the responder keeps a pending Machine, found by its
local index like the initiator's today; this is the multi-message responder support the
manager lacks now. A repeat msg1 maps to it through the answered-msg1 cache and gets msg2
resent with fresh symbols, exactly as in phase 1. Until msg3 verifies, data for that
index is dropped silently: it must not draw a RecvError, which would tear down the
initiator's new tunnel. On completing it sends an empty `Test`/`TestReply` under the
new keys, so the initiator hears back even when there is no traffic.
- **Initiator.** Writing msg3 yields the transport keys, so the hostinfo moves to the main
hostmap and data flows at once, readable only by the real responder. The handshake entry
stays behind in a confirming state: each retry tick resends msg3 with fresh symbols until
any packet from the responder decrypts on the new tunnel, and if that never happens
within the handshake timeout the tunnel is torn down. Losing msg3 costs the data sent
behind it, which the overlay's own transport retransmits.
- msg3's fragments use the responder's index as RemoteIndex and land on the responder's
pending entry, so they never enter the pre-authentication table.
The initiator never caches the responder's certificate, so every handshake has three
messages and carries both full certificates. With ML-DSA-87:
| Message | Size | Data fragments (k) |
|---|---|---|
| msg1 | ~8.2 KB | 8 |
| msg2 | ~9.8 KB | 9 |
| msg3 | ~1.6 KB | 2 |
flynn/noise cannot mix multiple KEM secrets into the chaining key, so phase 2 needs a
KEM-aware Noise symmetric state written in this repository. Both subtypes coexist.
Cryptographic review of the hybrid composition will be done by in-house cryptographers.
## Fragment transport
### Requirements
- No IP fragmentation: every datagram's UDP payload is at most 1232 bytes **after** relay
wrapping (16 B header + 16 B tag), the IPv6 minimum MTU budget also used by DNS Flag Day
2020 and Solana's Turbine.
- Every fragment of a handshake uses the same 5-tuple.
- Bounded pre-authentication state, bounded amplification, selective loss recovery,
tolerance of reordering and duplication, and a strictly validated, fuzzable parser.
### Wire format
Every packet of a fragmented handshake subtype is:
```
Nebula header (16) Version | Type=Handshake | Subtype | Reserved=0 | RemoteIndex | Counter
Fragment header (8) FragID u32 | MsgLen u16 | Index u8 | Flags u8
Cookie (16) only when Flags has 0x01
Symbol SymbolSize bytes
```
- `Counter` is the Noise message number (1, 2, 3). `RemoteIndex` is 0 for msg1, as today.
- `FragID` is the sender's local index (random, non-zero).
- `MsgLen` is the length of the whole message. Both ends derive the layout from it:
`K = ceil(MsgLen / 1160)`, `SymbolSize = ceil(MsgLen / K)`. Every symbol has the same
size. Room for a cookie is always reserved, so a packet is at most
16 + 8 + 16 + 1160 = 1200 bytes with or without one.
- `Index < K` is a data symbol (a slice of the message, the last one zero padded);
`Index >= K` is a repair symbol.
- `Flags` bit 0x01 means a cookie follows the fragment header. Other bits must be zero.
A cookie reply, sent by a responder under load, is a bare 32 byte packet:
```
Nebula header (16) Version | Type=Handshake | Subtype | Reserved=0 | RemoteIndex=FragID | Counter=0
Cookie (16)
```
It names the initiator's FragID, so the initiator finds its attempt the same way it finds
msg2.
### Forward error correction
Messages are protected with a systematic Reed-Solomon code over GF(2^8), implemented in
`handshake/fec`:
- The repair rows form a Cauchy matrix, `coefficient(row, col) = 1 / (row XOR col)` with
`row >= K > col`, so every square submatrix is invertible and **any K distinct symbols
reconstruct the message**. Naive identity-over-Vandermonde constructions are not MDS
(the PAR1 flaw and the 1997 Plank tutorial erratum).
- A repair row depends only on its index, so repair symbols are generated on demand and
their indexes never need to be agreed in advance. Up to 256 symbols exist per message.
- Known-answer vectors in `fec_test.go` pin the encoding and were cross-checked against an
independent implementation; changing them is a wire format change.
- Cost for a 9.8 KB message (k=9): about 13 µs for three repair symbols, about 19 µs to
decode with three data symbols missing.
**The sender chooses m**, the number of repair symbols per attempt. The receiver only needs
any K symbols, so this is policy and can change without a protocol change. The starting
rule is `m = max(2, ceil(K/3))`.
**Retries send fresh symbols.** Attempt 0 sends indexes `0 .. K+m-1`; each retry sends
the next `K+m` unused indexes. The receiver merges all distinct symbols across attempts,
so no received fragment is wasted. Indexes are a deterministic function of the message,
so retransmits stay idempotent: the responder caches msg2 itself and regenerates any
symbol.
**Repair symbols trail the data.** Data symbols go out back to back; repair symbols follow
after a short delay, so a queue tail-drop at the end of the data run does not also take the
repair. In simulation this lifts small messages substantially (k=2 at 10% loss, burst 8:
91.4% to 98.2% first-attempt success). Pacing every fragment was also evaluated; it gains
about 5 more points for k=7..9 on bursty paths, which is not worth the extra machinery,
and the retry with fresh symbols covers the remainder.
### Receiver rules
Implemented in `handshake/fragment` (`Sender`, `Assembly`, and the capped `Table`).
- Fragments are stored individually as they arrive and charged against a memory cap.
Nothing is allocated from a claimed size alone (FragmentSmack, game-engine "fragment
bombs").
- `MsgLen` must be identical for every fragment of a `FragID`; a mismatch drops the whole
entry. msg1 is capped at `MsgLen <= 16384` (K <= 15).
- The decoder keeps at most K distinct symbols, so an entry never exceeds about MsgLen bytes
however many repair symbols arrive.
- When K symbols are present the message is decoded and handed to the handshake `Machine`
as a synthetic `[header | message]`. The Machine never sees fragments.
- A completed entry is forgotten immediately. A late duplicate fragment starts a new entry
and, once enough arrive, reconstructs the same message again; the handshake layer's
duplicate detection recognises it.
- If Noise rejects a decoded message, the entry is discarded and the next attempt
supplies fresh symbols. (Possible refinement: keep a few spare symbols so an injected
bad symbol can be identified by re-encoding instead of waiting for a retry.)
- Pre-authentication msg1 entries live in a table keyed by (source address or relay index,
FragID) with a total memory cap, per-source quotas (/32 for IPv4, /64 for IPv6), a
sliding TTL of about 2 s refreshed by progress, and random eviction under pressure.
msg2 and msg3 reassembly attaches to the existing pending handshake found by RemoteIndex.
### Retransmission and amplification
- Only timers cause transmission. Reactive resends (the responder resending its cached
msg2 when a duplicate msg1 completes) are rate limited to one per handshake per retry
interval, to avoid TFTP's Sorcerer's Apprentice amplification.
- Duplicate msg1 detection compares the decoded message to the cached one **before**
running Noise or certificate verification.
- Before an address is validated, a responder sends at most 3x the bytes it received from
it (QUIC's rule). Normal handshakes sit near 1.1-1.2x.
- In phase 2 the initiator retransmits msg3 until any authenticated packet arrives from
the responder; the responder sends a small confirmation on completion and holds tunnel
traffic until msg3 verifies.
### Denial of service
The responder can no longer complete statelessly on msg1, since msg1 arrives in pieces.
Defenses are the capped table above plus a **cookie under load**, in the style of
WireGuard's mac2, QUIC Retry and the SCTP state cookie:
- Once the msg1 table is at least half full (by bytes or by entries), every msg1 fragment
must carry a valid cookie. One without is answered with a cookie reply and leaves no
state. Fragments that already belong to an entry need one too, so an attacker cannot
hold old entries open.
- A cookie is `HMAC-SHA256(secret, source address and port, relay channel, FragID)`
truncated to 16 bytes. The secret rotates every 2 minutes and the previous one is still
accepted, so a cookie is good for 2 to 4 minutes.
- Replies are never larger than the packet that prompted them (no amplification) and are
rate limited as a whole (a burst of 256, then 1000 per second), so spoofed floods cannot
turn the responder into a reflector.
- The initiator attaches the cookie to its PQ attempt, so every later packet carries it,
and immediately resends fresh symbols to the address the reply came from, at most once
per retry interval. Under load a PQ handshake costs one extra round trip, which in
practice still beats the classical attempt on the next retry tick.
- An attacker on the path can forge a cookie reply; the bogus cookie is simply rejected
and answered with a real one.
A CA-keyed pre-filter MAC was considered and rejected: networks commonly trust many CAs.
#### Injected symbols
The decoder keeps the first K distinct symbols it sees, so one forged symbol in those K
spoils that attempt's decode; Noise rejects the result and the next attempt brings fresh
symbols. Forging one needs the attempt's FragID or index, which are random 32 bit values,
and for msg1 the initiator's exact address and port too, so in practice only an attacker
who sees the traffic can do it. An attacker who can also drop packets gains nothing new.
One who can only inject, and races a forged symbol into every attempt, can stop a PQ
handshake where it could not stop a classical one, since classical msg2s are tried one
packet at a time. Keeping spare symbols to find a consistent subset only raises the number
of forgeries needed per attempt, and authenticating symbols is impossible for msg1, which
travels before any key exists. So the decoder deliberately does neither. A non-sticky
initiator still gets a classical tunnel from its next attempt; a sticky one, by design,
does not downgrade.
## Rollout
- Separate settings for initiating and accepting PQ handshakes, both off by default:
`handshakes.pq.initiate`, `handshakes.pq.accept`. `handshakes.pq.repair_delay` (default
10ms) sets how long repair fragments trail the data fragments.
- `handshakes.pq.require` makes every handshake PQ: it implies `initiate` and `accept`,
never offers a classical attempt, and drops classical msg1s.
- Old nodes silently drop unknown subtypes, so discovery is speculative: an initiator with
`initiate` enabled alternates PQ and classical attempts on successive timer retries,
keeping both pending, and the first to complete wins. A path nothing has been sent over
yet gets the PQ attempt first: an early resend triggered by a lighthouse reply is PQ, and
a relay that comes up pushes the PQ attempt through at once, counting as the PQ turn.
Without that, relays set up during the first attempt handed every relayed handshake to
the classical retry (seen on real hosts: 11 of 11). Once a msg2 fragment or cookie reply
arrives for the PQ attempt the peer evidently speaks PQ, and classical stops getting
turns.
- All `handshakes.pq` settings reload without a restart. Handshakes in flight carry on;
the new settings apply to the next attempt and the next packet.
- `handshakes.pq.sticky` (default off): once a peer has completed a PQ handshake, never fall
back to classical for it again during this process lifetime. This resists active
downgrade after first contact.
- No lighthouse capability advertisement.
## Simulation results
Gilbert-Elliott channel (bad state drops everything), losses expressed as average rate and
mean burst length in packets. First-attempt success for the whole handshake with
`m = max(2, ceil(K/3))` and repair delayed 16 packet slots, compared with no FEC:
| Handshake | Loss | Burst 1 | Burst 2 | Burst 4 | Burst 8 |
|---|---|---|---|---|---|
| phase 1 (2, 2) no FEC | 5% | 81.4% | 85.6% | 87.9% | 89.0% |
| phase 1 (2, 2) FEC | 5% | 99.9% | 99.4% | 99.3% | 98.6% |
| phase 1 (2, 2) FEC | 10% | 99.3% | 97.5% | 97.4% | 96.3% |
| phase 2 (8, 9, 2) no FEC | 5% | 37.8% | 56.0% | 69.3% | 77.1% |
| phase 2 (8, 9, 2) FEC | 5% | 99.6% | 93.7% | 89.9% | 88.4% |
| phase 2 (8, 9, 2) FEC | 10% | 95.3% | 83.5% | 77.9% | 76.7% |
With fresh repair symbols, phase 2 completes within two attempts at least 99.1% of the time
in every scenario above, and within three at least 99.98%.
The simulators and their outputs are in [pq-handshake-sim](pq-handshake-sim/README.md),
including a comparison with classical handshakes and a model of time to a working tunnel
under Nebula's retry schedule.
## Prior art
- **WireGuard / Rosenpass**: fit every message in one packet; cookies and biscuits for
statelessness. Not possible at our sizes.
- **IKEv2**: RFC 7383 fragments only after keys exist, with each fragment authenticated;
RFC 9242/9370 move PQ key exchange into intermediate exchanges for that reason.
- **QUIC / DTLS 1.3**: offset-based reassembly with ACKs; QUIC's 3x amplification limit and
1200-byte floor.
- **Solana Turbine**: Reed-Solomon over 1232-byte UDP packets through untrusted relays;
moved to Merkle-authenticated shreds after unauthenticated repair symbols enabled
poisoning.
- **kcp-go, UDPspeeder, CM256**: RS FEC over UDP; parity must scale up for small groups.
- **gQUIC FEC (removed)**: FEC does not pay for bulk transfer under bursty loss; it does
for short, latency-critical transfers such as handshakes.
- **TFTP, NFS duplicate request cache, memcached UDP, IP reassembly bugs**: retransmit
storms, ID-wrap misassembly, reflection amplification, and allocation from claimed sizes.
## Implementation (phase 1)
- `handshake`: the `ix_psk2_mlkem1024` subtype (`patterns.go`, `kem.go`), the `KEMKey`
payload field, and the psk2 plumbing in `Machine`.
- `handshake/fec`, `handshake/fragment`: the code and the fragment transport.
- `handshake_manager_pq.go` and the manager:
- **Initiator.** Attempts alternate PQ and classical, starting with PQ (sticky peers get
only PQ). The PQ attempt has its own local index registered alongside the classical
one; whichever response completes first wins, and the other index is released
(`settleIndex`). A timed out or deleted pending handshake releases both.
- **Responder.** msg1 fragments go into the capped pre-authentication table. A decoded
msg1 is looked up by hash among recently answered ones before any Noise, KEM or
certificate work: the trailing repair fragments of an attempt rebuild a msg1 its data
fragments already completed, and a retrying initiator offers fresh symbols for the
same msg1. A repeat goes straight to the cached-response resend, which offers fresh
msg2 symbols, is rate limited to one per retry interval and is held to the 3x
amplification budget.
- **Relays** carry fragments unchanged; repair fragments trail via a timer.
- The pre-authentication table is capped at 4 MiB, 1024 entries, 4 per source (IPv4 /32,
IPv6 /64, or relay channel), 16 KiB messages and a 3 s idle timeout. Cookies are
demanded from half full.
- Tests: Noise-level KEM tests, codec and fragment tests and fuzzers, manager unit tests
for the response budget and rate limit, and e2e tests for a clean handshake, loss of
data fragments in both directions, fallback to a peer that does not accept PQ, a lost
msg2 recovered by fresh symbols, sticky mode under an attacker dropping PQ packets,
relays, index cleanup on completion, fallback and timeout, and a responder flooded from
spoofed sources demanding and accepting a cookie. Unit tests cover cookie binding and
rotation, the never-amplify rule, the initiator's rate limited resend, and reload. ML-KEM and HKDF also
pass under `GOEXPERIMENT=boringcrypto` and `GODEBUG=fips140=on`.
## Open items
None for phase 1. Phase 2 needs certificate v3, the KEM-aware Noise symmetric state, and
the msg3 design above.
+164
View File
@@ -0,0 +1,164 @@
//go:build e2e_testing
package e2e
import (
"bytes"
"fmt"
"path/filepath"
"regexp"
"strings"
"testing"
"time"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/sshd"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/require"
)
// volatile lists everything in command output that legitimately differs between runs.
// Each run mints fresh certificates and indexes, so these are replaced with a marker
// before comparison. Keep the list short: anything added here is something the golden
// stops guarding.
var volatile = []struct {
re *regexp.Regexp
with string
}{
{regexp.MustCompile(`[0-9a-f]{40,}`), "<hex>"}, // fingerprints, keys, signatures
{regexp.MustCompile(`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})`), "<time>"}, // RFC 3339
{regexp.MustCompile(`\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}(?:\.\d+)? [+-]\d{4} \w+`), "<time>"}, // time.Time.String()
{regexp.MustCompile(`(?s)(-----BEGIN [A-Z ]+-----\n).*?(\n-----END [A-Z ]+-----)`), "${1}<pem>${2}"}, // print-cert -raw
{regexp.MustCompile(`(-----BEGIN [A-Z ]+-----\\n).*?(\\n-----END [A-Z ]+-----)`), "${1}<pem>${2}"}, // the same, inside a JSON string
{regexp.MustCompile(`"(localIndex|remoteIndex|messageCounter)":\d+`), `"$1":<n>`},
{regexp.MustCompile(`(?m)^(\s*"?(?:LocalIndex|RemoteIndex|localIndex|remoteIndex|messageCounter)"?:?\s*)\d+`), "${1}<n>"}, // and the -pretty form
}
func normalise(s string) string {
for _, v := range volatile {
s = v.re.ReplaceAllString(s, v.with)
}
return s
}
// outputCase is one command line to run through the ssh dispatch path.
type outputCase struct {
name string
args []string
}
// TestCommandOutput pins the bytes every command produces, in every format, through the
// ssh dispatch path. It is the safety net for moving rendering out of the commands: a
// change here is a change someone at an ssh prompt would see. Regenerate with:
//
// go test -tags e2e_testing ./e2e -run TestCommandOutput -update
func TestCommandOutput(t *testing.T) {
ca, _, caKey, _ := cert_test.NewTestCaCert(
cert.Version1, cert.Curve_CURVE25519,
time.Now(), time.Now().Add(10*time.Minute),
nil, nil, []string{},
)
control, _, _, _ := newSimpleServer(cert.Version1, ca, caKey, "output-golden", "10.222.0.1/24", nil)
control.Start()
t.Cleanup(func() { control.Stop() })
// As an ssh session does: a clone of the daemon's registry with logout added, which
// help on the clone lists and describes. There is no session to end here.
reg := control.Commands().Clone()
reg.Register(sshd.LogoutCommand(func() {}))
// run dispatches as the ssh exec path does and returns the output and the error, which
// over ssh is exit status 1.
run := func(args ...string) (string, error) {
buf := &bytes.Buffer{}
err := reg.DispatchArgs(args, commands.NewStringWriter(buf))
return buf.String(), err
}
// Seed a pending tunnel so the listing and per-address commands have something to show.
seed, err := run("create-tunnel", "10.222.0.9")
require.NoError(t, err)
require.Equal(t, "Created\n", seed, "seeding a tunnel should succeed before anything else is recorded")
cases := []outputCase{
{"version", nil},
{"device-info", nil},
{"list-hostmap", nil},
{"list-hostmap", []string{"-by-index"}},
{"list-pending-hostmap", nil},
{"list-pending-hostmap", []string{"-by-index"}},
{"list-lighthouse-addrmap", nil},
{"print-cert", nil},
{"print-cert", []string{"-raw"}},
{"print-cert", []string{"10.222.0.9"}},
{"print-cert", []string{"10.222.0.250"}},
{"print-cert", []string{"not-an-address"}},
{"print-tunnel", nil},
{"print-tunnel", []string{"10.222.0.9"}},
{"print-tunnel", []string{"10.222.0.250"}},
{"print-relays", nil},
{"query-lighthouse", nil},
{"query-lighthouse", []string{"10.222.0.9"}},
{"query-lighthouse", []string{"not-an-address"}},
// The test logger cannot change level or format, so these cases record the
// refusal each command prints on it, not a successful change.
{"log-level", nil},
{"log-level", []string{"debug"}},
{"log-level", []string{"bogus"}},
{"log-level", []string{"info"}},
{"log-format", nil},
{"log-format", []string{"json"}},
{"log-format", []string{"text"}},
{"change-remote", nil},
{"change-remote", []string{"10.222.0.9"}},
{"change-remote", []string{"-address", "10.94.0.10:4242", "10.222.0.9"}},
{"change-remote", []string{"-address", "10.94.0.10:4242", "10.222.0.250"}},
{"create-tunnel", nil},
{"create-tunnel", []string{"10.222.0.9"}},
// A fresh address, so this exercises the handshake start with a remote attached
// rather than the "already handshaking" path above.
{"create-tunnel", []string{"-address", "10.94.0.10:4242", "10.222.0.10"}},
{"create-tunnel", []string{"not-an-address"}},
{"close-tunnel", nil},
{"close-tunnel", []string{"10.222.0.250"}},
{"close-tunnel", []string{"not-an-address"}},
{"help", nil},
{"help", []string{"print-cert"}},
{"help", []string{"logout"}},
{"logout", []string{"-h"}},
{"not-a-command", []string{"10.222.0.9"}},
// reload is last so that whatever a reload does to the daemon cannot leak into an
// earlier case. In this harness it fails: the config was loaded from a string and
// has no path to re-read, which is what the recorded result says.
{"reload", nil},
}
got := &strings.Builder{}
for _, c := range cases {
for _, fmtFlags := range [][]string{nil, {"-json"}, {"-pretty"}} {
args := append([]string{c.name}, fmtFlags...)
args = append(args, c.args...)
fmt.Fprintf(got, "$ %s\n", strings.Join(args, " "))
out, err := run(args...)
out = normalise(out)
got.WriteString(out)
// Output that does not end in a newline is recorded as such, the way diff marks
// it, so the golden sees that byte too and the next prompt still starts a line.
if out != "" && !strings.HasSuffix(out, "\n") {
got.WriteString("\\ No newline at end of output\n")
}
// A failed command is recorded as the exit status an ssh caller sees, so the
// golden pins which outcomes count as failures.
if err != nil {
got.WriteString("exit status 1\n")
}
got.WriteString("\n")
}
}
test.Golden(t, filepath.Join("testdata", "commands_output.golden"), got.String())
}
+1 -1
View File
@@ -1513,7 +1513,7 @@ func TestLighthouseUpdateOnReload(t *testing.T) {
// Reload the config. The lighthouse.hosts change triggers TriggerUpdate,
// which wakes the update worker. It calls SendUpdate, initiating a
// handshake to the new lighthouse and caching the HostUpdateNotification.
require.NoError(t, myConfig.ReloadConfigString(string(newCfg)))
require.NoError(t, myConfig.ReloadConfigString(string(newCfg)).Err)
// Route until the lighthouse receives the HostUpdateNotification.
// This covers: handshake stage 1, stage 2, then the cached update.
-476
View File
@@ -1,476 +0,0 @@
//go:build e2e_testing
package e2e
import (
"net/netip"
"sync"
"testing"
"time"
"github.com/slackhq/nebula"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/e2e/router"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func pqRequire() m {
return m{"handshakes": m{"pq": m{"require": true, "repair_delay": "0s"}}}
}
func pqConfig(initiate, accept, sticky bool) m {
return m{"handshakes": m{"pq": m{
"initiate": initiate,
"accept": accept,
"sticky": sticky,
"repair_delay": "0s",
}}}
}
// handshakeOf returns the handshake header p carries, looking inside relay
// wrapping, plus the fragment header when the handshake is fragmented.
func handshakeOf(p *udp.Packet) (*header.H, *fragment.Header) {
b := p.Data
h := &header.H{}
if h.Parse(b) != nil {
return nil, nil
}
if h.Type == header.Message && h.Subtype == header.MessageRelay {
// Relays authenticate but do not encrypt the inner packet.
b = b[header.Len:]
if h.Parse(b) != nil {
return nil, nil
}
}
if h.Type != header.Handshake {
return nil, nil
}
if !header.IsFragmented(h.Subtype) {
return h, nil
}
// A relayed packet still has the relay's AEAD tag on the end.
for _, tag := range []int{0, 16} {
if len(b) < header.Len+tag {
break
}
if fh, _, err := fragment.Parse(b[header.Len : len(b)-tag]); err == nil {
return h, &fh
}
}
return h, nil
}
// handshakeTally counts handshake packets by style as the router moves them.
type handshakeTally struct {
sync.Mutex
pq, classical int
maxPQLen int
}
func (ht *handshakeTally) observe(p *udp.Packet) {
h, _ := handshakeOf(p)
if h == nil {
return
}
ht.Lock()
defer ht.Unlock()
if h.Subtype == header.HandshakeIXPSK2MLKEM1024 {
ht.pq++
ht.maxPQLen = max(ht.maxPQLen, len(p.Data))
} else {
ht.classical++
}
}
// isTunnelData reports whether p is ordinary tunnel traffic headed for to,
// directly or as the last hop of a relay.
func isTunnelData(p *udp.Packet, receiver, to *nebula.Control) bool {
h := &header.H{}
if receiver != to || h.Parse(p.Data) != nil || h.Type != header.Message {
return false
}
if h.Subtype == header.MessageRelay {
if h.Parse(p.Data[header.Len:]) != nil {
return false
}
}
return h.Type == header.Message && h.Subtype == header.MessageNone
}
func newPQPair(t *testing.T, mine, theirs m) (myControl, theirControl *nebula.Control, myVpn, theirVpn []netip.Prefix, myUdp, theirUdp netip.AddrPort, r *router.R) {
t.Helper()
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
myControl, myVpn, myUdp, _ = newSimpleServer(cert.Version2, ca, caKey, "me", "10.128.0.1/24", mine)
theirControl, theirVpn, theirUdp, _ = newSimpleServer(cert.Version2, ca, caKey, "them", "10.128.0.2/24", theirs)
myControl.InjectLightHouseAddr(theirVpn[0].Addr(), theirUdp)
r = router.NewR(t, myControl, theirControl)
myControl.Start()
theirControl.Start()
return
}
func TestPQHandshake(t *testing.T) {
t.Parallel()
// They only accept, which is all a PQ initiator needs.
myControl, theirControl, myVpn, theirVpn, myUdp, theirUdp, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Zero(t, tally.classical, "no classical handshake should be needed")
assert.GreaterOrEqual(t, tally.pq, 4, "both messages travel as several fragments")
assert.LessOrEqual(t, tally.maxPQLen, fragment.MaxPacketLen)
assertHostInfoPair(t, myUdp, theirUdp, myVpn, theirVpn, myControl, theirControl)
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
assert.Empty(t, myControl.ListHostmapIndexes(true), "no pending index may outlive the handshake")
assert.Empty(t, theirControl.ListHostmapIndexes(true))
myControl.Stop()
theirControl.Stop()
}
// TestPQHandshakeLoss drops data fragments of both messages; the repair
// fragments must carry the handshake without a retry.
func TestPQHandshakeLoss(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
var tally handshakeTally
dropped := 0
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
if h, fh := handshakeOf(p); fh != nil {
// Every data fragment of msg1, and the first of msg2.
if (h.MessageCounter == 1 && int(fh.Index) < 2) || (h.MessageCounter == 2 && fh.Index == 0) {
dropped++
return router.Drop
}
}
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Equal(t, 3, dropped)
assert.Zero(t, tally.classical, "the first attempt should succeed from repair fragments")
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
}
// TestPQFallback has a peer that does not accept PQ handshakes. The initiator's
// next attempt is classical and the tunnel comes up anyway.
func TestPQFallback(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, myUdp, theirUdp, r := newPQPair(t, pqConfig(true, true, false), nil)
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Positive(t, tally.pq, "a PQ attempt comes first")
assert.Positive(t, tally.classical, "then a classical one")
assertHostInfoPair(t, myUdp, theirUdp, myVpn, theirVpn, myControl, theirControl)
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
assert.Empty(t, myControl.ListHostmapIndexes(true), "the losing PQ attempt's index must be released")
myControl.Stop()
theirControl.Stop()
}
// TestPQResendFreshSymbols loses the responder's whole first msg2. The
// initiator's next PQ attempt offers fresh msg1 symbols, the responder
// recognises the repeat and resends msg2 with symbols it has not sent before.
func TestPQResendFreshSymbols(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
var firstMsg2, freshMsg2, freshMsg1 int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(10*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
h, fh := handshakeOf(p)
switch {
case h != nil && fh == nil:
// Keep the classical fallback out of the way.
return router.Drop
case fh != nil && h.MessageCounter == 2 && fh.Index < 4:
firstMsg2++
return router.Drop
case fh != nil && h.MessageCounter == 2:
freshMsg2++
case fh != nil && h.MessageCounter == 1 && fh.Index >= 4:
freshMsg1++
}
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Equal(t, 4, firstMsg2, "the first msg2 is 2 data and 2 repair fragments")
assert.Positive(t, freshMsg1, "the retry offers fresh msg1 symbols")
assert.Positive(t, freshMsg2, "the resend offers fresh msg2 symbols")
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
}
// TestPQSticky checks a sticky initiator never offers a classical handshake to a
// peer it has already completed a PQ handshake with.
func TestPQSticky(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqConfig(true, true, true), pqConfig(false, true, false))
defer r.RenderFlow()
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
theirControl.GetFromTun(true)
t.Log("Rehandshake while an attacker drops every PQ packet")
var tally handshakeTally
myControl.ReHandshake(theirVpn[0].Addr())
r.RouteForAllExitFuncOrTimeout(time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if h, _ := handshakeOf(p); h != nil && h.Subtype == header.HandshakeIXPSK2MLKEM1024 {
return router.Drop
}
return router.KeepRouting
})
assert.Positive(t, tally.pq, "PQ attempts keep being made")
assert.Zero(t, tally.classical, "sticky must never fall back")
myControl.Stop()
theirControl.Stop()
}
func TestPQRelays(t *testing.T) {
t.Parallel()
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version1, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
useRelays := pqConfig(true, true, false)
useRelays["relay"] = m{"use_relays": true}
amRelay := pqConfig(true, true, false)
amRelay["relay"] = m{"am_relay": true}
myControl, myVpnIpNet, _, _ := newSimpleServer(cert.Version1, ca, caKey, "me ", "10.128.0.1/24", useRelays)
relayControl, relayVpnIpNet, relayUdpAddr, _ := newSimpleServer(cert.Version1, ca, caKey, "relay ", "10.128.0.128/24", amRelay)
theirControl, theirVpnIpNet, theirUdpAddr, _ := newSimpleServer(cert.Version1, ca, caKey, "them ", "10.128.0.2/24", useRelays)
myControl.InjectLightHouseAddr(relayVpnIpNet[0].Addr(), relayUdpAddr)
myControl.InjectRelays(theirVpnIpNet[0].Addr(), []netip.Addr{relayVpnIpNet[0].Addr()})
relayControl.InjectLightHouseAddr(theirVpnIpNet[0].Addr(), theirUdpAddr)
r := router.NewR(t, myControl, relayControl, theirControl)
defer r.RenderFlow()
myControl.Start()
relayControl.Start()
theirControl.Start()
var tally handshakeTally
var relayedPQ, relayedClassical int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpnIpNet[0].Addr(), 80, myVpnIpNet[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(10*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
outer := &header.H{}
if outer.Parse(p.Data) == nil && outer.Type == header.Message && outer.Subtype == header.MessageRelay {
if h, fh := handshakeOf(p); h != nil && fh != nil {
relayedPQ++
assert.LessOrEqual(t, len(p.Data), fragment.MaxPacketLen+32, "a relayed fragment fits the IPv6 minimum MTU")
} else if h != nil {
relayedClassical++
}
}
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpnIpNet[0].Addr(), theirVpnIpNet[0].Addr(), 80, 80)
assert.Positive(t, relayedPQ, "the end to end handshake is PQ through the relay")
assert.Zero(t, relayedClassical, "the relay coming up carries the PQ attempt, not the classical retry")
r.RenderHostmaps("Final hostmaps", myControl, relayControl, theirControl)
myControl.Stop()
relayControl.Stop()
theirControl.Stop()
}
// TestPQTimeoutReleasesIndexes lets a handshake with both a PQ and a classical
// attempt pending time out, and checks neither index is left behind.
func TestPQTimeoutReleasesIndexes(t *testing.T) {
t.Parallel()
mine := pqConfig(true, true, false)
mine["handshakes"].(m)["retries"] = 2
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, mine, pqConfig(false, true, false))
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
r.RouteForAllExitFuncOrTimeout(1500*time.Millisecond, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
return router.Drop
})
assert.Positive(t, tally.pq)
assert.Positive(t, tally.classical)
assert.Nil(t, myControl.GetHostInfoByVpnAddr(theirVpn[0].Addr(), true), "the handshake should have timed out")
assert.Empty(t, myControl.ListHostmapIndexes(true))
myControl.Stop()
theirControl.Stop()
}
// TestPQCookieUnderLoad floods the responder with msg1 fragments from spoofed
// sources until it demands cookies, then checks a real initiator picks up the
// cookie and still completes a PQ handshake.
func TestPQCookieUnderLoad(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, theirUdp, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
// Half of the 1024 entry table is the cookie threshold. Stopping exactly there
// keeps them from answering the spoofed sources, which the router cannot reach.
t.Log("Flood them with the first fragment of msg1s nobody will finish")
for i := range 512 {
from := netip.AddrPortFrom(netip.AddrFrom4([4]byte{198, 18, byte(i >> 8), byte(i)}), 4242)
s, err := fragment.NewSender(make([]byte, 2000), header.HandshakeIXPSK2MLKEM1024, 0, 1, uint32(i+1))
require.NoError(t, err)
pkts, _, err := s.Attempt(0)
require.NoError(t, err)
theirControl.InjectUDPPacket(&udp.Packet{From: from, To: theirUdp, Data: pkts[0]})
}
var tally handshakeTally
var cookieReplies, msg2, withCookie int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if h, fh := handshakeOf(p); h != nil && h.Subtype == header.HandshakeIXPSK2MLKEM1024 {
switch {
case h.MessageCounter == 0:
cookieReplies++
case h.MessageCounter == 2:
msg2++
case fh != nil && fh.HasCookie():
withCookie++
}
}
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Positive(t, cookieReplies, "a loaded responder asks for a cookie")
assert.Positive(t, withCookie, "the initiator resends with it")
assert.Positive(t, msg2, "and the handshake completes post-quantum")
assert.Zero(t, tally.classical, "the cookie round trip beats the classical fallback")
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
}
// TestPQRequire checks require mode never handshakes classically, in either
// direction.
func TestPQRequire(t *testing.T) {
t.Parallel()
t.Run("initiator", func(t *testing.T) {
// They neither speak PQ nor require it; we never fall back.
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqRequire(), nil)
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
r.RouteForAllExitFuncOrTimeout(time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
return router.KeepRouting
})
assert.Positive(t, tally.pq)
assert.Zero(t, tally.classical)
assert.Nil(t, theirControl.GetHostInfoByVpnAddr(myVpn[0].Addr(), false), "no tunnel without PQ")
myControl.Stop()
theirControl.Stop()
})
t.Run("responder", func(t *testing.T) {
// We only speak classical; they refuse it.
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, nil, pqRequire())
defer r.RenderFlow()
var answers int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
r.RouteForAllExitFuncOrTimeout(time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
if h, _ := handshakeOf(p); h != nil && h.MessageCounter == 2 {
answers++
}
return router.KeepRouting
})
assert.Zero(t, answers, "a classical msg1 gets no answer")
assert.Nil(t, theirControl.GetHostInfoByVpnAddr(myVpn[0].Addr(), false))
myControl.Stop()
theirControl.Stop()
})
t.Run("both", func(t *testing.T) {
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqRequire(), pqRequire())
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Zero(t, tally.classical)
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
})
}
+720
View File
@@ -0,0 +1,720 @@
$ version
e2e-test
$ version -json
e2e-test
$ version -pretty
e2e-test
$ device-info
name= cidr=[10.222.0.1/24]
$ device-info -json
{"name":"","cidr":["10.222.0.1/24"]}
$ device-info -pretty
{
"name": "",
"cidr": [
"10.222.0.1/24"
]
}
$ list-hostmap
$ list-hostmap -json
[]
$ list-hostmap -pretty
[]
$ list-hostmap -by-index
$ list-hostmap -json -by-index
[]
$ list-hostmap -pretty -by-index
[]
$ list-pending-hostmap
[10.222.0.9]: []
$ list-pending-hostmap -json
[{"vpnAddrs":["10.222.0.9"],"localIndex":<n>,"remoteIndex":<n>,"remoteAddrs":[],"cert":null,"messageCounter":<n>,"currentRemote":"","currentRelaysToMe":[],"currentRelaysThroughMe":[]}]
$ list-pending-hostmap -pretty
[
{
"vpnAddrs": [
"10.222.0.9"
],
"localIndex": <n>,
"remoteIndex": <n>,
"remoteAddrs": [],
"cert": null,
"messageCounter": <n>,
"currentRemote": "",
"currentRelaysToMe": [],
"currentRelaysThroughMe": []
}
]
$ list-pending-hostmap -by-index
$ list-pending-hostmap -json -by-index
[]
$ list-pending-hostmap -pretty -by-index
[]
$ list-lighthouse-addrmap
10.222.0.9: {}
$ list-lighthouse-addrmap -json
[{"vpnAddr":"10.222.0.9","addrs":{}}]
$ list-lighthouse-addrmap -pretty
[
{
"vpnAddr": "10.222.0.9",
"addrs": {}
}
]
$ print-cert
{
"details": {
"curve": "CURVE25519",
"groups": [],
"isCa": false,
"issuer": "<hex>",
"name": "output-golden",
"networks": [
"10.222.0.1/24"
],
"notAfter": "<time>",
"notBefore": "<time>",
"publicKey": "<hex>",
"unsafeNetworks": []
},
"fingerprint": "<hex>",
"signature": "<hex>",
"version": 1
}
$ print-cert -json
{"details":{"curve":"CURVE25519","groups":[],"isCa":false,"issuer":"<hex>","name":"output-golden","networks":["10.222.0.1/24"],"notAfter":"<time>","notBefore":"<time>","publicKey":"<hex>","unsafeNetworks":[]},"fingerprint":"<hex>","signature":"<hex>","version":1}\ No newline at end of output
$ print-cert -pretty
{
"details": {
"curve": "CURVE25519",
"groups": [],
"isCa": false,
"issuer": "<hex>",
"name": "output-golden",
"networks": [
"10.222.0.1/24"
],
"notAfter": "<time>",
"notBefore": "<time>",
"publicKey": "<hex>",
"unsafeNetworks": []
},
"fingerprint": "<hex>",
"signature": "<hex>",
"version": 1
}\ No newline at end of output
$ print-cert -raw
-----BEGIN NEBULA CERTIFICATE-----
<pem>
-----END NEBULA CERTIFICATE-----
$ print-cert -json -raw
{"details":{"curve":"CURVE25519","groups":[],"isCa":false,"issuer":"<hex>","name":"output-golden","networks":["10.222.0.1/24"],"notAfter":"<time>","notBefore":"<time>","publicKey":"<hex>","unsafeNetworks":[]},"fingerprint":"<hex>","signature":"<hex>","version":1}\ No newline at end of output
$ print-cert -pretty -raw
{
"details": {
"curve": "CURVE25519",
"groups": [],
"isCa": false,
"issuer": "<hex>",
"name": "output-golden",
"networks": [
"10.222.0.1/24"
],
"notAfter": "<time>",
"notBefore": "<time>",
"publicKey": "<hex>",
"unsafeNetworks": []
},
"fingerprint": "<hex>",
"signature": "<hex>",
"version": 1
}\ No newline at end of output
$ print-cert 10.222.0.9
Could not find tunnel for vpn addr: 10.222.0.9
exit status 1
$ print-cert -json 10.222.0.9
Could not find tunnel for vpn addr: 10.222.0.9
exit status 1
$ print-cert -pretty 10.222.0.9
Could not find tunnel for vpn addr: 10.222.0.9
exit status 1
$ print-cert 10.222.0.250
Could not find tunnel for vpn addr: 10.222.0.250
exit status 1
$ print-cert -json 10.222.0.250
Could not find tunnel for vpn addr: 10.222.0.250
exit status 1
$ print-cert -pretty 10.222.0.250
Could not find tunnel for vpn addr: 10.222.0.250
exit status 1
$ print-cert not-an-address
The provided vpn addr could not be parsed: not-an-address
exit status 1
$ print-cert -json not-an-address
The provided vpn addr could not be parsed: not-an-address
exit status 1
$ print-cert -pretty not-an-address
The provided vpn addr could not be parsed: not-an-address
exit status 1
$ print-tunnel
No vpn address was provided
exit status 1
$ print-tunnel -json
flag provided but not defined: -json
Usage:
-pretty
pretty prints json
exit status 1
$ print-tunnel -pretty
No vpn address was provided
exit status 1
$ print-tunnel 10.222.0.9
Could not find tunnel for vpn addr: 10.222.0.9
exit status 1
$ print-tunnel -json 10.222.0.9
flag provided but not defined: -json
Usage:
-pretty
pretty prints json
exit status 1
$ print-tunnel -pretty 10.222.0.9
Could not find tunnel for vpn addr: 10.222.0.9
exit status 1
$ print-tunnel 10.222.0.250
Could not find tunnel for vpn addr: 10.222.0.250
exit status 1
$ print-tunnel -json 10.222.0.250
flag provided but not defined: -json
Usage:
-pretty
pretty prints json
exit status 1
$ print-tunnel -pretty 10.222.0.250
Could not find tunnel for vpn addr: 10.222.0.250
exit status 1
$ print-relays
{"Relays":null}
$ print-relays -json
flag provided but not defined: -json
Usage:
-pretty
pretty prints json
exit status 1
$ print-relays -pretty
{
"Relays": null
}
$ query-lighthouse
No vpn address was provided
exit status 1
$ query-lighthouse -json
The provided vpn address could not be parsed: -json
exit status 1
$ query-lighthouse -pretty
The provided vpn address could not be parsed: -pretty
exit status 1
$ query-lighthouse 10.222.0.9
{}
$ query-lighthouse -json 10.222.0.9
The provided vpn address could not be parsed: -json
exit status 1
$ query-lighthouse -pretty 10.222.0.9
The provided vpn address could not be parsed: -pretty
exit status 1
$ query-lighthouse not-an-address
The provided vpn address could not be parsed: not-an-address
exit status 1
$ query-lighthouse -json not-an-address
The provided vpn address could not be parsed: -json
exit status 1
$ query-lighthouse -pretty not-an-address
The provided vpn address could not be parsed: -pretty
exit status 1
$ log-level
Log level is not reconfigurable on this logger
exit status 1
$ log-level -json
Log level is not reconfigurable on this logger
exit status 1
$ log-level -pretty
Log level is not reconfigurable on this logger
exit status 1
$ log-level debug
Log level is not reconfigurable on this logger
exit status 1
$ log-level -json debug
Log level is not reconfigurable on this logger
exit status 1
$ log-level -pretty debug
Log level is not reconfigurable on this logger
exit status 1
$ log-level bogus
Log level is not reconfigurable on this logger
exit status 1
$ log-level -json bogus
Log level is not reconfigurable on this logger
exit status 1
$ log-level -pretty bogus
Log level is not reconfigurable on this logger
exit status 1
$ log-level info
Log level is not reconfigurable on this logger
exit status 1
$ log-level -json info
Log level is not reconfigurable on this logger
exit status 1
$ log-level -pretty info
Log level is not reconfigurable on this logger
exit status 1
$ log-format
Log format is not reconfigurable on this logger
exit status 1
$ log-format -json
Log format is not reconfigurable on this logger
exit status 1
$ log-format -pretty
Log format is not reconfigurable on this logger
exit status 1
$ log-format json
Log format is not reconfigurable on this logger
exit status 1
$ log-format -json json
Log format is not reconfigurable on this logger
exit status 1
$ log-format -pretty json
Log format is not reconfigurable on this logger
exit status 1
$ log-format text
Log format is not reconfigurable on this logger
exit status 1
$ log-format -json text
Log format is not reconfigurable on this logger
exit status 1
$ log-format -pretty text
Log format is not reconfigurable on this logger
exit status 1
$ change-remote
No vpn address was provided
exit status 1
$ change-remote -json
flag provided but not defined: -json
Usage:
-address string
The new remote address, ip:port
exit status 1
$ change-remote -pretty
flag provided but not defined: -pretty
Usage:
-address string
The new remote address, ip:port
exit status 1
$ change-remote 10.222.0.9
No address was provided
exit status 1
$ change-remote -json 10.222.0.9
flag provided but not defined: -json
Usage:
-address string
The new remote address, ip:port
exit status 1
$ change-remote -pretty 10.222.0.9
flag provided but not defined: -pretty
Usage:
-address string
The new remote address, ip:port
exit status 1
$ change-remote -address 10.94.0.10:4242 10.222.0.9
Could not find tunnel for vpn address: 10.222.0.9
exit status 1
$ change-remote -json -address 10.94.0.10:4242 10.222.0.9
flag provided but not defined: -json
Usage:
-address string
The new remote address, ip:port
exit status 1
$ change-remote -pretty -address 10.94.0.10:4242 10.222.0.9
flag provided but not defined: -pretty
Usage:
-address string
The new remote address, ip:port
exit status 1
$ change-remote -address 10.94.0.10:4242 10.222.0.250
Could not find tunnel for vpn address: 10.222.0.250
exit status 1
$ change-remote -json -address 10.94.0.10:4242 10.222.0.250
flag provided but not defined: -json
Usage:
-address string
The new remote address, ip:port
exit status 1
$ change-remote -pretty -address 10.94.0.10:4242 10.222.0.250
flag provided but not defined: -pretty
Usage:
-address string
The new remote address, ip:port
exit status 1
$ create-tunnel
No vpn address was provided
exit status 1
$ create-tunnel -json
flag provided but not defined: -json
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ create-tunnel -pretty
flag provided but not defined: -pretty
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ create-tunnel 10.222.0.9
Tunnel already handshaking
exit status 1
$ create-tunnel -json 10.222.0.9
flag provided but not defined: -json
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ create-tunnel -pretty 10.222.0.9
flag provided but not defined: -pretty
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ create-tunnel -address 10.94.0.10:4242 10.222.0.10
Created
$ create-tunnel -json -address 10.94.0.10:4242 10.222.0.10
flag provided but not defined: -json
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ create-tunnel -pretty -address 10.94.0.10:4242 10.222.0.10
flag provided but not defined: -pretty
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ create-tunnel not-an-address
The provided vpn address could not be parsed: not-an-address
exit status 1
$ create-tunnel -json not-an-address
flag provided but not defined: -json
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ create-tunnel -pretty not-an-address
flag provided but not defined: -pretty
Usage:
-address string
Optionally provide a real remote address, ip:port
exit status 1
$ close-tunnel
No vpn address was provided
exit status 1
$ close-tunnel -json
flag provided but not defined: -json
Usage:
-local-only
Disables notifying the remote that the tunnel is shutting down
exit status 1
$ close-tunnel -pretty
flag provided but not defined: -pretty
Usage:
-local-only
Disables notifying the remote that the tunnel is shutting down
exit status 1
$ close-tunnel 10.222.0.250
Could not find tunnel for vpn address: 10.222.0.250
exit status 1
$ close-tunnel -json 10.222.0.250
flag provided but not defined: -json
Usage:
-local-only
Disables notifying the remote that the tunnel is shutting down
exit status 1
$ close-tunnel -pretty 10.222.0.250
flag provided but not defined: -pretty
Usage:
-local-only
Disables notifying the remote that the tunnel is shutting down
exit status 1
$ close-tunnel not-an-address
The provided vpn address could not be parsed: not-an-address
exit status 1
$ close-tunnel -json not-an-address
flag provided but not defined: -json
Usage:
-local-only
Disables notifying the remote that the tunnel is shutting down
exit status 1
$ close-tunnel -pretty not-an-address
flag provided but not defined: -pretty
Usage:
-local-only
Disables notifying the remote that the tunnel is shutting down
exit status 1
$ help
Available commands:
change-remote - Changes the remote address used in the tunnel for the provided vpn addr
close-tunnel - Closes a tunnel for the provided vpn addr
create-tunnel - Creates a tunnel for the provided vpn address
device-info - Prints information about the network device.
help - prints available commands or help <command> for specific usage info
list-hostmap - List all known previously connected hosts
list-lighthouse-addrmap - List all lighthouse map entries
list-pending-hostmap - List all handshaking hosts
log-format - Gets or sets the current log format
log-level - Gets or sets the current log level
logout - Ends the current session
print-cert - Prints the current certificate being used or the certificate for the provided vpn addr
print-relays - Prints json details about all relay info
print-tunnel - Prints json details about a tunnel for the provided vpn addr
query-lighthouse - Query the lighthouses for the provided vpn address
reload - Reloads configuration from disk, same as sending HUP to the process
version - Prints the currently running version of nebula
$ help -json
Command not available -json
exit status 1
$ help -pretty
Command not available -pretty
exit status 1
$ help print-cert
print-cert - Prints the current certificate being used or the certificate for the provided vpn addr
-json
outputs as json
-pretty
pretty prints json, assumes -json
-raw
raw prints the PEM encoded certificate, not compatible with -json or -pretty
$ help -json print-cert
Command not available -json
exit status 1
$ help -pretty print-cert
Command not available -pretty
exit status 1
$ help logout
logout - Ends the current session
$ help -json logout
Command not available -json
exit status 1
$ help -pretty logout
Command not available -pretty
exit status 1
$ logout -h
logout - Ends the current session
$ logout -json -h
logout - Ends the current session
$ logout -pretty -h
logout - Ends the current session
$ not-a-command 10.222.0.9
did not understand: not-a-command
Available commands:
change-remote - Changes the remote address used in the tunnel for the provided vpn addr
close-tunnel - Closes a tunnel for the provided vpn addr
create-tunnel - Creates a tunnel for the provided vpn address
device-info - Prints information about the network device.
help - prints available commands or help <command> for specific usage info
list-hostmap - List all known previously connected hosts
list-lighthouse-addrmap - List all lighthouse map entries
list-pending-hostmap - List all handshaking hosts
log-format - Gets or sets the current log format
log-level - Gets or sets the current log level
logout - Ends the current session
print-cert - Prints the current certificate being used or the certificate for the provided vpn addr
print-relays - Prints json details about all relay info
print-tunnel - Prints json details about a tunnel for the provided vpn addr
query-lighthouse - Query the lighthouses for the provided vpn address
reload - Reloads configuration from disk, same as sending HUP to the process
version - Prints the currently running version of nebula
exit status 1
$ not-a-command -json 10.222.0.9
did not understand: not-a-command
Available commands:
change-remote - Changes the remote address used in the tunnel for the provided vpn addr
close-tunnel - Closes a tunnel for the provided vpn addr
create-tunnel - Creates a tunnel for the provided vpn address
device-info - Prints information about the network device.
help - prints available commands or help <command> for specific usage info
list-hostmap - List all known previously connected hosts
list-lighthouse-addrmap - List all lighthouse map entries
list-pending-hostmap - List all handshaking hosts
log-format - Gets or sets the current log format
log-level - Gets or sets the current log level
logout - Ends the current session
print-cert - Prints the current certificate being used or the certificate for the provided vpn addr
print-relays - Prints json details about all relay info
print-tunnel - Prints json details about a tunnel for the provided vpn addr
query-lighthouse - Query the lighthouses for the provided vpn address
reload - Reloads configuration from disk, same as sending HUP to the process
version - Prints the currently running version of nebula
exit status 1
$ not-a-command -pretty 10.222.0.9
did not understand: not-a-command
Available commands:
change-remote - Changes the remote address used in the tunnel for the provided vpn addr
close-tunnel - Closes a tunnel for the provided vpn addr
create-tunnel - Creates a tunnel for the provided vpn address
device-info - Prints information about the network device.
help - prints available commands or help <command> for specific usage info
list-hostmap - List all known previously connected hosts
list-lighthouse-addrmap - List all lighthouse map entries
list-pending-hostmap - List all handshaking hosts
log-format - Gets or sets the current log format
log-level - Gets or sets the current log level
logout - Ends the current session
print-cert - Prints the current certificate being used or the certificate for the provided vpn addr
print-relays - Prints json details about all relay info
print-tunnel - Prints json details about a tunnel for the provided vpn addr
query-lighthouse - Query the lighthouses for the provided vpn address
reload - Reloads configuration from disk, same as sending HUP to the process
version - Prints the currently running version of nebula
exit status 1
$ reload
Reloading config
Config reload failed, the new config did not load: no config files found at
exit status 1
$ reload -json
Reloading config
Config reload failed, the new config did not load: no config files found at
exit status 1
$ reload -pretty
Reloading config
Config reload failed, the new config did not load: no config files found at
exit status 1
+2 -2
View File
@@ -129,7 +129,7 @@ func TestCertUpgrade(t *testing.T) {
}
r.Logf("reload new v2-only config")
err = myC.ReloadConfigString(string(cb))
err = myC.ReloadConfigString(string(cb)).Err
assert.NoError(t, err)
r.Log("yay, spin until their sees it")
waitStart := time.Now()
@@ -222,7 +222,7 @@ func TestCertDowngrade(t *testing.T) {
}
r.Logf("reload new v1-only config")
err = myC.ReloadConfigString(string(cb))
err = myC.ReloadConfigString(string(cb)).Err
assert.NoError(t, err)
r.Log("yay, spin until their sees it")
waitStart := time.Now()
+20 -32
View File
@@ -8,6 +8,7 @@ pki:
cert: /etc/nebula/host.crt
key: /etc/nebula/host.key
# blocklist is a list of certificate fingerprints that we will refuse to talk to
# Entries are the hex fingerprints shown by `nebula-cert print`, an invalid entry fails startup and reload.
#blocklist:
# - c99d4e650533b92061b09918e838a5a0a6aaee21eed1d12fd937682865936c72
# disconnect_invalid is a toggle to force a client to be disconnected if the certificate is expired or invalid.
@@ -43,7 +44,7 @@ static_host_map:
lighthouse:
# am_lighthouse is used to enable lighthouse functionality for a node. This should ONLY be true on nodes
# you have configured to be lighthouses in your network
# you have configured to be lighthouses in your network. Not reloadable.
am_lighthouse: false
# serve_dns optionally starts a dns listener that responds to various queries and can even be
# delegated to for resolution
@@ -126,8 +127,9 @@ lighthouse:
# Port Nebula will be listening on. The default here is 4242. For a lighthouse node, the port should be defined,
# however using port 0 will dynamically assign a port and is recommended for roaming nodes.
listen:
# To listen on only ipv4, use "0.0.0.0"
# To listen on only ipv4, use "0.0.0.0". Not reloadable.
host: "::"
# Not reloadable.
port: 4242
# Sets the max number of packets to pull from the kernel for each syscall (under systems that support recvmmsg)
# default is 64, does not support reload
@@ -135,7 +137,8 @@ listen:
# superpacket, so the receive scratch is batch * 64KiB per listening socket (~4MiB per routine at the
# default of 64). Lower this to trade peak per-syscall throughput for memory on constrained hosts.
#batch: 64
# Linux only. Configure socket buffers for the udp side (outside), leave unset to use the system defaults. Values will be doubled by the kernel
# Linux only. Reloadable, but removing a value leaves the socket at the old size until Nebula restarts.
# Configure socket buffers for the udp side (outside), leave unset to use the system defaults. Values will be doubled by the kernel
# Default is net.core.rmem_default and net.core.wmem_default (/proc/sys/net/core/rmem_default and /proc/sys/net/core/rmem_default)
# Maximum is limited by memory in the system, SO_RCVBUFFORCE and SO_SNDBUFFORCE is used to avoid having to raise the system wide
# max, net.core.rmem_max and net.core.wmem_max
@@ -179,7 +182,7 @@ listen:
# Currently, this defaults to 1 which means we have 1 tun queue reader and 1
# UDP queue reader. Setting this above one will set IFF_MULTI_QUEUE on the tun
# device and SO_REUSEPORT on the UDP socket to allow multiple queues.
# This option is only supported on Linux.
# This option is only supported on Linux. Not reloadable.
#routines: 1
punchy:
@@ -203,6 +206,7 @@ punchy:
# Cipher allows you to choose between the available ciphers for your network. Options are chachapoly or aes
# IMPORTANT: this value must be identical on ALL NODES/LIGHTHOUSES. We do not/will not support use of different ciphers simultaneously!
# Not reloadable.
#cipher: aes
# Preferred ranges is used to define a hint about the local network ranges, which speeds up discovering the fastest
@@ -229,12 +233,6 @@ punchy:
# Trusted SSH CA public keys. These are the public keys of the CAs that are allowed to sign SSH keys for access.
#trusted_cas:
#- "ssh public key string"
# sandbox_dir restricts file paths for profiling commands (start-cpu-profile, save-heap-profile,
# save-mutex-profile) to the specified directory. Relative paths will be resolved within this directory,
# and absolute paths outside of it will be rejected. Default is $TMP/nebula-debug.
# The directory is NOT automatically created.
# Overriding this to "" is the same as "/" and will allow overwriting any path on the host.
#sandbox_dir: /var/tmp/nebula-debug
# EXPERIMENTAL: relay support for networks that can't establish direct connections.
relay:
@@ -253,19 +251,25 @@ relay:
# Configure the private interface. Note: addr is baked into the nebula certificate
tun:
# When tun is disabled, a lighthouse can be started without a local tun interface (and therefore without root)
# Not reloadable.
disabled: false
# Name of the device. If not set, a default will be chosen by the OS.
# For macOS: if set, must be in the form `utun[0-9]+`.
# For NetBSD: Required to be set, must be in the form `tun[0-9]+`
# For OpenBSD and NetBSD: if set, must be in the form `tun[0-9]+`, otherwise the first free one is used.
# Not reloadable.
dev: nebula1
# Toggles forwarding of local broadcast packets, the address of which depends on the ip/mask encoded in pki.cert
# Not reloadable.
drop_local_broadcast: false
# Toggles forwarding of multicast packets
# Toggles forwarding of multicast packets. Not reloadable.
drop_multicast: false
# Sets the transmit queue length, if you notice lots of transmit drops on the tun it may help to raise this number. Default is 5000
# Not reloadable.
tx_queue: 5000
# Default MTU for every packet, safe setting is (and the default) 1300 for internet based traffic
# Anything above 8937, here or on a route below, is capped to 8937 with a warning
# Reloadable on Linux only. Where the tun is handed to nebula by a host app (iOS, Android, macOS network extensions)
# the host app owns the MTU.
mtu: 1300
# the use_offloads setting controls if Nebula will attempt to enable GSO and GRO for the tun device. Linux only, not reloadable.
@@ -358,7 +362,7 @@ tun:
#use_system_route_table: false
# Buffer size for reading routes updates. 0 means default system buffer size. (/proc/sys/net/core/rmem_default).
# If using massive routes updates, for example BGP, you may need to increase this value to avoid packet loss.
# SO_RCVBUFFORCE is used to avoid having to raise the system wide max
# SO_RCVBUFFORCE is used to avoid having to raise the system wide max. Not reloadable.
#use_system_route_table_buffer_size: 0
# Configure logging level
@@ -408,34 +412,18 @@ logging:
# Handshakes are sent to all known addresses at each interval with a linear backoff,
# Wait try_interval after the 1st attempt, 2 * try_interval after the 2nd, etc, until the handshake is older than timeout
# A 100ms interval with the default 10 retries will give a handshake 5.5 seconds to resolve before timing out
# Not reloadable.
#try_interval: 100ms
#retries: 20
# query_buffer is the size of the buffer channel for querying lighthouses, drops are recorded in the lighthouse.query.dropped metric.
# Not reloadable.
#query_buffer: 1024
# trigger_buffer is the size of the buffer channel for quickly sending handshakes
# after receiving the response for lighthouse queries
# after receiving the response for lighthouse queries. Not reloadable.
#trigger_buffer: 64
# pq enables handshakes that mix an ephemeral ML-KEM-1024 key exchange into the session keys, protecting
# recorded traffic against a future quantum computer. Authentication is unchanged. These handshakes are too
# large for one packet and travel as fragments with forward error correction.
#pq:
# initiate starts post-quantum handshakes. Attempts alternate between post-quantum and classical until one is
# answered, so peers that do not support them still connect.
#initiate: false
# accept answers post-quantum handshakes from peers.
#accept: false
# require makes every handshake post-quantum: it implies initiate and accept, never offers a classical handshake
# and refuses classical handshakes from peers. Only enable it once every host can speak post-quantum handshakes.
#require: false
# sticky never falls back to a classical handshake with a peer that has completed a post-quantum one since
# nebula started, which defeats an attacker who drops post-quantum handshakes to force a downgrade.
#sticky: false
# repair_delay is how long the forward error correction fragments trail the rest of a handshake message.
#repair_delay: 10ms
# Tunnel manager settings
#tunnels:
# drop_inactive controls whether inactive tunnels are maintained or dropped after the inactive_timeout period has
+9 -9
View File
@@ -1,13 +1,12 @@
module github.com/slackhq/nebula
go 1.26.0
go 1.27.2
require (
dario.cat/mergo v1.0.2
filippo.io/bigmod v0.1.0
github.com/DefinedNet/netlink v1.3.2-dn.1
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be
github.com/armon/go-radix v1.0.0
github.com/flynn/noise v1.1.0
github.com/gaissmai/bart v0.29.0
github.com/gogo/protobuf v1.3.2
@@ -23,17 +22,17 @@ require (
github.com/stretchr/testify v1.12.1
go.uber.org/goleak v1.3.0
go.yaml.in/yaml/v3 v3.0.5
golang.org/x/crypto v0.57.0
golang.org/x/net v0.59.0
golang.org/x/sync v0.23.0
golang.org/x/sys v0.48.0
golang.org/x/term v0.46.0
golang.org/x/crypto v0.58.0
golang.org/x/net v0.60.0
golang.org/x/sync v0.24.0
golang.org/x/sys v0.49.0
golang.org/x/term v0.47.0
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2
golang.zx2c4.com/wireguard v0.0.0-20230325221338-052af4a8072b
golang.zx2c4.com/wireguard/windows v1.0.1
google.golang.org/protobuf v1.36.12
gopkg.in/yaml.v3 v3.0.1
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe
gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980
)
require (
@@ -45,5 +44,6 @@ require (
github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect
github.com/vishvananda/netns v0.0.5 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc // indirect
golang.org/x/time v0.15.0 // indirect
)
+16 -16
View File
@@ -12,8 +12,6 @@ github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRF
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be/go.mod h1:ySMOLuWl6zY27l47sB3qLNK6tF2fkHG55UZxx8oIVo4=
github.com/armon/go-radix v1.0.0 h1:F4z6KzEeeQIMeLFa97iZU6vupzoecKdU5TX24SNppXI=
github.com/armon/go-radix v1.0.0/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgIH9cCH8=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
@@ -155,8 +153,10 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc h1:TS73t7x3KarrNd5qAipmspBDS1rkMcgVG/fS1aRb4Rc=
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc/go.mod h1:A+z0yzpGtvnG90cToK5n2tu8UJVP2XUATh+r+sfOOOc=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
@@ -171,8 +171,8 @@ golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLL
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.59.0 h1:5zfYln+w5XCxwrnMMJPufRgNoXEaGxl0wo5GqPXyues=
golang.org/x/net v0.59.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -180,8 +180,8 @@ golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -195,16 +195,16 @@ golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=
golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
@@ -243,5 +243,5 @@ gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe h1:fre4i6mv4iBuz5lCMOzHD1rH1ljqHWSICFmZRbbgp3g=
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe/go.mod h1:sxc3Uvk/vHcd3tj7/DHVBoR5wvWT/MmRq2pj7HRJnwU=
gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980 h1:ribyrTYQ/QYeUIs8x8bVLYwGyepa7RnHLqJzegT5YYk=
gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980/go.mod h1:pK/dapF5Bf3KGW7/kd5CI5IC4lYh9FMFY1qETAIGLXk=
+2 -4
View File
@@ -35,9 +35,7 @@ func NewCredential(
}
// buildHandshakeState creates a noise.HandshakeState from this credential.
// pskPlacement selects the noise psk modifier; 0 means none. A psk for
// placements 2 and up is supplied later with SetPresharedKey.
func (hc *Credential) buildHandshakeState(initiator bool, pattern noise.HandshakePattern, pskPlacement int) (*noise.HandshakeState, error) {
func (hc *Credential) buildHandshakeState(initiator bool, pattern noise.HandshakePattern) (*noise.HandshakeState, error) {
return noise.NewHandshakeState(noise.Config{
CipherSuite: hc.cipherSuite,
Random: rand.Reader,
@@ -45,7 +43,7 @@ func (hc *Credential) buildHandshakeState(initiator bool, pattern noise.Handshak
Initiator: initiator,
StaticKeypair: noise.DHKey{Private: hc.privateKey, Public: hc.Cert.PublicKey()},
PresharedKey: []byte{},
PresharedKeyPlacement: pskPlacement,
PresharedKeyPlacement: 0,
})
}
-216
View File
@@ -1,216 +0,0 @@
// Package fec implements the systematic Reed-Solomon erasure code used to carry
// handshake messages that do not fit in a single UDP packet.
//
// A message is split into K equally sized data symbols (the last one zero
// padded). Symbol indexes 0..K-1 are the data symbols themselves; indexes
// K..MaxSymbols-1 are repair symbols. Any K distinct symbols reconstruct the
// message. Repair symbols are generated on demand, so a sender can emit fresh
// ones on every retransmit and a receiver can merge everything it has seen.
package fec
import (
"errors"
"fmt"
)
// MaxSymbols is the number of distinct symbols, data and repair together, one
// message can be encoded into. It is bounded by the size of GF(2^8).
const MaxSymbols = 256
var (
ErrInvalidLayout = errors.New("invalid fec layout")
ErrIndexRange = errors.New("symbol index out of range")
ErrSymbolSize = errors.New("symbol has the wrong size")
ErrNotEnough = errors.New("not enough symbols to decode")
ErrCorrupt = errors.New("decoded message has non-zero padding")
)
// Layout describes how a message is split into symbols. Both ends derive it
// from the message length alone, so it never has to be sent.
type Layout struct {
MsgLen int // length of the message in bytes
K int // number of data symbols
SymbolSize int // size of every symbol, data and repair
}
// NewLayout returns the layout for a message of msgLen bytes carried in symbols
// of at most maxSymbolSize bytes. Symbols are balanced, so padding is always
// less than K bytes.
func NewLayout(msgLen, maxSymbolSize int) (Layout, error) {
if msgLen <= 0 || maxSymbolSize <= 0 {
return Layout{}, ErrInvalidLayout
}
k := (msgLen + maxSymbolSize - 1) / maxSymbolSize
if k >= MaxSymbols {
return Layout{}, fmt.Errorf("%w: %d data symbols", ErrInvalidLayout, k)
}
return Layout{
MsgLen: msgLen,
K: k,
SymbolSize: (msgLen + k - 1) / k,
}, nil
}
// Encoder produces symbols for one message.
type Encoder struct {
layout Layout
data [][]byte
}
// NewEncoder copies msg into padded data symbols.
func NewEncoder(msg []byte, maxSymbolSize int) (*Encoder, error) {
l, err := NewLayout(len(msg), maxSymbolSize)
if err != nil {
return nil, err
}
buf := make([]byte, l.K*l.SymbolSize)
copy(buf, msg)
data := make([][]byte, l.K)
for i := range data {
data[i] = buf[i*l.SymbolSize : (i+1)*l.SymbolSize]
}
return &Encoder{layout: l, data: data}, nil
}
func (e *Encoder) Layout() Layout {
return e.layout
}
// Symbol appends the symbol with the given index to dst and returns the result.
func (e *Encoder) Symbol(dst []byte, index int) ([]byte, error) {
if index < 0 || index >= MaxSymbols {
return nil, ErrIndexRange
}
if index < e.layout.K {
return append(dst, e.data[index]...), nil
}
start := len(dst)
dst = append(dst, make([]byte, e.layout.SymbolSize)...)
out := dst[start:]
for col, d := range e.data {
mulAdd(out, d, coefficient(index, col))
}
return dst, nil
}
// Decoder collects symbols for one message until it can reconstruct it.
// It stores at most K symbols, so its memory is bounded by the layout no matter
// how many symbols arrive.
type Decoder struct {
layout Layout
have [MaxSymbols]bool
syms map[int][]byte
}
// NewDecoder returns a decoder for a message of msgLen bytes that was encoded
// with the same maxSymbolSize.
func NewDecoder(msgLen, maxSymbolSize int) (*Decoder, error) {
l, err := NewLayout(msgLen, maxSymbolSize)
if err != nil {
return nil, err
}
return &Decoder{layout: l, syms: make(map[int][]byte, l.K)}, nil
}
func (d *Decoder) Layout() Layout {
return d.layout
}
// Stored returns the number of symbols held.
func (d *Decoder) Stored() int {
return len(d.syms)
}
// Has reports whether the symbol with the given index is held.
func (d *Decoder) Has(index int) bool {
return index >= 0 && index < MaxSymbols && d.have[index]
}
// Ready reports whether enough symbols have arrived to decode.
func (d *Decoder) Ready() bool {
return len(d.syms) >= d.layout.K
}
// Add stores a copy of the symbol. Duplicates, and anything beyond the first K
// distinct symbols, are ignored. It returns whether the decoder is ready.
func (d *Decoder) Add(index int, sym []byte) (bool, error) {
if index < 0 || index >= MaxSymbols {
return d.Ready(), ErrIndexRange
}
if len(sym) != d.layout.SymbolSize {
return d.Ready(), ErrSymbolSize
}
if d.have[index] || d.Ready() {
return d.Ready(), nil
}
d.have[index] = true
d.syms[index] = append([]byte(nil), sym...)
return d.Ready(), nil
}
// Decode reconstructs the message from the stored symbols.
func (d *Decoder) Decode() ([]byte, error) {
if !d.Ready() {
return nil, ErrNotEnough
}
l := d.layout
out := make([]byte, l.K*l.SymbolSize)
var missing, repair []int
for i := range l.K {
if s, ok := d.syms[i]; ok {
copy(out[i*l.SymbolSize:], s)
} else {
missing = append(missing, i)
}
}
if len(missing) > 0 {
for i := l.K; i < MaxSymbols && len(repair) < len(missing); i++ {
if d.have[i] {
repair = append(repair, i)
}
}
if len(repair) < len(missing) {
return nil, ErrNotEnough
}
// Each repair symbol r satisfies r = sum_col coefficient(r, col) * data[col].
// Move the known data columns to the left hand side, leaving a square
// Cauchy system in the missing columns.
rhs := make([][]byte, len(repair))
m := make([][]byte, len(repair))
for ri, row := range repair {
rhs[ri] = append([]byte(nil), d.syms[row]...)
m[ri] = make([]byte, len(missing))
mi := 0
for col := range l.K {
if mi < len(missing) && missing[mi] == col {
m[ri][mi] = coefficient(row, col)
mi++
continue
}
mulAdd(rhs[ri], out[col*l.SymbolSize:(col+1)*l.SymbolSize], coefficient(row, col))
}
}
if !invert(m) {
// Unreachable for a Cauchy matrix; kept so a broken invariant fails loudly.
return nil, errors.New("fec: singular decode matrix")
}
for mi, col := range missing {
dst := out[col*l.SymbolSize : (col+1)*l.SymbolSize]
for ri := range repair {
mulAdd(dst, rhs[ri], m[mi][ri])
}
}
}
for _, b := range out[l.MsgLen:] {
if b != 0 {
return nil, ErrCorrupt
}
}
return out[:l.MsgLen], nil
}
-318
View File
@@ -1,318 +0,0 @@
package fec
import (
"crypto/sha256"
"encoding/hex"
"math/rand/v2"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// slowMul is carry-less multiplication reduced by gfPoly, used to check the tables.
func slowMul(a, b byte) byte {
var p int
x, y := int(a), int(b)
for y > 0 {
if y&1 != 0 {
p ^= x
}
x <<= 1
if x&0x100 != 0 {
x ^= gfPoly
}
y >>= 1
}
return byte(p)
}
func TestGF(t *testing.T) {
for a := range 256 {
for b := range 256 {
require.Equal(t, slowMul(byte(a), byte(b)), gfMul(byte(a), byte(b)), "%d*%d", a, b)
}
if a != 0 {
require.Equal(t, byte(1), gfMul(byte(a), gfInv(byte(a))), "inv(%d)", a)
}
}
src := make([]byte, 256)
for i := range src {
src[i] = byte(i)
}
for c := range 256 {
dst := make([]byte, 256)
mulAdd(dst, src, byte(c))
for i := range src {
require.Equal(t, gfMul(byte(c), src[i]), dst[i])
}
}
}
func TestLayout(t *testing.T) {
for msgLen := 1; msgLen <= 20000; msgLen++ {
l, err := NewLayout(msgLen, 1176)
require.NoError(t, err)
require.Equal(t, (msgLen+1175)/1176, l.K)
require.LessOrEqual(t, l.SymbolSize, 1176)
require.GreaterOrEqual(t, l.K*l.SymbolSize, msgLen)
require.Less(t, l.K*l.SymbolSize-msgLen, l.K, "padding must be less than one byte per symbol")
}
l, err := NewLayout(65535, 1176)
require.NoError(t, err)
assert.Equal(t, 56, l.K)
_, err = NewLayout(0, 1176)
require.ErrorIs(t, err, ErrInvalidLayout)
_, err = NewLayout(100, 0)
require.ErrorIs(t, err, ErrInvalidLayout)
_, err = NewLayout(256, 1)
require.ErrorIs(t, err, ErrInvalidLayout)
_, err = NewLayout(255, 1)
require.NoError(t, err)
}
func randMsg(r *rand.Rand, n int) []byte {
b := make([]byte, n)
for i := range b {
b[i] = byte(r.Uint32())
}
return b
}
func decodeWith(t *testing.T, enc *Encoder, msgLen, maxSym int, indexes []int) []byte {
t.Helper()
dec, err := NewDecoder(msgLen, maxSym)
require.NoError(t, err)
for _, idx := range indexes {
sym, err := enc.Symbol(nil, idx)
require.NoError(t, err)
_, err = dec.Add(idx, sym)
require.NoError(t, err)
}
require.True(t, dec.Ready())
out, err := dec.Decode()
require.NoError(t, err)
return out
}
// TestExhaustiveSubsets decodes every k-subset of the first k+m symbols for
// small codes, which checks the MDS property directly.
func TestExhaustiveSubsets(t *testing.T) {
r := rand.New(rand.NewPCG(1, 2))
const maxSym = 16
for k := 1; k <= 10; k++ {
for m := 1; m <= 5; m++ {
msgLen := (k-1)*maxSym + 1 + r.IntN(maxSym)
msg := randMsg(r, msgLen)
enc, err := NewEncoder(msg, maxSym)
require.NoError(t, err)
require.Equal(t, k, enc.Layout().K)
n := k + m
for mask := 0; mask < 1<<n; mask++ {
var idx []int
for i := range n {
if mask&(1<<i) != 0 {
idx = append(idx, i)
}
}
if len(idx) != k {
continue
}
require.Equal(t, msg, decodeWith(t, enc, msgLen, maxSym, idx), "k=%d m=%d idx=%v", k, m, idx)
}
}
}
}
// TestRandomHighIndexes draws symbols from the whole index space, the shape
// retransmits produce once fresh repair symbols are in play.
func TestRandomHighIndexes(t *testing.T) {
r := rand.New(rand.NewPCG(3, 4))
for range 2000 {
msgLen := 1 + r.IntN(65535)
msg := randMsg(r, msgLen)
enc, err := NewEncoder(msg, 1176)
require.NoError(t, err)
k := enc.Layout().K
idx := r.Perm(MaxSymbols)[:k]
require.Equal(t, msg, decodeWith(t, enc, msgLen, 1176, idx), "k=%d idx=%v", k, idx)
}
}
func TestDecoderBounds(t *testing.T) {
r := rand.New(rand.NewPCG(5, 6))
msg := randMsg(r, 3000)
enc, err := NewEncoder(msg, 1176)
require.NoError(t, err)
l := enc.Layout()
require.Equal(t, 3, l.K)
require.Equal(t, 1000, l.SymbolSize)
dec, err := NewDecoder(3000, 1176)
require.NoError(t, err)
_, err = dec.Decode()
require.ErrorIs(t, err, ErrNotEnough)
_, err = dec.Add(-1, make([]byte, 1000))
require.ErrorIs(t, err, ErrIndexRange)
_, err = dec.Add(MaxSymbols, make([]byte, 1000))
require.ErrorIs(t, err, ErrIndexRange)
_, err = dec.Add(0, make([]byte, 999))
require.ErrorIs(t, err, ErrSymbolSize)
s5, _ := enc.Symbol(nil, 5)
ready, err := dec.Add(5, s5)
require.NoError(t, err)
require.False(t, ready)
// A duplicate index is ignored, even with different contents.
ready, err = dec.Add(5, make([]byte, 1000))
require.NoError(t, err)
require.False(t, ready)
s0, _ := enc.Symbol(nil, 0)
s9, _ := enc.Symbol(nil, 9)
_, err = dec.Add(0, s0)
require.NoError(t, err)
ready, err = dec.Add(9, s9)
require.NoError(t, err)
require.True(t, ready)
// Symbols beyond K are not stored.
ready, err = dec.Add(1, make([]byte, 1000))
require.NoError(t, err)
require.True(t, ready)
require.Len(t, dec.syms, 3)
out, err := dec.Decode()
require.NoError(t, err)
require.Equal(t, msg, out)
}
func TestCorruptPadding(t *testing.T) {
// 3001 bytes in 3 symbols of 1001 leaves 2 bytes of padding in the last one.
dec, err := NewDecoder(3001, 1176)
require.NoError(t, err)
require.Equal(t, 1001, dec.Layout().SymbolSize)
last := make([]byte, 1001)
last[1000] = 1
for i, s := range [][]byte{make([]byte, 1001), make([]byte, 1001), last} {
_, err = dec.Add(i, s)
require.NoError(t, err)
}
_, err = dec.Decode()
require.ErrorIs(t, err, ErrCorrupt)
}
// TestKnownAnswer pins the encoding. Both ends of a handshake must produce
// identical repair symbols, so any change here is a wire format change.
func TestKnownAnswer(t *testing.T) {
msg := make([]byte, 3000)
for i := range msg {
msg[i] = byte(i*7 + 3)
}
enc, err := NewEncoder(msg, 1176)
require.NoError(t, err)
// Cross-checked against an independent bit-serial GF(2^8) implementation.
want := map[int]string{
3: "52124f58276ec6338d3b3f4eed273bc7cabfe4f0f467928962b046ad1b348f50",
4: "a50928b04738981985a7d4a17ffaf8d7e30014ca64fc53c99800ffb61aec4aa7",
5: "ada4c95a7facbc542aaef584e6f0d081b7b0d0eed69ded03b57d0753a119cfc9",
255: "839066cdf94f963b223fb1c1c1360f698c14f6349b25653ff5b521791a4021b1",
}
for idx, h := range want {
sym, err := enc.Symbol(nil, idx)
require.NoError(t, err)
sum := sha256.Sum256(sym)
assert.Equal(t, h, hex.EncodeToString(sum[:]), "symbol %d", idx)
}
}
func FuzzRoundTrip(f *testing.F) {
f.Add([]byte("hello"), uint16(2), uint64(0))
f.Add(make([]byte, 9000), uint16(1176), uint64(7))
f.Fuzz(func(t *testing.T, msg []byte, maxSym uint16, seed uint64) {
if len(msg) == 0 || maxSym == 0 {
return
}
enc, err := NewEncoder(msg, int(maxSym))
if err != nil {
return
}
r := rand.New(rand.NewPCG(seed, seed))
idx := r.Perm(MaxSymbols)[:enc.Layout().K]
require.Equal(t, msg, decodeWith(t, enc, len(msg), int(maxSym), idx))
})
}
// FuzzDecoder feeds arbitrary symbols to a decoder; it must never panic and
// must either fail or return a message of the declared length.
func FuzzDecoder(f *testing.F) {
f.Add(uint16(3000), []byte{0, 5, 9, 200}, []byte("some symbol bytes"))
f.Fuzz(func(t *testing.T, msgLen uint16, indexes []byte, fill []byte) {
dec, err := NewDecoder(int(msgLen), 1176)
if err != nil {
return
}
size := dec.Layout().SymbolSize
for i, idx := range indexes {
sym := make([]byte, size)
for j := range sym {
if len(fill) > 0 {
sym[j] = fill[(i+j)%len(fill)]
}
}
if _, err := dec.Add(int(idx), sym); err != nil {
t.Fatal(err)
}
}
out, err := dec.Decode()
if err == nil {
require.Len(t, out, int(msgLen))
}
})
}
// BenchmarkRepair and BenchmarkDecode use a v3-sized msg2 (~9.8KB, k=9) with
// three repair symbols, decoding with three data symbols missing.
func BenchmarkRepair(b *testing.B) {
msg := randMsg(rand.New(rand.NewPCG(7, 8)), 9800)
enc, err := NewEncoder(msg, 1176)
require.NoError(b, err)
buf := make([]byte, 0, 1176)
b.SetBytes(int64(len(msg)))
for b.Loop() {
for idx := 9; idx < 12; idx++ {
buf, _ = enc.Symbol(buf[:0], idx)
}
}
}
func BenchmarkDecode(b *testing.B) {
msg := randMsg(rand.New(rand.NewPCG(7, 8)), 9800)
enc, err := NewEncoder(msg, 1176)
require.NoError(b, err)
idx := []int{0, 1, 2, 3, 5, 6, 9, 10, 11}
syms := make([][]byte, len(idx))
for i, x := range idx {
syms[i], _ = enc.Symbol(nil, x)
}
b.SetBytes(int64(len(msg)))
for b.Loop() {
dec, _ := NewDecoder(len(msg), 1176)
for i, x := range idx {
_, _ = dec.Add(x, syms[i])
}
if _, err := dec.Decode(); err != nil {
b.Fatal(err)
}
}
}
-115
View File
@@ -1,115 +0,0 @@
package fec
// Arithmetic over GF(2^8) with the reducing polynomial x^8+x^4+x^3+x^2+1 (0x11d).
// Addition is XOR. Multiplication uses log/exp tables; the exp table is doubled so
// the sum of two logs never needs reducing mod 255.
const gfPoly = 0x11d
var (
gfExp [510]byte
gfLog [256]byte
)
func init() {
x := 1
for i := range 255 {
gfExp[i] = byte(x)
gfExp[i+255] = byte(x)
gfLog[x] = byte(i)
x <<= 1
if x&0x100 != 0 {
x ^= gfPoly
}
}
}
func gfMul(a, b byte) byte {
if a == 0 || b == 0 {
return 0
}
return gfExp[int(gfLog[a])+int(gfLog[b])]
}
// gfInv returns the multiplicative inverse of a. a must be non-zero.
func gfInv(a byte) byte {
return gfExp[255-int(gfLog[a])]
}
// mulAdd computes dst ^= c*src element-wise. dst must be at least as long as src.
func mulAdd(dst, src []byte, c byte) {
switch c {
case 0:
return
case 1:
for i, s := range src {
dst[i] ^= s
}
return
}
var t [256]byte
lc := int(gfLog[c])
for s := 1; s < 256; s++ {
t[s] = gfExp[int(gfLog[s])+lc]
}
for i, s := range src {
dst[i] ^= t[s]
}
}
// coefficient returns the generator matrix entry for repair symbol index row and
// data symbol index col. Repair rows are a Cauchy matrix 1/(x_row + y_col) with
// x_row = row and y_col = col: rows are always >= k and columns < k so the two
// sets are disjoint, which makes every square submatrix invertible and the
// systematic code [I; C] MDS. A row depends only on its index, never on how many
// repair symbols exist, so new repair symbols can be generated at any time.
func coefficient(row, col int) byte {
return gfInv(byte(row) ^ byte(col))
}
// invert inverts the n x n matrix m in place with Gauss-Jordan elimination.
// It returns false if m is singular.
func invert(m [][]byte) bool {
n := len(m)
inv := make([][]byte, n)
for i := range inv {
inv[i] = make([]byte, n)
inv[i][i] = 1
}
for col := range n {
pivot := -1
for r := col; r < n; r++ {
if m[r][col] != 0 {
pivot = r
break
}
}
if pivot < 0 {
return false
}
m[col], m[pivot] = m[pivot], m[col]
inv[col], inv[pivot] = inv[pivot], inv[col]
scale := gfInv(m[col][col])
for c := range n {
m[col][c] = gfMul(m[col][c], scale)
inv[col][c] = gfMul(inv[col][c], scale)
}
for r := 0; r < n; r++ {
if r == col || m[r][col] == 0 {
continue
}
f := m[r][col]
for c := range n {
m[r][c] ^= gfMul(f, m[col][c])
inv[r][c] ^= gfMul(f, inv[col][c])
}
}
}
copy(m, inv)
return true
}
-67
View File
@@ -1,67 +0,0 @@
package fragment
import (
"errors"
"github.com/slackhq/nebula/handshake/fec"
"github.com/slackhq/nebula/header"
)
// ErrMismatch is returned when a fragment disagrees with the message it claims to
// belong to. The caller should discard the whole assembly.
var ErrMismatch = errors.New("fragment does not match its message")
// Assembly collects the fragments of one incoming message. Its memory is bounded
// by the message length: at most K symbols are kept however many arrive.
//
// An Assembly is not safe for concurrent use.
type Assembly struct {
fragID uint32
msgLen uint16
dec *fec.Decoder
received int
}
// NewAssembly starts an assembly for the message described by h. The fragment
// itself still has to be passed to Add.
func NewAssembly(h Header) (*Assembly, error) {
dec, err := fec.NewDecoder(int(h.MsgLen), MaxSymbolSize)
if err != nil {
return nil, err
}
return &Assembly{fragID: h.FragID, msgLen: h.MsgLen, dec: dec}, nil
}
// Add records a fragment parsed by Parse. It returns true once enough symbols
// are held to call Decode.
func (a *Assembly) Add(h Header, sym []byte) (bool, error) {
if h.FragID != a.fragID || h.MsgLen != a.msgLen {
return a.dec.Ready(), ErrMismatch
}
a.received += header.Len + HeaderLen + len(sym)
return a.dec.Add(int(h.Index), sym)
}
// wouldStore returns how many bytes Add would retain for this fragment.
func (a *Assembly) wouldStore(h Header, sym []byte) int {
if h.FragID != a.fragID || h.MsgLen != a.msgLen || a.dec.Ready() || a.dec.Has(int(h.Index)) {
return 0
}
return len(sym)
}
// Decode reconstructs the message. It requires Add to have reported ready.
func (a *Assembly) Decode() ([]byte, error) {
return a.dec.Decode()
}
// Stored returns the number of symbol bytes held.
func (a *Assembly) Stored() int {
return a.dec.Stored() * a.dec.Layout().SymbolSize
}
// Received returns the number of packet bytes received for this message,
// duplicates included. It is the basis for the amplification budget.
func (a *Assembly) Received() int {
return a.received
}
-515
View File
@@ -1,515 +0,0 @@
package fragment
import (
"bytes"
"math/rand/v2"
"sync"
"testing"
"time"
"github.com/slackhq/nebula/header"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const testSubtype = header.MessageSubType(9)
type testKey struct {
src string
fragID uint32
}
func randMsg(r *rand.Rand, n int) []byte {
b := make([]byte, n)
for i := range b {
b[i] = byte(r.Uint32())
}
return b
}
func testConfig() TableConfig {
return TableConfig{
MaxBytes: 1 << 20,
MaxEntries: 64,
MaxPerSource: 4,
MaxMsgLen: 16384,
TTL: 2 * time.Second,
}
}
// checkInvariants verifies the table's bookkeeping against its contents.
func checkInvariants[K, S comparable](t *testing.T, tb *Table[K, S]) {
t.Helper()
tb.mu.Lock()
defer tb.mu.Unlock()
require.Len(t, tb.order, len(tb.entries))
require.LessOrEqual(t, len(tb.entries), tb.cfg.MaxEntries)
bytes := 0
perSource := map[S]int{}
for i, key := range tb.order {
e, ok := tb.entries[key]
require.True(t, ok)
require.Equal(t, i, e.pos)
bytes += e.a.Stored()
perSource[e.source]++
}
require.Equal(t, bytes, tb.bytes)
require.LessOrEqual(t, tb.bytes, tb.cfg.MaxBytes)
require.Equal(t, perSource, tb.perSource)
for _, n := range perSource {
require.LessOrEqual(t, n, tb.cfg.MaxPerSource)
}
}
func parse(t *testing.T, p []byte) (header.H, Header, []byte) {
t.Helper()
var nh header.H
require.NoError(t, nh.Parse(p))
h, sym, err := Parse(p[header.Len:])
require.NoError(t, err)
return nh, h, sym
}
func TestParse(t *testing.T) {
h := Header{FragID: 0xdeadbeef, MsgLen: 3000, Index: 7}
b := h.Encode(nil)
require.Len(t, b, HeaderLen)
b = append(b, make([]byte, 1000)...)
got, sym, err := Parse(b)
require.NoError(t, err)
assert.Equal(t, h, got)
assert.Len(t, sym, 1000)
_, _, err = Parse(b[:HeaderLen-1])
require.ErrorIs(t, err, ErrShortFragment)
_, _, err = Parse(b[:len(b)-1])
require.ErrorIs(t, err, ErrSymbolSize)
_, _, err = Parse(append(b, 0))
require.ErrorIs(t, err, ErrSymbolSize)
bad := Header{FragID: 0, MsgLen: 3000}.Encode(nil)
_, _, err = Parse(append(bad, make([]byte, 1000)...))
require.ErrorIs(t, err, ErrFragID)
bad = Header{FragID: 1, MsgLen: 0}.Encode(nil)
_, _, err = Parse(bad)
require.ErrorIs(t, err, ErrMsgLen)
bad = Header{FragID: 1, MsgLen: 3000, Flags: 0x02}.Encode(nil)
_, _, err = Parse(append(bad, make([]byte, 1000)...))
require.ErrorIs(t, err, ErrFlags)
t.Run("cookie", func(t *testing.T) {
h := Header{FragID: 7, MsgLen: 3000, Index: 4, Flags: FlagCookie}
for i := range h.Cookie {
h.Cookie[i] = byte(i + 1)
}
b := h.Encode(nil)
require.Len(t, b, HeaderLen+CookieLen)
b = append(b, make([]byte, 1000)...)
got, sym, err := Parse(b)
require.NoError(t, err)
assert.Equal(t, h, got)
assert.True(t, got.HasCookie())
assert.Len(t, sym, 1000)
_, _, err = Parse(b[:HeaderLen+CookieLen-1])
require.ErrorIs(t, err, ErrNoCookie)
})
}
func TestDefaultRepair(t *testing.T) {
for k, want := range map[int]int{1: 2, 2: 2, 6: 2, 7: 3, 9: 3, 10: 4, 14: 5} {
assert.Equal(t, want, DefaultRepair(k), "k=%d", k)
}
}
func TestSender(t *testing.T) {
_, err := NewSender(nil, testSubtype, 0, 1, 1)
require.ErrorIs(t, err, ErrMsgLen)
_, err = NewSender(make([]byte, 65536), testSubtype, 0, 1, 1)
require.ErrorIs(t, err, ErrMsgLen)
_, err = NewSender(make([]byte, 10), testSubtype, 0, 1, 0)
require.ErrorIs(t, err, ErrFragID)
msg := randMsg(rand.New(rand.NewPCG(1, 1)), 9800)
s, err := NewSender(msg, testSubtype, 77, 2, 1234)
require.NoError(t, err)
require.Equal(t, 9, s.Layout().K)
indexes := func(ps [][]byte) []int {
var out []int
for _, p := range ps {
require.LessOrEqual(t, len(p), MaxPacketLen)
nh, h, _ := parse(t, p)
assert.Equal(t, header.Handshake, nh.Type)
assert.Equal(t, testSubtype, nh.Subtype)
assert.Equal(t, uint32(77), nh.RemoteIndex)
assert.Equal(t, uint64(2), nh.MessageCounter)
assert.Equal(t, uint32(1234), h.FragID)
assert.Equal(t, uint16(9800), h.MsgLen)
out = append(out, int(h.Index))
}
return out
}
now, later, err := s.Attempt(3)
require.NoError(t, err)
assert.Equal(t, []int{0, 1, 2, 3, 4, 5, 6, 7, 8}, indexes(now))
assert.Equal(t, []int{9, 10, 11}, indexes(later))
// A retry offers only indexes the receiver has not seen.
now, later, err = s.Attempt(3)
require.NoError(t, err)
assert.Equal(t, []int{12, 13, 14, 15, 16, 17, 18, 19, 20}, indexes(now))
assert.Equal(t, []int{21, 22, 23}, indexes(later))
// A sender rebuilt from the same message resumes where it was told to.
s2, err := NewSender(msg, testSubtype, 77, 2, 1234)
require.NoError(t, err)
s2.SetNext(s.Next())
now2, _, err := s2.Attempt(0)
require.NoError(t, err)
assert.Equal(t, []int{24, 25, 26, 27, 28, 29, 30, 31, 32}, indexes(now2))
s2.SetNext(-1)
assert.Equal(t, 255, s2.Next())
// A cookie rides on every later packet, which still fits MaxPacketLen even
// with a full size symbol.
full, err := NewSender(make([]byte, 2*MaxSymbolSize), testSubtype, 0, 1, 9)
require.NoError(t, err)
full.SetCookie(Cookie{1, 2, 3})
pkts, _, err := full.Attempt(1)
require.NoError(t, err)
for _, p := range pkts {
assert.Len(t, p, MaxPacketLen)
_, h, _ := parse(t, p)
assert.True(t, h.HasCookie())
assert.Equal(t, Cookie{1, 2, 3}, h.Cookie)
}
// Indexes wrap once the space is used up.
for range 19 {
_, _, err = s.Attempt(3)
require.NoError(t, err)
}
now, _, err = s.Attempt(3)
require.NoError(t, err)
assert.Equal(t, []int{252, 253, 254, 255, 0, 1, 2, 3, 4}, indexes(now))
}
// TestLossy runs whole messages through heavy loss, reordering and duplication,
// merging across attempts, and checks the table ends empty.
func TestLossy(t *testing.T) {
r := rand.New(rand.NewPCG(2, 2))
tb := NewTable[testKey, string](testConfig())
now := time.Unix(1000, 0)
for i := range 300 {
msg := randMsg(r, 1+r.IntN(16384))
fragID := uint32(i + 1)
s, err := NewSender(msg, testSubtype, 0, 1, fragID)
require.NoError(t, err)
var got *Result
for attempt := 0; got == nil; attempt++ {
require.Less(t, attempt, 20, "message never completed")
a, b, err := s.Attempt(DefaultRepair(s.Layout().K))
require.NoError(t, err)
var pkts [][]byte
for _, p := range append(a, b...) {
if r.Float64() < 0.3 {
continue
}
pkts = append(pkts, p)
if r.Float64() < 0.2 {
pkts = append(pkts, p)
}
}
r.Shuffle(len(pkts), func(i, j int) { pkts[i], pkts[j] = pkts[j], pkts[i] })
for _, p := range pkts {
_, h, sym := parse(t, p)
res, err := tb.Add(testKey{"a", h.FragID}, "a", now, h, sym)
require.NoError(t, err)
if res != nil {
// The table keeps no memory of completed messages, so a late
// duplicate can complete one again; the handshake layer
// recognises the repeat.
require.True(t, bytes.Equal(msg, res.Msg))
got = res
}
checkInvariants(t, tb)
}
now = now.Add(100 * time.Millisecond)
}
require.Positive(t, got.Received)
// Late fragments of a completed message start a fresh entry; drop it.
tb.Expire(now.Add(time.Hour))
}
assert.Zero(t, tb.Len())
assert.Zero(t, tb.Bytes())
}
// TestConcurrent spreads each message's fragments across goroutines, as
// multiple UDP reader routines would.
func TestConcurrent(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
now := time.Unix(1000, 0)
r := rand.New(rand.NewPCG(3, 3))
const msgs = 50
want := make(map[uint32][]byte, msgs)
var pkts [][]byte
for id := uint32(1); id <= msgs; id++ {
msg := randMsg(r, 1+r.IntN(16384))
want[id] = msg
s, err := NewSender(msg, testSubtype, 0, 1, id)
require.NoError(t, err)
a, b, err := s.Attempt(DefaultRepair(s.Layout().K))
require.NoError(t, err)
pkts = append(pkts, a...)
pkts = append(pkts, b...)
}
r.Shuffle(len(pkts), func(i, j int) { pkts[i], pkts[j] = pkts[j], pkts[i] })
var mu sync.Mutex
done := map[uint32]bool{}
var wg sync.WaitGroup
for w := range 8 {
wg.Go(func() {
for i := w; i < len(pkts); i += 8 {
h, sym, err := Parse(pkts[i][16:])
if err != nil {
t.Error(err)
return
}
src := string(rune('a' + h.FragID))
res, err := tb.Add(testKey{src, h.FragID}, src, now, h, sym)
if err != nil {
t.Error(err)
return
}
if res != nil {
if !bytes.Equal(want[h.FragID], res.Msg) {
t.Errorf("message %d decoded wrong", h.FragID)
}
mu.Lock()
done[h.FragID] = true
mu.Unlock()
}
}
})
}
wg.Wait()
assert.Len(t, done, msgs)
checkInvariants(t, tb)
}
func firstFragment(t *testing.T, msgLen int, fragID uint32) (Header, []byte) {
t.Helper()
s, err := NewSender(make([]byte, msgLen), testSubtype, 0, 1, fragID)
require.NoError(t, err)
now, _, err := s.Attempt(0)
require.NoError(t, err)
_, h, sym := parse(t, now[0])
return h, sym
}
func TestTableLimits(t *testing.T) {
now := time.Unix(1000, 0)
t.Run("max message length", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
h, sym := firstFragment(t, 16385, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.ErrorIs(t, err, ErrTooLarge)
assert.Zero(t, tb.Len())
})
t.Run("per source quota", func(t *testing.T) {
cfg := testConfig()
cfg.MaxPerSource = 2
tb := NewTable[testKey, string](cfg)
for id := uint32(1); id <= 3; id++ {
h, sym := firstFragment(t, 5000, id)
_, err := tb.Add(testKey{"a", id}, "a", now, h, sym)
if id <= 2 {
require.NoError(t, err)
} else {
require.ErrorIs(t, err, ErrSourceQuota)
}
}
h, sym := firstFragment(t, 5000, 4)
_, err := tb.Add(testKey{"b", 4}, "b", now, h, sym)
require.NoError(t, err)
assert.Equal(t, 3, tb.Len())
checkInvariants(t, tb)
})
t.Run("max entries evicts", func(t *testing.T) {
cfg := testConfig()
cfg.MaxEntries = 4
tb := NewTable[testKey, string](cfg)
for id := uint32(1); id <= 10; id++ {
h, sym := firstFragment(t, 5000, id)
src := string(rune('a' + id))
_, err := tb.Add(testKey{src, id}, src, now, h, sym)
require.NoError(t, err)
checkInvariants(t, tb)
}
assert.Equal(t, 4, tb.Len())
})
t.Run("max bytes evicts others", func(t *testing.T) {
cfg := testConfig()
cfg.MaxBytes = 3 * 1000
tb := NewTable[testKey, string](cfg)
for id := uint32(1); id <= 10; id++ {
h, sym := firstFragment(t, 5000, id) // 5 symbols of 1000
src := string(rune('a' + id))
_, err := tb.Add(testKey{src, id}, src, now, h, sym)
require.NoError(t, err)
checkInvariants(t, tb)
}
assert.Equal(t, 3000, tb.Bytes())
})
t.Run("no room removes new entry", func(t *testing.T) {
cfg := testConfig()
cfg.MaxBytes = 500
tb := NewTable[testKey, string](cfg)
h, sym := firstFragment(t, 5000, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.ErrorIs(t, err, ErrNoRoom)
assert.Zero(t, tb.Len())
checkInvariants(t, tb)
})
t.Run("duplicates are not charged", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
h, sym := firstFragment(t, 5000, 1)
for range 5 {
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
}
assert.Equal(t, 1000, tb.Bytes())
})
t.Run("mismatch drops the entry", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
h, sym := firstFragment(t, 5000, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
h2, sym2 := firstFragment(t, 6000, 1)
_, err = tb.Add(testKey{"a", 1}, "a", now, h2, sym2)
require.ErrorIs(t, err, ErrMismatch)
assert.Zero(t, tb.Len())
assert.Zero(t, tb.Bytes())
checkInvariants(t, tb)
})
t.Run("load", func(t *testing.T) {
cfg := testConfig()
cfg.MaxEntries = 4
cfg.MaxBytes = 10000
tb := NewTable[testKey, string](cfg)
assert.Zero(t, tb.Load())
h, sym := firstFragment(t, 5000, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
assert.InDelta(t, 0.25, tb.Load(), 0.001, "one of four entries")
h, sym = firstFragment(t, 5000, 2)
_, err = tb.Add(testKey{"b", 2}, "b", now, h, sym)
require.NoError(t, err)
assert.InDelta(t, 0.5, tb.Load(), 0.001)
})
t.Run("ttl", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
s, err := NewSender(make([]byte, 5000), testSubtype, 0, 1, 1)
require.NoError(t, err)
pkts, _, err := s.Attempt(0)
require.NoError(t, err)
_, h, sym := parse(t, pkts[0])
_, err = tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
// Activity within the TTL keeps the entry and slides its deadline.
_, h, sym = parse(t, pkts[1])
_, err = tb.Add(testKey{"a", 1}, "a", now.Add(1500*time.Millisecond), h, sym)
require.NoError(t, err)
assert.Equal(t, 2000, tb.Bytes())
tb.Expire(now.Add(3 * time.Second))
assert.Equal(t, 1, tb.Len())
// A stale entry is restarted rather than merged.
_, h, sym = parse(t, pkts[2])
_, err = tb.Add(testKey{"a", 1}, "a", now.Add(10*time.Second), h, sym)
require.NoError(t, err)
assert.Equal(t, 1000, tb.Bytes())
tb.Expire(now.Add(time.Minute))
assert.Zero(t, tb.Len())
checkInvariants(t, tb)
})
}
func FuzzParse(f *testing.F) {
f.Add(append(Header{FragID: 1, MsgLen: 3000}.Encode(nil), make([]byte, 1000)...))
f.Add([]byte{0, 0, 0, 1, 0, 1, 0, 0, 7})
f.Fuzz(func(t *testing.T, b []byte) {
h, sym, err := Parse(b)
if err != nil {
return
}
require.Equal(t, b, append(h.Encode(nil), sym...))
l, err := h.Layout()
require.NoError(t, err)
require.LessOrEqual(t, l.SymbolSize, MaxSymbolSize)
})
}
// FuzzTable drives a small table with arbitrary fragments and checks its
// bookkeeping after every step.
func FuzzTable(f *testing.F) {
f.Add([]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12})
f.Fuzz(func(t *testing.T, ops []byte) {
tb := NewTable[testKey, string](TableConfig{
MaxBytes: 4000,
MaxEntries: 3,
MaxPerSource: 2,
MaxMsgLen: 6000,
TTL: time.Second,
})
now := time.Unix(0, 0)
for len(ops) >= 4 {
src := string(rune('a' + ops[0]%4))
fragID := uint32(ops[1]%5) + 1
msgLen := uint16(ops[2])*32 + 1
index := ops[3]
ops = ops[4:]
h := Header{FragID: fragID, MsgLen: msgLen, Index: index}
l, err := h.Layout()
require.NoError(t, err)
sym := make([]byte, l.SymbolSize)
if index%7 == 0 {
now = now.Add(600 * time.Millisecond)
tb.Expire(now)
}
res, err := tb.Add(testKey{src, fragID}, src, now, h, sym)
if res != nil {
require.NoError(t, err)
require.Len(t, res.Msg, int(msgLen))
}
checkInvariants(t, tb)
}
})
}
-118
View File
@@ -1,118 +0,0 @@
package fragment
import (
"errors"
"math"
"github.com/slackhq/nebula/handshake/fec"
"github.com/slackhq/nebula/header"
)
// DefaultRepair is the starting number of repair symbols per attempt for a
// message of k data symbols. Small messages get proportionally more repair.
func DefaultRepair(k int) int {
return max(2, (k+2)/3)
}
// Sender produces the fragment packets of one outgoing handshake message.
//
// The first attempt sends the K data symbols followed by repair symbols. Every
// later attempt sends symbols with indexes the receiver has not been offered yet,
// so whatever it already holds still counts toward K. Symbols are a deterministic
// function of the message, which keeps retransmits idempotent. Once all
// fec.MaxSymbols indexes are used the sequence wraps and repeats.
//
// A Sender is not safe for concurrent use.
type Sender struct {
enc *fec.Encoder
subtype header.MessageSubType
remoteIndex uint32
counter uint64
fragID uint32
next int
cookie *Cookie
}
// NewSender prepares msg for sending. subtype, remoteIndex and counter fill the
// Nebula header of every fragment; fragID is the sender's local index.
func NewSender(msg []byte, subtype header.MessageSubType, remoteIndex uint32, counter uint64, fragID uint32) (*Sender, error) {
if len(msg) == 0 || len(msg) > math.MaxUint16 {
return nil, ErrMsgLen
}
if fragID == 0 {
return nil, ErrFragID
}
enc, err := fec.NewEncoder(msg, MaxSymbolSize)
if err != nil {
return nil, err
}
return &Sender{
enc: enc,
subtype: subtype,
remoteIndex: remoteIndex,
counter: counter,
fragID: fragID,
}, nil
}
func (s *Sender) Layout() fec.Layout {
return s.enc.Layout()
}
// SetNext sets the index the next attempt starts from, so a sender rebuilt from
// a cached message can carry on offering fresh symbols.
func (s *Sender) SetNext(index int) {
s.next = ((index % fec.MaxSymbols) + fec.MaxSymbols) % fec.MaxSymbols
}
// SetCookie attaches a cookie from a responder under load to every later packet.
func (s *Sender) SetCookie(c Cookie) {
s.cookie = &c
}
// Next returns the index the next attempt starts from.
func (s *Sender) Next() int {
return s.next
}
// Attempt returns the packets for one transmission attempt carrying K+repair
// symbols. now should be sent immediately and later after the repair delay, so
// a tail drop at the end of the first burst does not also take the repair.
func (s *Sender) Attempt(repair int) (now, later [][]byte, err error) {
if repair < 0 {
return nil, nil, errors.New("negative repair count")
}
k := s.enc.Layout().K
now = make([][]byte, 0, k)
later = make([][]byte, 0, repair)
for i := range k + repair {
p, err := s.packet(s.next)
if err != nil {
return nil, nil, err
}
s.next = (s.next + 1) % fec.MaxSymbols
if i < k {
now = append(now, p)
} else {
later = append(later, p)
}
}
return now, later, nil
}
func (s *Sender) packet(index int) ([]byte, error) {
l := s.enc.Layout()
p := make([]byte, header.Len, header.Len+HeaderLen+CookieLen+l.SymbolSize)
header.Encode(p, header.Version, header.Handshake, s.subtype, s.remoteIndex, s.counter)
h := Header{
FragID: s.fragID,
MsgLen: uint16(l.MsgLen),
Index: uint8(index),
}
if s.cookie != nil {
h.Flags |= FlagCookie
h.Cookie = *s.cookie
}
p = h.Encode(p)
return s.enc.Symbol(p, index)
}
-220
View File
@@ -1,220 +0,0 @@
package fragment
import (
"errors"
"math/rand/v2"
"sync"
"time"
)
var (
ErrTooLarge = errors.New("fragmented message exceeds the size limit")
ErrSourceQuota = errors.New("source has too many messages in progress")
ErrNoRoom = errors.New("no room for fragment")
)
// TableConfig bounds a Table.
type TableConfig struct {
MaxBytes int // symbol bytes held across all entries
MaxEntries int // messages in progress across all sources
MaxPerSource int // messages in progress per source
MaxMsgLen int // largest message accepted
TTL time.Duration // an entry idle this long is dropped
}
// Table reassembles messages from senders that have not authenticated yet, the
// responder's view of msg1. Every resource is capped. When a cap is hit a
// random entry is evicted; random eviction is harder to aim than LRU, which a
// flood can use to flush specific entries.
//
// K identifies one message, typically the source plus FragID. S identifies the
// source for quotas, typically the address masked to /32 or /64.
//
// A Table is safe for concurrent use.
type Table[K, S comparable] struct {
mu sync.Mutex
cfg TableConfig
entries map[K]*tableEntry[K, S]
order []K
perSource map[S]int
bytes int
}
type tableEntry[K, S comparable] struct {
a *Assembly
source S
lastSeen time.Time
pos int
}
func NewTable[K, S comparable](cfg TableConfig) *Table[K, S] {
return &Table[K, S]{
cfg: cfg,
entries: make(map[K]*tableEntry[K, S]),
perSource: make(map[S]int),
}
}
// Result describes a completed message.
type Result struct {
Msg []byte // the decoded message
Received int // packet bytes received for it, for the amplification budget
}
// Add records a fragment parsed by Parse. When the message is complete its entry
// is removed and the result returned; otherwise the result is nil. A decode
// error means the message is lost and must be retransmitted.
func (t *Table[K, S]) Add(key K, source S, now time.Time, h Header, sym []byte) (*Result, error) {
if int(h.MsgLen) > t.cfg.MaxMsgLen {
return nil, ErrTooLarge
}
t.mu.Lock()
e, ok := t.entries[key]
if ok && now.Sub(e.lastSeen) > t.cfg.TTL {
t.remove(key, e)
ok = false
}
if !ok {
var err error
e, err = t.insert(key, source, h)
if err != nil {
t.mu.Unlock()
return nil, err
}
}
e.lastSeen = now
if !t.makeRoom(key, e.a.wouldStore(h, sym)) {
if e.a.Stored() == 0 {
t.remove(key, e)
}
t.mu.Unlock()
return nil, ErrNoRoom
}
before := e.a.Stored()
ready, err := e.a.Add(h, sym)
t.bytes += e.a.Stored() - before
if err != nil || !ready {
if errors.Is(err, ErrMismatch) || (err != nil && e.a.Stored() == 0) {
t.remove(key, e)
}
t.mu.Unlock()
return nil, err
}
t.remove(key, e)
t.mu.Unlock()
msg, err := e.a.Decode()
if err != nil {
return nil, err
}
return &Result{Msg: msg, Received: e.a.Received()}, nil
}
// Expire drops every entry idle for longer than the TTL.
func (t *Table[K, S]) Expire(now time.Time) {
t.mu.Lock()
defer t.mu.Unlock()
for i := 0; i < len(t.order); {
key := t.order[i]
e := t.entries[key]
if now.Sub(e.lastSeen) > t.cfg.TTL {
// remove swaps the last entry into position i, so look at i again.
t.remove(key, e)
continue
}
i++
}
}
// Len returns the number of messages in progress.
func (t *Table[K, S]) Len() int {
t.mu.Lock()
defer t.mu.Unlock()
return len(t.entries)
}
// Load returns how full the table is, from 0 to 1, by whichever of its byte and
// entry limits is closer.
func (t *Table[K, S]) Load() float64 {
t.mu.Lock()
defer t.mu.Unlock()
return max(float64(t.bytes)/float64(t.cfg.MaxBytes), float64(len(t.entries))/float64(t.cfg.MaxEntries))
}
// Bytes returns the number of symbol bytes held.
func (t *Table[K, S]) Bytes() int {
t.mu.Lock()
defer t.mu.Unlock()
return t.bytes
}
func (t *Table[K, S]) insert(key K, source S, h Header) (*tableEntry[K, S], error) {
if t.perSource[source] >= t.cfg.MaxPerSource {
return nil, ErrSourceQuota
}
a, err := NewAssembly(h)
if err != nil {
return nil, err
}
for len(t.entries) >= t.cfg.MaxEntries {
if !t.evictRandom(nil) {
return nil, ErrNoRoom
}
}
e := &tableEntry[K, S]{a: a, source: source, pos: len(t.order)}
t.entries[key] = e
t.order = append(t.order, key)
t.perSource[source]++
return e, nil
}
// makeRoom evicts random entries other than keep until n more bytes fit.
func (t *Table[K, S]) makeRoom(keep K, n int) bool {
for t.bytes+n > t.cfg.MaxBytes {
if !t.evictRandom(&keep) {
return false
}
}
return true
}
func (t *Table[K, S]) evictRandom(keep *K) bool {
candidates := len(t.order)
if keep != nil {
if _, ok := t.entries[*keep]; ok {
candidates--
}
}
if candidates <= 0 {
return false
}
for {
key := t.order[rand.IntN(len(t.order))]
if keep != nil && key == *keep {
continue
}
t.remove(key, t.entries[key])
return true
}
}
func (t *Table[K, S]) remove(key K, e *tableEntry[K, S]) {
last := len(t.order) - 1
moved := t.order[last]
t.order[e.pos] = moved
t.entries[moved].pos = e.pos
t.order = t.order[:last]
delete(t.entries, key)
t.bytes -= e.a.Stored()
if t.perSource[e.source] <= 1 {
delete(t.perSource, e.source)
} else {
t.perSource[e.source]--
}
}
-121
View File
@@ -1,121 +0,0 @@
// Package fragment carries handshake messages that do not fit in one UDP
// packet. A message is Reed-Solomon coded into equally sized symbols (see
// handshake/fec) and each symbol travels in its own packet:
//
// Nebula header (16) | Fragment header (8) | [cookie (16)] | symbol
//
// The fragment header is FragID u32 | MsgLen u16 | Index u8 | Flags u8, all big
// endian. Both ends derive the symbol layout from MsgLen, so any MsgLen-derived
// value never has to be sent. FlagCookie marks a cookie a responder under load
// asked for; room for it is always reserved, so a fragment never outgrows
// MaxPacketLen.
package fragment
import (
"encoding/binary"
"errors"
"github.com/slackhq/nebula/handshake/fec"
"github.com/slackhq/nebula/header"
)
const (
// HeaderLen is the size of the fragment header that follows the Nebula header.
HeaderLen = 8
// CookieLen is the size of the cookie a fragment carries when FlagCookie is set.
CookieLen = 16
// FlagCookie marks a fragment that carries a cookie after its header.
FlagCookie uint8 = 0x01
// MaxPacketLen bounds every fragment packet, Nebula header included. A relayed
// fragment gains a 16 byte header and a 16 byte tag, which lands exactly on
// the 1232 byte UDP payload available on an IPv6 minimum MTU path.
MaxPacketLen = 1200
// MaxSymbolSize is the largest symbol that fits in MaxPacketLen alongside a cookie.
MaxSymbolSize = MaxPacketLen - header.Len - HeaderLen - CookieLen
)
var (
ErrShortFragment = errors.New("fragment is too short")
ErrFragID = errors.New("fragment has a zero FragID")
ErrMsgLen = errors.New("fragment has a zero MsgLen")
ErrFlags = errors.New("fragment has reserved flags set")
ErrNoCookie = errors.New("fragment is too short for its cookie")
ErrSymbolSize = errors.New("fragment symbol size does not match its MsgLen")
)
// Cookie proves a sender received a responder's reply at its address.
type Cookie [CookieLen]byte
// Header is the fragment header.
type Header struct {
FragID uint32 // the sender's local index for this handshake, never zero
MsgLen uint16 // length of the whole message
Index uint8 // symbol index, data below K and repair from K
Flags uint8 // FlagCookie, other bits reserved and zero
Cookie Cookie // valid when Flags has FlagCookie
}
func (h Header) HasCookie() bool {
return h.Flags&FlagCookie != 0
}
// Encode appends the header, and its cookie if it has one, to b.
func (h Header) Encode(b []byte) []byte {
b = binary.BigEndian.AppendUint32(b, h.FragID)
b = binary.BigEndian.AppendUint16(b, h.MsgLen)
b = append(b, h.Index, h.Flags)
if h.HasCookie() {
b = append(b, h.Cookie[:]...)
}
return b
}
// Layout returns the symbol layout the header's MsgLen implies.
func (h Header) Layout() (fec.Layout, error) {
return fec.NewLayout(int(h.MsgLen), MaxSymbolSize)
}
// Parse validates the bytes following the Nebula header and splits them into
// the fragment header and the symbol. The symbol aliases b.
func Parse(b []byte) (Header, []byte, error) {
if len(b) < HeaderLen {
return Header{}, nil, ErrShortFragment
}
h := Header{
FragID: binary.BigEndian.Uint32(b[0:4]),
MsgLen: binary.BigEndian.Uint16(b[4:6]),
Index: b[6],
Flags: b[7],
}
if h.FragID == 0 {
return Header{}, nil, ErrFragID
}
if h.MsgLen == 0 {
return Header{}, nil, ErrMsgLen
}
if h.Flags&^FlagCookie != 0 {
return Header{}, nil, ErrFlags
}
b = b[HeaderLen:]
if h.HasCookie() {
if len(b) < CookieLen {
return Header{}, nil, ErrNoCookie
}
copy(h.Cookie[:], b)
b = b[CookieLen:]
}
l, err := h.Layout()
if err != nil {
return Header{}, nil, err
}
sym := b
if len(sym) != l.SymbolSize {
return Header{}, nil, ErrSymbolSize
}
return h, sym, nil
}
-3
View File
@@ -24,9 +24,6 @@ message NebulaHandshakeDetails {
uint64 Cookie = 4 [deprecated = true];
uint64 Time = 5;
uint32 CertVersion = 8;
// KEMKey is the initiator's ephemeral ML-KEM-1024 encapsulation key. It is
// only present in msg1 of the ix_psk2_mlkem1024 subtype.
bytes KEMKey = 9;
// reserved for WIP multiport
reserved 6, 7;
}
-68
View File
@@ -1,68 +0,0 @@
package handshake
import (
"crypto/hkdf"
"crypto/mlkem"
"crypto/sha256"
"errors"
"fmt"
)
// kemPSKSalt separates the psk derivation from any other use of the secret.
const kemPSKSalt = "nebula ix_psk2_mlkem1024 v1"
var ErrInvalidKEMKey = errors.New("peer sent an invalid ML-KEM encapsulation key")
// kemPSK derives the noise psk from an ML-KEM shared secret. The ciphertext is
// bound in because, unlike the encapsulation key, it travels outside the noise
// transcript.
func kemPSK(sharedKey, ciphertext []byte) ([]byte, error) {
secret := make([]byte, 0, len(sharedKey)+len(ciphertext))
secret = append(secret, sharedKey...)
secret = append(secret, ciphertext...)
return hkdf.Extract(sha256.New, secret, []byte(kemPSKSalt))
}
// kemEncapsulate is the responder half: encapsulate to the initiator's key,
// install the psk, and remember the ciphertext to prefix the response with.
func (m *Machine) kemEncapsulate(encapsulationKey []byte) error {
ek, err := mlkem.NewEncapsulationKey1024(encapsulationKey)
if err != nil {
return fmt.Errorf("%w: %w", ErrInvalidKEMKey, err)
}
sharedKey, ct := ek.Encapsulate()
psk, err := kemPSK(sharedKey, ct)
if err != nil {
return err
}
if err := m.hs.SetPresharedKey(psk); err != nil {
return err
}
m.kemCiphertext = ct
return nil
}
// kemDecapsulate is the initiator half: recover the shared secret from the
// ciphertext prefixing the response and install the psk. It returns the rest of
// the message, which is the noise message.
func (m *Machine) kemDecapsulate(msg []byte) ([]byte, error) {
if len(msg) < mlkem.CiphertextSize1024 {
return nil, ErrPacketTooShort
}
if m.kemKey == nil {
return nil, errors.New("no ML-KEM decapsulation key for this handshake")
}
ct := msg[:mlkem.CiphertextSize1024]
sharedKey, err := m.kemKey.Decapsulate(ct)
if err != nil {
return nil, err
}
psk, err := kemPSK(sharedKey, ct)
if err != nil {
return nil, err
}
if err := m.hs.SetPresharedKey(psk); err != nil {
return nil, err
}
return msg[mlkem.CiphertextSize1024:], nil
}
-208
View File
@@ -1,208 +0,0 @@
package handshake
import (
"bytes"
"crypto/mlkem"
"net/netip"
"testing"
"time"
"github.com/flynn/noise"
"github.com/slackhq/nebula/cert"
ct "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/header"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func newKEMMachine(t *testing.T, cs *testCertState, v CertVerifier, initiator bool, index uint32) *Machine {
t.Helper()
m, err := NewMachine(
cs.version, cs.getCredential, v,
func() (uint32, error) { return index, nil },
initiator, header.HandshakeIXPSK2MLKEM1024,
)
require.NoError(t, err)
return m
}
type kemFixture struct {
initCS, respCS *testCertState
v CertVerifier
}
func newKEMFixture(t *testing.T, cipher noise.CipherFunc) kemFixture {
ca, _, caKey, _ := ct.NewTestCaCert(
cert.Version2, cert.Curve_CURVE25519, time.Time{}, time.Time{}, nil, nil, nil,
)
return kemFixture{
initCS: newTestCertStateWithCipher(t, ca, caKey, "initiator", []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")}, cipher),
respCS: newTestCertStateWithCipher(t, ca, caKey, "responder", []netip.Prefix{netip.MustParsePrefix("10.0.0.2/24")}, cipher),
v: testVerifier(ct.NewTestCAPool(ca)),
}
}
func requireKeysMatch(t *testing.T, a, b *Result) {
t.Helper()
c, err := a.EKey.Encrypt(nil, nil, []byte("hello"))
require.NoError(t, err)
p, err := b.DKey.Decrypt(nil, nil, c)
require.NoError(t, err)
assert.Equal(t, []byte("hello"), p)
c, err = b.EKey.Encrypt(nil, nil, []byte("world"))
require.NoError(t, err)
p, err = a.DKey.Decrypt(nil, nil, c)
require.NoError(t, err)
assert.Equal(t, []byte("world"), p)
}
func TestKEMHandshake(t *testing.T) {
for _, cipher := range []noise.CipherFunc{noise.CipherChaChaPoly, noise.CipherAESGCM} {
t.Run(cipher.CipherName(), func(t *testing.T) {
f := newKEMFixture(t, cipher)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newKEMMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
assert.Equal(t, header.HandshakeIXPSK2MLKEM1024, header.MessageSubType(msg1[1]))
assert.Greater(t, len(msg1), header.Len+mlkem.EncapsulationKeySize1024)
msg2, respR, err := respM.ProcessPacket(nil, msg1)
require.NoError(t, err)
require.NotNil(t, respR)
assert.Greater(t, len(msg2), header.Len+mlkem.CiphertextSize1024)
_, initR, err := initM.ProcessPacket(nil, msg2)
require.NoError(t, err)
require.NotNil(t, initR)
assert.Equal(t, "responder", initR.RemoteCert.Certificate.Name())
assert.Equal(t, "initiator", respR.RemoteCert.Certificate.Name())
assert.Equal(t, uint32(2000), initR.RemoteIndex)
assert.Equal(t, uint32(1000), respR.RemoteIndex)
requireKeysMatch(t, initR, respR)
})
}
}
// TestKEMSecretMatters checks the ML-KEM secret really reaches the transport
// keys: an initiator holding a different decapsulation key cannot complete even
// though every DH input is genuine.
func TestKEMSecretMatters(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newKEMMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
msg2, _, err := respM.ProcessPacket(nil, msg1)
require.NoError(t, err)
other, err := mlkem.GenerateKey1024()
require.NoError(t, err)
initM.kemKey = other
_, res, err := initM.ProcessPacket(nil, msg2)
require.Error(t, err)
assert.Nil(t, res)
}
func TestKEMTamperedCiphertext(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newKEMMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
msg2, respR, err := respM.ProcessPacket(nil, msg1)
require.NoError(t, err)
t.Run("truncated", func(t *testing.T) {
_, res, err := initM.ProcessPacket(nil, msg2[:header.Len+mlkem.CiphertextSize1024-1])
require.ErrorIs(t, err, ErrPacketTooShort)
assert.Nil(t, res)
assert.False(t, initM.Failed())
})
t.Run("flipped ciphertext bit", func(t *testing.T) {
bad := bytes.Clone(msg2)
bad[header.Len+10] ^= 1
_, res, err := initM.ProcessPacket(nil, bad)
require.Error(t, err)
assert.Nil(t, res)
assert.False(t, initM.Failed(), "a forged ciphertext must not kill the handshake")
})
t.Run("genuine response still completes", func(t *testing.T) {
_, initR, err := initM.ProcessPacket(nil, msg2)
require.NoError(t, err)
require.NotNil(t, initR)
requireKeysMatch(t, initR, respR)
})
}
// TestKEMBadMsg1 drives processPayload by hand. With the psk modifier msg1's
// payload is encrypted under a key derived from the public ephemeral, so a
// forged payload would need re-encrypting; the parsing rules are what matter.
func TestKEMBadMsg1(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
msg1Flags := subtypeInfos[header.HandshakeIXPSK2MLKEM1024].msgs[0]
cert := f.initCS.creds[cert.Version2]
payload := func(kemKey []byte) []byte {
return MarshalPayload(nil, Payload{
Cert: cert.Bytes,
CertVersion: uint32(cert.Cert.Version()),
InitiatorIndex: 1000,
Time: 1,
KEMKey: kemKey,
})
}
dk, err := mlkem.GenerateKey1024()
require.NoError(t, err)
t.Run("missing KEM key", func(t *testing.T) {
m := newKEMMachine(t, f.respCS, f.v, false, 2000)
err := m.processPayload(payload(nil), msg1Flags)
require.ErrorIs(t, err, ErrUnexpectedContent)
assert.True(t, m.Failed())
})
t.Run("invalid KEM key", func(t *testing.T) {
m := newKEMMachine(t, f.respCS, f.v, false, 2000)
// Coefficients of 0xfff exceed the modulus, which ML-KEM must reject.
bad := bytes.Repeat([]byte{0xff}, mlkem.EncapsulationKeySize1024)
err := m.kemEncapsulate(bad)
require.ErrorIs(t, err, ErrInvalidKEMKey)
err = m.kemEncapsulate(bad[:100])
require.ErrorIs(t, err, ErrInvalidKEMKey)
})
t.Run("valid KEM key", func(t *testing.T) {
m := newKEMMachine(t, f.respCS, f.v, false, 2000)
require.NoError(t, m.kemEncapsulate(dk.EncapsulationKey().Bytes()))
assert.Len(t, m.kemCiphertext, mlkem.CiphertextSize1024)
})
t.Run("KEM key on the classical subtype", func(t *testing.T) {
m := newTestMachine(t, f.respCS, f.v, false, 2000)
classical := subtypeInfos[header.HandshakeIXPSK0].msgs[0]
err := m.processPayload(payload(dk.EncapsulationKey().Bytes()), classical)
require.ErrorIs(t, err, ErrUnexpectedContent)
assert.True(t, m.Failed())
})
}
// TestKEMDistinctFromClassical makes sure the two subtypes cannot be mixed: a
// classical responder cannot answer a KEM initiator.
func TestKEMDistinctFromClassical(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newTestMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
_, _, err = respM.ProcessPacket(nil, msg1)
require.ErrorIs(t, err, ErrSubtypeMismatch)
}
+5 -54
View File
@@ -2,8 +2,6 @@ package handshake
import (
"bytes"
"crypto/mlkem"
"errors"
"fmt"
"slices"
"time"
@@ -69,9 +67,6 @@ type Machine struct {
remoteCertSet bool
payloadSet bool
failed bool
kemKey *mlkem.DecapsulationKey1024 // initiator's ephemeral ML-KEM key, until completion
kemCiphertext []byte // responder's ciphertext to prefix its response with
}
// NewMachine creates a handshake state machine. The subtype determines both
@@ -96,7 +91,7 @@ func NewMachine(
return nil, fmt.Errorf("%w: %v", ErrNoCredential, version)
}
hs, err := cred.buildHandshakeState(initiator, info.pattern, info.pskPlacement)
hs, err := cred.buildHandshakeState(initiator, info.pattern)
if err != nil {
return nil, fmt.Errorf("build noise state: %w", err)
}
@@ -142,8 +137,7 @@ func (m *Machine) requireComplete() error {
return nil
}
// myMsgFlags returns the flags for the current outgoing message, which are
// also the flags of the next incoming message before it is read.
// myMsgFlags returns the flags for the current outgoing message.
func (m *Machine) myMsgFlags() msgFlags {
idx := m.hs.MessageIndex()
if idx < len(m.msgs) {
@@ -231,18 +225,7 @@ func (m *Machine) ProcessPacket(out, packet []byte) ([]byte, *Result, error) {
// noise returns (cs1, cs2) where cs1 is the initiator->responder cipher.
// For 3-message patterns where a responder finishes by reading the final
// message, this ordering would be wrong; revisit when XX/pqIX lands.
noiseMsg := packet[header.Len:]
if m.myMsgFlags().kemCiphertext {
var err error
// Installing a psk derived from a forged ciphertext is harmless: ReadMessage
// then fails without advancing, and the next packet installs its own.
noiseMsg, err = m.kemDecapsulate(noiseMsg)
if err != nil {
return nil, nil, err
}
}
msg, eKey, dKey, err := m.hs.ReadMessage(nil, noiseMsg)
msg, eKey, dKey, err := m.hs.ReadMessage(nil, packet[header.Len:])
if err != nil {
// Noise ReadMessage failed. The noise library checkpoints and rolls back
// on failure, so the Machine is still alive. The caller can retry with
@@ -293,8 +276,6 @@ func (m *Machine) ProcessPacket(out, packet []byte) ([]byte, *Result, error) {
}
func (m *Machine) completed(eKey, dKey *noise.CipherState) *Result {
m.kemKey = nil
m.kemCiphertext = nil
m.result.EKey = eKey
m.result.DKey = dKey
m.result.MessageIndex = uint64(m.hs.MessageIndex())
@@ -303,7 +284,7 @@ func (m *Machine) completed(eKey, dKey *noise.CipherState) *Result {
func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
if len(msg) == 0 {
if flags.expectsPayload || flags.expectsCert || flags.kemKey {
if flags.expectsPayload || flags.expectsCert {
m.failed = true
return ErrMissingContent
}
@@ -329,12 +310,6 @@ func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
return ErrUnexpectedContent
}
hasKEMKey := len(payload.KEMKey) > 0
if hasKEMKey != flags.kemKey {
m.failed = true
return ErrUnexpectedContent
}
// Process payload
if flags.expectsPayload {
var remoteIndex uint32
@@ -361,13 +336,6 @@ func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
}
}
if flags.kemKey {
if err := m.kemEncapsulate(payload.KEMKey); err != nil {
m.failed = true
return err
}
}
return nil
}
@@ -412,21 +380,11 @@ func (m *Machine) validateCert(payload Payload) error {
}
func (m *Machine) marshalOutgoing(flags msgFlags) ([]byte, error) {
if !flags.expectsPayload && !flags.expectsCert && !flags.kemKey {
if !flags.expectsPayload && !flags.expectsCert {
return nil, nil
}
var p Payload
if flags.kemKey {
if m.kemKey == nil {
dk, err := mlkem.GenerateKey1024()
if err != nil {
return nil, fmt.Errorf("generate ML-KEM key: %w", err)
}
m.kemKey = dk
}
p.KEMKey = m.kemKey.EncapsulationKey().Bytes()
}
if flags.expectsPayload {
if !m.indexAllocated {
index, err := m.allocIndex()
@@ -478,13 +436,6 @@ func (m *Machine) buildResponse(out []byte) ([]byte, *noise.CipherState, *noise.
uint64(m.hs.MessageIndex()+1),
)
if flags.kemCiphertext {
if m.kemCiphertext == nil {
return nil, nil, nil, errors.New("no ML-KEM ciphertext to send")
}
out = append(out, m.kemCiphertext...)
}
// noise.WriteMessage appends the encrypted handshake message to out,
// reusing capacity when present.
//
+2 -21
View File
@@ -11,16 +11,13 @@ import (
type msgFlags struct {
expectsPayload bool // message carries indexes and time
expectsCert bool // message carries the certificate
kemKey bool // payload carries an ML-KEM encapsulation key
kemCiphertext bool // message is prefixed with the ML-KEM ciphertext for the peer's key
}
// subtypeInfo bundles the noise pattern with the per-message flags for a
// given handshake subtype.
type subtypeInfo struct {
pattern noise.HandshakePattern
pskPlacement int // noise psk modifier position, 0 means no psk token
msgs []msgFlags
pattern noise.HandshakePattern
msgs []msgFlags
}
// subtypeInfos defines the noise pattern and message content layout for each
@@ -35,22 +32,6 @@ var subtypeInfos = map[header.MessageSubType]subtypeInfo{
},
},
// IX with an ephemeral ML-KEM-1024 exchange:
// msg1 (I->R): payload + cert + encapsulation key
// msg2 (R->I): ciphertext || noise(payload + cert)
// The responder encapsulates after reading msg1 and mixes the shared secret in
// through the psk2 token at the end of msg2. The ciphertext travels outside the
// noise message because the initiator needs the secret before it can process
// that token.
header.HandshakeIXPSK2MLKEM1024: {
pattern: noise.HandshakeIX,
pskPlacement: 2,
msgs: []msgFlags{
{expectsPayload: true, expectsCert: true, kemKey: true},
{expectsPayload: true, expectsCert: true, kemCiphertext: true},
},
},
// XX: 3 messages
// msg1 (I->R): payload only
// msg2 (R->I): payload + cert
-16
View File
@@ -20,7 +20,6 @@ type Payload struct {
ResponderIndex uint32
Time uint64
CertVersion uint32
KEMKey []byte // ML-KEM encapsulation key, only in subtypes that carry one
}
// Proto field numbers for NebulaHandshakeDetails
@@ -30,7 +29,6 @@ const (
fieldResponderIndex = 3 // uint32
fieldTime = 5 // uint64
fieldCertVersion = 8 // uint32
fieldKEMKey = 9 // bytes
)
// MarshalPayload encodes a handshake payload in protobuf wire format compatible
@@ -59,10 +57,6 @@ func MarshalPayload(out []byte, p Payload) []byte {
details = protowire.AppendTag(details, fieldCertVersion, protowire.VarintType)
details = protowire.AppendVarint(details, uint64(p.CertVersion))
}
if len(p.KEMKey) > 0 {
details = protowire.AppendTag(details, fieldKEMKey, protowire.BytesType)
details = protowire.AppendBytes(details, p.KEMKey)
}
out = protowire.AppendTag(out, 1, protowire.BytesType)
out = protowire.AppendBytes(out, details)
@@ -167,16 +161,6 @@ func unmarshalPayloadDetails(p *Payload, b []byte) error {
}
p.CertVersion = uint32(v)
b = b[n:]
case fieldKEMKey:
if typ != protowire.BytesType {
return errInvalidHandshakeDetails
}
v, n := protowire.ConsumeBytes(b)
if n < 0 {
return errInvalidHandshakeDetails
}
p.KEMKey = append([]byte(nil), v...)
b = b[n:]
default:
n := protowire.ConsumeFieldValue(num, typ, b)
if n < 0 {
+46 -151
View File
@@ -10,15 +10,12 @@ import (
"net/netip"
"slices"
"sync"
"sync/atomic"
"time"
"github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/ratelimit"
"github.com/slackhq/nebula/udp"
)
@@ -52,7 +49,6 @@ type HandshakeConfig struct {
tryInterval time.Duration
retries int64
triggerBuffer int
pq pqHandshakeConfig
messageMetrics *MessageMetrics
}
@@ -78,18 +74,6 @@ type HandshakeManager struct {
// can be used to trigger outbound handshake for the given vpnIp
trigger chan netip.Addr
// pq holds the post-quantum settings, which change on reload.
pq atomic.Pointer[pqHandshakeConfig]
// pqTable reassembles post-quantum msg1s from senders that have not authenticated yet.
pqTable *fragment.Table[pqFragKey, pqFragSource]
// pqPeers holds vpn addresses that completed a post-quantum handshake, for sticky mode.
pqPeers sync.Map
// pqAnswered recognises repeats of post-quantum msg1s already answered.
pqAnswered *pqAnswered
// pqCookies and pqCookieLimiter serve cookies while pqTable is under load.
pqCookies *pqCookies
pqCookieLimiter *ratelimit.Limiter
}
type HandshakeHostInfo struct {
@@ -105,8 +89,6 @@ type HandshakeHostInfo struct {
hostinfo *HostInfo
machine *handshake.Machine // The handshake state machine, set during stage 0 (initiator) or beginHandshake (responder multi-message)
pq *pqAttempt // The initiator's post-quantum attempt, pending alongside the classical one
lastPQ bool // Whether the latest attempt was post-quantum
}
func (hh *HandshakeHostInfo) cachePacket(l *slog.Logger, t header.MessageType, st header.MessageSubType, packet []byte, f packetCallback, m *cachedPacketMetrics) {
@@ -135,7 +117,7 @@ func (hh *HandshakeHostInfo) cachePacket(l *slog.Logger, t header.MessageType, s
}
func NewHandshakeManager(l *slog.Logger, mainHostMap *HostMap, lightHouse *LightHouse, outside udp.Conn, config HandshakeConfig) *HandshakeManager {
hm := &HandshakeManager{
return &HandshakeManager{
vpnIps: map[netip.Addr]*HandshakeHostInfo{},
indexes: map[uint32]*HandshakeHostInfo{},
mainHostMap: mainHostMap,
@@ -148,16 +130,7 @@ func NewHandshakeManager(l *slog.Logger, mainHostMap *HostMap, lightHouse *Light
metricInitiated: metrics.GetOrRegisterCounter("handshake_manager.initiated", nil),
metricTimedOut: metrics.GetOrRegisterCounter("handshake_manager.timed_out", nil),
l: l,
pqTable: fragment.NewTable[pqFragKey, pqFragSource](pqTableConfig),
pqAnswered: newPQAnswered(hsTimeout(config.retries, config.tryInterval)),
}
pq := config.pq
hm.pq.Store(&pq)
hm.pqCookies = newPQCookies(time.Now())
// Cookie replies are tiny, but a flood of spoofed fragments must not turn
// into a flood of replies.
hm.pqCookieLimiter, _ = ratelimit.New(256, time.Millisecond)
return hm
}
func (hm *HandshakeManager) Run(ctx context.Context) {
@@ -172,8 +145,6 @@ func (hm *HandshakeManager) Run(ctx context.Context) {
hm.handleOutbound(vpnIP, true)
case now := <-clockSource.C:
hm.NextOutboundHandshakeTimerTick(now)
hm.pqTable.Expire(now)
hm.pqAnswered.expire(now)
}
}
}
@@ -183,7 +154,7 @@ func (hm *HandshakeManager) HandleIncoming(via ViaSender, packet []byte, h *head
// don't yet support) are dropped here rather than silently routed through
// the IX path. Add a case when introducing a new pattern.
switch h.Subtype {
case header.HandshakeIXPSK0, pqSubtype:
case header.HandshakeIXPSK0:
// supported
default:
hm.l.Debug("dropping handshake with unsupported subtype",
@@ -199,11 +170,6 @@ func (hm *HandshakeManager) HandleIncoming(via ViaSender, packet []byte, h *head
}
}
if header.IsFragmented(h.Subtype) {
hm.handleFragment(via, packet, h)
return
}
// First message of a new handshake. The wire format requires RemoteIndex
// to be zero here (the initiator has no responder index to fill in yet),
// and generateIndex never allocates 0, so any non-zero RemoteIndex on a
@@ -215,11 +181,7 @@ func (hm *HandshakeManager) HandleIncoming(via ViaSender, packet []byte, h *head
"from", via, "remoteIndex", h.RemoteIndex)
return
}
if hm.pqConfig().require {
hm.l.Debug("dropping classical handshake, post-quantum handshakes are required", "from", via)
return
}
hm.beginHandshake(via, packet, h, 0)
hm.beginHandshake(via, packet, h)
return
}
@@ -268,9 +230,6 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
"style": header.SubTypeName(header.Handshake, hh.machine.Subtype()),
})
}
if hh.pq != nil {
fields = append(fields, "pqInitiatorIndex", hh.pq.index)
}
hh.hostinfo.logger(hm.l).Info("Handshake timed out", fields...)
hm.metricTimedOut.Inc(1)
hm.DeleteHostInfo(hostinfo)
@@ -280,45 +239,25 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
// Increment the counter to increase our delay, linear backoff
hh.counter++
// Pick this attempt's style and make sure its first message exists. A PQ
// attempt sends msg1 as fragments: now goes out immediately, later after the
// repair delay, and every attempt offers fresh repair symbols.
var now, later [][]byte
var machine *handshake.Machine
hh.lastPQ = hm.usePQ(hh, vpnIp, lighthouseTriggered)
if hh.lastPQ {
if hh.pq == nil && !hm.buildPQAttempt(hh) {
// Check if we have a handshake packet to transmit yet
if !hh.ready {
if !hm.buildStage0Packet(hh) {
hm.OutboundHandshakeTimer.Add(vpnIp, hm.config.tryInterval*time.Duration(hh.counter))
return
}
var err error
now, later, err = hh.pq.sender.Attempt(fragment.DefaultRepair(hh.pq.sender.Layout().K))
if err != nil {
hostinfo.logger(hm.l).Error("Failed to fragment handshake", "error", err)
hm.OutboundHandshakeTimer.Add(vpnIp, hm.config.tryInterval*time.Duration(hh.counter))
return
}
machine = hh.pq.machine
} else {
if !hh.ready {
if !hm.buildStage0Packet(hh) {
hm.OutboundHandshakeTimer.Add(vpnIp, hm.config.tryInterval*time.Duration(hh.counter))
return
}
}
// TODO: this hardcodes "always retransmit stage 0", which is correct for
// IX (the initiator only ever sends one packet, msg1) but wrong the
// moment a 3+ message pattern lands. The retry loop should resend the
// most recent outgoing message, not always stage 0. That implies
// HandshakeHostInfo tracking a single "currentOutbound" packet (bytes +
// header metadata) that gets replaced as the handshake progresses,
// instead of indexing into HandshakePacket.
now = [][]byte{hostinfo.HandshakePacket[handshakePacketStage0]}
machine = hh.machine
}
// TODO: this hardcodes "always retransmit stage 0", which is correct for
// IX (the initiator only ever sends one packet, msg1) but wrong the
// moment a 3+ message pattern lands. The retry loop should resend the
// most recent outgoing message, not always stage 0. That implies
// HandshakeHostInfo tracking a single "currentOutbound" packet (bytes +
// header metadata) that gets replaced as the handshake progresses,
// instead of indexing into HandshakePacket.
stage0 := hostinfo.HandshakePacket[handshakePacketStage0]
hsFields := m{
"stage": uint64(machine.MessageIndex()),
"style": header.SubTypeName(header.Handshake, machine.Subtype()),
"stage": uint64(hh.machine.MessageIndex()),
"style": header.SubTypeName(header.Handshake, hh.machine.Subtype()),
}
remotes := hostinfo.remotes.CopyAddrs(hm.mainHostMap.GetPreferredRanges())
@@ -346,10 +285,8 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
// Send the handshake to all known ips, stage 2 takes care of assigning the hostinfo.remote based on the first to reply
var sentTo []netip.AddrPort
hostinfo.remotes.ForEach(hm.mainHostMap.GetPreferredRanges(), func(addr netip.AddrPort, _ bool) {
hm.messageMetrics.Tx(header.Handshake, machine.Subtype(), int64(len(now)+len(later)))
err := hm.sendFragmented(now, later, func(p []byte) error {
return hm.outside.WriteTo(p, addr)
})
hm.messageMetrics.Tx(header.Handshake, hh.machine.Subtype(), 1)
err := hm.outside.WriteTo(stage0, addr)
if err != nil {
// These repeat every attempt, so match the success log below and only shout when the remotes changed
level := slog.LevelDebug
@@ -385,7 +322,7 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
)
}
hm.f.relayManager.StartRelays(hm.f, vpnIp, hh, now, later)
hm.f.relayManager.StartRelays(hm.f, vpnIp, hh, stage0)
// If a lighthouse triggered this attempt then we are still in the timer wheel and do not need to re-add
if !lighthouseTriggered {
@@ -595,10 +532,6 @@ func (hm *HandshakeManager) DeleteHostInfo(hostinfo *HostInfo) {
func (hm *HandshakeManager) unlockedDeleteHostInfo(hostinfo *HostInfo) {
for _, addr := range hostinfo.vpnAddrs {
if cur, ok := hm.vpnIps[addr]; ok && cur.hostinfo == hostinfo {
// A PQ attempt holds its own index alongside the hostinfo's.
if idx := cur.pqIndex(); idx != 0 && hm.indexes[idx] == cur {
delete(hm.indexes, idx)
}
delete(hm.vpnIps, addr)
}
}
@@ -709,8 +642,9 @@ func hsTimeout(tries int64, interval time.Duration) time.Duration {
return time.Duration(tries / 2 * ((2 * int64(interval)) + (tries-1)*int64(interval)))
}
// initiatingVersion picks the certificate version to initiate a handshake with.
func (hm *HandshakeManager) initiatingVersion(hh *HandshakeHostInfo, cs *CertState) cert.Version {
// buildStage0Packet creates the initial handshake packet for the initiator.
func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
cs := hm.f.pki.getCertState()
v := cs.DefaultVersion()
if hh.initiatingVersionOverride != cert.VersionPre1 {
v = hh.initiatingVersionOverride
@@ -722,13 +656,6 @@ func (hm *HandshakeManager) initiatingVersion(hh *HandshakeHostInfo, cs *CertSta
}
}
}
return v
}
// buildStage0Packet creates the initial handshake packet for the initiator.
func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
cs := hm.f.pki.getCertState()
v := hm.initiatingVersion(hh, cs)
cred := cs.GetCredential(v)
if cred == nil {
@@ -767,10 +694,7 @@ func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
// beginHandshake handles an incoming handshake packet that doesn't match any
// existing pending handshake. It creates a new responder Machine and processes
// the first message.
//
// received is the number of bytes a fragmented msg1 arrived in, which bounds the
// response sent back before the address is validated; 0 for unfragmented ones.
func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *header.H, received int) {
func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *header.H) {
f := hm.f
cs := f.pki.getCertState()
@@ -784,7 +708,7 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
machine, err := handshake.NewMachine(
v, cs.GetCredential,
hm.certVerifier(), func() (uint32, error) { return generateIndex(f.l) },
false, h.Subtype,
false, header.HandshakeIXPSK0,
)
if err != nil {
f.l.Error("Failed to create handshake machine", "from", via, "error", err)
@@ -874,15 +798,11 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
existing, err := hm.CheckAndComplete(hostinfo, handshakePacketStage0, f)
if err != nil {
hm.handleCheckAndCompleteError(err, existing, hostinfo, via, machine.Subtype(), received)
hm.handleCheckAndCompleteError(err, existing, hostinfo, via)
return
}
hm.sendHandshakeResponse(via, response, hostinfo, false, received)
if header.IsFragmented(machine.Subtype()) {
hm.pqAnswered.add(packet[header.Len:], hostinfo, time.Now())
}
hm.notePQPeer(machine.Subtype(), vpnAddrs)
hm.sendHandshakeResponse(via, response, hostinfo, false)
hostinfo.remotes.RefreshFromHandshake(vpnAddrs)
// Don't wait for UpdateWorker
@@ -893,32 +813,23 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
// continueHandshake feeds an incoming packet to an existing pending handshake Machine.
func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostInfo, packet []byte) {
f := hm.f
hh.Lock()
defer hh.Unlock()
if !hm.stillPending(hh, hh.hostinfo.localIndexId) {
// Re-verify hh is still tracked. Between queryIndex returning and us taking
// hh.Lock, handleOutbound may have timed out and deleted it. Once we hold
// hh.Lock no other deleter can race our index: handleOutbound also takes
// hh.Lock first, and handleRecvError targets a main-hostmap entry with a
// different localIndexId.
hm.RLock()
cur, ok := hm.indexes[hh.hostinfo.localIndexId]
hm.RUnlock()
if !ok || cur != hh {
return
}
hm.processResponse(via, hh, hh.machine, packet)
}
// stillPending re-verifies hh is still tracked under index. Between queryIndex
// returning and the caller taking hh.Lock, handleOutbound may have timed out and
// deleted it. Once hh.Lock is held no other deleter can race the index:
// handleOutbound also takes hh.Lock first, and handleRecvError targets a
// main-hostmap entry with a different localIndexId. The caller must hold hh.Lock.
func (hm *HandshakeManager) stillPending(hh *HandshakeHostInfo, index uint32) bool {
hm.RLock()
defer hm.RUnlock()
cur, ok := hm.indexes[index]
return ok && cur == hh
}
// processResponse runs a responder's message through one of the initiator's
// pending attempts and completes the tunnel when the handshake finishes. The
// caller must hold hh.Lock.
func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo, machine *handshake.Machine, packet []byte) {
f := hm.f
hostinfo := hh.hostinfo
if !via.IsRelayed {
if !f.lightHouse.GetRemoteAllowList().AllowAll(hostinfo.vpnAddrs, via.UdpAddr.Addr()) {
@@ -928,6 +839,7 @@ func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo
}
}
machine := hh.machine
if machine == nil {
f.l.Error("No handshake machine available for continuation",
"vpnAddrs", hostinfo.vpnAddrs, "from", via)
@@ -950,7 +862,7 @@ func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo
}
if response != nil {
hm.sendHandshakeResponse(via, response, hostinfo, false, 0)
hm.sendHandshakeResponse(via, response, hostinfo, false)
}
if result == nil {
@@ -1063,10 +975,7 @@ func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo
hostinfo.vpnAddrs = vpnAddrs
hostinfo.buildNetworks(f.myVpnNetworksTable, remoteCert.Certificate)
// The classical and PQ attempts each hold an index; keep the one that won.
hm.settleIndex(hh, result.LocalIndex)
hm.Complete(hostinfo, f)
hm.notePQPeer(machine.Subtype(), vpnAddrs)
if len(hh.packetStore) > 0 {
if f.l.Enabled(context.Background(), slog.LevelDebug) {
@@ -1145,23 +1054,20 @@ func (hm *HandshakeManager) validatePeerCert(via ViaSender, remoteCert *cert.Cac
// cached is true when msg is a stored response being retransmitted because
// the peer's stage-1 retransmit landed (the ErrAlreadySeen path); false on a
// fresh response.
//
// received is the number of bytes a fragmented msg1 arrived in; it bounds what a
// fragmented response may send back. It is ignored for unfragmented subtypes.
func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hostinfo *HostInfo, cached bool, received int) {
func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hostinfo *HostInfo, cached bool) {
if msg == nil {
return
}
f := hm.f
subtype := header.MessageSubType(msg[1])
f.messageMetrics.Tx(header.Handshake, header.MessageSubType(msg[1]), 1)
// Common log fields. peerCert may be nil during intermediate
// multi-message flows (handshake hasn't completed yet); skip the cert
// block if so.
logFields := []any{
"vpnAddrs", hostinfo.vpnAddrs,
"handshake", m{"stage": uint64(2), "style": header.SubTypeName(header.Handshake, subtype)},
"handshake", m{"stage": uint64(2), "style": header.SubTypeName(header.Handshake, header.HandshakeIXPSK0)},
"cached", cached,
"initiatorIndex", hostinfo.remoteIndexId,
"responderIndex", hostinfo.localIndexId,
@@ -1175,17 +1081,6 @@ func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hos
)
}
if header.IsFragmented(subtype) {
fields := append(logFields, "from", via)
if err := hm.sendFragmentedResponse(via, msg, hostinfo, cached, received); err != nil {
f.l.Debug("Did not send handshake message", append(fields, "error", err)...)
} else {
f.l.Info("Handshake message sent", fields...)
}
return
}
f.messageMetrics.Tx(header.Handshake, subtype, 1)
if !via.IsRelayed {
fields := append(logFields, "from", via)
err := f.outside.WriteTo(msg, via.UdpAddr)
@@ -1212,10 +1107,10 @@ func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hos
// This only fires from the responder-side beginHandshake path, after the
// peer cert has been validated and ConnectionState populated, so peerCert
// is always non-nil for the cases that log it.
func (hm *HandshakeManager) handleCheckAndCompleteError(err error, existing, hostinfo *HostInfo, via ViaSender, subtype header.MessageSubType, received int) {
func (hm *HandshakeManager) handleCheckAndCompleteError(err error, existing, hostinfo *HostInfo, via ViaSender) {
f := hm.f
peerCert := hostinfo.ConnectionState.peerCert
hsFields := m{"stage": uint64(1), "style": header.SubTypeName(header.Handshake, subtype)}
hsFields := m{"stage": uint64(1), "style": header.SubTypeName(header.Handshake, header.HandshakeIXPSK0)}
switch err {
case ErrAlreadySeen:
@@ -1226,7 +1121,7 @@ func (hm *HandshakeManager) handleCheckAndCompleteError(err error, existing, hos
// ephemeral keys; a freshly-built one would have different keys and break
// the tunnel even though both sides "completed" the handshake.
if msg := existing.HandshakePacket[handshakePacketStage2]; msg != nil {
hm.sendHandshakeResponse(via, msg, existing, true, received)
hm.sendHandshakeResponse(via, msg, existing, true)
}
case ErrExistingHostInfo:
-699
View File
@@ -1,699 +0,0 @@
package nebula
import (
"context"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/binary"
"errors"
"log/slog"
"net/netip"
"sync"
"time"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header"
)
const (
// pqSubtype is the post-quantum handshake this node speaks.
pqSubtype = header.HandshakeIXPSK2MLKEM1024
// DefaultPQRepairDelay is how long repair fragments trail the data fragments.
DefaultPQRepairDelay = 10 * time.Millisecond
// pqAmplification bounds what a responder sends in reply to an unvalidated
// address, as a multiple of what it received from it.
pqAmplification = 3
)
// pqTableConfig bounds the responder's reassembly of msg1 fragments from senders
// that have not authenticated yet.
var pqTableConfig = fragment.TableConfig{
MaxBytes: 4 << 20,
MaxEntries: 1024,
MaxPerSource: 4,
MaxMsgLen: 16384,
TTL: 3 * time.Second,
}
type pqHandshakeConfig struct {
initiate bool // start PQ handshakes
accept bool // answer PQ handshakes
require bool // only PQ handshakes, in either direction
sticky bool // never fall back to classical for a peer that completed PQ
repairDelay time.Duration // delay between data and repair fragments
}
func pqHandshakeConfigFrom(c *config.C) pqHandshakeConfig {
cfg := pqHandshakeConfig{
initiate: c.GetBool("handshakes.pq.initiate", false),
accept: c.GetBool("handshakes.pq.accept", false),
require: c.GetBool("handshakes.pq.require", false),
sticky: c.GetBool("handshakes.pq.sticky", false),
repairDelay: c.GetDuration("handshakes.pq.repair_delay", DefaultPQRepairDelay),
}
if cfg.require {
cfg.initiate = true
cfg.accept = true
}
return cfg
}
// pqConfig returns the post-quantum settings in effect.
func (hm *HandshakeManager) pqConfig() *pqHandshakeConfig {
return hm.pq.Load()
}
// reloadPQ applies changed post-quantum settings. Handshakes already in flight
// keep going; the new settings apply to the next attempt and packet.
func (hm *HandshakeManager) reloadPQ(c *config.C) {
if !c.HasChanged("handshakes.pq") {
return
}
old := hm.pqConfig()
cfg := pqHandshakeConfigFrom(c)
hm.pq.Store(&cfg)
hm.l.Info("Post-quantum handshake settings changed",
"initiate", cfg.initiate, "oldInitiate", old.initiate,
"accept", cfg.accept, "oldAccept", old.accept,
"require", cfg.require, "oldRequire", old.require,
"sticky", cfg.sticky, "oldSticky", old.sticky,
"repairDelay", cfg.repairDelay, "oldRepairDelay", old.repairDelay,
)
}
// pqAttempt is the initiator's post-quantum attempt for a pending handshake. It
// lives alongside the classical attempt in HandshakeHostInfo and has its own
// local index, so whichever answer arrives first wins.
type pqAttempt struct {
machine *handshake.Machine
index uint32
sender *fragment.Sender
msg2 *fragment.Assembly
cookieResendAt time.Time // last resend prompted by a cookie reply
heard bool // a msg2 fragment or cookie reply arrived, so the peer speaks PQ
}
// pqFragKey identifies one msg1 being reassembled.
type pqFragKey struct {
addr netip.AddrPort
relay uint32
fragID uint32
}
// pqFragSource groups msg1 reassemblies for the per-source quota: an IPv4 /32,
// an IPv6 /64, or one relay channel.
type pqFragSource struct {
prefix netip.Prefix
relay uint32
}
func pqFragKeyFor(via ViaSender, fragID uint32) (pqFragKey, pqFragSource) {
if via.IsRelayed {
return pqFragKey{relay: via.relay.LocalIndex, fragID: fragID},
pqFragSource{relay: via.relay.LocalIndex}
}
addr := via.UdpAddr.Addr().Unmap()
bits := 64
if addr.Is4() {
bits = 32
}
prefix, _ := addr.Prefix(bits)
return pqFragKey{addr: via.UdpAddr, fragID: fragID}, pqFragSource{prefix: prefix}
}
// reassembled rebuilds the packet the handshake Machine expects from the header
// of a fragment and the decoded message.
func reassembled(h *header.H, msg []byte) []byte {
p := make([]byte, header.Len, header.Len+len(msg))
header.Encode(p, header.Version, header.Handshake, h.Subtype, h.RemoteIndex, h.MessageCounter)
return append(p, msg...)
}
// usePQ decides whether this outbound attempt is post-quantum. A peer that does
// not speak PQ silently drops it, so timer retries alternate between PQ and
// classical, starting with PQ. An attempt triggered early by a lighthouse reply
// is sending to addresses nothing has been sent to yet, and a new path gets the
// PQ attempt first. With require, or for a peer that completed a PQ handshake
// when sticky is on, every attempt is PQ. Once anything has come back for the PQ
// attempt the peer evidently speaks PQ, and a classical attempt would only race
// it, so classical stops getting turns.
func (hm *HandshakeManager) usePQ(hh *HandshakeHostInfo, vpnIp netip.Addr, triggered bool) bool {
cfg := hm.pqConfig()
switch {
case !cfg.initiate:
return false
case cfg.require:
return true
case hh.pq != nil && hh.pq.heard:
return true
case cfg.sticky:
if _, ok := hm.pqPeers.Load(vpnIp); ok {
return true
}
}
return hh.counter <= 1 || triggered || !hh.lastPQ
}
// relayEstablished pushes the pending handshake with target through a relay that
// just came up, rather than leaving it for the next retry. Nothing has gone over
// the relay yet, so like any new path it gets the PQ attempt first, which then
// counts as the PQ turn: the next retry is classical, as if it had been sent on
// schedule. Without this, relays coming up while the first attempt is in flight
// handed every relayed handshake to the classical retry.
func (hm *HandshakeManager) relayEstablished(target netip.Addr, relayHI *HostInfo, relay *Relay) {
hh := hm.queryVpnIp(target)
if hh == nil {
return
}
hh.Lock()
defer hh.Unlock()
if hm.queryVpnIp(target) != hh {
return
}
var now, later [][]byte
switch {
case hh.pq != nil:
var err error
now, later, err = hh.pq.sender.Attempt(fragment.DefaultRepair(hh.pq.sender.Layout().K))
if err != nil {
return
}
hh.lastPQ = true
case hh.ready:
now = [][]byte{hh.hostinfo.HandshakePacket[handshakePacketStage0]}
default:
return
}
f := hm.f
hh.hostinfo.logger(hm.l).Info("Send handshake via new relay", "relay", relayHI.vpnAddrs[0])
_ = hm.sendFragmented(now, later, func(p []byte) error {
f.SendVia(relayHI, relay, p, make([]byte, 12), make([]byte, mtu), false, 0)
return nil
})
}
// buildPQAttempt creates the initiator's PQ Machine and msg1 sender.
func (hm *HandshakeManager) buildPQAttempt(hh *HandshakeHostInfo) bool {
cs := hm.f.pki.getCertState()
v := hm.initiatingVersion(hh, cs)
if cs.GetCredential(v) == nil {
hm.f.l.Error("Unable to handshake with host because no certificate is available",
"vpnAddrs", hh.hostinfo.vpnAddrs, "certVersion", v)
return false
}
pq := &pqAttempt{}
machine, err := handshake.NewMachine(
v, cs.GetCredential,
hm.certVerifier(), func() (uint32, error) { return hm.allocatePQIndex(hh, pq) },
true, pqSubtype,
)
if err != nil {
hm.f.l.Error("Failed to create handshake machine",
"vpnAddrs", hh.hostinfo.vpnAddrs, "error", err)
return false
}
msg, err := machine.Initiate(nil)
if err != nil {
hm.f.l.Error("Failed to initiate handshake",
"vpnAddrs", hh.hostinfo.vpnAddrs, "error", err)
hm.releasePQIndex(hh, pq)
return false
}
sender, err := fragment.NewSender(msg[header.Len:], pqSubtype, 0, 1, pq.index)
if err != nil {
hm.f.l.Error("Failed to fragment handshake",
"vpnAddrs", hh.hostinfo.vpnAddrs, "error", err)
hm.releasePQIndex(hh, pq)
return false
}
pq.machine = machine
pq.sender = sender
hh.pq = pq
return true
}
// allocatePQIndex registers a local index for the PQ attempt. Unlike
// allocateIndex it leaves hostinfo.localIndexId alone; that is settled when one
// of the attempts completes.
func (hm *HandshakeManager) allocatePQIndex(hh *HandshakeHostInfo, pq *pqAttempt) (uint32, error) {
hm.mainHostMap.RLock()
defer hm.mainHostMap.RUnlock()
hm.Lock()
defer hm.Unlock()
for range 32 {
index, err := generateIndex(hm.l)
if err != nil {
return 0, err
}
_, inPending := hm.indexes[index]
_, inMain := hm.mainHostMap.Indexes[index]
if !inMain && !inPending {
pq.index = index
hm.indexes[index] = hh
return index, nil
}
}
return 0, errors.New("failed to generate unique localIndexId")
}
func (hm *HandshakeManager) releasePQIndex(hh *HandshakeHostInfo, pq *pqAttempt) {
hm.Lock()
defer hm.Unlock()
if pq.index != 0 && hm.indexes[pq.index] == hh {
delete(hm.indexes, pq.index)
}
}
// settleIndex makes winner the hostinfo's local index and releases the index of
// the attempt that lost.
func (hm *HandshakeManager) settleIndex(hh *HandshakeHostInfo, winner uint32) {
hm.Lock()
defer hm.Unlock()
for _, idx := range []uint32{hh.hostinfo.localIndexId, hh.pqIndex()} {
if idx != 0 && idx != winner && hm.indexes[idx] == hh {
delete(hm.indexes, idx)
}
}
hh.hostinfo.localIndexId = winner
}
func (hh *HandshakeHostInfo) pqIndex() uint32 {
if hh.pq == nil {
return 0
}
return hh.pq.index
}
// handleFragment routes one fragment of a PQ handshake message.
func (hm *HandshakeManager) handleFragment(via ViaSender, packet []byte, h *header.H) {
if h.MessageCounter == pqCookieCounter {
hm.handleCookie(via, packet, h)
return
}
fh, sym, err := fragment.Parse(packet[header.Len:])
if err != nil {
hm.l.Debug("dropping malformed handshake fragment", "from", via, "error", err)
return
}
switch h.MessageCounter {
case 1:
if !hm.pqConfig().accept {
hm.l.Debug("dropping post-quantum handshake, not accepting them", "from", via)
return
}
if h.RemoteIndex != 0 {
hm.l.Debug("dropping stage-1 handshake with non-zero RemoteIndex",
"from", via, "remoteIndex", h.RemoteIndex)
return
}
now := time.Now()
key, src := pqFragKeyFor(via, fh.FragID)
if hm.pqTable.Load() >= pqCookieLoad && !(fh.HasCookie() && hm.pqCookies.valid(key, fh.Cookie, now)) {
hm.sendCookie(via, fh, key, len(packet), now)
return
}
res, err := hm.pqTable.Add(key, src, now, fh, sym)
if err != nil {
hm.l.Debug("dropping handshake fragment", "from", via, "error", err)
return
}
if res == nil {
return
}
if existing := hm.pqAnswered.lookup(res.Msg, time.Now()); existing != nil {
// Already answered: resend, rate limited, with fresh symbols.
if msg := existing.HandshakePacket[handshakePacketStage2]; msg != nil {
hm.sendHandshakeResponse(via, msg, existing, true, res.Received)
}
return
}
hm.beginHandshake(via, reassembled(h, res.Msg), h, res.Received)
case 2:
if hh := hm.queryIndex(h.RemoteIndex); hh != nil {
hm.continuePQ(via, hh, h, fh, sym)
}
}
}
// continuePQ collects msg2 fragments for the initiator's PQ attempt.
func (hm *HandshakeManager) continuePQ(via ViaSender, hh *HandshakeHostInfo, h *header.H, fh fragment.Header, sym []byte) {
hh.Lock()
defer hh.Unlock()
pq := hh.pq
if pq == nil || pq.index != h.RemoteIndex || !hm.stillPending(hh, pq.index) {
return
}
pq.heard = true
if pq.msg2 == nil {
a, err := fragment.NewAssembly(fh)
if err != nil {
return
}
pq.msg2 = a
}
ready, err := pq.msg2.Add(fh, sym)
if errors.Is(err, fragment.ErrMismatch) {
// The responder's FragID and MsgLen never change, so this is a stray or a
// forgery. Start over from this fragment rather than wedge.
pq.msg2 = nil
return
}
if err != nil || !ready {
return
}
msg, err := pq.msg2.Decode()
pq.msg2 = nil
if err != nil {
hm.l.Debug("failed to decode handshake fragments", "from", via, "error", err)
return
}
hm.processResponse(via, hh, pq.machine, reassembled(h, msg))
}
// sendFragmented sends packets to a peer: now immediately and later after the
// repair delay. It returns the first error from the immediate sends; errors from
// delayed sends are only logged.
func (hm *HandshakeManager) sendFragmented(now, later [][]byte, send func([]byte) error) error {
var first error
for _, p := range now {
if err := send(p); err != nil && first == nil {
first = err
}
}
sendLater := func() {
for _, p := range later {
if err := send(p); err != nil {
hm.l.Debug("Failed to send handshake repair fragment", "error", err)
}
}
}
delay := hm.pqConfig().repairDelay
switch {
case len(later) == 0:
case delay <= 0:
sendLater()
default:
time.AfterFunc(delay, sendLater)
}
return first
}
// sendFragmentedResponse sends the responder's msg2 as fragments. A resend of a
// cached response (cached) offers fresh symbols and is rate limited to one per
// retry interval. received is how many bytes the triggering msg1 took, which
// bounds what may be sent back before the address is validated.
func (hm *HandshakeManager) sendFragmentedResponse(via ViaSender, msg []byte, hostinfo *HostInfo, cached bool, received int) error {
now := time.Now()
if cached {
last := hostinfo.handshakeResendAt.Load()
if now.UnixNano()-last < int64(hm.config.tryInterval) ||
!hostinfo.handshakeResendAt.CompareAndSwap(last, now.UnixNano()) {
return errors.New("handshake response resend rate limited")
}
} else {
hostinfo.handshakeResendAt.Store(now.UnixNano())
}
var h header.H
if err := h.Parse(msg); err != nil {
return err
}
s, err := fragment.NewSender(msg[header.Len:], h.Subtype, h.RemoteIndex, h.MessageCounter, hostinfo.localIndexId)
if err != nil {
return err
}
l := s.Layout()
repair := fragment.DefaultRepair(l.K)
if received > 0 {
pktLen := header.Len + fragment.HeaderLen + l.SymbolSize
budget := pqAmplification * received / pktLen
if budget < l.K {
return errors.New("handshake response exceeds the amplification budget")
}
repair = min(repair, budget-l.K)
}
n := uint32(l.K + repair)
s.SetNext(int(hostinfo.handshakeNextSymbol.Add(n) - n))
pkts, later, err := s.Attempt(repair)
if err != nil {
return err
}
if via.IsRelayed {
if via.relay == nil {
return errors.New("both addr and via.relay are nil")
}
// We received a valid handshake on this relay, so make sure the relay
// state reflects that, in case it had been marked Disestablished.
hostinfo.relayState.InsertRelayTo(via.relayHI.vpnAddrs[0])
via.relayHI.relayState.UpdateRelayForByIdxState(via.relay.LocalIndex, Established)
}
hm.f.messageMetrics.Tx(header.Handshake, h.Subtype, int64(len(pkts)+len(later)))
return hm.sendFragmented(pkts, later, hm.sendTo(via))
}
// notePQPeer remembers peers that completed a PQ handshake, for sticky mode.
func (hm *HandshakeManager) notePQPeer(subtype header.MessageSubType, vpnAddrs []netip.Addr) {
if !hm.pqConfig().sticky || subtype != pqSubtype {
return
}
for _, a := range vpnAddrs {
hm.pqPeers.Store(a, struct{}{})
}
if hm.l.Enabled(context.Background(), slog.LevelDebug) {
hm.l.Debug("Remembering post-quantum peer", "vpnAddrs", vpnAddrs)
}
}
// pqAnswered remembers the msg1s a responder recently answered, keyed by hash,
// so a repeat is recognised before any noise, KEM or certificate work. Repeats
// are routine: the trailing repair fragments of an attempt rebuild a msg1 that
// its data fragments already completed, and a retrying initiator offers fresh
// symbols for the same msg1. Entries only exist for completed handshakes, which
// need a valid certificate, and live as long as an initiator keeps retrying.
type pqAnswered struct {
sync.Mutex
ttl time.Duration
entries map[[sha256.Size]byte]pqAnsweredEntry
}
type pqAnsweredEntry struct {
hostinfo *HostInfo
at time.Time
}
const pqAnsweredMax = 4096
func newPQAnswered(ttl time.Duration) *pqAnswered {
return &pqAnswered{ttl: ttl, entries: map[[sha256.Size]byte]pqAnsweredEntry{}}
}
func (a *pqAnswered) add(msg1 []byte, hostinfo *HostInfo, now time.Time) {
a.Lock()
defer a.Unlock()
if len(a.entries) >= pqAnsweredMax {
for k := range a.entries {
delete(a.entries, k)
break
}
}
a.entries[sha256.Sum256(msg1)] = pqAnsweredEntry{hostinfo: hostinfo, at: now}
}
func (a *pqAnswered) lookup(msg1 []byte, now time.Time) *HostInfo {
a.Lock()
defer a.Unlock()
e, ok := a.entries[sha256.Sum256(msg1)]
if !ok || now.Sub(e.at) > a.ttl {
return nil
}
return e.hostinfo
}
func (a *pqAnswered) expire(now time.Time) {
a.Lock()
defer a.Unlock()
for k, e := range a.entries {
if now.Sub(e.at) > a.ttl {
delete(a.entries, k)
}
}
}
// A responder whose msg1 table is under load stops keeping state for senders
// that have not shown they receive at their address. It answers a fragment
// without a valid cookie with a cookie reply and keeps nothing; the initiator
// resends with the cookie attached. This is WireGuard's under-load cookie, QUIC's
// Retry and SCTP's state cookie: one extra round trip, only while under attack.
const (
// pqCookieLoad is the table load above which msg1 fragments need a cookie.
pqCookieLoad = 0.5
// pqCookieCounter marks a cookie reply in the header's message counter.
// Handshake messages count from 1.
pqCookieCounter = 0
// pqCookieRotate is how often the cookie secret changes. A cookie stays
// valid for between one and two rotations.
pqCookieRotate = 2 * time.Minute
pqCookieReplyLen = header.Len + fragment.CookieLen
)
// pqCookies mints and checks cookies: an HMAC over the sender's address (or relay
// channel) and FragID under a rotating secret, with the previous secret still
// accepted.
type pqCookies struct {
sync.Mutex
secrets [2][32]byte
rotatedAt time.Time
}
func newPQCookies(now time.Time) *pqCookies {
c := &pqCookies{rotatedAt: now}
_, _ = rand.Read(c.secrets[0][:])
_, _ = rand.Read(c.secrets[1][:])
return c
}
func (c *pqCookies) rotate(now time.Time) {
if now.Sub(c.rotatedAt) < pqCookieRotate {
return
}
c.secrets[1] = c.secrets[0]
_, _ = rand.Read(c.secrets[0][:])
c.rotatedAt = now
}
func cookieMAC(secret []byte, key pqFragKey) fragment.Cookie {
mac := hmac.New(sha256.New, secret)
addr := key.addr.Addr().As16()
mac.Write(addr[:])
var b [10]byte
binary.BigEndian.PutUint16(b[0:], key.addr.Port())
binary.BigEndian.PutUint32(b[2:], key.relay)
binary.BigEndian.PutUint32(b[6:], key.fragID)
mac.Write(b[:])
var c fragment.Cookie
copy(c[:], mac.Sum(nil))
return c
}
func (c *pqCookies) make(key pqFragKey, now time.Time) fragment.Cookie {
c.Lock()
defer c.Unlock()
c.rotate(now)
return cookieMAC(c.secrets[0][:], key)
}
func (c *pqCookies) valid(key pqFragKey, cookie fragment.Cookie, now time.Time) bool {
c.Lock()
defer c.Unlock()
c.rotate(now)
for _, s := range c.secrets {
want := cookieMAC(s[:], key)
if hmac.Equal(want[:], cookie[:]) {
return true
}
}
return false
}
// sendCookie answers a msg1 fragment that arrived without a valid cookie while
// under load. The reply is never larger than what triggered it, and replies are
// rate limited as a whole.
func (hm *HandshakeManager) sendCookie(via ViaSender, fh fragment.Header, key pqFragKey, received int, now time.Time) {
if received < pqCookieReplyLen {
return
}
if _, ok := hm.pqCookieLimiter.Allow(); !ok {
return
}
c := hm.pqCookies.make(key, now)
reply := make([]byte, header.Len, pqCookieReplyLen)
header.Encode(reply, header.Version, header.Handshake, pqSubtype, fh.FragID, pqCookieCounter)
reply = append(reply, c[:]...)
hm.f.messageMetrics.Tx(header.Handshake, pqSubtype, 1)
if err := hm.sendTo(via)(reply); err != nil {
hm.l.Debug("Failed to send handshake cookie", "to", via, "error", err)
}
}
// handleCookie attaches a responder's cookie to the PQ attempt it names and
// resends right away, at most once per retry interval, rather than waiting for
// the next attempt.
func (hm *HandshakeManager) handleCookie(via ViaSender, packet []byte, h *header.H) {
if len(packet) != pqCookieReplyLen {
return
}
hh := hm.queryIndex(h.RemoteIndex)
if hh == nil {
return
}
hh.Lock()
defer hh.Unlock()
pq := hh.pq
if pq == nil || pq.index != h.RemoteIndex || !hm.stillPending(hh, pq.index) {
return
}
var c fragment.Cookie
copy(c[:], packet[header.Len:])
pq.sender.SetCookie(c)
pq.heard = true
now := time.Now()
if now.Sub(pq.cookieResendAt) < hm.config.tryInterval {
return
}
pq.cookieResendAt = now
pkts, later, err := pq.sender.Attempt(fragment.DefaultRepair(pq.sender.Layout().K))
if err != nil {
return
}
hm.messageMetrics.Tx(header.Handshake, pqSubtype, int64(len(pkts)+len(later)))
if err := hm.sendFragmented(pkts, later, hm.sendTo(via)); err != nil {
hm.l.Debug("Failed to resend handshake with cookie", "to", via, "error", err)
}
}
// sendTo returns a function sending a packet back the way via came: directly, or
// through the relay it arrived on.
func (hm *HandshakeManager) sendTo(via ViaSender) func([]byte) error {
f := hm.f
if !via.IsRelayed {
return func(p []byte) error {
return f.outside.WriteTo(p, via.UdpAddr)
}
}
return func(p []byte) error {
if via.relay == nil {
return errors.New("both addr and via.relay are nil")
}
f.SendVia(via.relayHI, via.relay, p, make([]byte, 12), make([]byte, mtu), false, 0)
return nil
}
}
-398
View File
@@ -1,398 +0,0 @@
package nebula
import (
"math/rand/v2"
"net/netip"
"sync"
"testing"
"time"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// recordingConn captures what is written to it.
type recordingConn struct {
udp.NoopConn
sync.Mutex
sent [][]byte
}
func (c *recordingConn) WriteTo(b []byte, _ netip.AddrPort) error {
c.Lock()
defer c.Unlock()
c.sent = append(c.sent, append([]byte(nil), b...))
return nil
}
func (c *recordingConn) take() [][]byte {
c.Lock()
defer c.Unlock()
s := c.sent
c.sent = nil
return s
}
func newPQTestManager(t *testing.T) (*HandshakeManager, *recordingConn) {
l := test.NewLogger()
conn := &recordingConn{}
cfg := defaultHandshakeConfig
cfg.pq = pqHandshakeConfig{initiate: true, accept: true}
hm := NewHandshakeManager(l, newHostMap(l), newTestLighthouse(), conn, cfg)
hm.f = &Interface{
handshakeManager: hm,
outside: conn,
messageMetrics: newMessageMetricsOnlyRecvError(),
l: l,
}
return hm, conn
}
// fragmentIndexes checks every packet is a fragment of the expected response and
// returns their symbol indexes.
func fragmentIndexes(t *testing.T, pkts [][]byte, fragID, remoteIndex uint32) []int {
t.Helper()
var out []int
for _, p := range pkts {
require.LessOrEqual(t, len(p), fragment.MaxPacketLen)
var h header.H
require.NoError(t, h.Parse(p))
assert.Equal(t, pqSubtype, h.Subtype)
assert.Equal(t, remoteIndex, h.RemoteIndex)
assert.Equal(t, uint64(2), h.MessageCounter)
fh, _, err := fragment.Parse(p[header.Len:])
require.NoError(t, err)
assert.Equal(t, fragID, fh.FragID)
out = append(out, int(fh.Index))
}
return out
}
func TestSendFragmentedResponse(t *testing.T) {
hm, conn := newPQTestManager(t)
via := ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}
hostinfo := &HostInfo{localIndexId: 77}
// A 2000 byte message is 2 symbols of 1000, so packets of 1024 bytes.
msg := make([]byte, header.Len, header.Len+2000)
header.Encode(msg, header.Version, header.Handshake, pqSubtype, 5, 2)
r := rand.New(rand.NewPCG(1, 1))
for range 2000 {
msg = append(msg, byte(r.Uint32()))
}
const pktLen = header.Len + fragment.HeaderLen + 1000
t.Run("first send", func(t *testing.T) {
require.NoError(t, hm.sendFragmentedResponse(via, msg, hostinfo, false, 0))
assert.Equal(t, []int{0, 1, 2, 3}, fragmentIndexes(t, conn.take(), 77, 5))
})
t.Run("resends are rate limited", func(t *testing.T) {
require.Error(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, 10*pktLen))
assert.Empty(t, conn.take())
})
allowResend := func() {
hostinfo.handshakeResendAt.Store(time.Now().Add(-time.Second).UnixNano())
}
t.Run("resend offers fresh symbols", func(t *testing.T) {
allowResend()
require.NoError(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, 10*pktLen))
assert.Equal(t, []int{4, 5, 6, 7}, fragmentIndexes(t, conn.take(), 77, 5))
})
t.Run("amplification budget trims repair", func(t *testing.T) {
allowResend()
// 3x of a little over one packet allows 3 packets: both data and 1 repair.
require.NoError(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, pktLen+10))
assert.Equal(t, []int{8, 9, 10}, fragmentIndexes(t, conn.take(), 77, 5))
})
t.Run("amplification budget below the data refuses", func(t *testing.T) {
allowResend()
require.Error(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, pktLen/2))
assert.Empty(t, conn.take())
})
}
func TestPQAnswered(t *testing.T) {
now := time.Unix(1000, 0)
a := newPQAnswered(5 * time.Second)
hi := &HostInfo{}
assert.Nil(t, a.lookup([]byte("msg1"), now))
a.add([]byte("msg1"), hi, now)
assert.Same(t, hi, a.lookup([]byte("msg1"), now.Add(time.Second)))
assert.Nil(t, a.lookup([]byte("other"), now))
assert.Nil(t, a.lookup([]byte("msg1"), now.Add(6*time.Second)), "stale entries are not answered")
a.expire(now.Add(6 * time.Second))
assert.Empty(t, a.entries)
for i := range pqAnsweredMax + 10 {
a.add([]byte{byte(i), byte(i >> 8)}, hi, now)
}
assert.Len(t, a.entries, pqAnsweredMax)
}
func TestPQFragKeyFor(t *testing.T) {
k1, s1 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}, 9)
k2, s2 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4243")}, 9)
assert.NotEqual(t, k1, k2, "different ports are different messages")
assert.Equal(t, s1, s2, "but share an IPv4 /32 quota")
_, s3 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("[2001:db8::1]:1")}, 9)
_, s4 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("[2001:db8::ffff]:1")}, 9)
assert.Equal(t, s3, s4, "an IPv6 /64 shares a quota")
_, s5 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("[::ffff:192.0.2.1]:1")}, 9)
assert.Equal(t, s1, s5, "v4-mapped addresses count as IPv4")
}
func TestPQCookies(t *testing.T) {
now := time.Unix(1000, 0)
c := newPQCookies(now)
key := pqFragKey{addr: netip.MustParseAddrPort("192.0.2.1:4242"), fragID: 9}
cookie := c.make(key, now)
assert.True(t, c.valid(key, cookie, now))
for _, other := range []pqFragKey{
{addr: netip.MustParseAddrPort("192.0.2.2:4242"), fragID: 9},
{addr: netip.MustParseAddrPort("192.0.2.1:4243"), fragID: 9},
{addr: netip.MustParseAddrPort("192.0.2.1:4242"), fragID: 10},
{addr: netip.MustParseAddrPort("192.0.2.1:4242"), relay: 1, fragID: 9},
} {
assert.False(t, c.valid(other, cookie, now), "%+v", other)
}
now = now.Add(pqCookieRotate)
assert.True(t, c.valid(key, cookie, now), "the previous secret is still accepted")
assert.NotEqual(t, cookie, c.make(key, now), "new cookies come from the new secret")
now = now.Add(pqCookieRotate)
assert.False(t, c.valid(key, cookie, now), "two rotations retire a cookie")
}
// pqMsg1Fragment builds the first fragment of a msg1, optionally with a cookie.
func pqMsg1Fragment(t *testing.T, fragID uint32, cookie *fragment.Cookie) ([]byte, *header.H) {
t.Helper()
s, err := fragment.NewSender(make([]byte, 2000), pqSubtype, 0, 1, fragID)
require.NoError(t, err)
if cookie != nil {
s.SetCookie(*cookie)
}
pkts, _, err := s.Attempt(0)
require.NoError(t, err)
var h header.H
require.NoError(t, h.Parse(pkts[0]))
return pkts[0], &h
}
// fillPQTable puts the responder's msg1 table at the cookie threshold.
func fillPQTable(t *testing.T, hm *HandshakeManager) {
t.Helper()
now := time.Now()
for i := 0; hm.pqTable.Load() < pqCookieLoad; i++ {
p, _ := pqMsg1Fragment(t, uint32(i+1), nil)
fh, sym, err := fragment.Parse(p[header.Len:])
require.NoError(t, err)
addr := netip.AddrPortFrom(netip.AddrFrom4([4]byte{198, 18, byte(i >> 8), byte(i)}), 4242)
key, src := pqFragKeyFor(ViaSender{UdpAddr: addr}, fh.FragID)
_, err = hm.pqTable.Add(key, src, now, fh, sym)
require.NoError(t, err)
}
}
func TestPQCookieUnderLoad(t *testing.T) {
hm, conn := newPQTestManager(t)
via := ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}
t.Run("no cookie needed below the threshold", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 5000, nil)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Empty(t, conn.take())
assert.Equal(t, before+1, hm.pqTable.Len())
})
fillPQTable(t, hm)
var cookie fragment.Cookie
t.Run("a fragment without a cookie gets one and leaves no state", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 7, nil)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Equal(t, before, hm.pqTable.Len())
sent := conn.take()
require.Len(t, sent, 1)
reply := sent[0]
require.Len(t, reply, pqCookieReplyLen)
var rh header.H
require.NoError(t, rh.Parse(reply))
assert.Equal(t, header.Handshake, rh.Type)
assert.Equal(t, pqSubtype, rh.Subtype)
assert.Equal(t, uint32(7), rh.RemoteIndex, "the reply names the initiator's FragID")
assert.Equal(t, uint64(pqCookieCounter), rh.MessageCounter)
copy(cookie[:], reply[header.Len:])
})
t.Run("a forged cookie gets a fresh one", func(t *testing.T) {
bad := cookie
bad[0] ^= 1
p, h := pqMsg1Fragment(t, 7, &bad)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Equal(t, before, hm.pqTable.Len())
assert.Len(t, conn.take(), 1)
})
t.Run("the cookie gets the fragment in", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 7, &cookie)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Equal(t, before+1, hm.pqTable.Len())
assert.Empty(t, conn.take())
})
t.Run("a cookie is bound to its address", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 7, &cookie)
hm.handleFragment(ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.99:4242")}, p, h)
assert.Len(t, conn.take(), 1, "another address gets its own cookie instead")
})
t.Run("a reply never outgrows what prompted it", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 8, nil)
hm.sendCookie(via, fragment.Header{FragID: 8}, pqFragKey{}, pqCookieReplyLen-1, time.Now())
assert.Empty(t, conn.take())
hm.handleFragment(via, p, h)
assert.Len(t, conn.take(), 1)
})
}
func TestPQHandleCookie(t *testing.T) {
hm, conn := newPQTestManager(t)
via := ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}
s, err := fragment.NewSender(make([]byte, 2000), pqSubtype, 0, 1, 99)
require.NoError(t, err)
_, _, err = s.Attempt(2)
require.NoError(t, err)
hh := &HandshakeHostInfo{hostinfo: &HostInfo{}, pq: &pqAttempt{index: 99, sender: s}}
hm.indexes[99] = hh
reply := make([]byte, header.Len, pqCookieReplyLen)
header.Encode(reply, header.Version, header.Handshake, pqSubtype, 99, pqCookieCounter)
reply = append(reply, bytes16(0xab)...)
var h header.H
require.NoError(t, h.Parse(reply))
hm.handleFragment(via, reply, &h)
assert.True(t, hh.pq.heard, "a cookie reply shows the peer speaks PQ")
sent := conn.take()
require.Len(t, sent, 4, "an immediate resend of fresh symbols")
for i, p := range sent {
fh, _, err := fragment.Parse(p[header.Len:])
require.NoError(t, err)
assert.True(t, fh.HasCookie())
assert.Equal(t, fragment.Cookie(bytes16(0xab)), fh.Cookie)
assert.Equal(t, uint8(4+i), fh.Index)
}
hm.handleFragment(via, reply, &h)
assert.Empty(t, conn.take(), "immediate resends are rate limited")
t.Run("replies for other indexes are ignored", func(t *testing.T) {
other := append([]byte(nil), reply...)
header.Encode(other, header.Version, header.Handshake, pqSubtype, 98, pqCookieCounter)
var oh header.H
require.NoError(t, oh.Parse(other))
hh.pq.cookieResendAt = time.Time{}
hm.handleFragment(via, other, &oh)
assert.Empty(t, conn.take())
})
}
func bytes16(b byte) []byte {
out := make([]byte, 16)
for i := range out {
out[i] = b
}
return out
}
func TestReloadPQ(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString(`handshakes: {pq: {initiate: false}}`))
cfg := defaultHandshakeConfig
cfg.pq = pqHandshakeConfigFrom(c)
hm := NewHandshakeManager(l, newHostMap(l), newTestLighthouse(), &udp.NoopConn{}, cfg)
c.RegisterReloadCallback(hm.reloadPQ)
assert.Equal(t, pqHandshakeConfig{repairDelay: DefaultPQRepairDelay}, *hm.pqConfig())
require.NoError(t, c.ReloadConfigString(`handshakes: {pq: {initiate: true, accept: true, sticky: true, repair_delay: 25ms}}`))
assert.Equal(t, pqHandshakeConfig{initiate: true, accept: true, sticky: true, repairDelay: 25 * time.Millisecond}, *hm.pqConfig())
before := hm.pqConfig()
require.NoError(t, c.ReloadConfigString(`handshakes: {try_interval: 1s, pq: {initiate: true, accept: true, sticky: true, repair_delay: 25ms}}`))
assert.Same(t, before, hm.pqConfig(), "an unrelated change leaves the settings alone")
require.NoError(t, c.ReloadConfigString(`handshakes: {}`))
assert.Equal(t, pqHandshakeConfig{repairDelay: DefaultPQRepairDelay}, *hm.pqConfig(), "removing the section restores the defaults")
require.NoError(t, c.ReloadConfigString(`handshakes: {pq: {require: true}}`))
assert.Equal(t, pqHandshakeConfig{initiate: true, accept: true, require: true, repairDelay: DefaultPQRepairDelay}, *hm.pqConfig(), "require implies initiate and accept")
}
func TestUsePQ(t *testing.T) {
hm, _ := newPQTestManager(t)
peer := netip.MustParseAddr("10.1.0.1")
set := func(cfg pqHandshakeConfig) { hm.pq.Store(&cfg) }
// attempts plays a run of attempts, given whether each was triggered early,
// and returns which were PQ.
attempts := func(triggered ...bool) []bool {
hh := &HandshakeHostInfo{}
var out []bool
for _, tr := range triggered {
hh.counter++
hh.lastPQ = hm.usePQ(hh, peer, tr)
out = append(out, hh.lastPQ)
}
return out
}
set(pqHandshakeConfig{})
assert.Equal(t, []bool{false, false, false}, attempts(false, false, false), "off")
set(pqHandshakeConfig{initiate: true})
assert.Equal(t, []bool{true, false, true, false}, attempts(false, false, false, false), "timer retries alternate, PQ first")
assert.Equal(t, []bool{true, true, false, true, false}, attempts(false, true, false, true, false), "an early trigger is a new path, which gets PQ first")
assert.Equal(t, []bool{true, true, false}, attempts(true, true, false), "a triggered first attempt is PQ")
t.Run("hearing back stops the classical turns", func(t *testing.T) {
hh := &HandshakeHostInfo{pq: &pqAttempt{}}
hh.counter, hh.lastPQ = 1, true
hh.counter++
assert.False(t, hm.usePQ(hh, peer, false), "unheard, the classical attempt gets its turn")
hh.pq.heard = true
assert.True(t, hm.usePQ(hh, peer, false))
})
set(pqHandshakeConfig{initiate: true, accept: true, require: true})
assert.Equal(t, []bool{true, true, true, true}, attempts(false, false, true, false), "require never offers classical")
set(pqHandshakeConfig{initiate: true, sticky: true})
assert.Equal(t, []bool{true, false}, attempts(false, false), "sticky alternates until the peer has done PQ")
hm.notePQPeer(pqSubtype, []netip.Addr{peer})
assert.Equal(t, []bool{true, true, true}, attempts(false, false, false), "then only PQ")
}
+2 -14
View File
@@ -67,19 +67,8 @@ const (
const (
HandshakeIXPSK0 MessageSubType = 0
HandshakeXXPSK0 MessageSubType = 1
// HandshakeIXPSK2MLKEM1024 is IX with an ephemeral ML-KEM-1024 exchange mixed
// in through the psk2 slot. Its messages exceed the MTU and travel as
// fragments (see handshake/fragment).
HandshakeIXPSK2MLKEM1024 MessageSubType = 2
)
// IsFragmented reports whether handshake messages of this subtype travel as
// fragments.
func IsFragmented(st MessageSubType) bool {
return st == HandshakeIXPSK2MLKEM1024
}
var ErrHeaderTooShort = errors.New("header is too short")
var subTypeTestMap = map[MessageSubType]string{
@@ -99,8 +88,7 @@ var subTypeMap = map[MessageType]*map[MessageSubType]string{
Test: &subTypeTestMap,
CloseTunnel: &subTypeNoneMap,
Handshake: {
HandshakeIXPSK0: "ix_psk0",
HandshakeIXPSK2MLKEM1024: "ix_psk2_mlkem1024",
HandshakeIXPSK0: "ix_psk0",
},
Control: &subTypeNoneMap,
}
@@ -211,7 +199,7 @@ func IsValidSubType(t MessageType, s MessageSubType) bool {
case Message:
return s == MessageNone || s == MessageRelay
case Handshake:
return s == HandshakeIXPSK0 || s == HandshakeIXPSK2MLKEM1024
return s == HandshakeIXPSK0
case Test:
return s == TestReply || s == TestRequest
case Control, CloseTunnel, RecvError, LightHouse:
+1 -3
View File
@@ -96,8 +96,7 @@ func TestTypeMap(t *testing.T) {
Test: &subTypeTestMap,
CloseTunnel: &subTypeNoneMap,
Handshake: {
HandshakeIXPSK0: "ix_psk0",
HandshakeIXPSK2MLKEM1024: "ix_psk2_mlkem1024",
HandshakeIXPSK0: "ix_psk0",
},
Control: &subTypeNoneMap,
}, subTypeMap)
@@ -123,7 +122,6 @@ func TestIsValidSubType(t *testing.T) {
assert.False(t, IsValidSubType(Message, 2))
assert.True(t, IsValidSubType(Handshake, HandshakeIXPSK0))
assert.True(t, IsValidSubType(Handshake, HandshakeIXPSK2MLKEM1024))
// HandshakeXXPSK0 is defined but not a wire-valid subtype.
assert.False(t, IsValidSubType(Handshake, HandshakeXXPSK0))
+7 -10
View File
@@ -260,11 +260,6 @@ type HostInfo struct {
// We need these to avoid replayed handshake packets creating new hostinfos which causes churn
HandshakePacket map[uint8][]byte
// A responder resending a fragmented handshake response offers symbols it has
// not sent before, at most once per retry interval.
handshakeResendAt atomic.Int64 // unix nanos of the last send
handshakeNextSymbol atomic.Uint32 // next fragment symbol index to offer
// nextLHQuery is the earliest we can ask the lighthouse for new information.
// This is used to limit lighthouse re-queries in chatty clients
nextLHQuery atomic.Int64
@@ -321,9 +316,9 @@ type cachedPacketMetrics struct {
func NewHostMapFromConfig(l *slog.Logger, c *config.C) *HostMap {
hm := newHostMap(l)
hm.reload(c, true)
c.RegisterReloadCallback(func(c *config.C) {
hm.reload(c, false)
hm.reload(c, true, nil)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
hm.reload(c, false, r)
})
l.Info("Main HostMap created", "preferredRanges", hm.GetPreferredRanges())
@@ -342,7 +337,8 @@ func newHostMap(l *slog.Logger) *HostMap {
}
}
func (hm *HostMap) reload(c *config.C, initial bool) {
// r is nil on the initial load
func (hm *HostMap) reload(c *config.C, initial bool, r *config.Reload) {
if initial || c.HasChanged("preferred_ranges") {
var preferredRanges []netip.Prefix
rawPreferredRanges := c.GetStringSlice("preferred_ranges", []string{})
@@ -353,8 +349,9 @@ func (hm *HostMap) reload(c *config.C, initial bool) {
if err != nil {
hm.l.Warn("Failed to parse preferred ranges, ignoring",
"error", err,
"range", rawPreferredRanges,
"range", rawPreferredRange,
)
r.Failed()
continue
}
+13
View File
@@ -447,3 +447,16 @@ func TestHostInfo_markOutOnly(t *testing.T) {
assert.True(t, h.markOut(6), "a relayed send must not consume the epoch edge")
assert.False(t, h.markOut(6))
}
// A preferred range that doesn't parse is skipped and the rest are used, the reload says one was skipped
func TestHostMap_reloadBadPreferredRangeFails(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("preferred_ranges: [10.0.0.0/8]\n"))
hm := NewHostMapFromConfig(l, c)
res := c.ReloadConfigString("preferred_ranges: [10.0.0.0/8, nope]\n")
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Equal(t, []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")}, hm.GetPreferredRanges())
}
+7 -5
View File
@@ -509,10 +509,11 @@ func (f *Interface) flushSendBatch(sb *batch.SendBatch, q int) {
func (f *Interface) RegisterConfigChangeCallbacks(c *config.C) {
c.RegisterReloadCallback(f.reloadFirewall)
c.RegisterReloadCallback(f.reloadSendRecvError)
c.RegisterReloadCallback(f.reloadAcceptRecvError)
c.RegisterReloadCallback(f.reloadDisconnectInvalid)
c.RegisterReloadCallback(f.reloadMisc)
for _, reload := range []func(*config.C){
f.reloadSendRecvError, f.reloadAcceptRecvError, f.reloadDisconnectInvalid, f.reloadMisc,
} {
c.RegisterReloadCallback(func(c *config.C, _ *config.Reload) { reload(c) })
}
for _, udpConn := range f.writers {
c.RegisterReloadCallback(udpConn.ReloadConfig)
@@ -529,7 +530,7 @@ func (f *Interface) reloadDisconnectInvalid(c *config.C) {
}
}
func (f *Interface) reloadFirewall(c *config.C) {
func (f *Interface) reloadFirewall(c *config.C, r *config.Reload) {
cs := f.pki.getCertState()
curCert := cs.getCertificate(cert.Version2)
if curCert == nil {
@@ -548,6 +549,7 @@ func (f *Interface) reloadFirewall(c *config.C) {
fw, err := NewFirewallFromConfig(f.l, cs, c)
if err != nil {
f.l.Error("Error while creating firewall during reload", "error", err)
r.Failed()
return
}
+26 -4
View File
@@ -65,10 +65,10 @@ func TestReloadFirewall_CertUnsafeNetworksChanged(t *testing.T) {
pki.cs.Store(&CertState{v2Cert: c2, initiatingVersion: cert.Version2})
// Reload with the same YAML so HasChanged("firewall") reports false.
require.NoError(t, cfg.ReloadConfigString(rawYAML))
require.NoError(t, cfg.ReloadConfigString(rawYAML).Err)
require.False(t, cfg.HasChanged("firewall"))
f.reloadFirewall(cfg)
f.reloadFirewall(cfg, nil)
assert.NotSame(t, fw, f.firewall, "firewall pointer should have been replaced")
assert.Equal(t, newUnsafe, f.firewall.unsafeNetworks)
@@ -113,8 +113,30 @@ func TestReloadFirewall_NoChange(t *testing.T) {
l: l,
}
require.NoError(t, cfg.ReloadConfigString(rawYAML))
f.reloadFirewall(cfg)
require.NoError(t, cfg.ReloadConfigString(rawYAML).Err)
f.reloadFirewall(cfg, nil)
assert.Same(t, fw, f.firewall, "firewall should not have been replaced")
}
// Rules that don't parse keep the old firewall, and the reload says it could not apply them
func TestReloadFirewall_BadRulesFailTheReload(t *testing.T) {
l := test.NewLogger()
pki := &PKI{}
pki.cs.Store(&CertState{v2Cert: &dummyCert{version: cert.Version2, networks: []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")}}, initiatingVersion: cert.Version2})
rules := func(proto string) string {
return "firewall:\n outbound:\n - port: any\n proto: any\n host: any\n inbound:\n - port: any\n proto: " + proto + "\n host: any\n"
}
cfg := config.NewC(l)
require.NoError(t, cfg.LoadString(rules("any")))
fw, err := NewFirewallFromConfig(l, pki.getCertState(), cfg)
require.NoError(t, err)
f := &Interface{pki: pki, firewall: fw, l: l}
cfg.RegisterReloadCallback(f.reloadFirewall)
res := cfg.ReloadConfigString(rules("nope"))
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Same(t, fw, f.firewall)
}
+10 -4
View File
@@ -138,18 +138,21 @@ func NewLightHouseFromConfig(ctx context.Context, l *slog.Logger, c *config.C, c
h.metrics = newLighthouseMetrics()
}
err := h.reload(c, true)
err := h.reload(c, true, nil)
if err != nil {
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := h.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("lighthouse.am_lighthouse", "listen.port", "handshakes.query_buffer", "stats.lighthouse_metrics")
err := h.reload(c, false, r)
switch v := err.(type) {
case *util.ContextualError:
v.Log(l)
r.Failed()
case error:
l.Error("failed to reload lighthouse", "error", err)
r.Failed()
}
})
@@ -190,7 +193,8 @@ func (lh *LightHouse) GetUpdateInterval() int64 {
return lh.interval.Load()
}
func (lh *LightHouse) reload(c *config.C, initial bool) error {
// r is nil on the initial load
func (lh *LightHouse) reload(c *config.C, initial bool, r *config.Reload) error {
if initial || c.HasChanged("lighthouse.advertise_addrs") {
rawAdvAddrs := c.GetStringSlice("lighthouse.advertise_addrs", []string{})
advAddrs := make([]netip.AddrPort, 0)
@@ -225,6 +229,7 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
"addr", rawAddr,
"entry", i+1,
)
r.Failed()
continue
}
@@ -383,6 +388,7 @@ func (lh *LightHouse) reload(c *config.C, initial bool) error {
"relay", v,
"error", err,
)
r.Failed()
} else {
lh.l.Info("Read relay from config", "relay", v)
relaysForMe = append(relaysForMe, configRIP)
+59 -14
View File
@@ -87,15 +87,15 @@ func TestReloadLighthouseInterval(t *testing.T) {
lh.ifce = &mockEncWriter{}
// The first one routine is kicked off by main.go currently, lets make sure that one dies
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 5"))
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 5").Err)
assert.Equal(t, int64(5), lh.interval.Load())
// Subsequent calls are killed off by the LightHouse.Reload function
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 10"))
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 10").Err)
assert.Equal(t, int64(10), lh.interval.Load())
// If this completes then nothing is stealing our reload routine
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 11"))
require.NoError(t, c.ReloadConfigString("lighthouse:\n interval: 11").Err)
assert.Equal(t, int64(11), lh.interval.Load())
}
@@ -296,7 +296,7 @@ func TestLighthouse_reload(t *testing.T) {
require.NoError(t, err)
c.ReloadConfigString(string(rc))
err = lh.reload(c, false)
err = lh.reload(c, false, nil)
require.NoError(t, err)
}
@@ -334,7 +334,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err := yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl := lh.Query(staticHost)
require.NotNil(t, rl)
@@ -350,7 +350,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl = lh.Query(staticHost)
require.NotNil(t, rl)
@@ -358,7 +358,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
assert.Equal(t, []netip.AddrPort{netip.MustParseAddrPort("1.1.1.1:4242")}, rl.CopyAddrs([]netip.Prefix{}))
// Reload with the same config. An unchanged entry must not duplicate.
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl = lh.Query(staticHost)
require.NotNil(t, rl)
@@ -373,7 +373,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
// Add a second host alongside the first. Both should be present, neither duplicated.
nc = map[string]any{
@@ -384,7 +384,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
rl = lh.Query(staticHost)
require.NotNil(t, rl)
@@ -406,7 +406,7 @@ func TestLighthouse_reloadStaticHostMap(t *testing.T) {
}
rc, err = yaml.Marshal(nc)
require.NoError(t, err)
require.NoError(t, c.ReloadConfigString(string(rc)))
require.NoError(t, c.ReloadConfigString(string(rc)).Err)
_, isStatic := lh.GetStaticHostList()[staticHost]
assert.False(t, isStatic)
@@ -966,13 +966,13 @@ func TestLighthouse_reloadRelaysTriggersUpdate(t *testing.T) {
require.NoError(t, err)
assert.Empty(t, lh.updateTrigger)
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6]\n"))
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6]\n").Err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.6")}, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
<-lh.updateTrigger
// The config changed but what we advertise did not, a bad entry is skipped
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6, nope]\n"))
require.NoError(t, c.ReloadConfigString(base+"relay:\n relays: [10.128.0.6, nope]\n").Err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.6")}, lh.GetRelaysForMe())
assert.Empty(t, lh.updateTrigger)
}
@@ -988,12 +988,57 @@ func TestLighthouse_reloadAmRelayAlone(t *testing.T) {
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.5")}, lh.GetRelaysForMe())
// Relays aren't allowed to have relays, flipping am_relay alone must drop them and tell the lighthouses
require.NoError(t, c.ReloadConfigString(base+" am_relay: true\n"))
require.NoError(t, c.ReloadConfigString(base+" am_relay: true\n").Err)
assert.Empty(t, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
<-lh.updateTrigger
require.NoError(t, c.ReloadConfigString(base+" am_relay: false\n"))
require.NoError(t, c.ReloadConfigString(base+" am_relay: false\n").Err)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.5")}, lh.GetRelaysForMe())
require.Len(t, lh.updateTrigger, 1)
}
func TestLighthouse_reloadRestartRequired(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
base := "lighthouse:\n am_lighthouse: true\n"
require.NoError(t, c.LoadString(base+"listen:\n port: 4242\n"))
_, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
res := c.ReloadConfigString(base + "listen:\n port: 4242\n")
require.NoError(t, res.Err)
assert.False(t, res.RestartRequired)
res = c.ReloadConfigString(base + "listen:\n port: 4243\n")
require.NoError(t, res.Err)
assert.True(t, res.RestartRequired)
}
// A relay that doesn't parse is skipped and the rest are used, the reload says one was skipped
func TestLighthouse_reloadBadRelayFails(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("listen:\n port: 4242\n"))
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
res := c.ReloadConfigString("listen:\n port: 4242\nrelay:\n relays: [10.128.0.2, nope]\n")
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Equal(t, []netip.Addr{netip.MustParseAddr("10.128.0.2")}, lh.GetRelaysForMe())
}
// An advertise_addrs entry inside our own network is skipped and the rest are used, the reload says one was skipped
func TestLighthouse_reloadAdvertiseAddrInNetworkFails(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("listen:\n port: 4242\n"))
lh, err := NewLightHouseFromConfig(t.Context(), l, c, testCertState(netip.MustParsePrefix("10.128.0.1/24")), nil, nil)
require.NoError(t, err)
res := c.ReloadConfigString("listen:\n port: 4242\nlighthouse:\n advertise_addrs: [\"1.2.3.4:4242\", \"10.128.0.5:4242\"]\n")
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.Equal(t, []netip.AddrPort{netip.MustParseAddrPort("1.2.3.4:4242")}, lh.GetAdvertiseAddrs())
}
+20 -6
View File
@@ -12,6 +12,7 @@ import (
"strings"
"time"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/cpupick"
"github.com/slackhq/nebula/noiseutil"
@@ -71,14 +72,18 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
}
l.Info("Firewall started", "firewallHashes", fw.GetRuleHashes())
ssh, err := sshd.NewSSHServer(ctx, l.With("subsystem", "sshd"))
// The command registry is shared by every surface that exposes these commands, so a
// command cannot reach one of them without reaching the others.
cmdRegistry := commands.New()
ssh, err := sshd.NewSSHServer(ctx, l.With("subsystem", "sshd"), cmdRegistry)
if err != nil {
return nil, util.ContextualizeIfNeeded("Error while creating SSH server", err)
}
wireSSHReload(l, ssh, c)
var sshStart func()
if c.GetBool("sshd.enabled", false) {
sshStart, err = configSSH(l, ssh, c)
sshStart, err = configSSH(l, ssh, c, nil)
if err != nil {
l.Warn("Failed to configure sshd, ssh debugging will not be available", "error", err)
sshStart = nil
@@ -197,7 +202,7 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
if err != nil {
return nil, util.NewContextualError("Failed to open udp listener", m{"queue": i}, err)
}
udpServer.ReloadConfig(c)
udpServer.ReloadConfig(c, nil)
udpConns[i] = udpServer
// If port is dynamic, discover it before the next pass through the for loop
@@ -232,11 +237,9 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
retries: int64(c.GetInt("handshakes.retries", DefaultHandshakeRetries)),
triggerBuffer: c.GetInt("handshakes.trigger_buffer", DefaultHandshakeTriggerBuffer),
messageMetrics: messageMetrics,
pq: pqHandshakeConfigFrom(c),
}
handshakeManager := NewHandshakeManager(l, hostMap, lightHouse, udpConns[0], handshakeConfig)
c.RegisterReloadCallback(handshakeManager.reloadPQ)
lightHouse.handshakeTrigger = handshakeManager.trigger
ds, err := newDnsServerFromConfig(ctx, l, pki, hostMap, c)
@@ -313,6 +316,16 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
lightHouse.ifce = ifce
ifce.RegisterConfigChangeCallbacks(c)
// These are only read at startup
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged(
"routines", "tun.routines", "listen.routines", "firewall.conntrack.routine_cache_timeout",
"listen.port", "listen.host", "stats.message_metrics",
"handshakes.try_interval", "handshakes.retries", "handshakes.trigger_buffer",
"tun.drop_local_broadcast", "tun.drop_multicast",
)
r.RestartRequiredIfChanged(osStartupOnlyKeys...)
})
ifce.reloadDisconnectInvalid(c)
ifce.reloadSendRecvError(c)
ifce.reloadAcceptRecvError(c)
@@ -335,13 +348,14 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
return nil, nil
}
attachCommands(l, c, ssh, ifce)
attachCommands(l, c, cmdRegistry, ifce)
networkChanges := udp.NewNetworkChangeMonitor(ctx, l, c)
return &Control{
state: StateReady,
f: ifce,
commands: cmdRegistry,
l: l,
ctx: ctx,
cancel: cancel,
+6
View File
@@ -0,0 +1,6 @@
//go:build linux && !android && !e2e_testing
package nebula
// osStartupOnlyKeys are startup only keys that only matter on linux: udp batch size, udp offloads and thread pinning
var osStartupOnlyKeys = []string{"listen.batch", "listen.udp_offloads", "tun.pin_threads", "tun.pin_threads_key", "tun.cpu_affinity"}
+6
View File
@@ -0,0 +1,6 @@
//go:build !linux || android || e2e_testing
package nebula
// osStartupOnlyKeys is empty, none of the OS specific startup keys matter here
var osStartupOnlyKeys []string
+3 -1
View File
@@ -476,7 +476,9 @@ func parseV4(data []byte, incoming bool, fp *firewall.ParsedPacket) error {
func (f *Interface) handleOutsideMessagePacket(hostinfo *HostInfo, messageCounter uint64, out []byte, rxc *rxContext) {
err := newPacket(out, true, rxc.fwPacket)
if err != nil {
hostinfo.logger(f.l).Warn("Error while validating inbound packet", "error", err, "packet", out)
if f.l.Enabled(context.Background(), slog.LevelDebug) {
hostinfo.logger(f.l).Debug("Error while validating inbound packet", "error", err, "packet", out)
}
return
}
+7
View File
@@ -41,9 +41,16 @@ func (e *NameError) Error() string {
type DeviceFactory func(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, routines int) (Device, error)
func NewDeviceFromConfig(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, routines int) (Device, error) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.disabled")
})
switch {
case c.GetBool("tun.disabled", false):
tun := newDisabledTun(vpnNetworks, c.GetInt("tun.tx_queue", 5000), c.GetBool("stats.message_metrics", false), l)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.tx_queue", "stats.message_metrics")
})
return tun, nil
default:
+5 -4
View File
@@ -38,16 +38,17 @@ func newTunFromFd(c *config.C, l *slog.Logger, deviceFd int, vpnNetworks []netip
l: l,
}
err := t.reload(c, true)
err := t.reload(c, true, nil)
if err != nil {
_ = file.Close()
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
@@ -67,7 +68,7 @@ func (t *tun) Activate() error {
return nil
}
func (t *tun) reload(c *config.C, initial bool) error {
func (t *tun) reload(c *config.C, initial bool, r *config.Reload) error {
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
+45
View File
@@ -0,0 +1,45 @@
//go:build (openbsd || netbsd) && !e2e_testing
package overlay
import (
"errors"
"fmt"
"log/slog"
"os"
"regexp"
"runtime"
"golang.org/x/sys/unix"
)
var deviceNameRE = regexp.MustCompile(`^tun[0-9]+$`)
// tunDevNeedsRestart is true when tun.dev asks for a particular tun that isn't the running one. Anything else takes the
// first free one, and the one we have will do
func tunDevNeedsRestart(name, running string) bool {
return deviceNameRE.MatchString(name) && name != running
}
// openTun opens /dev/name, or the first /dev/tunN nothing else has open when name is empty or not tunN
func openTun(l *slog.Logger, name string) (int, string, error) {
if name != "" && !deviceNameRE.MatchString(name) {
// The name doesn't mean anything here, so like darwin a bad one isn't worth refusing to start over
l.Warn("interface name must be tun[0-9]+ on "+runtime.GOOS+", ignoring", "name", name)
name = ""
}
if name != "" {
fd, err := unix.Open("/dev/"+name, os.O_RDWR, 0)
return fd, name, err
}
for i := 0; ; i++ {
name = fmt.Sprintf("tun%d", i)
fd, err := unix.Open("/dev/"+name, os.O_RDWR, 0)
if err == nil {
return fd, name, nil
}
if !errors.Is(err, unix.EBUSY) {
return -1, "", fmt.Errorf("no free tun device, %s: %w", name, err)
}
}
}
+80
View File
@@ -0,0 +1,80 @@
//go:build (openbsd || netbsd) && !e2e_testing
package overlay
import (
"net/netip"
"testing"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// A tun.dev that isn't tunN is ignored like darwin does, and the first free tun is used. Only naming a different tun
// asks for a restart
func TestTunReloadDevRestart(t *testing.T) {
newWithDev := func(dev string) (*config.C, *tun, error) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString("tun:\n dev: "+dev+"\n"))
tn, err := newTun(c, test.NewLogger(), []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")}, false)
if err == nil {
t.Cleanup(func() { _ = tn.Close() })
}
return c, tn, err
}
if _, _, err := newWithDev(""); err != nil {
t.Skipf("can't create a tun device here: %v", err)
}
c, tn, err := newWithDev("nope")
require.NoError(t, err, "a bad name should fall back to a free tun")
require.Regexp(t, `^tun[0-9]+$`, tn.Device)
for _, dev := range []string{"nope", "", tn.Device, "nope"} {
res := c.ReloadConfigString("tun:\n dev: " + dev + "\n")
require.NoError(t, res.Err)
assert.False(t, res.Failed, dev)
assert.False(t, res.RestartRequired, dev)
}
other := "tun0"
if tn.Device == other {
other = "tun1"
}
res := c.ReloadConfigString("tun:\n dev: " + other + "\n")
require.NoError(t, res.Err)
assert.Equal(t, []string{"tun.dev"}, res.RestartKeys)
}
// An empty tun.dev takes the first tun nothing else has open. Reloading it empty, or naming the one we have, needs
// no restart
func TestTunEmptyDevPicksAFreeOne(t *testing.T) {
newEmpty := func() (*config.C, *tun, error) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString("tun: {}\n"))
tn, err := newTun(c, test.NewLogger(), []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")}, false)
if err == nil {
t.Cleanup(func() { _ = tn.Close() })
}
return c, tn, err
}
c, first, err := newEmpty()
if err != nil {
t.Skipf("can't create a tun device here: %v", err)
}
_, second, err := newEmpty()
require.NoError(t, err, "the second tun should skip the one the first has open")
assert.NotEqual(t, first.Device, second.Device)
for _, cfg := range []string{
"tun: {}\npunchy:\n punch: true\n",
"tun:\n dev: " + first.Device + "\n",
"tun: {}\n",
} {
res := c.ReloadConfigString(cfg)
require.NoError(t, res.Err)
assert.False(t, res.Failed)
assert.False(t, res.RestartRequired, cfg)
}
}
+3
View File
@@ -10,6 +10,9 @@ import (
"github.com/slackhq/nebula/wfp"
)
// interfaceBypassKeys are only read when the interface bypass is installed
var interfaceBypassKeys = []string{"tun.windows_bypass_wdf"}
// installInterfaceBypass installs a WFP PERMIT filter scoped to the wintun interface LUID so inbound traffic on the
// nebula adapter bypasses Windows Defender Firewall.
func installInterfaceBypass(l *slog.Logger, luid uint64) closer {
+3
View File
@@ -4,6 +4,9 @@ package overlay
import "log/slog"
// interfaceBypassKeys is empty, windows-386 has no bypass
var interfaceBypassKeys []string
// installInterfaceBypass is a no-op on windows-386 because we don't currently build for it.
func installInterfaceBypass(_ *slog.Logger, _ uint64) closer {
return nil
+149 -38
View File
@@ -6,8 +6,10 @@ import (
"errors"
"fmt"
"log/slog"
"math/bits"
"net/netip"
"os"
"sync"
"sync/atomic"
"syscall"
"unsafe"
@@ -28,7 +30,7 @@ type tun struct {
DefaultMTU int
Routes atomic.Pointer[[]Route]
routeTree atomic.Pointer[bart.Table[routing.Gateways]]
linkAddr *netroute.LinkAddr
linkAddr atomic.Pointer[netroute.LinkAddr]
// hostOwned means the fd arrived from the OS, which has already configured addressing, mtu
// and routes for it. NEPacketTunnelProvider on darwin does this.
hostOwned bool
@@ -78,17 +80,31 @@ type ifreqAlias6 struct {
Lifetime addrLifetime
}
// utunIndex is the utun tun.dev asks for, or -1 for whichever is free. ok is false for a name that isn't utunN, which
// also gets whichever is free
func utunIndex(name string) (index int, ok bool) {
if name == "" || name == "utun" {
return -1, true
}
if _, err := fmt.Sscanf(name, "utun%d", &index); err != nil || index < 0 {
return -1, false
}
return index, true
}
// tunDevNeedsRestart is true when tun.dev asks for a particular utun that isn't the running one. Anything else takes
// whichever is free, and the one we have will do
func tunDevNeedsRestart(name, running string) bool {
index, _ := utunIndex(name)
return index >= 0 && fmt.Sprintf("utun%d", index) != running
}
func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*tun, error) {
name := c.GetString("tun.dev", "")
ifIndex := -1
if name != "" && name != "utun" {
_, err := fmt.Sscanf(name, "utun%d", &ifIndex)
if err != nil || ifIndex < 0 {
// NOTE: we don't make this error so we don't break existing
// configs that set a name before it was used.
l.Warn("interface name must be utun[0-9]+ on Darwin, ignoring")
ifIndex = -1
}
ifIndex, ok := utunIndex(c.GetString("tun.dev", ""))
if !ok {
// NOTE: we don't make this error so we don't break existing
// configs that set a name before it was used.
l.Warn("interface name must be utun[0-9]+ on Darwin, ignoring")
}
fd, err := unix.Socket(unix.AF_SYSTEM, unix.SOCK_DGRAM, unix.AF_SYS_CONTROL)
@@ -112,7 +128,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
return nil, fmt.Errorf("SYS_CONNECT: %v", err)
}
name, err = unix.GetsockoptString(fd, unix.AF_SYS_CONTROL, _UTUN_OPT_IFNAME)
name, err := unix.GetsockoptString(fd, unix.AF_SYS_CONTROL, _UTUN_OPT_IFNAME)
if err != nil {
return nil, fmt.Errorf("failed to retrieve tun name: %w", err)
}
@@ -130,15 +146,20 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
l: l,
}
err = t.reload(c, true)
err = t.reload(c, true, nil)
if err != nil {
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.mtu")
if tunDevNeedsRestart(c.GetString("tun.dev", ""), t.Device) {
r.RestartRequiredIfChanged("tun.dev")
}
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
@@ -172,14 +193,15 @@ func newTunFromFd(c *config.C, l *slog.Logger, deviceFd int, vpnNetworks []netip
l: l,
}
if err := t.reload(c, true); err != nil {
if err := t.reload(c, true, nil); err != nil {
_ = file.Close()
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
if err := t.reload(c, false); err != nil {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
if err := t.reload(c, false, r); err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
@@ -243,7 +265,7 @@ func (t *tun) Activate() error {
if linkAddr == nil {
return fmt.Errorf("unable to discover link_addr for tun interface")
}
t.linkAddr = linkAddr
t.linkAddr.Store(linkAddr)
for _, network := range t.vpnNetworks {
if network.Addr().Is4() {
@@ -266,9 +288,10 @@ func (t *tun) Activate() error {
}
// Unsafe path routes
return t.addRoutes(false)
return t.addRoutes(false, nil)
}
// activate4 assigns our vpn address as a /32 and routes the vpn network to the tun explicitly.
func (t *tun) activate4(network netip.Prefix) error {
s, err := unix.Socket(
unix.AF_INET,
@@ -295,22 +318,32 @@ func (t *tun) activate4(network netip.Prefix) error {
MaskAddr: unix.RawSockaddrInet4{
Len: unix.SizeofSockaddrInet4,
Family: unix.AF_INET,
Addr: prefixToMask(network).As4(),
Addr: [4]byte{255, 255, 255, 255}, //always make the address a /32
},
}
// On Darwin, VPNs don't edit the routing table directly.
// A network extension hands its routes to the system, and configd installs them.
// configd replaces any route already holding the same prefix, including ours.
// As an example, Tailscale with an exit node treats any interface prefix wider than one address as a LAN.
// With LAN access off, it routes each LAN into Tailscale's utun.
// With LAN access on, it excludes each LAN from its tunnel.
// configd routes an excluded prefix like its own routes would, ignoring ours, so it goes out the default route.
// Either way our route is replaced, and traffic for the vpn network no longer reaches nebula.
// A single address isn't a LAN to Tailscale, so our route is left alone.
// We route the vpn network to the tun explicitly instead.
if err := ioctl(uintptr(s), unix.SIOCAIFADDR, uintptr(unsafe.Pointer(&ifr))); err != nil {
return fmt.Errorf("failed to set tun v4 address: %s", err)
}
err = addRoute(network, t.linkAddr)
if err != nil {
return err
if err := addRoute(network, t.linkAddr.Load()); err != nil {
return fmt.Errorf("failed to add route for %v: %w", network.Masked(), err)
}
return nil
}
// activate6 is activate4 for IPv6
func (t *tun) activate6(network netip.Prefix) error {
s, err := unix.Socket(
unix.AF_INET6,
@@ -332,24 +365,28 @@ func (t *tun) activate6(network netip.Prefix) error {
PrefixMask: unix.RawSockaddrInet6{
Len: unix.SizeofSockaddrInet6,
Family: unix.AF_INET6,
Addr: prefixToMask(network).As16(),
// Always set the netmask to be a /128. See activate4 for why we do this.
Addr: [16]byte{255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255},
},
Lifetime: addrLifetime{
// never expires
Vltime: 0xffffffff,
Pltime: 0xffffffff,
},
Flags: _IN6_IFF_NODAD,
Flags: _IN6_IFF_NODAD, // No DAD, a utun has no neighbors.
}
if err := ioctl(uintptr(s), _SIOCAIFADDR_IN6, uintptr(unsafe.Pointer(&ifr))); err != nil {
return fmt.Errorf("failed to set tun address: %s", err)
}
if err := addRoute(network, t.linkAddr.Load()); err != nil {
return fmt.Errorf("failed to add route for %v: %w", network.Masked(), err)
}
return nil
}
func (t *tun) reload(c *config.C, initial bool) error {
func (t *tun) reload(c *config.C, initial bool, r *config.Reload) error {
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
@@ -370,16 +407,18 @@ func (t *tun) reload(c *config.C, initial bool) error {
if !initial {
// Remove first, if the system removes a wanted route hopefully it will be re-added next
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes))
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes), r)
if err != nil {
util.LogWithContextIfNeeded("Failed to remove routes", err, t.l)
r.Failed()
}
// Ensure any routes we actually want are installed
err = t.addRoutes(true)
err = t.addRoutes(true, r)
if err != nil {
// Catch any stray logs
util.LogWithContextIfNeeded("Failed to add routes", err, t.l)
r.Failed()
}
}
@@ -394,6 +433,56 @@ func (t *tun) RoutesFor(ip netip.Addr) routing.Gateways {
return routing.Gateways{}
}
func fetchRoutes() ([]netroute.Message, error) {
rib, err := netroute.FetchRIB(unix.AF_UNSPEC, unix.NET_RT_DUMP, 0)
if err != nil {
return nil, err
}
return netroute.ParseRIB(unix.NET_RT_DUMP, rib)
}
// routeInterfaceIn returns the index of the interface prefix is routed through, or 0 if msgs has no route for exactly
// that prefix. Routes scoped to an interface are skipped, they don't stop us adding ours.
func routeInterfaceIn(msgs []netroute.Message, prefix netip.Prefix) int {
for _, msg := range msgs {
m, ok := msg.(*netroute.RouteMessage)
if !ok || len(m.Addrs) <= unix.RTAX_NETMASK || m.Flags&unix.RTF_IFSCOPE != 0 {
continue
}
if p, ok := routePrefix(m.Addrs[unix.RTAX_DST], m.Addrs[unix.RTAX_NETMASK]); ok && p == prefix.Masked() {
return m.Index
}
}
return 0
}
// routePrefix returns the prefix a route covers. Host routes have no netmask.
func routePrefix(dst, mask netroute.Addr) (netip.Prefix, bool) {
switch d := dst.(type) {
case *netroute.Inet4Addr:
bits := 32
if m, ok := mask.(*netroute.Inet4Addr); ok {
bits = maskBits(m.IP[:])
}
return netip.PrefixFrom(netip.AddrFrom4(d.IP), bits), true
case *netroute.Inet6Addr:
bits := 128
if m, ok := mask.(*netroute.Inet6Addr); ok {
bits = maskBits(m.IP[:])
}
return netip.PrefixFrom(netip.AddrFrom16(d.IP), bits), true
}
return netip.Prefix{}, false
}
func maskBits(mask []byte) int {
n := 0
for _, b := range mask {
n += bits.OnesCount8(b)
}
return n
}
// Get the LinkAddr for the interface of the given name
// Is there an easier way to fetch this when we create the interface?
// Maybe SIOCGIFINDEX? but this doesn't appear to exist in the darwin headers.
@@ -422,13 +511,20 @@ func getLinkAddr(name string) (*netroute.LinkAddr, error) {
return nil, nil
}
func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) addRoutes(logErrors bool, rl *config.Reload) error {
// The route tree is still ours, the system routing table is not
if t.hostOwned {
return nil
}
// Activate hasn't run yet, it will install the routes
linkAddr := t.linkAddr.Load()
if linkAddr == nil {
return nil
}
routes := *t.Routes.Load()
systemRoutes := sync.OnceValues(fetchRoutes)
for _, r := range routes {
if len(r.Via) == 0 || !r.Install {
@@ -436,39 +532,54 @@ func (t *tun) addRoutes(logErrors bool) error {
continue
}
err := addRoute(r.Cidr, t.linkAddr)
err := addRoute(r.Cidr, linkAddr)
if err != nil {
if errors.Is(err, unix.EEXIST) {
t.l.Warn("unable to add unsafe_route, identical route already exists", "route", r.Cidr)
// Every route we added already exists on a reload, only warn about someone else's
msgs, err := systemRoutes()
if err != nil {
t.l.Warn("unable to add unsafe_route, a route already exists but its interface could not be determined", "route", r.Cidr, "error", err)
} else if routeInterfaceIn(msgs, r.Cidr) != linkAddr.Index {
t.l.Warn("unable to add unsafe_route, the destination is already routed through another interface", "route", r.Cidr)
}
} else {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors {
retErr.Log(t.l)
rl.Failed()
} else {
return retErr
}
}
} else {
t.l.Info("Added route", "route", r)
// A cidr listed twice would otherwise find our own route missing from an older fetch
systemRoutes = sync.OnceValues(fetchRoutes)
}
}
return nil
}
func (t *tun) removeRoutes(routes []Route) error {
if t.hostOwned {
func (t *tun) removeRoutes(routes []Route, rl *config.Reload) error {
linkAddr := t.linkAddr.Load()
if t.hostOwned || linkAddr == nil {
return nil
}
for _, r := range routes {
if !r.Install {
if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue
}
err := delRoute(r.Cidr, t.linkAddr)
if err != nil {
err := delRoute(r.Cidr, linkAddr)
if errors.Is(err, unix.ESRCH) {
// Already gone, which is what we wanted
t.l.Info("Route was already removed", "route", r)
} else if err != nil {
t.l.Error("Failed to remove route", "error", err, "route", r)
rl.Failed()
} else {
t.l.Info("Removed route", "route", r)
}
+118
View File
@@ -0,0 +1,118 @@
//go:build darwin && !ios && !e2e_testing
package overlay
import (
"net"
"net/netip"
"os"
"os/exec"
"strings"
"testing"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
netroute "golang.org/x/net/route"
"golang.org/x/sys/unix"
)
// Every mac routes loopback through lo0
func TestRouteInterfaceIn(t *testing.T) {
msgs, err := fetchRoutes()
require.NoError(t, err)
routeInterface := func(s string) int { return routeInterfaceIn(msgs, netip.MustParsePrefix(s)) }
lo0, err := net.InterfaceByName("lo0")
require.NoError(t, err)
assert.Equal(t, lo0.Index, routeInterface("127.0.0.0/8"))
assert.Equal(t, lo0.Index, routeInterface("127.0.0.1/32"), "a host route has no netmask")
assert.Equal(t, lo0.Index, routeInterface("::1/128"))
assert.Equal(t, 0, routeInterface("127.0.0.0/16"), "only an exact prefix counts")
// Most macs also list the default route scoped to their other interfaces
out, err := exec.Command("route", "-n", "get", "default").Output()
require.NoError(t, err)
var name string
for _, line := range strings.Split(string(out), "\n") {
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "interface: "); ok {
name = v
}
}
if name == "" {
t.Skip("no default route")
}
def, err := net.InterfaceByName(name)
require.NoError(t, err)
assert.Equal(t, def.Index, routeInterface("0.0.0.0/0"))
}
func TestRouteInterfaceIn_SkipsScopedCopies(t *testing.T) {
route := func(index, flags int) netroute.Message {
addrs := make([]netroute.Addr, unix.RTAX_MAX)
addrs[unix.RTAX_DST] = &netroute.Inet4Addr{IP: [4]byte{192, 168, 1, 0}}
addrs[unix.RTAX_NETMASK] = &netroute.Inet4Addr{IP: [4]byte{255, 255, 255, 0}}
return &netroute.RouteMessage{Index: index, Flags: flags, Addrs: addrs}
}
p := netip.MustParsePrefix("192.168.1.0/24")
assert.Equal(t, 7, routeInterfaceIn([]netroute.Message{route(4, unix.RTF_UP|unix.RTF_IFSCOPE), route(7, unix.RTF_UP)}, p))
assert.Equal(t, 0, routeInterfaceIn([]netroute.Message{route(4, unix.RTF_UP|unix.RTF_IFSCOPE)}, p))
}
// A reload can happen before Activate
func TestTunRoutesBeforeActivate(t *testing.T) {
tn := &tun{l: test.NewLogger()}
routes := []Route{{
Cidr: netip.MustParsePrefix("10.253.1.0/24"),
Via: routing.Gateways{routing.NewGateway(netip.MustParseAddr("10.250.0.2"), 1)},
Install: true,
}}
tn.Routes.Store(&routes)
assert.NoError(t, tn.addRoutes(true, nil))
assert.NoError(t, tn.removeRoutes(routes, nil))
}
// A tun.routes entry with no via is never installed, so removing it must not delete a route for the same cidr that
// belongs to someone else
func TestTunReloadLeavesRoutesItNeverInstalled(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("creating a utun and adding routes needs root")
}
lo0, err := net.InterfaceByName("lo0")
require.NoError(t, err)
cidr := "10.250.0.128/25"
require.NoError(t, exec.Command("route", "-q", "-n", "add", "-net", cidr, "-interface", "lo0").Run())
t.Cleanup(func() { _ = exec.Command("route", "-q", "-n", "delete", "-net", cidr).Run() })
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("tun:\n routes:\n - route: "+cidr+"\n mtu: 1300\n"))
tn, err := newTun(c, l, []netip.Prefix{netip.MustParsePrefix("10.250.0.1/24")}, false)
require.NoError(t, err)
t.Cleanup(func() { _ = tn.Close() })
require.NoError(t, tn.Activate())
res := c.ReloadConfigString("tun: {}\n")
require.NoError(t, res.Err)
assert.False(t, res.Failed, "removing a route we never installed failed the reload")
msgs, err := fetchRoutes()
require.NoError(t, err)
assert.Equal(t, lo0.Index, routeInterfaceIn(msgs, netip.MustParsePrefix(cidr)))
}
// Only naming a different utun asks for a restart. Empty, plain utun and names that aren't utunN take whichever is free
func TestTunDevNeedsRestart(t *testing.T) {
for name, want := range map[string]bool{
"": false,
"utun": false,
"nope": false,
"utun7": false,
"utun3": true,
} {
assert.Equal(t, want, tunDevNeedsRestart(name, "utun7"), name)
}
}
+121
View File
@@ -0,0 +1,121 @@
//go:build !ios && !e2e_testing
package overlay
import (
"bytes"
"os"
"syscall"
"testing"
"github.com/stretchr/testify/require"
)
func TestTun_WriteFraming(t *testing.T) {
r, w, err := os.Pipe()
require.NoError(t, err)
defer r.Close()
defer w.Close()
tn := &tun{f: w}
for _, tc := range []struct {
name string
pkt []byte
af byte
}{
// Literal families rather than syscall.AF_*, so a wrong constant in Write can't also fix the expectation.
{"ipv4", []byte{0x45, 1, 2, 3, 4, 5}, 2},
{"ipv6", []byte{0x60, 1, 2, 3, 4, 5}, 30},
} {
t.Run(tc.name, func(t *testing.T) {
n, err := tn.Write(tc.pkt)
require.NoError(t, err)
require.Equal(t, len(tc.pkt), n)
got := make([]byte, 64)
n, err = r.Read(got)
require.NoError(t, err)
require.Equal(t, append([]byte{0, 0, 0, tc.af}, tc.pkt...), got[:n])
})
}
_, err = tn.Write([]byte{0x10, 1, 2, 3})
require.Error(t, err)
_, err = tn.Write([]byte{})
require.ErrorIs(t, err, syscall.EIO)
// The rejected writes must not have put anything on the fd, so the next read sees only this packet.
pkt := []byte{0x45, 9, 9}
_, err = tn.Write(pkt)
require.NoError(t, err)
got := make([]byte, 64)
n, err := r.Read(got)
require.NoError(t, err)
require.Equal(t, append([]byte{0, 0, 0, 2}, pkt...), got[:n])
}
func TestTun_ReadStripsHeader(t *testing.T) {
r, w, err := os.Pipe()
require.NoError(t, err)
defer r.Close()
defer w.Close()
tn := &tun{f: r}
pkt := []byte{0x45, 1, 2, 3, 4, 5}
_, err = w.Write(append([]byte{0, 0, 0, syscall.AF_INET}, pkt...))
require.NoError(t, err)
got := make([]byte, 64)
n, err := tn.Read(got)
require.NoError(t, err)
require.Equal(t, pkt, got[:n])
}
func TestTun_ReadShort(t *testing.T) {
r, w, err := os.Pipe()
require.NoError(t, err)
defer r.Close()
defer w.Close()
tn := &tun{f: r}
// Fewer bytes than the protocol header is not a packet.
_, err = w.Write([]byte{0, 0})
require.NoError(t, err)
n, err := tn.Read(make([]byte, 64))
require.NoError(t, err)
require.Zero(t, n)
// The next packet reads cleanly only if the short read consumed exactly what was written.
pkt := []byte{0x45, 1, 2, 3}
_, err = w.Write(append([]byte{0, 0, 0, syscall.AF_INET}, pkt...))
require.NoError(t, err)
got := make([]byte, 64)
n, err = tn.Read(got)
require.NoError(t, err)
require.Equal(t, pkt, got[:n])
}
// A heap allocation per packet rarely shows in a short throughput run, only across GC pauses, so
// this pins Read and Write to zero allocations directly.
func TestTun_ReadWriteDoNotAllocate(t *testing.T) {
r, w, err := os.Pipe()
require.NoError(t, err)
defer r.Close()
defer w.Close()
tw := &tun{f: w}
tr := &tun{f: r}
pkts := [][]byte{bytes.Repeat([]byte{0x45}, 1300), bytes.Repeat([]byte{0x60}, 1300)}
buf := make([]byte, 1500)
i := 0
allocs := testing.AllocsPerRun(1000, func() {
i++
if _, err := tw.Write(pkts[i%len(pkts)]); err != nil {
t.Fatal(err)
}
if _, err := tr.Read(buf); err != nil {
t.Fatal(err)
}
})
require.Zero(t, allocs)
}
+21 -11
View File
@@ -385,16 +385,18 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
},
}
err = t.reload(c, true)
err = t.reload(c, true, nil)
if err != nil {
return nil, err
}
closeOnErr = false
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.dev", "tun.mtu")
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
@@ -493,7 +495,7 @@ func (t *tun) Activate() error {
}
}
return t.addRoutes(false)
return t.addRoutes(false, nil)
}
func (t *tun) setMTU() error {
@@ -509,7 +511,7 @@ func (t *tun) setMTU() error {
return err
}
func (t *tun) reload(c *config.C, initial bool) error {
func (t *tun) reload(c *config.C, initial bool, r *config.Reload) error {
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
@@ -530,16 +532,18 @@ func (t *tun) reload(c *config.C, initial bool) error {
if !initial {
// Remove first, if the system removes a wanted route hopefully it will be re-added next
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes))
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes), r)
if err != nil {
util.LogWithContextIfNeeded("Failed to remove routes", err, t.l)
r.Failed()
}
// Ensure any routes we actually want are installed
err = t.addRoutes(true)
err = t.addRoutes(true, r)
if err != nil {
// Catch any stray logs
util.LogWithContextIfNeeded("Failed to add routes", err, t.l)
r.Failed()
}
}
@@ -563,7 +567,7 @@ func (t *tun) Queues(int) ([]tio.Queue, error) {
return []tio.Queue{tio.NewSingleQueue(t, defaultBatchBufSize)}, nil
}
func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) addRoutes(logErrors bool, rl *config.Reload) error {
routes := *t.Routes.Load()
for _, r := range routes {
if len(r.Via) == 0 || !r.Install {
@@ -576,6 +580,7 @@ func (t *tun) addRoutes(logErrors bool) error {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors {
retErr.Log(t.l)
rl.Failed()
} else {
return retErr
}
@@ -587,15 +592,20 @@ func (t *tun) addRoutes(logErrors bool) error {
return nil
}
func (t *tun) removeRoutes(routes []Route) error {
func (t *tun) removeRoutes(routes []Route, rl *config.Reload) error {
for _, r := range routes {
if !r.Install {
if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue
}
err := delRoute(r.Cidr, t.linkAddr)
if err != nil {
if errors.Is(err, unix.ESRCH) {
// Already gone, which is what we wanted
t.l.Info("Route was already removed", "route", r)
} else if err != nil {
t.l.Error("Failed to remove route", "error", err, "route", r)
rl.Failed()
} else {
t.l.Info("Removed route", "route", r)
}
+5 -4
View File
@@ -47,16 +47,17 @@ func newTunFromFd(c *config.C, l *slog.Logger, deviceFd int, vpnNetworks []netip
l: l,
}
err := t.reload(c, true)
err := t.reload(c, true, nil)
if err != nil {
_ = file.Close()
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
@@ -67,7 +68,7 @@ func (t *tun) Activate() error {
return nil
}
func (t *tun) reload(c *config.C, initial bool) error {
func (t *tun) reload(c *config.C, initial bool, r *config.Reload) error {
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
+28 -14
View File
@@ -185,6 +185,10 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, multiqueue
}
t.Device = name
// Only read when we create the device, an fd handed to us already has its name and offloads
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.dev", "tun.use_offloads")
})
return t, nil
}
@@ -228,22 +232,24 @@ func newTunGeneric(c *config.C, l *slog.Logger, fd int, vnetHdr bool, offloadFla
l: l,
}
if err = t.reload(c, true); err != nil {
if err = t.reload(c, true, nil); err != nil {
_ = t.Close()
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.tx_queue", "tun.use_system_route_table", "tun.use_system_route_table_buffer_size")
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
return t, nil
}
func (t *tun) reload(c *config.C, initial bool) error {
func (t *tun) reload(c *config.C, initial bool, r *config.Reload) error {
routeChange, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
@@ -297,7 +303,7 @@ func (t *tun) reload(c *config.C, initial bool) error {
if !initial {
if oldMaxMTU != newMaxMTU {
t.setMTU()
t.setMTU(r)
t.l.Info("Set max MTU", "mtu", t.MaxMTU, "oldMTU", oldMaxMTU)
}
@@ -305,7 +311,8 @@ func (t *tun) reload(c *config.C, initial bool) error {
for i := range t.vpnNetworks {
err := t.setDefaultRoute(t.vpnNetworks[i])
if err != nil {
t.l.Warn(err.Error())
t.l.Error(err.Error())
r.Failed()
} else {
t.l.Info("Set default MTU", "mtu", t.DefaultMTU, "oldMTU", oldDefaultMTU)
}
@@ -313,13 +320,14 @@ func (t *tun) reload(c *config.C, initial bool) error {
}
// Remove first, if the system removes a wanted route hopefully it will be re-added next
t.removeRoutes(findRemovedRoutes(routes, *oldRoutes))
t.removeRoutes(findRemovedRoutes(routes, *oldRoutes), r)
// Ensure any routes we actually want are installed
err = t.addRoutes(true)
err = t.addRoutes(true, r)
if err != nil {
// This should never be called since addRoutes should log its own errors in a reload condition
util.LogWithContextIfNeeded("Failed to refresh routes", err, t.l)
r.Failed()
}
}
@@ -474,7 +482,7 @@ func (t *tun) Activate() error {
t.deviceIndex = link.Attrs().Index
// Setup our default MTU
t.setMTU()
t.setMTU(nil)
// Set the transmit queue length
ifrq := ifreqQLEN{Name: devName, Value: int32(t.TXQueueLen)}
@@ -506,7 +514,7 @@ func (t *tun) Activate() error {
}
// Set the routes
if err = t.addRoutes(false); err != nil {
if err = t.addRoutes(false, nil); err != nil {
return err
}
@@ -519,12 +527,13 @@ func (t *tun) Activate() error {
return nil
}
func (t *tun) setMTU() {
func (t *tun) setMTU(r *config.Reload) {
// Set the MTU on the device
ifm := ifreqMTU{Name: t.deviceBytes(), MTU: int32(t.MaxMTU)}
if err := ioctl(t.ioctlFd, unix.SIOCSIFMTU, uintptr(unsafe.Pointer(&ifm))); err != nil {
// This is currently a non fatal condition because the route table must have the MTU set appropriately as well
t.l.Error("Failed to set tun mtu", "error", err)
r.Failed()
}
}
@@ -577,7 +586,7 @@ func (t *tun) setDefaultRoute(cidr netip.Prefix) error {
return nil
}
func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) addRoutes(logErrors bool, rl *config.Reload) error {
// Path routes
routes := *t.Routes.Load()
for _, r := range routes {
@@ -607,6 +616,7 @@ func (t *tun) addRoutes(logErrors bool) error {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors {
retErr.Log(t.l)
rl.Failed()
} else {
return retErr
}
@@ -618,7 +628,7 @@ func (t *tun) addRoutes(logErrors bool) error {
return nil
}
func (t *tun) removeRoutes(routes []Route) {
func (t *tun) removeRoutes(routes []Route, rl *config.Reload) {
for _, r := range routes {
if !r.Install {
continue
@@ -642,8 +652,12 @@ func (t *tun) removeRoutes(routes []Route) {
}
err := netlink.RouteDel(&nr)
if err != nil {
if errors.Is(err, unix.ESRCH) {
// Already gone, which is what we wanted
t.l.Info("Route was already removed", "route", r)
} else if err != nil {
t.l.Error("Failed to remove route", "error", err, "route", r)
rl.Failed()
} else {
t.l.Info("Removed route", "route", r)
}
+25 -25
View File
@@ -8,7 +8,6 @@ import (
"log/slog"
"net/netip"
"os"
"regexp"
"sync/atomic"
"syscall"
"unsafe"
@@ -67,24 +66,12 @@ type tun struct {
fd int
}
var deviceNameRE = regexp.MustCompile(`^tun[0-9]+$`)
func newTunFromFd(_ *config.C, _ *slog.Logger, _ int, _ []netip.Prefix) (*tun, error) {
return nil, fmt.Errorf("newTunFromFd not supported in NetBSD")
}
func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*tun, error) {
// Try to open tun device
var err error
deviceName := c.GetString("tun.dev", "")
if deviceName == "" {
return nil, fmt.Errorf("a device name in the format of /dev/tunN must be specified")
}
if !deviceNameRE.MatchString(deviceName) {
return nil, fmt.Errorf("a device name in the format of /dev/tunN must be specified")
}
fd, err := unix.Open("/dev/"+deviceName, os.O_RDWR, 0)
fd, deviceName, err := openTun(l, c.GetString("tun.dev", ""))
if err != nil {
return nil, err
}
@@ -103,15 +90,20 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
l: l,
}
err = t.reload(c, true)
err = t.reload(c, true, nil)
if err != nil {
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.mtu")
if tunDevNeedsRestart(c.GetString("tun.dev", ""), t.Device) {
r.RestartRequiredIfChanged("tun.dev")
}
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
@@ -324,7 +316,7 @@ func (t *tun) Activate() error {
}
}
return t.addRoutes(false)
return t.addRoutes(false, nil)
}
func (t *tun) doIoctlByName(ctl uintptr, value uint32) error {
@@ -339,7 +331,7 @@ func (t *tun) doIoctlByName(ctl uintptr, value uint32) error {
return err
}
func (t *tun) reload(c *config.C, initial bool) error {
func (t *tun) reload(c *config.C, initial bool, r *config.Reload) error {
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
@@ -360,16 +352,18 @@ func (t *tun) reload(c *config.C, initial bool) error {
if !initial {
// Remove first, if the system removes a wanted route hopefully it will be re-added next
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes))
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes), r)
if err != nil {
util.LogWithContextIfNeeded("Failed to remove routes", err, t.l)
r.Failed()
}
// Ensure any routes we actually want are installed
err = t.addRoutes(true)
err = t.addRoutes(true, r)
if err != nil {
// Catch any stray logs
util.LogWithContextIfNeeded("Failed to add routes", err, t.l)
r.Failed()
}
}
@@ -393,7 +387,7 @@ func (t *tun) Queues(int) ([]tio.Queue, error) {
return []tio.Queue{tio.NewSingleQueue(t, defaultBatchBufSize)}, nil
}
func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) addRoutes(logErrors bool, rl *config.Reload) error {
routes := *t.Routes.Load()
for _, r := range routes {
@@ -407,6 +401,7 @@ func (t *tun) addRoutes(logErrors bool) error {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors {
retErr.Log(t.l)
rl.Failed()
} else {
return retErr
}
@@ -418,15 +413,20 @@ func (t *tun) addRoutes(logErrors bool) error {
return nil
}
func (t *tun) removeRoutes(routes []Route) error {
func (t *tun) removeRoutes(routes []Route, rl *config.Reload) error {
for _, r := range routes {
if !r.Install {
if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue
}
err := delRoute(r.Cidr, t.vpnNetworks)
if err != nil {
if errors.Is(err, unix.ESRCH) {
// Already gone, which is what we wanted
t.l.Info("Route was already removed", "route", r)
} else if err != nil {
t.l.Error("Failed to remove route", "error", err, "route", r)
rl.Failed()
} else {
t.l.Info("Removed route", "route", r)
}
+25 -25
View File
@@ -8,7 +8,6 @@ import (
"log/slog"
"net/netip"
"os"
"regexp"
"sync/atomic"
"syscall"
"unsafe"
@@ -58,24 +57,12 @@ type tun struct {
fd int
}
var deviceNameRE = regexp.MustCompile(`^tun[0-9]+$`)
func newTunFromFd(_ *config.C, _ *slog.Logger, _ int, _ []netip.Prefix) (*tun, error) {
return nil, fmt.Errorf("newTunFromFd not supported in openbsd")
}
func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*tun, error) {
// Try to open tun device
var err error
deviceName := c.GetString("tun.dev", "")
if deviceName == "" {
return nil, fmt.Errorf("a device name in the format of /dev/tunN must be specified")
}
if !deviceNameRE.MatchString(deviceName) {
return nil, fmt.Errorf("a device name in the format of /dev/tunN must be specified")
}
fd, err := unix.Open("/dev/"+deviceName, os.O_RDWR, 0)
fd, deviceName, err := openTun(l, c.GetString("tun.dev", ""))
if err != nil {
return nil, err
}
@@ -94,15 +81,20 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*t
l: l,
}
err = t.reload(c, true)
err = t.reload(c, true, nil)
if err != nil {
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.mtu")
if tunDevNeedsRestart(c.GetString("tun.dev", ""), t.Device) {
r.RestartRequiredIfChanged("tun.dev")
}
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
@@ -303,7 +295,7 @@ func (t *tun) Activate() error {
}
}
return t.addRoutes(false)
return t.addRoutes(false, nil)
}
func (t *tun) doIoctlByName(ctl uintptr, value uint32) error {
@@ -318,7 +310,7 @@ func (t *tun) doIoctlByName(ctl uintptr, value uint32) error {
return err
}
func (t *tun) reload(c *config.C, initial bool) error {
func (t *tun) reload(c *config.C, initial bool, r *config.Reload) error {
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
@@ -339,16 +331,18 @@ func (t *tun) reload(c *config.C, initial bool) error {
if !initial {
// Remove first, if the system removes a wanted route hopefully it will be re-added next
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes))
err := t.removeRoutes(findRemovedRoutes(routes, *oldRoutes), r)
if err != nil {
util.LogWithContextIfNeeded("Failed to remove routes", err, t.l)
r.Failed()
}
// Ensure any routes we actually want are installed
err = t.addRoutes(true)
err = t.addRoutes(true, r)
if err != nil {
// Catch any stray logs
util.LogWithContextIfNeeded("Failed to add routes", err, t.l)
r.Failed()
}
}
@@ -372,7 +366,7 @@ func (t *tun) Queues(int) ([]tio.Queue, error) {
return []tio.Queue{tio.NewSingleQueue(t, defaultBatchBufSize)}, nil
}
func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) addRoutes(logErrors bool, rl *config.Reload) error {
routes := *t.Routes.Load()
for _, r := range routes {
@@ -386,6 +380,7 @@ func (t *tun) addRoutes(logErrors bool) error {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors {
retErr.Log(t.l)
rl.Failed()
} else {
return retErr
}
@@ -397,15 +392,20 @@ func (t *tun) addRoutes(logErrors bool) error {
return nil
}
func (t *tun) removeRoutes(routes []Route) error {
func (t *tun) removeRoutes(routes []Route, rl *config.Reload) error {
for _, r := range routes {
if !r.Install {
if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue
}
err := delRoute(r.Cidr, t.vpnNetworks)
if err != nil {
if errors.Is(err, unix.ESRCH) {
// Already gone, which is what we wanted
t.l.Info("Route was already removed", "route", r)
} else if err != nil {
t.l.Error("Failed to remove route", "error", err, "route", r)
rl.Failed()
} else {
t.l.Info("Removed route", "route", r)
}
+55 -4
View File
@@ -4,6 +4,7 @@ package overlay
import (
"net/netip"
"os"
"testing"
"github.com/slackhq/nebula/config"
@@ -25,10 +26,10 @@ func TestReloadMTUOnlyKeepsRoutes(t *testing.T) {
ioctlFd: ^uintptr(0),
routesFromSystem: map[netip.Prefix]routing.Gateways{},
}
require.NoError(t, tn.reload(c, true))
require.NoError(t, tn.reload(c, true, nil))
require.NoError(t, c.ReloadConfigString("tun:\n mtu: 1400\n"+routes))
_ = tn.reload(c, false)
require.NoError(t, c.ReloadConfigString("tun:\n mtu: 1400\n"+routes).Err)
_ = tn.reload(c, false, nil)
got := *tn.Routes.Load()
require.Len(t, got, 1)
@@ -50,7 +51,57 @@ func TestReloadMaxMTUIsTheLargestRoute(t *testing.T) {
ioctlFd: ^uintptr(0),
routesFromSystem: map[netip.Prefix]routing.Gateways{},
}
require.NoError(t, tn.reload(c, true))
require.NoError(t, tn.reload(c, true, nil))
assert.Equal(t, 1500, tn.MaxMTU)
assert.Equal(t, 1300, tn.DefaultMTU)
}
// There is no device behind this tun, so adding a route always fails. Removing one is refused without root, as root the
// kernel answers that the route isn't there, which is what we wanted
func TestReloadRouteFailures(t *testing.T) {
route := "tun:\n mtu: 1300\n unsafe_routes:\n - route: 192.168.100.0/24\n via: 10.1.0.1\n"
cases := []struct {
name string
from, to string
wantFails bool
}{
{"added", "tun:\n mtu: 1300\n", route, true},
{"removed", route, "tun:\n mtu: 1300\n", os.Geteuid() != 0},
{"unchanged", route, route + "punchy:\n punch: true\n", false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString(tc.from))
tn := &tun{
l: test.NewLogger(),
vpnNetworks: []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")},
ioctlFd: ^uintptr(0),
routesFromSystem: map[netip.Prefix]routing.Gateways{},
}
require.NoError(t, tn.reload(c, true, nil))
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
require.NoError(t, tn.reload(c, false, r))
})
res := c.ReloadConfigString(tc.to)
require.NoError(t, res.Err)
assert.Equal(t, tc.wantFails, res.Failed)
})
}
}
// Keys only read when the device is created need a restart to apply
func TestTunReloadRestartKeys(t *testing.T) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString("tun:\n dev: nebrk0\n"))
tn, err := NewDeviceFromConfig(c, test.NewLogger(), []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")}, 1)
if err != nil {
t.Skipf("can't create a tun device here: %v", err)
}
t.Cleanup(func() { _ = tn.Close() })
res := c.ReloadConfigString("tun:\n disabled: true\n dev: nebrk1\n tx_queue: 100\n")
require.NoError(t, res.Err)
assert.ElementsMatch(t, []string{"tun.disabled", "tun.dev", "tun.tx_queue"}, res.RestartKeys)
}
+24
View File
@@ -0,0 +1,24 @@
package overlay
import (
"net/netip"
"testing"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// tun.disabled and the disabled tun's settings are only read when the device is made
func TestDisabledTunRestartKeys(t *testing.T) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString("tun:\n disabled: true\n"))
d, err := NewDeviceFromConfig(c, test.NewLogger(), []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")}, 1)
require.NoError(t, err)
t.Cleanup(func() { _ = d.Close() })
res := c.ReloadConfigString("tun:\n disabled: false\n tx_queue: 100\n")
require.NoError(t, res.Err)
assert.ElementsMatch(t, []string{"tun.disabled", "tun.tx_queue"}, res.RestartKeys)
}
+28 -10
View File
@@ -84,7 +84,7 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*w
l: l,
}
err = t.reload(c, true)
err = t.reload(c, true, nil)
if err != nil {
return nil, err
}
@@ -108,17 +108,28 @@ func newTun(c *config.C, l *slog.Logger, vpnNetworks []netip.Prefix, _ bool) (*w
// Disable registration before Activate assigns addresses and triggers it.
disableDNSRegistration(l, *guid)
c.RegisterReloadCallback(func(c *config.C) {
err := t.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("tun.dev", "tun.mtu")
// Compare with the running category, not the last config. A restart with one we can't parse would fail, so say
// that now rather than ask for one, and going back to the running one after a bad reload needs no restart
if cat, setCat, err := parseNetworkCategory(c.GetString("tun.network_category", "private")); err != nil {
t.l.Error("Invalid tun.network_category, a restart with it would fail", "error", err)
r.Failed()
} else if cat != t.networkCategory || setCat != t.setCategory {
r.RestartRequiredIfChanged("tun.network_category")
}
r.RestartRequiredIfChanged(interfaceBypassKeys...)
err := t.reload(c, false, r)
if err != nil {
util.LogWithContextIfNeeded("failed to reload tun device", err, t.l)
r.Failed()
}
})
return t, nil
}
func (t *winTun) reload(c *config.C, initial bool) error {
func (t *winTun) reload(c *config.C, initial bool, r *config.Reload) error {
change, routes, err := getAllRoutesFromConfig(t.l, c, t.vpnNetworks, initial)
if err != nil {
return err
@@ -139,16 +150,18 @@ func (t *winTun) reload(c *config.C, initial bool) error {
if !initial {
// Remove first, if the system removes a wanted route hopefully it will be re-added next
err := t.removeRoutes(findRemovedSystemRoutes(routes, *oldRoutes))
err := t.removeRoutes(findRemovedSystemRoutes(routes, *oldRoutes), r)
if err != nil {
util.LogWithContextIfNeeded("Failed to remove routes", err, t.l)
r.Failed()
}
// Ensure any routes we actually want are installed
err = t.addRoutes(true)
err = t.addRoutes(true, r)
if err != nil {
// Catch any stray logs
util.LogWithContextIfNeeded("Failed to add routes", err, t.l)
r.Failed()
}
}
@@ -163,7 +176,7 @@ func (t *winTun) Activate() error {
return fmt.Errorf("failed to set address: %w", err)
}
err = t.addRoutes(false)
err = t.addRoutes(false, nil)
if err != nil {
return err
}
@@ -182,7 +195,7 @@ func (t *winTun) Activate() error {
return nil
}
func (t *winTun) addRoutes(logErrors bool) error {
func (t *winTun) addRoutes(logErrors bool, rl *config.Reload) error {
luid := winipcfg.LUID(t.tun.LUID())
routes := *t.Routes.Load()
foundDefault4 := false
@@ -213,6 +226,7 @@ func (t *winTun) addRoutes(logErrors bool) error {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors {
retErr.Log(t.l)
rl.Failed()
continue
} else {
return retErr
@@ -274,7 +288,7 @@ func (t *winTun) setMTU(luid winipcfg.LUID, foundDefault4, carriesV6 bool) error
return nil
}
func (t *winTun) removeRoutes(routes []Route) error {
func (t *winTun) removeRoutes(routes []Route, rl *config.Reload) error {
luid := winipcfg.LUID(t.tun.LUID())
for _, r := range routes {
@@ -284,8 +298,12 @@ func (t *winTun) removeRoutes(routes []Route) error {
// See comment on luid.AddRoute
err := luid.DeleteRoute(r.Cidr, unspecifiedNextHop(r.Cidr))
if err != nil {
if errors.Is(err, windows.ERROR_NOT_FOUND) {
// Already gone, which is what we wanted
t.l.Info("Route was already removed", "route", r)
} else if err != nil {
t.l.Error("Failed to remove route", "error", err, "route", r)
rl.Failed()
} else {
t.l.Info("Removed route", "route", r)
}
+46 -5
View File
@@ -53,6 +53,14 @@ func (rt *routeTest) metric(t *testing.T, cidr string) uint32 {
return row.Metric
}
// reload applies cfg and expects nothing in it to fail
func (rt *routeTest) reload(t *testing.T, cfg string) {
t.Helper()
res := rt.c.ReloadConfigString(cfg)
require.NoError(t, res.Err)
assert.False(t, res.Failed, "the reload reported a failure")
}
func unsafeRoutes(routes ...string) string {
var b strings.Builder
b.WriteString("tun:\n dev: nebroutetest\n unsafe_routes:\n")
@@ -77,7 +85,7 @@ func TestWinTun_ReloadKeepsInstalledRoutes(t *testing.T) {
rt := newRouteTest(t, unsafeRoutes(a))
require.NoError(t, rt.tun.Activate())
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a, b)))
rt.reload(t, unsafeRoutes(a, b))
assert.NotContains(t, rt.log.String(), "Failed to add route")
rt.metric(t, a)
rt.metric(t, b)
@@ -92,7 +100,7 @@ func TestWinTun_ReloadMetricChangeApplies(t *testing.T) {
require.NoError(t, rt.tun.Activate())
require.Equal(t, uint32(100), rt.metric(t, a))
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a+"@200")))
rt.reload(t, unsafeRoutes(a+"@200"))
assert.NotContains(t, rt.log.String(), "Failed to add route")
assert.NotContains(t, rt.log.String(), "Removed route")
assert.Equal(t, uint32(200), rt.metric(t, a))
@@ -120,7 +128,7 @@ func TestWinTun_DuplicateCidrInstallsTheLastEntry(t *testing.T) {
require.NoError(t, rt.tun.Activate())
assert.Equal(t, uint32(200), rt.metric(t, a))
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a+"@100", a+"@200", b)))
rt.reload(t, unsafeRoutes(a+"@100", a+"@200", b))
assert.Equal(t, uint32(200), rt.metric(t, a))
assert.NotContains(t, rt.log.String(), "Updated route metric")
assert.NotContains(t, rt.log.String(), "Failed to add route")
@@ -134,7 +142,7 @@ func TestWinTun_DroppingAnUninstalledDuplicateKeepsTheRoute(t *testing.T) {
rt := newRouteTest(t, unsafeRoutes(a+"@100", a+"@200"))
require.NoError(t, rt.tun.Activate())
require.NoError(t, rt.c.ReloadConfigString(unsafeRoutes(a+"@200")))
rt.reload(t, unsafeRoutes(a+"@200"))
assert.NotContains(t, rt.log.String(), "Removed route")
assert.Equal(t, uint32(200), rt.metric(t, a))
})
@@ -155,12 +163,45 @@ func TestWinTun_ReloadMatchesAFreshStart(t *testing.T) {
rt := newRouteTest(t, unsafeRoutes(a+"@100"))
require.NoError(t, rt.tun.Activate())
require.NoError(t, rt.c.ReloadConfigString(routes))
rt.reload(t, routes)
_, err = rt.luid.Route(cidr, unspecifiedNextHop(cidr))
assert.Error(t, err, "the reload left it installed")
})
}
// Removing a route that is already gone is what we wanted, not a failure
func TestWinTun_ReloadRouteAlreadyGone(t *testing.T) {
forFamilies(t, func(t *testing.T, a, b string) {
rt := newRouteTest(t, unsafeRoutes(a, b))
require.NoError(t, rt.tun.Activate())
p := netip.MustParsePrefix(b)
require.NoError(t, rt.luid.DeleteRoute(p, unspecifiedNextHop(p)))
rt.reload(t, unsafeRoutes(a))
})
}
// A network category we can't parse would fail the restart, so it fails the reload instead of asking for one
func TestWinTun_ReloadInvalidNetworkCategoryFails(t *testing.T) {
rt := newRouteTest(t, unsafeRoutes())
for range 2 {
res := rt.c.ReloadConfigString(unsafeRoutes() + " network_category: nope\n")
require.NoError(t, res.Err)
assert.True(t, res.Failed, "a bad category that didn't change since the last reload is still bad")
assert.False(t, res.RestartRequired)
}
// Fixing it back to the running category needs nothing
res := rt.c.ReloadConfigString(unsafeRoutes() + " network_category: private\n")
require.NoError(t, res.Err)
assert.False(t, res.Failed)
assert.False(t, res.RestartRequired)
res = rt.c.ReloadConfigString(unsafeRoutes() + " network_category: public\n")
require.NoError(t, res.Err)
assert.Equal(t, []string{"tun.network_category"}, res.RestartKeys)
}
// forFamilies runs f with two unsafe_routes CIDRs of each address family, the tun's own network stays IPv4
func forFamilies(t *testing.T, f func(t *testing.T, a, b string)) {
t.Run("ipv4", func(t *testing.T) { f(t, "10.251.1.0/24", "10.251.2.0/24") })
+70 -18
View File
@@ -2,7 +2,9 @@ package nebula
import (
"crypto/fips140"
"crypto/sha256"
"encoding/binary"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
@@ -50,15 +52,30 @@ type CertState struct {
myVpnBroadcastAddrsTable *bart.Lite
}
// errCertNeedsRestart marks a new cert that is refused on reload because only a restart can apply it
var errCertNeedsRestart = errors.New("a restart is required to use the new cert")
func NewPKIFromConfig(l *slog.Logger, c *config.C) (*PKI, error) {
pki := &PKI{l: l}
err := pki.reload(c, true)
err := pki.reload(c, true, nil)
if err != nil {
return nil, err
}
c.RegisterReloadCallback(func(c *config.C) {
rErr := pki.reload(c, false)
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
// Compare with the running cipher, not the last config. A restart with one we can't run would fail, so say that
// now rather than ask for one, and going back to the running one after a bad reload needs no restart
if cipher := c.GetString("cipher", "aes"); !validCipher(cipher) {
l.Error("Unknown cipher, a restart with it would fail", "cipher", cipher)
r.Failed()
} else if cipher == "chachapoly" && fipsOnly() {
// newCipherSuite refuses it at startup
l.Error("ChaChaPoly is not allowed in FIPS 140-only mode, a restart with it would fail", "cipher", cipher)
r.Failed()
} else if cipher != pki.getCertState().cipher {
r.RestartRequiredIfChanged("cipher")
}
rErr := pki.reload(c, false, r)
if rErr != nil {
util.LogWithContextIfNeeded("Failed to reload PKI from config", rErr, l)
}
@@ -75,13 +92,29 @@ func (p *PKI) getCertState() *CertState {
return p.cs.Load()
}
func (p *PKI) reload(c *config.C, initial bool) error {
// r is nil on the initial load
func (p *PKI) reload(c *config.C, initial bool, r *config.Reload) error {
before := p.cs.Load()
err := p.reloadCerts(c, initial)
if err != nil {
if initial {
return err
}
err.Log(p.l)
if errors.Is(err, errCertNeedsRestart) {
r.RestartRequired()
} else {
r.Failed()
}
} else if !initial {
// A cert for a version we weren't running is accepted without comparing its networks to ours. The tun and the
// lighthouse keep the networks they started with
after := p.cs.Load()
if !slices.Equal(before.myVpnNetworks, after.myVpnNetworks) {
p.l.Warn("The new cert changes our networks, a restart is required to use them",
"old_networks", before.myVpnNetworks, "new_networks", after.myVpnNetworks)
r.RestartRequired()
}
}
err = p.reloadCAPool(c)
@@ -90,21 +123,27 @@ func (p *PKI) reload(c *config.C, initial bool) error {
return err
}
err.Log(p.l)
r.Failed()
}
return nil
}
// fipsOnly is fips140.Enforced, a variable so a test can run the FIPS 140-only reload check without GODEBUG
var fipsOnly = fips140.Enforced
// validCipher is true for the ciphers nebula can run. Each post-handshake CipherState in noiseutil hardcodes its own
// nonce endianness, so there's nothing to set up for either.
func validCipher(cipher string) bool {
return cipher == "aes" || cipher == "chachapoly"
}
func (p *PKI) reloadCerts(c *config.C, initial bool) *util.ContextualError {
var cipher string
var currentState *CertState
if initial {
cipher = c.GetString("cipher", "aes")
switch cipher {
case "aes", "chachapoly":
// Each post-handshake CipherState in noiseutil hardcodes its own
// nonce endianness now, so there's nothing to set up here.
default:
if !validCipher(cipher) {
return util.NewContextualError(
"unknown cipher",
m{"cipher": cipher},
@@ -132,7 +171,7 @@ func (p *PKI) reloadCerts(c *config.C, initial bool) *util.ContextualError {
return util.NewContextualError(
"Networks in new cert was different from old",
m{"new_networks": newState.v1Cert.Networks(), "old_networks": currentState.v1Cert.Networks(), "cert_version": cert.Version1},
nil,
errCertNeedsRestart,
)
}
@@ -140,7 +179,7 @@ func (p *PKI) reloadCerts(c *config.C, initial bool) *util.ContextualError {
return util.NewContextualError(
"Curve in new v1 cert was different from old",
m{"new_curve": newState.v1Cert.Curve(), "old_curve": currentState.v1Cert.Curve(), "cert_version": cert.Version1},
nil,
errCertNeedsRestart,
)
}
}
@@ -155,7 +194,7 @@ func (p *PKI) reloadCerts(c *config.C, initial bool) *util.ContextualError {
return util.NewContextualError(
"Networks in new cert was different from old",
m{"new_networks": newState.v2Cert.Networks(), "old_networks": currentState.v2Cert.Networks(), "cert_version": cert.Version2},
nil,
errCertNeedsRestart,
)
}
@@ -163,7 +202,7 @@ func (p *PKI) reloadCerts(c *config.C, initial bool) *util.ContextualError {
return util.NewContextualError(
"Curve in new cert was different from old",
m{"new_curve": newState.v2Cert.Curve(), "old_curve": currentState.v2Cert.Curve(), "cert_version": cert.Version2},
nil,
errCertNeedsRestart,
)
}
}
@@ -178,7 +217,7 @@ func (p *PKI) reloadCerts(c *config.C, initial bool) *util.ContextualError {
return util.NewContextualError(
"Removing a V2 cert is not permitted unless it has identical networks to the new V1 cert",
m{"new_v1_networks": newState.v1Cert.Networks(), "old_v2_networks": currentState.v2Cert.Networks()},
nil,
errCertNeedsRestart,
)
}
}
@@ -196,12 +235,14 @@ func (p *PKI) reloadCerts(c *config.C, initial bool) *util.ContextualError {
func (p *PKI) reloadCAPool(c *config.C) *util.ContextualError {
caPool, err := loadCAPoolFromConfig(p.l, c)
if caPool != nil {
p.caPool.Store(caPool)
p.l.Debug("Trusted CA fingerprints", "fingerprints", caPool.GetFingerprints())
}
if err != nil {
return util.NewContextualError("Failed to load ca from config", nil, err)
}
p.caPool.Store(caPool)
p.l.Debug("Trusted CA fingerprints", "fingerprints", caPool.GetFingerprints())
return nil
}
@@ -576,11 +617,22 @@ func loadCAPoolFromConfig(l *slog.Logger, c *config.C) (*cert.CAPool, error) {
bl := c.GetStringSlice("pki.blocklist", []string{})
if len(bl) > 0 {
for _, fp := range bl {
var bad []string
for i, raw := range bl {
// Fingerprints are compared as lowercase hex strings, a malformed entry would silently block nothing
fp := strings.ToLower(strings.TrimSpace(raw))
if b, err := hex.DecodeString(fp); err != nil || len(b) != sha256.Size {
bad = append(bad, fmt.Sprintf("entry %d %q", i+1, raw))
continue
}
caPool.BlocklistFingerprint(fp)
}
l.Info("Blocklisted certificates", "fingerprintCount", len(bl))
l.Info("Blocklisted certificates", "fingerprintCount", len(bl)-len(bad))
if len(bad) > 0 {
// The valid entries are still returned so a failed reload blocks them, startup refuses the error
return caPool, fmt.Errorf("pki.blocklist entries are not hex certificate fingerprints: %s", strings.Join(bad, ", "))
}
}
return caPool, nil
+168
View File
@@ -1,10 +1,17 @@
package nebula
import (
"crypto/fips140"
"fmt"
"net/netip"
"strings"
"testing"
"time"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -92,3 +99,164 @@ func TestNewCipherSuiteUnsupportedCurve(t *testing.T) {
assert.True(t, strings.HasPrefix(err.Error(), "unsupported curve:"), "got: %v", err)
assert.Nil(t, cs)
}
func TestPKIReloadRestartRequired(t *testing.T) {
ca, _, caKey, caPEM := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Time{}, time.Time{}, nil, nil, nil)
pkiConfig := func(cipher, network string) string {
_, _, keyPEM, certPEM := cert_test.NewTestCert(cert.Version2, cert.Curve_CURVE25519, ca, caKey, "host", time.Time{}, time.Time{}, []netip.Prefix{netip.MustParsePrefix(network)}, nil, nil)
return fmt.Sprintf("cipher: %s\npki:\n ca: %q\n cert: %q\n key: %q\n", cipher, caPEM, certPEM, keyPEM)
}
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString(pkiConfig("aes", "10.1.0.2/24")))
p, err := NewPKIFromConfig(test.NewLogger(), c)
require.NoError(t, err)
// A cert that changes our networks is refused and needs a restart, it is not a failure
res := c.ReloadConfigString(pkiConfig("aes", "10.1.0.3/24"))
require.NoError(t, res.Err)
assert.True(t, res.RestartRequired)
assert.False(t, res.Failed)
assert.Equal(t, []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")}, p.getCertState().myVpnNetworks)
// The cipher only applies at startup
c2 := config.NewC(test.NewLogger())
require.NoError(t, c2.LoadString(pkiConfig("aes", "10.1.0.2/24")))
p, err = NewPKIFromConfig(test.NewLogger(), c2)
require.NoError(t, err)
res = c2.ReloadConfigString(pkiConfig("chachapoly", "10.1.0.2/24"))
require.NoError(t, res.Err)
assert.True(t, res.RestartRequired)
assert.Equal(t, "aes", p.getCertState().cipher)
// A cipher we can't run would fail the restart, so it fails the reload instead of asking for one
c4 := config.NewC(test.NewLogger())
require.NoError(t, c4.LoadString(pkiConfig("aes", "10.1.0.2/24")))
_, err = NewPKIFromConfig(test.NewLogger(), c4)
require.NoError(t, err)
for range 2 {
res = c4.ReloadConfigString(pkiConfig("nope", "10.1.0.2/24"))
require.NoError(t, res.Err)
assert.True(t, res.Failed, "a bad cipher that didn't change since the last reload is still bad")
assert.False(t, res.RestartRequired)
}
// Fixing it back to the running cipher needs nothing
res = c4.ReloadConfigString(pkiConfig("aes", "10.1.0.2/24"))
require.NoError(t, res.Err)
assert.False(t, res.Failed)
assert.False(t, res.RestartRequired)
// A cert that keeps our networks applies without a restart
c3 := config.NewC(test.NewLogger())
require.NoError(t, c3.LoadString(pkiConfig("aes", "10.1.0.2/24")))
_, err = NewPKIFromConfig(test.NewLogger(), c3)
require.NoError(t, err)
res = c3.ReloadConfigString(pkiConfig("aes", "10.1.0.2/24"))
require.NoError(t, res.Err)
assert.False(t, res.RestartRequired)
// A cert that can't be read is a failure, and a restart wouldn't help
res = c3.ReloadConfigString(strings.Replace(pkiConfig("aes", "10.1.0.2/24"), "cert: \"", "cert: \"garbage", 1))
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.False(t, res.RestartRequired)
}
// A cert for a version we weren't running is accepted without comparing networks, so switching to one with different
// networks needs a restart. One with the same networks doesn't
func TestPKIReloadNewVersionWithOtherNetworks(t *testing.T) {
ca1, _, ca1Key, ca1PEM := cert_test.NewTestCaCert(cert.Version1, cert.Curve_CURVE25519, time.Time{}, time.Time{}, nil, nil, nil)
ca2, _, ca2Key, ca2PEM := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Time{}, time.Time{}, nil, nil, nil)
caPEM := string(ca1PEM) + string(ca2PEM)
pkiConfig := func(v cert.Version, network string) string {
ca, caKey := ca1, ca1Key
if v == cert.Version2 {
ca, caKey = ca2, ca2Key
}
_, _, keyPEM, certPEM := cert_test.NewTestCert(v, cert.Curve_CURVE25519, ca, caKey, "host", time.Time{}, time.Time{}, []netip.Prefix{netip.MustParsePrefix(network)}, nil, nil)
return fmt.Sprintf("pki:\n ca: %q\n cert: %q\n key: %q\n", caPEM, certPEM, keyPEM)
}
for _, tc := range []struct {
network string
restart bool
}{
{"10.9.0.7/24", true},
{"10.1.0.2/24", false},
} {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString(pkiConfig(cert.Version1, "10.1.0.2/24")))
p, err := NewPKIFromConfig(test.NewLogger(), c)
require.NoError(t, err)
res := c.ReloadConfigString(pkiConfig(cert.Version2, tc.network))
require.NoError(t, res.Err)
assert.False(t, res.Failed, tc.network)
assert.Equal(t, tc.restart, res.RestartRequired, tc.network)
assert.Equal(t, []netip.Prefix{netip.MustParsePrefix(tc.network)}, p.getCertState().myVpnNetworks, "accepted either way, as before")
}
}
// FIPS 140-only mode refuses chachapoly at startup, so reloading to it fails rather than asks for a restart
func TestPKIReloadChachapolyInFIPSOnlyFails(t *testing.T) {
fipsOnly = func() bool { return true }
t.Cleanup(func() { fipsOnly = fips140.Enforced })
ca, _, caKey, caPEM := cert_test.NewTestCaCert(cert.Version2, cert.Curve_P256, time.Time{}, time.Time{}, nil, nil, nil)
_, _, keyPEM, certPEM := cert_test.NewTestCert(cert.Version2, cert.Curve_P256, ca, caKey, "host", time.Time{}, time.Time{}, []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")}, nil, nil)
pkiConfig := func(cipher string) string {
return fmt.Sprintf("cipher: %s\npki:\n ca: %q\n cert: %q\n key: %q\n", cipher, caPEM, certPEM, keyPEM)
}
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString(pkiConfig("aes")))
_, err := NewPKIFromConfig(test.NewLogger(), c)
require.NoError(t, err)
res := c.ReloadConfigString(pkiConfig("chachapoly"))
require.NoError(t, res.Err)
assert.True(t, res.Failed)
assert.False(t, res.RestartRequired)
}
func TestLoadCAPoolBlocklist(t *testing.T) {
ca, _, caKey, caPEM := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Time{}, time.Time{}, nil, nil, nil)
crt, _, _, _ := cert_test.NewTestCert(cert.Version2, cert.Curve_CURVE25519, ca, caKey, "host", time.Time{}, time.Time{}, []netip.Prefix{netip.MustParsePrefix("10.1.0.2/24")}, nil, nil)
fp, err := crt.Fingerprint()
require.NoError(t, err)
load := func(entry string) (*cert.CAPool, error) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString(fmt.Sprintf("pki:\n ca: %q\n blocklist:\n - %q\n", caPEM, entry)))
return loadCAPoolFromConfig(test.NewLogger(), c)
}
for _, entry := range []string{fp, strings.ToUpper(fp), " " + fp + "\t"} {
pool, err := load(entry)
require.NoError(t, err, entry)
assert.True(t, pool.IsBlocklisted(fp), entry)
}
for _, entry := range []string{fp[1:], fp + "00", "sha256:" + fp, "zz" + fp[2:], ""} {
_, err := load(entry)
require.Error(t, err, entry)
}
// A bad entry on reload fails it, the valid entries are still blocked
_, _, keyPEM, certPEM := cert_test.NewTestCert(cert.Version2, cert.Curve_CURVE25519, ca, caKey, "me", time.Time{}, time.Time{}, []netip.Prefix{netip.MustParsePrefix("10.1.0.1/24")}, nil, nil)
pkiConfig := func(entry string) string {
return fmt.Sprintf("pki:\n ca: %q\n cert: %q\n key: %q\n blocklist:\n - %q\n", caPEM, certPEM, keyPEM, entry)
}
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString(pkiConfig(fp)))
p, err := NewPKIFromConfig(test.NewLogger(), c)
require.NoError(t, err)
other, _, _, _ := cert_test.NewTestCert(cert.Version2, cert.Curve_CURVE25519, ca, caKey, "other", time.Time{}, time.Time{}, []netip.Prefix{netip.MustParsePrefix("10.1.0.3/24")}, nil, nil)
otherFp, err := other.Fingerprint()
require.NoError(t, err)
res := c.ReloadConfigString(pkiConfig(fp) + fmt.Sprintf(" - %q\n - %q\n", otherFp, fp[1:]))
assert.True(t, res.Failed)
assert.True(t, p.GetCAPool().IsBlocklisted(fp))
assert.True(t, p.GetCAPool().IsBlocklisted(otherFp))
}
+2 -1
View File
@@ -67,7 +67,8 @@ func NewPunchyFromConfig(l *slog.Logger, c *config.C, punchConn udp.Conn) *Punch
}
p.reload(c, true)
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("stats.lighthouse_metrics")
p.reload(c, false)
})
+16 -6
View File
@@ -76,7 +76,7 @@ punchy:
punch: true
delay: 10m
respond: true
`))
`).Err)
p.reload(c, false)
assert.True(t, p.punch.Load())
assert.Equal(t, newDelay, time.Duration(p.delay.Load()))
@@ -170,7 +170,7 @@ func TestPunchy_LogFormat_ReloadPunch(t *testing.T) {
NewPunchyFromConfig(l, c, nil)
hook.entries = nil
require.NoError(t, c.ReloadConfigString(`punchy: {punch: true}`))
require.NoError(t, c.ReloadConfigString(`punchy: {punch: true}`).Err)
entry := findEntry(t, hook.entries, "punchy.punch changed")
assert.Equal(t, slog.LevelInfo, entry.Level)
@@ -184,7 +184,7 @@ func TestPunchy_LogFormat_ReloadRespond(t *testing.T) {
NewPunchyFromConfig(l, c, nil)
hook.entries = nil
require.NoError(t, c.ReloadConfigString(`punchy: {respond: true}`))
require.NoError(t, c.ReloadConfigString(`punchy: {respond: true}`).Err)
entry := findEntry(t, hook.entries, "punchy.respond changed")
assert.Equal(t, slog.LevelInfo, entry.Level)
@@ -198,7 +198,7 @@ func TestPunchy_LogFormat_ReloadDelay(t *testing.T) {
NewPunchyFromConfig(l, c, nil)
hook.entries = nil
require.NoError(t, c.ReloadConfigString(`punchy: {delay: 10s}`))
require.NoError(t, c.ReloadConfigString(`punchy: {delay: 10s}`).Err)
entry := findEntry(t, hook.entries, "punchy.delay changed")
assert.Equal(t, slog.LevelInfo, entry.Level)
@@ -212,7 +212,7 @@ func TestPunchy_LogFormat_ReloadTargetAllRemotes(t *testing.T) {
NewPunchyFromConfig(l, c, nil)
hook.entries = nil
require.NoError(t, c.ReloadConfigString(`punchy: {target_all_remotes: true}`))
require.NoError(t, c.ReloadConfigString(`punchy: {target_all_remotes: true}`).Err)
entry := findEntry(t, hook.entries, "punchy.target_all_remotes changed")
assert.Equal(t, slog.LevelInfo, entry.Level)
@@ -226,9 +226,19 @@ func TestPunchy_LogFormat_ReloadRespondDelay(t *testing.T) {
NewPunchyFromConfig(l, c, nil)
hook.entries = nil
require.NoError(t, c.ReloadConfigString(`punchy: {respond_delay: 15s}`))
require.NoError(t, c.ReloadConfigString(`punchy: {respond_delay: 15s}`).Err)
entry := findEntry(t, hook.entries, "punchy.respond_delay changed")
assert.Equal(t, slog.LevelInfo, entry.Level)
assert.Equal(t, map[string]any{"respond_delay": 15 * time.Second}, entry.Attrs)
}
func TestPunchyReloadRestartRequired(t *testing.T) {
c := config.NewC(test.NewLogger())
require.NoError(t, c.LoadString("stats:\n lighthouse_metrics: false\n"))
NewPunchyFromConfig(test.NewLogger(), c, nil)
res := c.ReloadConfigString("stats:\n lighthouse_metrics: true\n")
require.NoError(t, res.Err)
assert.True(t, res.RestartRequired)
}
+5 -10
View File
@@ -28,10 +28,11 @@ func NewRelayManager(ctx context.Context, l *slog.Logger, hostmap *HostMap, c *c
hostmap: hostmap,
}
rm.reload(c, true)
c.RegisterReloadCallback(func(c *config.C) {
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
err := rm.reload(c, false)
if err != nil {
rm.l.Error("Failed to reload relay_manager", "error", err)
r.Failed()
}
})
return rm
@@ -57,9 +58,8 @@ func (rm *relayManager) GetUseRelays() bool {
// StartRelays drives the relay-establishment side of an outbound handshake attempt.
// For each candidate relay it either kicks off a handshake to the relay, sends a CreateRelayRequest, retransmits
// one that may have been lost, or, once the relay is Established, forwards the in-progress
// stage 0 handshake packets for vpnIp through it: now immediately and later after the
// handshake repair delay.
func (rm *relayManager) StartRelays(f *Interface, vpnIp netip.Addr, hh *HandshakeHostInfo, now, later [][]byte) {
// stage 0 handshake packet for vpnIp through it.
func (rm *relayManager) StartRelays(f *Interface, vpnIp netip.Addr, hh *HandshakeHostInfo, stage0 []byte) {
hostinfo := hh.hostinfo
if !rm.GetUseRelays() || len(hostinfo.remotes.relays) == 0 {
hh.lastRelays = nil
@@ -162,10 +162,7 @@ func (rm *relayManager) StartRelays(f *Interface, vpnIp netip.Addr, hh *Handshak
switch existingRelay.State {
case Established:
hl.Log(context.Background(), level, "Send handshake via relay", "relay", relay.String())
_ = f.handshakeManager.sendFragmented(now, later, func(p []byte) error {
f.SendVia(relayHostInfo, existingRelay, p, make([]byte, 12), make([]byte, mtu), false, 0)
return nil
})
f.SendVia(relayHostInfo, existingRelay, stage0, make([]byte, 12), make([]byte, mtu), false, 0)
case Disestablished:
// Mark this relay as 'requested'
relayHostInfo.relayState.UpdateRelayForByIpState(vpnIp, Requested)
@@ -364,8 +361,6 @@ func (rm *relayManager) handleCreateRelayResponse(v cert.Version, h *HostInfo, f
}
// Do I need to complete the relays now?
if relay.Type == TerminalType {
// The relay I asked for is up; a handshake waiting on it can go now.
f.handshakeManager.relayEstablished(relay.PeerAddr, h, relay)
return
}
// I'm the middle man. Let the initiator know that the I've established the relay they requested.
+6 -6
View File
@@ -50,7 +50,7 @@ func TestStartRelaysLogDedupe(t *testing.T) {
var buf bytes.Buffer
rm := newRM(&buf)
hh := newHH()
rm.StartRelays(f, vpnIp, hh, nil, nil)
rm.StartRelays(f, vpnIp, hh, nil)
assert.Equal(t, []netip.Addr{vpnIp}, hh.lastRelays, "lastRelays should record the relay set we just attempted")
assert.Contains(t, buf.String(), "level=INFO "+msg, "expected Info level on first attempt")
})
@@ -59,10 +59,10 @@ func TestStartRelaysLogDedupe(t *testing.T) {
var buf bytes.Buffer
rm := newRM(&buf)
hh := newHH()
rm.StartRelays(f, vpnIp, hh, nil, nil)
rm.StartRelays(f, vpnIp, hh, nil)
first := append([]netip.Addr(nil), hh.lastRelays...)
buf.Reset()
rm.StartRelays(f, vpnIp, hh, nil, nil)
rm.StartRelays(f, vpnIp, hh, nil)
assert.Equal(t, first, hh.lastRelays)
assert.Contains(t, buf.String(), "level=DEBUG "+msg, "expected Debug level on identical retry")
assert.NotContains(t, buf.String(), "level=INFO "+msg, "Info should not fire on identical retry")
@@ -72,13 +72,13 @@ func TestStartRelaysLogDedupe(t *testing.T) {
var buf bytes.Buffer
rm := newRM(&buf)
hh := newHH()
rm.StartRelays(f, vpnIp, hh, nil, nil)
rm.StartRelays(f, vpnIp, hh, nil)
buf.Reset()
// The lighthouse handed us a new set this round.
hh.hostinfo.remotes.relays = []netip.Addr{vpnIp, otherRelay}
rm.StartRelays(f, vpnIp, hh, nil, nil)
rm.StartRelays(f, vpnIp, hh, nil)
assert.Equal(t, []netip.Addr{vpnIp, otherRelay}, hh.lastRelays)
assert.Contains(t, buf.String(), "level=INFO "+msg, "expected Info when the relay list changes")
})
@@ -90,7 +90,7 @@ func TestStartRelaysLogDedupe(t *testing.T) {
hh := newHH()
hh.lastRelays = []netip.Addr{vpnIp}
rm.StartRelays(f, vpnIp, hh, nil, nil)
rm.StartRelays(f, vpnIp, hh, nil)
assert.Nil(t, hh.lastRelays, "with relays disabled lastRelays should be cleared")
assert.NotContains(t, buf.String(), msg, "should not log when we shortcut out")
})
+27 -10
View File
@@ -1,8 +1,8 @@
package service
import (
"io"
"net"
"sync"
)
type tcpListener struct {
@@ -10,23 +10,40 @@ type tcpListener struct {
s *Service
addr *net.TCPAddr
accept chan net.Conn
// closed is closed instead of accept so a handler blocked handing off a
// connection can't send on a closed channel.
closed chan struct{}
closeOnce sync.Once
}
func (l *tcpListener) Accept() (net.Conn, error) {
conn, ok := <-l.accept
if !ok {
return nil, io.EOF
// select picks randomly among ready cases, so without this a handler still
// waiting to hand off could win over closed after Close has returned.
select {
case <-l.closed:
return nil, net.ErrClosed
default:
}
select {
case conn := <-l.accept:
return conn, nil
case <-l.closed:
return nil, net.ErrClosed
}
return conn, nil
}
func (l *tcpListener) Close() error {
l.s.mu.Lock()
defer l.s.mu.Unlock()
delete(l.s.mu.listeners, uint16(l.addr.Port))
close(l.accept)
l.closeOnce.Do(func() {
// Close before leaving the map so a concurrent closeListeners that
// no longer sees this listener knows Accept is already released.
close(l.closed)
l.s.mu.Lock()
delete(l.s.mu.listeners, l.port)
l.s.mu.Unlock()
})
return nil
}
+79 -10
View File
@@ -32,13 +32,18 @@ import (
const nicID = 1
type Service struct {
eg *errgroup.Group
control *nebula.Control
ipstack *stack.Stack
// done is closed once the forwarding goroutines have exited and the
// netstack is torn down, at which point err holds the first fatal error.
done chan struct{}
err error
mu struct {
sync.Mutex
closed bool
listeners map[uint16]*tcpListener
}
}
@@ -65,7 +70,6 @@ func New(control *nebula.Control) (_ *Service, reterr error) {
ctx := control.Context()
eg, ctx := errgroup.WithContext(ctx)
s := Service{
eg: eg,
control: control,
}
s.mu.listeners = map[uint16]*tcpListener{}
@@ -74,6 +78,12 @@ func New(control *nebula.Control) (_ *Service, reterr error) {
NetworkProtocols: []stack.NetworkProtocolFactory{ipv4.NewProtocol, ipv6.NewProtocol},
TransportProtocols: []stack.TransportProtocolFactory{tcp.NewProtocol, udp.NewProtocol, icmp.NewProtocol4, icmp.NewProtocol6},
})
// The teardown goroutine only owns the stack once New succeeds
defer func() {
if reterr != nil {
s.ipstack.Destroy()
}
}()
sackEnabledOpt := tcpip.TCPSACKEnabled(true) // TCP SACK is disabled by default
tcpipErr := s.ipstack.SetTransportProtocolOption(tcp.ProtocolNumber, &sackEnabledOpt)
if tcpipErr != nil {
@@ -123,6 +133,12 @@ func New(control *nebula.Control) (_ *Service, reterr error) {
// this will read exactly one packet
n, err := reader.Read(buf)
if err != nil {
// An error after shutdown began is noise from the device pipes
// closing, anything else is fatal and must stop nebula too
if ctx.Err() != nil {
return nil
}
s.control.Stop()
return err
}
packetBuf := stack.NewPacketBuffer(stack.PacketBufferOptions{
@@ -130,8 +146,8 @@ func New(control *nebula.Control) (_ *Service, reterr error) {
})
linkEP.InjectInbound(header.IPv4ProtocolNumber, packetBuf)
if err := ctx.Err(); err != nil {
return err
if ctx.Err() != nil {
return nil
}
}
})
@@ -139,13 +155,18 @@ func New(control *nebula.Control) (_ *Service, reterr error) {
for {
packet := linkEP.ReadContext(ctx)
if packet == nil {
if err := ctx.Err(); err != nil {
return err
if ctx.Err() != nil {
return nil
}
continue
}
bufView := packet.ToView()
if _, err := bufView.WriteTo(writer); err != nil {
// Same shutdown noise handling as the inbound reader
if ctx.Err() != nil {
return nil
}
s.control.Stop()
return err
}
bufView.Release()
@@ -158,6 +179,22 @@ func New(control *nebula.Control) (_ *Service, reterr error) {
return control.Wait()
})
// Tear down here rather than in Close so it also happens when nebula stops
// on its own. The forwarding goroutines must exit before the stack they feed
// is destroyed.
s.done = make(chan struct{})
go func() {
s.err = eg.Wait()
s.closeListeners()
// Destroy aborts the endpoints that exist when it starts, then waits on
// every endpoint, so a Dial registering in between would hold Wait until
// its connect timed out. Without the NIC new dials fail route lookup.
// RemoveNIC also closes linkEP.
_ = s.ipstack.RemoveNIC(nicID)
s.ipstack.Destroy()
close(s.done)
}()
return &s, nil
}
@@ -231,11 +268,15 @@ func (s *Service) Listen(network, address string) (net.Listener, error) {
s: s,
addr: addr,
accept: make(chan net.Conn),
closed: make(chan struct{}),
}
s.mu.Lock()
defer s.mu.Unlock()
if s.mu.closed {
return nil, net.ErrClosed
}
if _, ok := s.mu.listeners[port]; ok {
return nil, fmt.Errorf("already listening on port %d", port)
}
@@ -244,22 +285,44 @@ func (s *Service) Listen(network, address string) (net.Listener, error) {
return l, nil
}
// Wait blocks until the service has fully stopped, either via Close or a fatal
// error, and returns the first fatal error if there was one.
func (s *Service) Wait() error {
return s.eg.Wait()
<-s.done
return s.err
}
// Close closes all listeners and stops nebula. Use Wait to block until
// shutdown has completed.
func (s *Service) Close() error {
s.closeListeners()
s.control.Stop()
return nil
}
// closeListeners unblocks pending Accept calls and rejects new listeners.
func (s *Service) closeListeners() {
// Every caller closes every listener rather than returning early once
// closed is set, so none returns while another is still closing them.
s.mu.Lock()
s.mu.closed = true
listeners := make([]*tcpListener, 0, len(s.mu.listeners))
for _, l := range s.mu.listeners {
listeners = append(listeners, l)
}
s.mu.Unlock()
for _, l := range listeners {
_ = l.Close()
}
}
func (s *Service) tcpHandler(r *tcp.ForwarderRequest) {
endpointID := r.ID()
s.mu.Lock()
defer s.mu.Unlock()
l, ok := s.mu.listeners[endpointID.LocalPort]
s.mu.Unlock()
if !ok {
r.Complete(true)
return
@@ -275,6 +338,12 @@ func (s *Service) tcpHandler(r *tcp.ForwarderRequest) {
r.Complete(false)
ep.SocketOptions().SetKeepAlive(true)
// The handoff blocks until Accept is called, so it must not hold s.mu or a
// listener nobody accepts from stalls every other port.
conn := gonet.NewTCPConn(&wq, ep)
l.accept <- conn
select {
case l.accept <- conn:
case <-l.closed:
_ = conn.Close()
}
}
+298
View File
@@ -188,3 +188,301 @@ func TestService(t *testing.T) {
t.Fatal(err)
}
}
// A listener that is never accepted from must not stall connections to other
// ports, and closing it must not block.
func TestServiceUnacceptedListener(t *testing.T) {
lighthousePort := ephemeralUDPPort(t)
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
a := newSimpleService(ca, caKey, "a", netip.MustParseAddr("10.0.0.1"), m{
"static_host_map": m{},
"lighthouse": m{
"am_lighthouse": true,
},
"listen": m{
"host": "0.0.0.0",
"port": lighthousePort,
},
})
t.Cleanup(func() { _ = a.Close() })
b := newSimpleService(ca, caKey, "b", netip.MustParseAddr("10.0.0.2"), m{
"static_host_map": m{
"10.0.0.1": []string{fmt.Sprintf("127.0.0.1:%d", lighthousePort)},
},
"lighthouse": m{
"hosts": []string{"10.0.0.1"},
"interval": 1,
},
})
t.Cleanup(func() { _ = b.Close() })
idle, err := a.Listen("tcp", ":1234")
if err != nil {
t.Fatal(err)
}
active, err := a.Listen("tcp", ":1235")
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
idleConn, err := b.DialContext(ctx, "tcp", "10.0.0.1:1234")
if err != nil {
t.Fatal(err)
}
defer idleConn.Close()
accepted := make(chan error, 1)
go func() {
conn, err := active.Accept()
if err == nil {
err = conn.Close()
}
accepted <- err
}()
ctx, cancel = context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
activeConn, err := b.DialContext(ctx, "tcp", "10.0.0.1:1235")
if err != nil {
t.Fatalf("dial to accepted port stalled behind idle listener: %v", err)
}
defer activeConn.Close()
select {
case err := <-accepted:
if err != nil {
t.Fatal(err)
}
case <-time.After(5 * time.Second):
t.Fatal("accept stalled behind idle listener")
}
closed := make(chan error, 1)
go func() { closed <- idle.Close() }()
select {
case err := <-closed:
if err != nil {
t.Fatal(err)
}
case <-time.After(5 * time.Second):
t.Fatal("closing idle listener blocked")
}
if _, err := idle.Accept(); !errors.Is(err, net.ErrClosed) {
t.Fatalf("expected net.ErrClosed from closed listener, got %v", err)
}
if err := idle.Close(); err != nil {
t.Fatalf("second close: %v", err)
}
}
// Accept must report net.ErrClosed after Close even while a handler is still
// trying to hand off a connection.
func TestListenerAcceptAfterClose(t *testing.T) {
s := &Service{}
s.mu.listeners = map[uint16]*tcpListener{}
l := &tcpListener{
port: 1234,
s: s,
accept: make(chan net.Conn),
closed: make(chan struct{}),
}
s.mu.listeners[l.port] = l
if err := l.Close(); err != nil {
t.Fatal(err)
}
// Keep a send pending so every Accept has a connection ready alongside closed.
stop := make(chan struct{})
done := make(chan struct{})
go func() {
defer close(done)
for {
server, client := net.Pipe()
_ = client.Close()
select {
case l.accept <- server:
case <-stop:
_ = server.Close()
return
}
}
}()
defer func() {
close(stop)
<-done
}()
for range 100 {
time.Sleep(time.Millisecond)
conn, err := l.Accept()
if conn != nil {
_ = conn.Close()
}
if !errors.Is(err, net.ErrClosed) {
t.Fatalf("Accept after Close: got err %v, want net.ErrClosed", err)
}
}
}
// Close must release blocked Accept calls and let Wait report a clean shutdown.
func TestServiceClose(t *testing.T) {
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
a := newSimpleService(ca, caKey, "a", netip.MustParseAddr("10.0.0.1"), m{
"static_host_map": m{},
"listen": m{
"host": "127.0.0.1",
"port": 0,
},
})
ln, err := a.Listen("tcp", ":1234")
if err != nil {
t.Fatal(err)
}
accepted := make(chan error, 1)
go func() {
_, err := ln.Accept()
accepted <- err
}()
if err := a.Close(); err != nil {
t.Fatal(err)
}
select {
case err := <-accepted:
if !errors.Is(err, net.ErrClosed) {
t.Fatalf("expected net.ErrClosed from Accept, got %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("Accept still blocked after Close")
}
waited := make(chan error, 1)
go func() { waited <- a.Wait() }()
select {
case err := <-waited:
if err != nil {
t.Fatalf("expected clean shutdown from Wait, got %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("Wait still blocked after Close")
}
if a.ipstack.HasNIC(nicID) {
t.Fatal("netstack not torn down after Wait")
}
if err := a.Close(); err != nil {
t.Fatalf("second close: %v", err)
}
if _, err := a.Listen("tcp", ":1235"); !errors.Is(err, net.ErrClosed) {
t.Fatalf("expected net.ErrClosed from Listen after Close, got %v", err)
}
}
// If nebula stops without Close being called, the service must still release
// Accept and tear down the netstack.
func TestServiceNebulaStopped(t *testing.T) {
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
a := newSimpleService(ca, caKey, "a", netip.MustParseAddr("10.0.0.1"), m{
"static_host_map": m{},
"listen": m{
"host": "127.0.0.1",
"port": 0,
},
})
ln, err := a.Listen("tcp", ":1234")
if err != nil {
t.Fatal(err)
}
accepted := make(chan error, 1)
go func() {
_, err := ln.Accept()
accepted <- err
}()
a.control.Stop()
waited := make(chan error, 1)
go func() { waited <- a.Wait() }()
select {
case err := <-waited:
if err != nil {
t.Fatalf("expected clean shutdown from Wait, got %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("Wait still blocked after nebula stopped")
}
select {
case err := <-accepted:
if !errors.Is(err, net.ErrClosed) {
t.Fatalf("expected net.ErrClosed from Accept, got %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("Accept still blocked after nebula stopped")
}
if a.ipstack.HasNIC(nicID) {
t.Fatal("netstack not torn down after Wait")
}
}
// Dials racing Close must not hold Wait until their connect times out.
func TestServiceDialDuringClose(t *testing.T) {
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
a := newSimpleService(ca, caKey, "a", netip.MustParseAddr("10.0.0.1"), m{
"static_host_map": m{},
"listen": m{
"host": "127.0.0.1",
"port": 0,
},
})
stop := make(chan struct{})
var eg errgroup.Group
for range 8 {
eg.Go(func() error {
for {
select {
case <-stop:
return nil
default:
}
// 10.0.0.2 has no tunnel, so each dial sits in SYN-SENT until aborted
if c, err := a.Dial("tcp", "10.0.0.2:80"); err == nil {
_ = c.Close()
}
}
})
}
time.Sleep(50 * time.Millisecond)
if err := a.Close(); err != nil {
t.Fatal(err)
}
waited := make(chan error, 1)
go func() { waited <- a.Wait() }()
select {
case err := <-waited:
if err != nil {
t.Fatalf("expected clean shutdown from Wait, got %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("Wait blocked behind a dial that raced Close")
}
close(stop)
dialersDone := make(chan error, 1)
go func() { dialersDone <- eg.Wait() }()
select {
case <-dialersDone:
case <-time.After(5 * time.Second):
t.Fatal("Dial still blocked after Wait")
}
}
+50 -925
View File
File diff suppressed because it is too large. Load diff
+57
View File
@@ -0,0 +1,57 @@
package nebula
import (
"bytes"
"fmt"
"path/filepath"
"strings"
"testing"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
)
// TestSSHCommandSurface is a characterization test for the ssh command surface.
// It pins the registered command set, their descriptions, help text and flag definitions
// so that moving the registry to a transport neutral package is provably a relocation
// rather than a rewrite. Regenerate with `go test . -run TestSSHCommandSurface -update`.
func TestSSHCommandSurface(t *testing.T) {
got := renderCommandSurface(registeredCommands(t).All())
test.Golden(t, filepath.Join("testdata", "ssh_commands.golden"), got)
}
// registeredCommands returns a registry holding every command nebula serves. The
// Interface is never dereferenced: attachCommands only closes over it.
func registeredCommands(t *testing.T) *commands.Registry {
t.Helper()
l := test.NewLogger()
reg := commands.New()
attachCommands(l, config.NewC(l), reg, &Interface{})
return reg
}
func renderCommandSurface(cmds []*commands.Command) string {
lines := make([]string, 0, len(cmds))
for _, cmd := range cmds {
b := &bytes.Buffer{}
fmt.Fprintf(b, "%s\n", cmd.Name)
fmt.Fprintf(b, " short: %s\n", cmd.ShortDescription)
fmt.Fprintf(b, " help: %s\n", cmd.Help)
if cmd.Flags != nil {
fl, _ := cmd.Flags()
if fl != nil {
fb := &bytes.Buffer{}
fl.SetOutput(fb)
fl.PrintDefaults()
fmt.Fprintf(b, " flags:\n%s", fb.String())
}
}
lines = append(lines, b.String())
}
// In the order All gives, so the golden pins display order too.
return strings.Join(lines, "")
}
+94
View File
@@ -0,0 +1,94 @@
package nebula
import (
"errors"
"io"
"os"
"path/filepath"
"testing"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type lineWriter struct {
lines []string
err error
}
func (w *lineWriter) WriteLine(s string) error {
w.lines = append(w.lines, s)
return w.err
}
func (w *lineWriter) Write(s string) error { return w.WriteLine(s) }
func (w *lineWriter) WriteBytes(b []byte) error { return w.WriteLine(string(b)) }
func (w *lineWriter) GetWriter() io.Writer { return io.Discard }
func TestReloadCommandReportsTheResult(t *testing.T) {
dir := t.TempDir()
write := func(body string) {
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.yml"), []byte(body), 0o600))
}
write("a: 1\nb: 1")
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.Load(dir))
fail := false
c.RegisterReloadCallback(func(c *config.C, r *config.Reload) {
r.RestartRequiredIfChanged("b")
if fail {
r.Failed()
}
})
// reload returns what the client was told. A reload that failed is a commands.Failure
// to the surface, exit 1 over ssh, with the result line as its message; one that
// merely needs a restart is not.
reload := func(failed bool) []string {
w := &lineWriter{}
err := reloadCommand(l, c, w)
if failed {
var f *commands.Failure
require.ErrorAs(t, err, &f)
assert.Equal(t, w.lines[len(w.lines)-1], f.Msg)
} else {
require.NoError(t, err)
}
return w.lines
}
write("a: 2\nb: 1")
assert.Equal(t, []string{"Reloading config", "Config reload complete"}, reload(false))
fail = true
write("a: 3\nb: 1")
assert.Equal(t, []string{"Reloading config", "Config reload completed with errors, review the logs"}, reload(true))
fail = false
write("a: 3\nb: 2")
assert.Equal(t, []string{"Reloading config", "Config reload was completed but requires a restart to fully apply, keys [b]"}, reload(false))
write("a: [")
lines := reload(true)
require.Len(t, lines, 2)
assert.Contains(t, lines[1], "Config reload failed, the new config did not load")
assert.Contains(t, lines[1], dir, "names the config that did not load")
}
// A client that went away still gets the reload it asked for
func TestReloadCommandReloadsWhenTheClientIsGone(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.yml"), []byte("a: 1"), 0o600))
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.Load(dir))
reloaded := false
c.RegisterReloadCallback(func(*config.C, *config.Reload) { reloaded = true })
gone := errors.New("client is gone")
require.ErrorIs(t, reloadCommand(l, c, &lineWriter{err: gone}), gone)
assert.True(t, reloaded)
}
-161
View File
@@ -1,161 +0,0 @@
package sshd
import (
"errors"
"flag"
"fmt"
"sort"
"strings"
"github.com/armon/go-radix"
)
// CommandFlags is a function called before help or command execution to parse command line flags
// It should return a flag.FlagSet instance and a pointer to the struct that will contain parsed flags
type CommandFlags func() (*flag.FlagSet, any)
// CommandCallback is the function called when your command should execute.
// fs will be a a pointer to the struct provided by Command.Flags callback, if there was one. -h and -help are reserved
// and handled automatically for you.
// a will be any unconsumed arguments, if no Command.Flags was available this will be all the flags passed in.
// w is the writer to use when sending messages back to the client.
// If an error is returned by the callback it is logged locally, the callback should handle messaging errors to the user
// where appropriate
type CommandCallback func(fs any, a []string, w StringWriter) error
type Command struct {
Name string
ShortDescription string
Help string
Flags CommandFlags
Callback CommandCallback
}
func execCommand(c *Command, args []string, w StringWriter) error {
var (
fl *flag.FlagSet
fs any
)
if c.Flags != nil {
fl, fs = c.Flags()
if fl != nil {
// SetOutput() here in case fl.Parse dumps usage.
fl.SetOutput(w.GetWriter())
err := fl.Parse(args)
if err != nil {
// fl.Parse has dumped error information to the user via the w writer.
return err
}
args = fl.Args()
}
}
return c.Callback(fs, args, w)
}
func dumpCommands(c *radix.Tree, w StringWriter) {
err := w.WriteLine("Available commands:")
if err != nil {
return
}
cmds := make([]string, 0)
for _, l := range allCommands(c) {
cmds = append(cmds, fmt.Sprintf("%s - %s", l.Name, l.ShortDescription))
}
sort.Strings(cmds)
_ = w.Write(strings.Join(cmds, "\n") + "\n\n")
}
func lookupCommand(c *radix.Tree, sCmd string) (*Command, error) {
cmd, ok := c.Get(sCmd)
if !ok {
return nil, nil
}
command, ok := cmd.(*Command)
if !ok {
return nil, errors.New("failed to cast command")
}
return command, nil
}
func matchCommand(c *radix.Tree, cmd string) []string {
cmds := make([]string, 0)
c.WalkPrefix(cmd, func(found string, v any) bool {
cmds = append(cmds, found)
return false
})
sort.Strings(cmds)
return cmds
}
func allCommands(c *radix.Tree) []*Command {
cmds := make([]*Command, 0)
c.WalkPrefix("", func(found string, v any) bool {
cmd, ok := v.(*Command)
if ok {
cmds = append(cmds, cmd)
}
return false
})
return cmds
}
func helpCallback(commands *radix.Tree, a []string, w StringWriter) (err error) {
// Just typed help
if len(a) == 0 {
dumpCommands(commands, w)
return nil
}
// We are printing a specific commands help text
cmd, err := lookupCommand(commands, a[0])
if err != nil {
return
}
if cmd != nil {
err = w.WriteLine(fmt.Sprintf("%s - %s", cmd.Name, cmd.ShortDescription))
if err != nil {
return err
}
if cmd.Help != "" {
err = w.WriteLine(fmt.Sprintf(" %s", cmd.Help))
if err != nil {
return err
}
}
if cmd.Flags != nil {
fs, _ := cmd.Flags()
if fs != nil {
fs.SetOutput(w.GetWriter())
fs.PrintDefaults()
}
}
return nil
}
err = w.WriteLine("Command not available " + a[0])
if err != nil {
return err
}
return nil
}
func checkHelpArgs(args []string) bool {
for _, a := range args {
if a == "-h" || a == "-help" {
return true
}
}
return false
}
+56 -35
View File
@@ -8,14 +8,18 @@ import (
"log/slog"
"net"
"sync"
"sync/atomic"
"github.com/armon/go-radix"
"github.com/slackhq/nebula/commands"
"golang.org/x/crypto/ssh"
)
type SSHServer struct {
config *ssh.ServerConfig
l *slog.Logger
// config is replaced whole when the host key changes, ssh.ServerConfig.AddHostKey only replaces a key of the same
// algorithm and appends any other
config atomic.Pointer[ssh.ServerConfig]
authenticate func(ssh.ConnMetadata, ssh.PublicKey) (*ssh.Permissions, error)
l *slog.Logger
certChecker *ssh.CertChecker
@@ -25,24 +29,28 @@ type SSHServer struct {
trustedKeys map[string]map[string]bool
trustedCAs []ssh.PublicKey
// List of available commands
helpCommand *Command
commands *radix.Tree
listener net.Listener
// Available commands.
commands *commands.Registry
listener atomic.Pointer[net.Listener]
// runs counts the Runs that are listening. A reload's Run can start before the one it replaced has returned
runs atomic.Int32
// addr is where the newest Run listens, nil when none is
addr atomic.Pointer[net.Addr]
// ctx parents per-Run contexts. Cancelling it (e.g. via Control.Stop) tears the server down even
// across reloads, since each Run derives a fresh child rather than reusing this one directly.
ctx context.Context
}
// NewSSHServer creates a new ssh server rigged with default commands and prepares to listen.
// The ssh server's context is parented off the supplied ctx so cancelling it
// (e.g. on Control.Stop) tears down active sessions and closes the listener.
func NewSSHServer(ctx context.Context, l *slog.Logger) (*SSHServer, error) {
// NewSSHServer creates a new ssh server serving the supplied command registry and
// prepares to listen. The ssh server's context is parented off the supplied ctx so
// cancelling it (e.g. on Control.Stop) tears down active sessions and closes the listener.
func NewSSHServer(ctx context.Context, l *slog.Logger, reg *commands.Registry) (*SSHServer, error) {
s := &SSHServer{
trustedKeys: make(map[string]map[string]bool),
l: l,
commands: radix.New(),
commands: reg,
ctx: ctx,
}
@@ -85,18 +93,8 @@ func NewSSHServer(ctx context.Context, l *slog.Logger) (*SSHServer, error) {
},
}
s.config = &ssh.ServerConfig{
PublicKeyCallback: cc.Authenticate,
ServerVersion: fmt.Sprintf("SSH-2.0-Nebula???"),
}
s.RegisterCommand(&Command{
Name: "help",
ShortDescription: "prints available commands or help <command> for specific usage info",
Callback: func(a any, args []string, w StringWriter) error {
return helpCallback(s.commands, args, w)
},
})
s.authenticate = cc.Authenticate
s.config.Store(s.newConfig())
return s, nil
}
@@ -107,10 +105,32 @@ func (s *SSHServer) SetHostKey(hostPrivateKey []byte) error {
return fmt.Errorf("failed to parse private key: %s", err)
}
s.config.AddHostKey(private)
config := s.newConfig()
config.AddHostKey(private)
s.config.Store(config)
return nil
}
func (s *SSHServer) newConfig() *ssh.ServerConfig {
return &ssh.ServerConfig{
PublicKeyCallback: s.authenticate,
ServerVersion: fmt.Sprintf("SSH-2.0-Nebula???"),
}
}
// Addr is the address the newest Run listens on, nil when none is.
func (s *SSHServer) Addr() net.Addr {
if a := s.addr.Load(); a != nil {
return *a
}
return nil
}
// Running is true while a Run is listening.
func (s *SSHServer) Running() bool {
return s.runs.Load() > 0
}
func (s *SSHServer) ClearTrustedCAs() {
s.authLock.Lock()
s.trustedCAs = []ssh.PublicKey{}
@@ -160,11 +180,6 @@ func (s *SSHServer) AddAuthorizedKey(user, pubKey string) error {
return nil
}
// RegisterCommand adds a command that can be run by a user, by default only `help` is available
func (s *SSHServer) RegisterCommand(c *Command) {
s.commands.Insert(c.Name, c)
}
// Run begins listening and accepting connections. Each invocation derives a fresh per-Run context
// from the constructor-supplied ctx so a Stop+Run sequence (used by config reload) starts clean
// rather than carrying a permanently-cancelled context across runs.
@@ -181,10 +196,16 @@ func (s *SSHServer) Run(addr string) error {
// this run owns. They start equal but a fast reload may overwrite s.listener with the next run's
// listener before this run's watcher fires, so each run must close its own listener via the local
// reference.
s.listener = listener
s.listener.Store(&listener)
s.runs.Add(1)
bound := listener.Addr()
s.addr.Store(&bound)
runCtx, cancel := context.WithCancel(s.ctx)
defer cancel()
// Deferred after cancel so they run first, a run on its way out no longer counts as listening
defer s.runs.Add(-1)
defer s.addr.CompareAndSwap(&bound, nil)
// Close the listener when this run's context is cancelled. That can come from the parent
// (Control.Stop), from Run returning normally (defer cancel above), or transitively when a sibling
@@ -197,7 +218,7 @@ func (s *SSHServer) Run(addr string) error {
}
}()
s.l.Info("SSH server is listening", "sshListener", addr)
s.l.Info("SSH server is listening", "sshListener", bound)
// Run loops until there is an error
s.run(runCtx, listener)
@@ -226,7 +247,7 @@ func (s *SSHServer) run(ctx context.Context, listener net.Listener) {
<-sessionContext.Done()
c.Close()
}()
conn, chans, reqs, err := ssh.NewServerConn(c, s.config)
conn, chans, reqs, err := ssh.NewServerConn(c, s.config.Load())
fp := ""
if conn != nil {
fp = conn.Permissions.Extensions["fp"]
@@ -264,8 +285,8 @@ func (s *SSHServer) run(ctx context.Context, listener net.Listener) {
}
func (s *SSHServer) Stop() {
if s.listener != nil {
if err := s.listener.Close(); err != nil {
if l := s.listener.Load(); l != nil {
if err := (*l).Close(); err != nil && !errors.Is(err, net.ErrClosed) {
s.l.Warn("Failed to close the sshd listener", "error", err)
}
}
+37 -46
View File
@@ -1,13 +1,11 @@
package sshd
import (
"fmt"
"errors"
"log/slog"
"sort"
"strings"
"github.com/anmitsu/go-shlex"
"github.com/armon/go-radix"
"github.com/slackhq/nebula/commands"
"golang.org/x/crypto/ssh"
"golang.org/x/term"
)
@@ -16,31 +14,37 @@ type session struct {
l *slog.Logger
c *ssh.ServerConn
term *term.Terminal
commands *radix.Tree
commands *commands.Registry
cancel func()
}
func NewSession(commands *radix.Tree, conn *ssh.ServerConn, chans <-chan ssh.NewChannel, cancel func(), l *slog.Logger) *session {
func NewSession(reg *commands.Registry, conn *ssh.ServerConn, chans <-chan ssh.NewChannel, cancel func(), l *slog.Logger) *session {
s := &session{
commands: radix.NewFromMap(commands.ToMap()),
commands: reg.Clone(),
l: l,
c: conn,
cancel: cancel,
}
s.commands.Insert("logout", &Command{
Name: "logout",
ShortDescription: "Ends the current session",
Callback: func(a any, args []string, w StringWriter) error {
s.Close()
return nil
},
})
s.commands.Register(LogoutCommand(s.Close))
go s.handleChannels(chans)
return s
}
// LogoutCommand is the logout command a session registers on its clone of the registry,
// with end as what it does. Exported so a test can serve what a session serves.
func LogoutCommand(end func()) *commands.Command {
return &commands.Command{
Name: commands.Logout,
ShortDescription: "Ends the current session",
Callback: func(fs any, a []string, w commands.StringWriter) error {
end()
return nil
},
}
}
func (s *session) handleChannels(chans <-chan ssh.NewChannel) {
defer s.Close()
for newChannel := range chans {
@@ -87,9 +91,13 @@ func (s *session) handleRequests(in <-chan *ssh.Request, channel ssh.Channel) {
}
req.Reply(true, nil)
s.dispatchCommand(payload.Value, &stringWriter{channel})
err := s.dispatchCommand(payload.Value, commands.NewStringWriter(channel))
// A command that failed exits 1, so `ssh host cmd` callers can tell.
status := struct{ Status uint32 }{uint32(0)}
if err != nil {
status.Status = 1
}
channel.SendRequest("exit-status", false, ssh.Marshal(status))
channel.Close()
return
@@ -111,12 +119,11 @@ func (s *session) createTerm(channel ssh.Channel) *term.Terminal {
term.AutoCompleteCallback = func(line string, pos int, key rune) (newLine string, newPos int, ok bool) {
// key 9 is tab
if key == 9 {
cmds := matchCommand(s.commands, line)
cmds := s.commands.Match(line)
if len(cmds) == 1 {
return cmds[0] + " ", len(cmds[0]) + 1, true
}
sort.Strings(cmds)
term.Write([]byte(strings.Join(cmds, "\n") + "\n\n"))
}
@@ -128,7 +135,7 @@ func (s *session) createTerm(channel ssh.Channel) *term.Terminal {
}
func (s *session) handleInput() {
w := &stringWriter{w: s.term}
w := commands.NewStringWriter(s.term)
for {
line, err := s.term.ReadLine()
if err != nil {
@@ -139,36 +146,20 @@ func (s *session) handleInput() {
}
}
func (s *session) dispatchCommand(line string, w StringWriter) {
args, err := shlex.Split(line, true)
if err != nil {
return
// dispatchCommand runs line and returns the command's error, which it has logged unless
// the command already told the user everything: the log line is for the operator reading
// the daemon's log, who learns nothing from a failure the user caused.
func (s *session) dispatchCommand(line string, w commands.StringWriter) error {
err := s.commands.Dispatch(line, w)
if err == nil {
return nil
}
if len(args) == 0 {
dumpCommands(s.commands, w)
return
var f *commands.Failure
if !errors.As(err, &f) {
s.l.Debug("Command returned an error", "line", line, "error", err)
}
c, err := lookupCommand(s.commands, args[0])
if err != nil {
return
}
if c == nil {
err := w.WriteLine(fmt.Sprintf("did not understand: %s", line))
_ = err
dumpCommands(s.commands, w)
return
}
if checkHelpArgs(args) {
s.dispatchCommand(fmt.Sprintf("%s %s", "help", c.Name), w)
return
}
_ = execCommand(c, args[1:], w)
return err
}
func (s *session) Close() {
+281
View File
@@ -0,0 +1,281 @@
package nebula
import (
"crypto"
"crypto/ed25519"
"crypto/rand"
"crypto/rsa"
"encoding/pem"
"errors"
"fmt"
"net"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/slackhq/nebula/commands"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/sshd"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/crypto/ssh"
)
type sshdReloadHarness struct {
c *config.C
reg *commands.Registry
server *sshd.SSHServer
keyFile string
signer ssh.Signer
listen string
users []string
extra string
}
func newSSHDReloadHarness(t *testing.T) *sshdReloadHarness {
t.Helper()
l := test.NewLogger()
h := &sshdReloadHarness{keyFile: filepath.Join(t.TempDir(), "host_key"), listen: "127.0.0.1:0", users: []string{"test"}}
h.writeHostKey(t, newEd25519Key(t))
_, userKey, err := ed25519.GenerateKey(rand.Reader)
require.NoError(t, err)
h.signer, err = ssh.NewSignerFromKey(userKey)
require.NoError(t, err)
h.c = config.NewC(l)
require.NoError(t, h.c.LoadString(h.config()))
h.reg = commands.New()
h.server, err = sshd.NewSSHServer(t.Context(), l, h.reg)
require.NoError(t, err)
wireSSHReload(l, h.server, h.c)
run, err := configSSH(l, h.server, h.c, nil)
require.NoError(t, err)
go run()
t.Cleanup(h.server.Stop)
require.Eventually(t, h.server.Running, 5*time.Second, 10*time.Millisecond)
return h
}
func newEd25519Key(t *testing.T) crypto.Signer {
_, key, err := ed25519.GenerateKey(rand.Reader)
require.NoError(t, err)
return key
}
// writeHostKey replaces the host key file and returns the public key a client should now see
func (h *sshdReloadHarness) writeHostKey(t *testing.T, key crypto.Signer) ssh.PublicKey {
t.Helper()
block, err := ssh.MarshalPrivateKey(key, "")
require.NoError(t, err)
require.NoError(t, os.WriteFile(h.keyFile, pem.EncodeToMemory(block), 0o600))
pub, err := ssh.NewPublicKey(key.Public())
require.NoError(t, err)
return pub
}
func (h *sshdReloadHarness) config() string {
users := ""
for _, u := range h.users {
users += fmt.Sprintf(" - user: %s\n keys: [%q]\n", u, string(ssh.MarshalAuthorizedKey(h.signer.PublicKey())))
}
return fmt.Sprintf("sshd:\n enabled: true\n listen: %s\n host_key: %s\n authorized_users:\n%s%s",
h.listen, h.keyFile, users, h.extra)
}
// dial connects to wherever the server listens now, a restart on :0 moves it. hostKeyAlgorithms limits which host keys
// the client accepts, none means any
func (h *sshdReloadHarness) dial(hostKeyAlgorithms ...string) (*ssh.Client, ssh.PublicKey, error) {
addr := h.server.Addr()
if addr == nil {
return nil, nil, fmt.Errorf("not listening")
}
var hostKey ssh.PublicKey
c, err := ssh.Dial("tcp", addr.String(), &ssh.ClientConfig{
User: "test",
Auth: []ssh.AuthMethod{ssh.PublicKeys(h.signer)},
HostKeyCallback: func(_ string, _ net.Addr, key ssh.PublicKey) error {
hostKey = key
return nil
},
HostKeyAlgorithms: hostKeyAlgorithms,
Timeout: time.Second,
})
return c, hostKey, err
}
func (h *sshdReloadHarness) connect(t *testing.T) (*ssh.Client, ssh.PublicKey) {
t.Helper()
var client *ssh.Client
var hostKey ssh.PublicKey
require.Eventually(t, func() bool {
var err error
client, hostKey, err = h.dial()
return err == nil
}, 5*time.Second, 20*time.Millisecond)
t.Cleanup(func() { _ = client.Close() })
return client, hostKey
}
// dropped is true when the server closed the client's connection within wait
func dropped(client *ssh.Client, wait time.Duration) bool {
done := make(chan struct{})
go func() {
_ = client.Wait()
close(done)
}()
select {
case <-done:
return true
case <-time.After(wait):
return false
}
}
// Restarting sshd drops every open session, the one that asked for the reload included. A reload that touches nothing
// sshd reads leaves it alone
func TestSSHDReload_UnchangedKeepsSessions(t *testing.T) {
h := newSSHDReloadHarness(t)
client, _ := h.connect(t)
require.NoError(t, h.c.ReloadConfigString(h.config()+"punchy:\n punch: true\n").Err)
assert.False(t, dropped(client, 500*time.Millisecond), "an unchanged sshd was restarted")
}
// Every key the server reads restarts it when it changes
func TestSSHDReload_ChangedConfigRestarts(t *testing.T) {
cases := map[string]func(t *testing.T, h *sshdReloadHarness){
"listen": func(t *testing.T, h *sshdReloadHarness) { h.listen = "localhost:0" },
"authorized_users": func(t *testing.T, h *sshdReloadHarness) { h.users = append(h.users, "other") },
"trusted_cas": func(t *testing.T, h *sshdReloadHarness) { h.extra = " trusted_cas: []\n" },
"host_key": func(t *testing.T, h *sshdReloadHarness) {
h.keyFile = filepath.Join(t.TempDir(), "other_host_key")
h.writeHostKey(t, newEd25519Key(t))
},
}
for name, change := range cases {
t.Run(name, func(t *testing.T) {
h := newSSHDReloadHarness(t)
client, _ := h.connect(t)
change(t, h)
require.NoError(t, h.c.ReloadConfigString(h.config()).Err)
assert.True(t, dropped(client, 5*time.Second), "a changed sshd config was not applied")
require.Eventually(t, h.server.Running, 5*time.Second, 10*time.Millisecond)
h.connect(t)
})
}
}
// The config can stay the same while the host key file it names changes. New connections get the new key, open
// sessions are left alone
func TestSSHDReload_ChangedHostKeyFile(t *testing.T) {
h := newSSHDReloadHarness(t)
client, _ := h.connect(t)
want := h.writeHostKey(t, newEd25519Key(t))
require.NoError(t, h.c.ReloadConfigString(h.config()).Err)
_, got := h.connect(t)
assert.Equal(t, want.Marshal(), got.Marshal(), "a new host key was not applied")
assert.False(t, dropped(client, 500*time.Millisecond), "a new host key restarted sshd")
}
// A host key of another algorithm replaces the old one rather than being offered next to it
func TestSSHDReload_HostKeyAlgorithmChange(t *testing.T) {
h := newSSHDReloadHarness(t)
h.connect(t)
rsaKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
want := h.writeHostKey(t, rsaKey)
require.NoError(t, h.c.ReloadConfigString(h.config()).Err)
_, got := h.connect(t)
assert.Equal(t, want.Marshal(), got.Marshal())
_, _, err = h.dial(ssh.KeyAlgoED25519)
assert.Error(t, err, "the old ed25519 host key is still offered")
}
// A server that isn't running, say its port was busy, is tried again on every reload as before
func TestSSHDReload_NotRunningRestarts(t *testing.T) {
h := newSSHDReloadHarness(t)
h.server.Stop()
require.Eventually(t, func() bool { return !h.server.Running() }, 5*time.Second, 10*time.Millisecond)
require.NoError(t, h.c.ReloadConfigString(h.config()).Err)
require.Eventually(t, h.server.Running, 5*time.Second, 10*time.Millisecond)
h.connect(t)
}
// A host key that no longer loads stops sshd, the same as a config that doesn't
func TestSSHDReload_BadHostKeyFileStops(t *testing.T) {
h := newSSHDReloadHarness(t)
h.connect(t)
require.NoError(t, os.WriteFile(h.keyFile, []byte("not a key"), 0o600))
assert.True(t, h.c.ReloadConfigString(h.config()).Failed)
require.Eventually(t, func() bool { return !h.server.Running() }, 5*time.Second, 10*time.Millisecond)
}
// A trusted CA or authorized user that is skipped with a warning fails the reload, the valid ones are still used
func TestSSHDReload_SkippedEntriesFail(t *testing.T) {
cases := map[string]string{
"bad trusted ca": " trusted_cas: [nope]\n",
"user entry not a map": " - nope\n",
"user missing": " - keys: [nope]\n",
"bad key": " - user: other\n keys: nope\n",
"key in a list not a string": " - user: other\n keys: [5]\n",
"bad key in a list": " - user: other\n keys: [nope]\n",
"keys missing": " - user: other\n",
}
for name, extra := range cases {
t.Run(name, func(t *testing.T) {
h := newSSHDReloadHarness(t)
h.extra = extra
res := h.c.ReloadConfigString(h.config())
require.NoError(t, res.Err)
assert.True(t, res.Failed)
require.Eventually(t, h.server.Running, 5*time.Second, 10*time.Millisecond)
h.connect(t)
})
}
}
// TestSSHExecExitStatusReflectsTheCommand: an exec request exits 1 when the command
// returned an error and 0 otherwise, with the command's output either way.
func TestSSHExecExitStatusReflectsTheCommand(t *testing.T) {
h := newSSHDReloadHarness(t)
h.reg.Register(&commands.Command{
Name: "fails",
Callback: func(fs any, a []string, w commands.StringWriter) error {
_ = w.WriteLine("nope")
return errors.New("failed")
},
})
client, _ := h.connect(t)
sess, err := client.NewSession()
require.NoError(t, err)
out, err := sess.CombinedOutput("fails")
var exit *ssh.ExitError
require.ErrorAs(t, err, &exit)
assert.Equal(t, 1, exit.ExitStatus())
assert.Equal(t, "nope\n", string(out))
sess, err = client.NewSession()
require.NoError(t, err)
out, err = sess.CombinedOutput("help fails")
require.NoError(t, err)
assert.Equal(t, "fails - \n", string(out))
// A word that is not a command is a failure too, with the command list as before.
sess, err = client.NewSession()
require.NoError(t, err)
out, err = sess.CombinedOutput("nope")
require.ErrorAs(t, err, &exit)
assert.Equal(t, 1, exit.ExitStatus())
assert.True(t, strings.HasPrefix(string(out), "did not understand: nope\nAvailable commands:\n"), string(out))
}
Loaded 100 of 115 files, more files were not shown because too many files have changed in this diff. Show more