Compare commits

..
Author SHA1 Message Date
JackDoan 5bcceac53c linux: Don't replace another interface's route with the same destination and metric 2026-10-15 10:01:05 -05:00
John Maguire afbdc87ed9 Update gvisor and bump to go1.27 (#1940)
gvisor's arm64 race-build compare-and-swap (pkg/sync/race_arm64.s)
loaded the new value into the register meant for the old one (R1
instead of R2), so it never succeeded. Sleepers could not commit to
parking and spun: under -race on arm64 every idle service netstack
busied about 14 cores, and TestService's static host lookup of
localhost missed its 250ms timeout and stalled for the 30s static_map
cadence. Fixed in gvisor 1b2946c4fa.

Go 1.27.2 is now the minimum because gvisor's go branch requires it.
golangci-lint is bumped to v2.14, the first release that can read
go1.27.2 export data.
2026-10-14 21:30:59 -04:00
Nate Brown 94372f4f51 Don't delete routes we never installed on darwin and the BSDs (#1933) 2026-10-13 16:34:38 -05:00
John Maguire 42280f04ce darwin: Assign host address to the utun and route vpn explicitly (#1931)
Tailscale with an exit node treats every interface address prefix as an
attached LAN and either claims it or excludes it, and configd then rewrites
the matching route away from nebula's utun. Interfaces holding a single
address are skipped, so assign /32 and /128 and add the network route
ourselves for both families.
2026-10-13 13:50:37 -04:00
Nate Brown 11c1917387 Warn when an unsafe_route is already routed through another interface on darwin (#1925) 2026-10-12 16:57:14 -05:00
Nate Brown 5ba7aff4b3 ssh: Only restart sshd on reload when its config or host key changed (#1923) 2026-10-12 16:56:51 -05:00
44 changed files with 802 additions and 4389 deletions

No files matched your search

+3 -3
View File
@@ -14,7 +14,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: Build - name: Build
@@ -40,7 +40,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: Build - name: Build
@@ -83,7 +83,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
# GitHub holds ARNs, not credentials, and ARNs outlive a rotation # GitHub holds ARNs, not credentials, and ARNs outlive a rotation
+3 -3
View File
@@ -34,7 +34,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: add hashicorp source - name: add hashicorp source
@@ -66,7 +66,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: add hashicorp source - name: add hashicorp source
@@ -92,7 +92,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
# WSL2 + Ubuntu so the smoke can run a real linux peer with its own # WSL2 + Ubuntu so the smoke can run a real linux peer with its own
+2 -2
View File
@@ -22,7 +22,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: Smoke Docker - name: Smoke Docker
@@ -51,7 +51,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: build - name: build
+4 -4
View File
@@ -22,7 +22,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: Install goimports - name: Install goimports
@@ -42,7 +42,7 @@ jobs:
- name: golangci-lint - name: golangci-lint
uses: golangci/golangci-lint-action@v9 uses: golangci/golangci-lint-action@v9
with: with:
version: v2.12 version: v2.14
test: test:
name: Test ${{ matrix.name }} name: Test ${{ matrix.name }}
@@ -87,7 +87,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: Build - name: Build
@@ -132,7 +132,7 @@ jobs:
- uses: actions/setup-go@v7 - uses: actions/setup-go@v7
with: with:
go-version: '1.26' go-version: '1.27'
check-latest: true check-latest: true
- name: Build ${{ matrix.name }} - name: Build ${{ matrix.name }}
-131
View File
@@ -1,131 +0,0 @@
# Post-quantum handshake transport compared with QUIC
Status: analysis, 2026-10-14. Compares the fragment transport in
[pq-handshake.md](pq-handshake.md) with how QUIC makes its handshake reliable. TLS is out of
scope; only the transport mechanics are compared.
## Summary
QUIC's handshake reliability is closed-loop: the receiver acknowledges, and the sender
repairs exactly what is missing. Ours is open-loop: repair symbols are sent up front, and a
timer sends fresh symbols. At handshake sizes (K <= 15) a QUIC-style acknowledgement
shrinks to a 16-bit bitmap, and with an MDS code to a single count. So the real comparison
is forward error correction versus acknowledgements, not QUIC versus us.
FEC is the right core. One piece of feedback is worth adding, initiator to responder and for
msg2 only (see [Worth taking from QUIC](#worth-taking-from-quic)).
## What QUIC does
- **Messages as a stream.** Handshake bytes travel as `CRYPTO` frames (offset and length)
and are reassembled in order. Initial and Handshake packets each have their own packet
number space.
- **Immediate acknowledgements.** Receivers must acknowledge Initial and Handshake packets
without delay. ACK frames list ranges of received packet numbers.
- **Loss detection** (RFC 9002). A packet is lost once 3 later packets are acknowledged, or
after 9/8 of the round-trip time. Lost bytes are resent in new packets.
- **Probe timeout (PTO).** If a whole flight vanishes, a probe timer fires. Before any RTT
sample it starts at about 1 s (333 ms initial RTT plus 4x its variance) and doubles each
time. Both endpoints run it.
- **Amplification limit and anti-deadlock.** The server may send at most 3x what it
received until the client's address is validated. The client must keep probing so a
server stuck at that limit cannot deadlock.
- **Retry.** A stateless round trip to validate the address. `NEW_TOKEN` lets later
connections skip it.
- **Congestion control from the first packet.** The initial window is 10 packets of 1200
bytes.
## Side by side
| | QUIC | pq-handshake |
|---|---|---|
| Receiver feedback | ACK ranges, sent immediately | None; the next handshake message is the only acknowledgement |
| Partial loss | +1 to 2 RTT (ACK, detect, resend) | 0 RTT while at most m symbols are lost |
| Total loss | PTO: about 1 s, then doubling | Next PQ tick: 300 ms, then 1 s (alternating), or 100/300/600 ms (sticky) |
| Spurious retransmit | Wasted bytes | Still useful: fresh repair indexes add to what the receiver holds |
| Reassembly | Every byte offset, in order | Any K of up to 256 symbols |
| Who retransmits | Both sides on their own timers | Initiator on a timer; the responder resends msg2 only when a repeat msg1 arrives (rate limited) |
| State from the first packet | Server buffers and acknowledges partial flights | Capped pre-authentication table that never acknowledges |
| Address validation | Retry, 1 RTT | Cookie under load, 1 RTT |
From the [simulation results](pq-handshake.md#simulation-results): phase 2 at 5% random
loss with no FEC completes on the first attempt 39.8% of the time. Under
acknowledgement-based recovery roughly 60% of handshakes would pay at least one extra round
trip. With FEC, 0.4% wait for a retry. Spending bandwidth to save round trips is the stated
design preference, and open-loop recovery delivers it.
## Where the designs already agree
- **1200-byte floor and the 3x amplification budget**, both taken directly from QUIC.
- **Cookie and Retry.** Both cost one round trip and both can be forged by an on-path
attacker. QUIC's Retry integrity tag uses a fixed, published key.
- **On-path injection.** QUIC's Initial keys are derived from the client's connection ID,
which an observer can read, so QUIC accepts the same handshake-blocking injection
weakness documented for our symbols in
[Injected symbols](pq-handshake.md#injected-symbols).
- **Phase 2 msg3 confirmation.** Resending msg3 until any authenticated packet arrives,
with a `Test` from the responder on completion, is QUIC's handshake confirmation and
`HANDSHAKE_DONE`.
## Worth taking from QUIC
### 1. A msg2 progress probe from the initiator
This is the main gap. Today, recovering a partially lost msg2 means the initiator resends a
full msg1 attempt; the responder decodes it, matches it in the answered-msg1 cache, and
resends msg2. In phase 2 that is about 12 fragments (~13 KB) uploaded just to act as an
acknowledgement, and recovery waits for the 300 ms or 1 s tick.
- The initiator already knows the responder's index: msg2 fragments carry it as `FragID`
(`sendFragmentedResponse` passes `hostinfo.localIndexId`).
- A one-packet probe, "msg2: have n of K", sent to that index would recover in about one
round trip. The responder finds it the way phase 2 plans to find msg3.
- Timing comes for free: msg1 send to first msg2 fragment is an RTT sample. Probe at
roughly RTT/8 plus the repair delay after the first msg2 fragment if decoding has not
completed.
- Forging a probe needs the 32-bit responder index, so only an on-path attacker can, and
the existing 3x budget bounds the reply.
- If no msg2 fragment arrived at all, the initiator keeps today's msg1 resend. That is the
same tail-loss case QUIC's PTO covers.
The reverse direction, the responder reporting progress on msg1, should not be added. It
needs per-entry timers on the pre-authentication path, which is exactly the state QUIC
servers carry and the reason QUIC needs Retry.
### 2. Buffer early data in phase 2
QUIC stacks hold a few packets they cannot decrypt yet. The phase 2 plan drops data that
reaches the responder before msg3 decodes. Because repair fragments trail by
`handshakes.pq.repair_delay` (10 ms), losing one msg3 data fragment drops about 10 ms of
the initiator's first data while the repair is in flight. A small queue on the pending
responder entry, replayed once msg3 verifies, avoids this.
### 3. Burst size across remotes
QUIC handshakes on a single path. `handleOutbound` sends the same `now` and `later` sets to
every known remote and to relays (`StartRelays`).
- Phase 1 is fine.
- In phase 2, four remotes means about 48 packets (~54 KB) leaving one uplink at once,
every PQ tick, about 4x QUIC's initial window.
- The Gilbert-Elliott simulation treats each path independently, so it does not capture
this self-inflicted burst on a constrained uplink such as cellular.
- Options: send the full FEC set only to the most likely remote and a reduced set to the
others, or interleave packets across remotes.
## Not worth taking
- **Packet numbers, ACK ranges, an RTT estimator, loss thresholds, congestion control.**
QUIC pays for this machinery because it also serves a data transport. Nebula carries
packets and leaves reliability to the overlay's own transports. Building it only for
handshakes of at most 15 fragments buys nothing a symbol count cannot.
- **Offset-based reassembly.** It gives up the any-K-of-256 property and makes spurious
retransmits wasteful again.
- **An RTT-adaptive retry timer.** Low value. The fixed 100 ms linear schedule already fires
sooner than QUIC's 1 s first PTO, and a spurious resend costs bandwidth, not correctness.
## Conclusion
Keep FEC as the core and add the msg2 progress probe. That gets QUIC's roughly one round
trip tail recovery without pre-authentication state, and cuts phase 2's retry cost on the
uplink by about 10x. Items 2 and 3 are phase 2 concerns; item 1 also helps phase 1.
-348
View File
@@ -1,348 +0,0 @@
# Post-quantum handshakes
Status: draft. Decisions below were made on 2026-10-13; open items are listed at the end.
Phase 1 is implemented (see [Implementation](#implementation-phase-1)).
## Goals
1. **Confidentiality against harvest-now-decrypt-later.** Traffic recorded today must stay
secret after a cryptographically relevant quantum computer exists. This needs a
post-quantum key exchange now.
2. **Post-quantum authentication, later.** Needs certificates whose CA signature and host
keys are post-quantum. This requires reissuing certificates, so it ships second.
3. **Never rely on IP fragmentation.** ML-KEM and ML-DSA do not fit in one packet, so the
handshake must carry messages larger than the path MTU itself.
Non-goals: identity hiding beyond what IX offers today, and signatures at handshake time.
The only signature in the system stays the CA signature inside the certificate.
## Primitives and sizes
| Primitive | Size |
|---|---|
| ML-KEM-1024 encapsulation key | 1568 B |
| ML-KEM-1024 ciphertext | 1568 B |
| ML-DSA-87 public key / signature | 2592 B / 4627 B |
| Today's IX msg1 / msg2 (v2 cert, 151 B) | 252 B / 287 B |
`crypto/mlkem` is available in Go 1.26 (the current `go.mod`); `crypto/mldsa` arrives in
Go 1.27 and is only needed for phase 2.
## Phase 1: ephemeral ML-KEM on today's certificates
A new handshake subtype, `ix_psk2_mlkem1024` (2), IX-shaped (2 messages; the responder
completes on msg1 as today), using the flynn/noise `psk2` modifier. The Noise protocol name
becomes `Noise_IXpsk2_...`, which separates it from the classical subtype.
- **msg1**: the initiator generates a fresh ML-KEM-1024 key pair and places the
encapsulation key in `NebulaHandshakeDetails.KEMKey` (field 9). msg1's payload is mixed
into the Noise transcript, so it cannot be stripped or replaced without msg2 failing.
With the psk modifier Noise also mixes `e` into the cipher key, so msg1's static key and
payload are encrypted, though only under a key derived from the public ephemeral.
- **Responder**: after `ReadMessage(msg1)` it encapsulates to that key, obtaining
`(ct, ss)`, and calls `HandshakeState.SetPresharedKey(psk)` before writing msg2, with
`psk = HKDF-Extract(SHA-256, salt = "nebula ix_psk2_mlkem1024 v1", ikm = ss || ct)`.
The ciphertext is bound in because, unlike the encapsulation key, it is not part of the
Noise transcript.
- **msg2** on the wire is `ct || noise_msg2`. The ciphertext travels outside the Noise
message because the initiator needs `ss` before processing msg2's `psk` token. Any change
to `ct` yields a different `ss` (ML-KEM implicit rejection) and the handshake fails.
Transport keys depend on both the X25519 DH outputs and the ML-KEM shared secret.
Authentication is unchanged and classical. With a minimal v2 certificate msg1 and msg2 are
about 1.84 KB each, so each travels as 2 data and 2 repair fragments of about 936 bytes.
A KEM key on the classical subtype, or a missing or malformed one on this subtype, is a fatal
protocol error; a forged ciphertext only fails that packet.
## Phase 2: pqIX hybrid with certificate v3
Certificate v3 is hybrid: the classical static key and CA signature plus a static
ML-KEM-1024 key and an ML-DSA-87 CA signature. Host authentication is by KEM rather than by
signature, following PQNoise's pqIX, while keeping the classical DH tokens:
- **msg1** (I to R): `e, s`, ephemeral ML-KEM key, initiator cert.
- **msg2** (R to I): `e, s`, ciphertext to I's ephemeral KEM key, ciphertext to I's static
KEM key, responder cert.
- **msg3** (I to R): ciphertext to R's static KEM key, plus first data. R authenticates I
when msg3 decrypts; I authenticates R implicitly when R's first reply decrypts.
msg3 is about 1.6 KB, 2 data and 2 repair fragments, and adds no new message types:
- **Responder.** After sending msg2 the responder keeps a pending Machine, found by its
local index like the initiator's today; this is the multi-message responder support the
manager lacks now. A repeat msg1 maps to it through the answered-msg1 cache and gets msg2
resent with fresh symbols, exactly as in phase 1. Until msg3 verifies, data for that
index is dropped silently: it must not draw a RecvError, which would tear down the
initiator's new tunnel. On completing it sends an empty `Test`/`TestReply` under the
new keys, so the initiator hears back even when there is no traffic.
- **Initiator.** Writing msg3 yields the transport keys, so the hostinfo moves to the main
hostmap and data flows at once, readable only by the real responder. The handshake entry
stays behind in a confirming state: each retry tick resends msg3 with fresh symbols until
any packet from the responder decrypts on the new tunnel, and if that never happens
within the handshake timeout the tunnel is torn down. Losing msg3 costs the data sent
behind it, which the overlay's own transport retransmits.
- msg3's fragments use the responder's index as RemoteIndex and land on the responder's
pending entry, so they never enter the pre-authentication table.
The initiator never caches the responder's certificate, so every handshake has three
messages and carries both full certificates. With ML-DSA-87:
| Message | Size | Data fragments (k) |
|---|---|---|
| msg1 | ~8.2 KB | 8 |
| msg2 | ~9.8 KB | 9 |
| msg3 | ~1.6 KB | 2 |
flynn/noise cannot mix multiple KEM secrets into the chaining key, so phase 2 needs a
KEM-aware Noise symmetric state written in this repository. Both subtypes coexist.
Cryptographic review of the hybrid composition will be done by in-house cryptographers.
## Fragment transport
### Requirements
- No IP fragmentation: every datagram's UDP payload is at most 1232 bytes **after** relay
wrapping (16 B header + 16 B tag), the IPv6 minimum MTU budget also used by DNS Flag Day
2020 and Solana's Turbine.
- Every fragment of a handshake uses the same 5-tuple.
- Bounded pre-authentication state, bounded amplification, selective loss recovery,
tolerance of reordering and duplication, and a strictly validated, fuzzable parser.
### Wire format
Every packet of a fragmented handshake subtype is:
```
Nebula header (16) Version | Type=Handshake | Subtype | Reserved=0 | RemoteIndex | Counter
Fragment header (8) FragID u32 | MsgLen u16 | Index u8 | Flags u8
Cookie (16) only when Flags has 0x01
Symbol SymbolSize bytes
```
- `Counter` is the Noise message number (1, 2, 3). `RemoteIndex` is 0 for msg1, as today.
- `FragID` is the sender's local index (random, non-zero).
- `MsgLen` is the length of the whole message. Both ends derive the layout from it:
`K = ceil(MsgLen / 1160)`, `SymbolSize = ceil(MsgLen / K)`. Every symbol has the same
size. Room for a cookie is always reserved, so a packet is at most
16 + 8 + 16 + 1160 = 1200 bytes with or without one.
- `Index < K` is a data symbol (a slice of the message, the last one zero padded);
`Index >= K` is a repair symbol.
- `Flags` bit 0x01 means a cookie follows the fragment header. Other bits must be zero.
A cookie reply, sent by a responder under load, is a bare 32 byte packet:
```
Nebula header (16) Version | Type=Handshake | Subtype | Reserved=0 | RemoteIndex=FragID | Counter=0
Cookie (16)
```
It names the initiator's FragID, so the initiator finds its attempt the same way it finds
msg2.
### Forward error correction
Messages are protected with a systematic Reed-Solomon code over GF(2^8), implemented in
`handshake/fec`:
- The repair rows form a Cauchy matrix, `coefficient(row, col) = 1 / (row XOR col)` with
`row >= K > col`, so every square submatrix is invertible and **any K distinct symbols
reconstruct the message**. Naive identity-over-Vandermonde constructions are not MDS
(the PAR1 flaw and the 1997 Plank tutorial erratum).
- A repair row depends only on its index, so repair symbols are generated on demand and
their indexes never need to be agreed in advance. Up to 256 symbols exist per message.
- Known-answer vectors in `fec_test.go` pin the encoding and were cross-checked against an
independent implementation; changing them is a wire format change.
- Cost for a 9.8 KB message (k=9): about 13 µs for three repair symbols, about 19 µs to
decode with three data symbols missing.
**The sender chooses m**, the number of repair symbols per attempt. The receiver only needs
any K symbols, so this is policy and can change without a protocol change. The starting
rule is `m = max(2, ceil(K/3))`.
**Retries send fresh symbols.** Attempt 0 sends indexes `0 .. K+m-1`; each retry sends
the next `K+m` unused indexes. The receiver merges all distinct symbols across attempts,
so no received fragment is wasted. Indexes are a deterministic function of the message,
so retransmits stay idempotent: the responder caches msg2 itself and regenerates any
symbol.
**Repair symbols trail the data.** Data symbols go out back to back; repair symbols follow
after a short delay, so a queue tail-drop at the end of the data run does not also take the
repair. In simulation this lifts small messages substantially (k=2 at 10% loss, burst 8:
91.4% to 98.2% first-attempt success). Pacing every fragment was also evaluated; it gains
about 5 more points for k=7..9 on bursty paths, which is not worth the extra machinery,
and the retry with fresh symbols covers the remainder.
### Receiver rules
Implemented in `handshake/fragment` (`Sender`, `Assembly`, and the capped `Table`).
- Fragments are stored individually as they arrive and charged against a memory cap.
Nothing is allocated from a claimed size alone (FragmentSmack, game-engine "fragment
bombs").
- `MsgLen` must be identical for every fragment of a `FragID`; a mismatch drops the whole
entry. msg1 is capped at `MsgLen <= 16384` (K <= 15).
- The decoder keeps at most K distinct symbols, so an entry never exceeds about MsgLen bytes
however many repair symbols arrive.
- When K symbols are present the message is decoded and handed to the handshake `Machine`
as a synthetic `[header | message]`. The Machine never sees fragments.
- A completed entry is forgotten immediately. A late duplicate fragment starts a new entry
and, once enough arrive, reconstructs the same message again; the handshake layer's
duplicate detection recognises it.
- If Noise rejects a decoded message, the entry is discarded and the next attempt
supplies fresh symbols. (Possible refinement: keep a few spare symbols so an injected
bad symbol can be identified by re-encoding instead of waiting for a retry.)
- Pre-authentication msg1 entries live in a table keyed by (source address or relay index,
FragID) with a total memory cap, per-source quotas (/32 for IPv4, /64 for IPv6), a
sliding TTL of about 2 s refreshed by progress, and random eviction under pressure.
msg2 and msg3 reassembly attaches to the existing pending handshake found by RemoteIndex.
### Retransmission and amplification
- Only timers cause transmission. Reactive resends (the responder resending its cached
msg2 when a duplicate msg1 completes) are rate limited to one per handshake per retry
interval, to avoid TFTP's Sorcerer's Apprentice amplification.
- Duplicate msg1 detection compares the decoded message to the cached one **before**
running Noise or certificate verification.
- Before an address is validated, a responder sends at most 3x the bytes it received from
it (QUIC's rule). Normal handshakes sit near 1.1-1.2x.
- In phase 2 the initiator retransmits msg3 until any authenticated packet arrives from
the responder; the responder sends a small confirmation on completion and holds tunnel
traffic until msg3 verifies.
### Denial of service
The responder can no longer complete statelessly on msg1, since msg1 arrives in pieces.
Defenses are the capped table above plus a **cookie under load**, in the style of
WireGuard's mac2, QUIC Retry and the SCTP state cookie:
- Once the msg1 table is at least half full (by bytes or by entries), every msg1 fragment
must carry a valid cookie. One without is answered with a cookie reply and leaves no
state. Fragments that already belong to an entry need one too, so an attacker cannot
hold old entries open.
- A cookie is `HMAC-SHA256(secret, source address and port, relay channel, FragID)`
truncated to 16 bytes. The secret rotates every 2 minutes and the previous one is still
accepted, so a cookie is good for 2 to 4 minutes.
- Replies are never larger than the packet that prompted them (no amplification) and are
rate limited as a whole (a burst of 256, then 1000 per second), so spoofed floods cannot
turn the responder into a reflector.
- The initiator attaches the cookie to its PQ attempt, so every later packet carries it,
and immediately resends fresh symbols to the address the reply came from, at most once
per retry interval. Under load a PQ handshake costs one extra round trip, which in
practice still beats the classical attempt on the next retry tick.
- An attacker on the path can forge a cookie reply; the bogus cookie is simply rejected
and answered with a real one.
A CA-keyed pre-filter MAC was considered and rejected: networks commonly trust many CAs.
#### Injected symbols
The decoder keeps the first K distinct symbols it sees, so one forged symbol in those K
spoils that attempt's decode; Noise rejects the result and the next attempt brings fresh
symbols. Forging one needs the attempt's FragID or index, which are random 32 bit values,
and for msg1 the initiator's exact address and port too, so in practice only an attacker
who sees the traffic can do it. An attacker who can also drop packets gains nothing new.
One who can only inject, and races a forged symbol into every attempt, can stop a PQ
handshake where it could not stop a classical one, since classical msg2s are tried one
packet at a time. Keeping spare symbols to find a consistent subset only raises the number
of forgeries needed per attempt, and authenticating symbols is impossible for msg1, which
travels before any key exists. So the decoder deliberately does neither. A non-sticky
initiator still gets a classical tunnel from its next attempt; a sticky one, by design,
does not downgrade.
## Rollout
- Separate settings for initiating and accepting PQ handshakes, both off by default:
`handshakes.pq.initiate`, `handshakes.pq.accept`. `handshakes.pq.repair_delay` (default
10ms) sets how long repair fragments trail the data fragments.
- `handshakes.pq.require` makes every handshake PQ: it implies `initiate` and `accept`,
never offers a classical attempt, and drops classical msg1s.
- Old nodes silently drop unknown subtypes, so discovery is speculative: an initiator with
`initiate` enabled alternates PQ and classical attempts on successive timer retries,
keeping both pending, and the first to complete wins. A path nothing has been sent over
yet gets the PQ attempt first: an early resend triggered by a lighthouse reply is PQ, and
a relay that comes up pushes the PQ attempt through at once, counting as the PQ turn.
Without that, relays set up during the first attempt handed every relayed handshake to
the classical retry (seen on real hosts: 11 of 11). Once a msg2 fragment or cookie reply
arrives for the PQ attempt the peer evidently speaks PQ, and classical stops getting
turns.
- All `handshakes.pq` settings reload without a restart. Handshakes in flight carry on;
the new settings apply to the next attempt and the next packet.
- `handshakes.pq.sticky` (default off): once a peer has completed a PQ handshake, never fall
back to classical for it again during this process lifetime. This resists active
downgrade after first contact.
- No lighthouse capability advertisement.
## Simulation results
Gilbert-Elliott channel (bad state drops everything), losses expressed as average rate and
mean burst length in packets. First-attempt success for the whole handshake with
`m = max(2, ceil(K/3))` and repair delayed 16 packet slots, compared with no FEC:
| Handshake | Loss | Burst 1 | Burst 2 | Burst 4 | Burst 8 |
|---|---|---|---|---|---|
| phase 1 (2, 2) no FEC | 5% | 81.4% | 85.6% | 87.9% | 89.0% |
| phase 1 (2, 2) FEC | 5% | 99.9% | 99.4% | 99.3% | 98.6% |
| phase 1 (2, 2) FEC | 10% | 99.3% | 97.5% | 97.4% | 96.3% |
| phase 2 (8, 9, 2) no FEC | 5% | 37.8% | 56.0% | 69.3% | 77.1% |
| phase 2 (8, 9, 2) FEC | 5% | 99.6% | 93.7% | 89.9% | 88.4% |
| phase 2 (8, 9, 2) FEC | 10% | 95.3% | 83.5% | 77.9% | 76.7% |
With fresh repair symbols, phase 2 completes within two attempts at least 99.1% of the time
in every scenario above, and within three at least 99.98%.
The simulators and their outputs are in [pq-handshake-sim](pq-handshake-sim/README.md),
including a comparison with classical handshakes and a model of time to a working tunnel
under Nebula's retry schedule.
## Prior art
- **WireGuard / Rosenpass**: fit every message in one packet; cookies and biscuits for
statelessness. Not possible at our sizes.
- **IKEv2**: RFC 7383 fragments only after keys exist, with each fragment authenticated;
RFC 9242/9370 move PQ key exchange into intermediate exchanges for that reason.
- **QUIC / DTLS 1.3**: offset-based reassembly with ACKs; QUIC's 3x amplification limit and
1200-byte floor.
- **Solana Turbine**: Reed-Solomon over 1232-byte UDP packets through untrusted relays;
moved to Merkle-authenticated shreds after unauthenticated repair symbols enabled
poisoning.
- **kcp-go, UDPspeeder, CM256**: RS FEC over UDP; parity must scale up for small groups.
- **gQUIC FEC (removed)**: FEC does not pay for bulk transfer under bursty loss; it does
for short, latency-critical transfers such as handshakes.
- **TFTP, NFS duplicate request cache, memcached UDP, IP reassembly bugs**: retransmit
storms, ID-wrap misassembly, reflection amplification, and allocation from claimed sizes.
## Implementation (phase 1)
- `handshake`: the `ix_psk2_mlkem1024` subtype (`patterns.go`, `kem.go`), the `KEMKey`
payload field, and the psk2 plumbing in `Machine`.
- `handshake/fec`, `handshake/fragment`: the code and the fragment transport.
- `handshake_manager_pq.go` and the manager:
- **Initiator.** Attempts alternate PQ and classical, starting with PQ (sticky peers get
only PQ). The PQ attempt has its own local index registered alongside the classical
one; whichever response completes first wins, and the other index is released
(`settleIndex`). A timed out or deleted pending handshake releases both.
- **Responder.** msg1 fragments go into the capped pre-authentication table. A decoded
msg1 is looked up by hash among recently answered ones before any Noise, KEM or
certificate work: the trailing repair fragments of an attempt rebuild a msg1 its data
fragments already completed, and a retrying initiator offers fresh symbols for the
same msg1. A repeat goes straight to the cached-response resend, which offers fresh
msg2 symbols, is rate limited to one per retry interval and is held to the 3x
amplification budget.
- **Relays** carry fragments unchanged; repair fragments trail via a timer.
- The pre-authentication table is capped at 4 MiB, 1024 entries, 4 per source (IPv4 /32,
IPv6 /64, or relay channel), 16 KiB messages and a 3 s idle timeout. Cookies are
demanded from half full.
- Tests: Noise-level KEM tests, codec and fragment tests and fuzzers, manager unit tests
for the response budget and rate limit, and e2e tests for a clean handshake, loss of
data fragments in both directions, fallback to a peer that does not accept PQ, a lost
msg2 recovered by fresh symbols, sticky mode under an attacker dropping PQ packets,
relays, index cleanup on completion, fallback and timeout, and a responder flooded from
spoofed sources demanding and accepting a cookie. Unit tests cover cookie binding and
rotation, the never-amplify rule, the initiator's rate limited resend, and reload. ML-KEM and HKDF also
pass under `GOEXPERIMENT=boringcrypto` and `GODEBUG=fips140=on`.
## Open items
None for phase 1. Phase 2 needs certificate v3, the KEM-aware Noise symmetric state, and
the msg3 design above.
-476
View File
@@ -1,476 +0,0 @@
//go:build e2e_testing
package e2e
import (
"net/netip"
"sync"
"testing"
"time"
"github.com/slackhq/nebula"
"github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/e2e/router"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func pqRequire() m {
return m{"handshakes": m{"pq": m{"require": true, "repair_delay": "0s"}}}
}
func pqConfig(initiate, accept, sticky bool) m {
return m{"handshakes": m{"pq": m{
"initiate": initiate,
"accept": accept,
"sticky": sticky,
"repair_delay": "0s",
}}}
}
// handshakeOf returns the handshake header p carries, looking inside relay
// wrapping, plus the fragment header when the handshake is fragmented.
func handshakeOf(p *udp.Packet) (*header.H, *fragment.Header) {
b := p.Data
h := &header.H{}
if h.Parse(b) != nil {
return nil, nil
}
if h.Type == header.Message && h.Subtype == header.MessageRelay {
// Relays authenticate but do not encrypt the inner packet.
b = b[header.Len:]
if h.Parse(b) != nil {
return nil, nil
}
}
if h.Type != header.Handshake {
return nil, nil
}
if !header.IsFragmented(h.Subtype) {
return h, nil
}
// A relayed packet still has the relay's AEAD tag on the end.
for _, tag := range []int{0, 16} {
if len(b) < header.Len+tag {
break
}
if fh, _, err := fragment.Parse(b[header.Len : len(b)-tag]); err == nil {
return h, &fh
}
}
return h, nil
}
// handshakeTally counts handshake packets by style as the router moves them.
type handshakeTally struct {
sync.Mutex
pq, classical int
maxPQLen int
}
func (ht *handshakeTally) observe(p *udp.Packet) {
h, _ := handshakeOf(p)
if h == nil {
return
}
ht.Lock()
defer ht.Unlock()
if h.Subtype == header.HandshakeIXPSK2MLKEM1024 {
ht.pq++
ht.maxPQLen = max(ht.maxPQLen, len(p.Data))
} else {
ht.classical++
}
}
// isTunnelData reports whether p is ordinary tunnel traffic headed for to,
// directly or as the last hop of a relay.
func isTunnelData(p *udp.Packet, receiver, to *nebula.Control) bool {
h := &header.H{}
if receiver != to || h.Parse(p.Data) != nil || h.Type != header.Message {
return false
}
if h.Subtype == header.MessageRelay {
if h.Parse(p.Data[header.Len:]) != nil {
return false
}
}
return h.Type == header.Message && h.Subtype == header.MessageNone
}
func newPQPair(t *testing.T, mine, theirs m) (myControl, theirControl *nebula.Control, myVpn, theirVpn []netip.Prefix, myUdp, theirUdp netip.AddrPort, r *router.R) {
t.Helper()
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version2, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
myControl, myVpn, myUdp, _ = newSimpleServer(cert.Version2, ca, caKey, "me", "10.128.0.1/24", mine)
theirControl, theirVpn, theirUdp, _ = newSimpleServer(cert.Version2, ca, caKey, "them", "10.128.0.2/24", theirs)
myControl.InjectLightHouseAddr(theirVpn[0].Addr(), theirUdp)
r = router.NewR(t, myControl, theirControl)
myControl.Start()
theirControl.Start()
return
}
func TestPQHandshake(t *testing.T) {
t.Parallel()
// They only accept, which is all a PQ initiator needs.
myControl, theirControl, myVpn, theirVpn, myUdp, theirUdp, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Zero(t, tally.classical, "no classical handshake should be needed")
assert.GreaterOrEqual(t, tally.pq, 4, "both messages travel as several fragments")
assert.LessOrEqual(t, tally.maxPQLen, fragment.MaxPacketLen)
assertHostInfoPair(t, myUdp, theirUdp, myVpn, theirVpn, myControl, theirControl)
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
assert.Empty(t, myControl.ListHostmapIndexes(true), "no pending index may outlive the handshake")
assert.Empty(t, theirControl.ListHostmapIndexes(true))
myControl.Stop()
theirControl.Stop()
}
// TestPQHandshakeLoss drops data fragments of both messages; the repair
// fragments must carry the handshake without a retry.
func TestPQHandshakeLoss(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
var tally handshakeTally
dropped := 0
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
if h, fh := handshakeOf(p); fh != nil {
// Every data fragment of msg1, and the first of msg2.
if (h.MessageCounter == 1 && int(fh.Index) < 2) || (h.MessageCounter == 2 && fh.Index == 0) {
dropped++
return router.Drop
}
}
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Equal(t, 3, dropped)
assert.Zero(t, tally.classical, "the first attempt should succeed from repair fragments")
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
}
// TestPQFallback has a peer that does not accept PQ handshakes. The initiator's
// next attempt is classical and the tunnel comes up anyway.
func TestPQFallback(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, myUdp, theirUdp, r := newPQPair(t, pqConfig(true, true, false), nil)
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Positive(t, tally.pq, "a PQ attempt comes first")
assert.Positive(t, tally.classical, "then a classical one")
assertHostInfoPair(t, myUdp, theirUdp, myVpn, theirVpn, myControl, theirControl)
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
assert.Empty(t, myControl.ListHostmapIndexes(true), "the losing PQ attempt's index must be released")
myControl.Stop()
theirControl.Stop()
}
// TestPQResendFreshSymbols loses the responder's whole first msg2. The
// initiator's next PQ attempt offers fresh msg1 symbols, the responder
// recognises the repeat and resends msg2 with symbols it has not sent before.
func TestPQResendFreshSymbols(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
var firstMsg2, freshMsg2, freshMsg1 int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(10*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
h, fh := handshakeOf(p)
switch {
case h != nil && fh == nil:
// Keep the classical fallback out of the way.
return router.Drop
case fh != nil && h.MessageCounter == 2 && fh.Index < 4:
firstMsg2++
return router.Drop
case fh != nil && h.MessageCounter == 2:
freshMsg2++
case fh != nil && h.MessageCounter == 1 && fh.Index >= 4:
freshMsg1++
}
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Equal(t, 4, firstMsg2, "the first msg2 is 2 data and 2 repair fragments")
assert.Positive(t, freshMsg1, "the retry offers fresh msg1 symbols")
assert.Positive(t, freshMsg2, "the resend offers fresh msg2 symbols")
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
}
// TestPQSticky checks a sticky initiator never offers a classical handshake to a
// peer it has already completed a PQ handshake with.
func TestPQSticky(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqConfig(true, true, true), pqConfig(false, true, false))
defer r.RenderFlow()
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
theirControl.GetFromTun(true)
t.Log("Rehandshake while an attacker drops every PQ packet")
var tally handshakeTally
myControl.ReHandshake(theirVpn[0].Addr())
r.RouteForAllExitFuncOrTimeout(time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if h, _ := handshakeOf(p); h != nil && h.Subtype == header.HandshakeIXPSK2MLKEM1024 {
return router.Drop
}
return router.KeepRouting
})
assert.Positive(t, tally.pq, "PQ attempts keep being made")
assert.Zero(t, tally.classical, "sticky must never fall back")
myControl.Stop()
theirControl.Stop()
}
func TestPQRelays(t *testing.T) {
t.Parallel()
ca, _, caKey, _ := cert_test.NewTestCaCert(cert.Version1, cert.Curve_CURVE25519, time.Now(), time.Now().Add(10*time.Minute), nil, nil, []string{})
useRelays := pqConfig(true, true, false)
useRelays["relay"] = m{"use_relays": true}
amRelay := pqConfig(true, true, false)
amRelay["relay"] = m{"am_relay": true}
myControl, myVpnIpNet, _, _ := newSimpleServer(cert.Version1, ca, caKey, "me ", "10.128.0.1/24", useRelays)
relayControl, relayVpnIpNet, relayUdpAddr, _ := newSimpleServer(cert.Version1, ca, caKey, "relay ", "10.128.0.128/24", amRelay)
theirControl, theirVpnIpNet, theirUdpAddr, _ := newSimpleServer(cert.Version1, ca, caKey, "them ", "10.128.0.2/24", useRelays)
myControl.InjectLightHouseAddr(relayVpnIpNet[0].Addr(), relayUdpAddr)
myControl.InjectRelays(theirVpnIpNet[0].Addr(), []netip.Addr{relayVpnIpNet[0].Addr()})
relayControl.InjectLightHouseAddr(theirVpnIpNet[0].Addr(), theirUdpAddr)
r := router.NewR(t, myControl, relayControl, theirControl)
defer r.RenderFlow()
myControl.Start()
relayControl.Start()
theirControl.Start()
var tally handshakeTally
var relayedPQ, relayedClassical int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpnIpNet[0].Addr(), 80, myVpnIpNet[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(10*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
outer := &header.H{}
if outer.Parse(p.Data) == nil && outer.Type == header.Message && outer.Subtype == header.MessageRelay {
if h, fh := handshakeOf(p); h != nil && fh != nil {
relayedPQ++
assert.LessOrEqual(t, len(p.Data), fragment.MaxPacketLen+32, "a relayed fragment fits the IPv6 minimum MTU")
} else if h != nil {
relayedClassical++
}
}
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpnIpNet[0].Addr(), theirVpnIpNet[0].Addr(), 80, 80)
assert.Positive(t, relayedPQ, "the end to end handshake is PQ through the relay")
assert.Zero(t, relayedClassical, "the relay coming up carries the PQ attempt, not the classical retry")
r.RenderHostmaps("Final hostmaps", myControl, relayControl, theirControl)
myControl.Stop()
relayControl.Stop()
theirControl.Stop()
}
// TestPQTimeoutReleasesIndexes lets a handshake with both a PQ and a classical
// attempt pending time out, and checks neither index is left behind.
func TestPQTimeoutReleasesIndexes(t *testing.T) {
t.Parallel()
mine := pqConfig(true, true, false)
mine["handshakes"].(m)["retries"] = 2
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, mine, pqConfig(false, true, false))
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
r.RouteForAllExitFuncOrTimeout(1500*time.Millisecond, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
return router.Drop
})
assert.Positive(t, tally.pq)
assert.Positive(t, tally.classical)
assert.Nil(t, myControl.GetHostInfoByVpnAddr(theirVpn[0].Addr(), true), "the handshake should have timed out")
assert.Empty(t, myControl.ListHostmapIndexes(true))
myControl.Stop()
theirControl.Stop()
}
// TestPQCookieUnderLoad floods the responder with msg1 fragments from spoofed
// sources until it demands cookies, then checks a real initiator picks up the
// cookie and still completes a PQ handshake.
func TestPQCookieUnderLoad(t *testing.T) {
t.Parallel()
myControl, theirControl, myVpn, theirVpn, _, theirUdp, r := newPQPair(t, pqConfig(true, true, false), pqConfig(false, true, false))
defer r.RenderFlow()
// Half of the 1024 entry table is the cookie threshold. Stopping exactly there
// keeps them from answering the spoofed sources, which the router cannot reach.
t.Log("Flood them with the first fragment of msg1s nobody will finish")
for i := range 512 {
from := netip.AddrPortFrom(netip.AddrFrom4([4]byte{198, 18, byte(i >> 8), byte(i)}), 4242)
s, err := fragment.NewSender(make([]byte, 2000), header.HandshakeIXPSK2MLKEM1024, 0, 1, uint32(i+1))
require.NoError(t, err)
pkts, _, err := s.Attempt(0)
require.NoError(t, err)
theirControl.InjectUDPPacket(&udp.Packet{From: from, To: theirUdp, Data: pkts[0]})
}
var tally handshakeTally
var cookieReplies, msg2, withCookie int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if h, fh := handshakeOf(p); h != nil && h.Subtype == header.HandshakeIXPSK2MLKEM1024 {
switch {
case h.MessageCounter == 0:
cookieReplies++
case h.MessageCounter == 2:
msg2++
case fh != nil && fh.HasCookie():
withCookie++
}
}
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Positive(t, cookieReplies, "a loaded responder asks for a cookie")
assert.Positive(t, withCookie, "the initiator resends with it")
assert.Positive(t, msg2, "and the handshake completes post-quantum")
assert.Zero(t, tally.classical, "the cookie round trip beats the classical fallback")
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
}
// TestPQRequire checks require mode never handshakes classically, in either
// direction.
func TestPQRequire(t *testing.T) {
t.Parallel()
t.Run("initiator", func(t *testing.T) {
// They neither speak PQ nor require it; we never fall back.
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqRequire(), nil)
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
r.RouteForAllExitFuncOrTimeout(time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
return router.KeepRouting
})
assert.Positive(t, tally.pq)
assert.Zero(t, tally.classical)
assert.Nil(t, theirControl.GetHostInfoByVpnAddr(myVpn[0].Addr(), false), "no tunnel without PQ")
myControl.Stop()
theirControl.Stop()
})
t.Run("responder", func(t *testing.T) {
// We only speak classical; they refuse it.
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, nil, pqRequire())
defer r.RenderFlow()
var answers int
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
r.RouteForAllExitFuncOrTimeout(time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
if h, _ := handshakeOf(p); h != nil && h.MessageCounter == 2 {
answers++
}
return router.KeepRouting
})
assert.Zero(t, answers, "a classical msg1 gets no answer")
assert.Nil(t, theirControl.GetHostInfoByVpnAddr(myVpn[0].Addr(), false))
myControl.Stop()
theirControl.Stop()
})
t.Run("both", func(t *testing.T) {
myControl, theirControl, myVpn, theirVpn, _, _, r := newPQPair(t, pqRequire(), pqRequire())
defer r.RenderFlow()
var tally handshakeTally
myControl.InjectTunPacket(BuildTunUDPPacket(theirVpn[0].Addr(), 80, myVpn[0].Addr(), 80, []byte("Hi from me")))
require.True(t, r.RouteForAllExitFuncOrTimeout(5*time.Second, func(p *udp.Packet, receiver *nebula.Control) router.ExitType {
tally.observe(p)
if isTunnelData(p, receiver, theirControl) {
return router.RouteAndExit
}
return router.KeepRouting
}), "tunnel never came up")
assertUdpPacket(t, []byte("Hi from me"), theirControl.GetFromTun(true), myVpn[0].Addr(), theirVpn[0].Addr(), 80, 80)
assert.Zero(t, tally.classical)
assertTunnel(t, myVpn[0].Addr(), theirVpn[0].Addr(), myControl, theirControl, r)
myControl.Stop()
theirControl.Stop()
})
}
-18
View File
@@ -418,24 +418,6 @@ logging:
# after receiving the response for lighthouse queries # after receiving the response for lighthouse queries
#trigger_buffer: 64 #trigger_buffer: 64
# pq enables handshakes that mix an ephemeral ML-KEM-1024 key exchange into the session keys, protecting
# recorded traffic against a future quantum computer. Authentication is unchanged. These handshakes are too
# large for one packet and travel as fragments with forward error correction.
#pq:
# initiate starts post-quantum handshakes. Attempts alternate between post-quantum and classical until one is
# answered, so peers that do not support them still connect.
#initiate: false
# accept answers post-quantum handshakes from peers.
#accept: false
# require makes every handshake post-quantum: it implies initiate and accept, never offers a classical handshake
# and refuses classical handshakes from peers. Only enable it once every host can speak post-quantum handshakes.
#require: false
# sticky never falls back to a classical handshake with a peer that has completed a post-quantum one since
# nebula started, which defeats an attacker who drops post-quantum handshakes to force a downgrade.
#sticky: false
# repair_delay is how long the forward error correction fragments trail the rest of a handshake message.
#repair_delay: 10ms
# Tunnel manager settings # Tunnel manager settings
#tunnels: #tunnels:
# drop_inactive controls whether inactive tunnels are maintained or dropped after the inactive_timeout period has # drop_inactive controls whether inactive tunnels are maintained or dropped after the inactive_timeout period has
+4 -3
View File
@@ -1,6 +1,6 @@
module github.com/slackhq/nebula module github.com/slackhq/nebula
go 1.26.0 go 1.27.2
require ( require (
dario.cat/mergo v1.0.2 dario.cat/mergo v1.0.2
@@ -33,7 +33,7 @@ require (
golang.zx2c4.com/wireguard/windows v1.0.1 golang.zx2c4.com/wireguard/windows v1.0.1
google.golang.org/protobuf v1.36.12 google.golang.org/protobuf v1.36.12
gopkg.in/yaml.v3 v3.0.1 gopkg.in/yaml.v3 v3.0.1
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980
) )
require ( require (
@@ -45,5 +45,6 @@ require (
github.com/prometheus/common v0.70.1 // indirect github.com/prometheus/common v0.70.1 // indirect
github.com/prometheus/procfs v0.21.1 // indirect github.com/prometheus/procfs v0.21.1 // indirect
github.com/vishvananda/netns v0.0.5 // indirect github.com/vishvananda/netns v0.0.5 // indirect
golang.org/x/time v0.5.0 // indirect golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc // indirect
golang.org/x/time v0.15.0 // indirect
) )
+6 -4
View File
@@ -157,6 +157,8 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA= golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc h1:TS73t7x3KarrNd5qAipmspBDS1rkMcgVG/fS1aRb4Rc=
golang.org/x/exp v0.0.0-20250711185948-6ae5c78190dc/go.mod h1:A+z0yzpGtvnG90cToK5n2tu8UJVP2XUATh+r+sfOOOc=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
@@ -203,8 +205,8 @@ golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
@@ -243,5 +245,5 @@ gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe h1:fre4i6mv4iBuz5lCMOzHD1rH1ljqHWSICFmZRbbgp3g= gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980 h1:ribyrTYQ/QYeUIs8x8bVLYwGyepa7RnHLqJzegT5YYk=
gvisor.dev/gvisor v0.0.0-20240423190808-9d7a357edefe/go.mod h1:sxc3Uvk/vHcd3tj7/DHVBoR5wvWT/MmRq2pj7HRJnwU= gvisor.dev/gvisor v0.0.0-20261014193016-04d31b385980/go.mod h1:pK/dapF5Bf3KGW7/kd5CI5IC4lYh9FMFY1qETAIGLXk=
+2 -4
View File
@@ -35,9 +35,7 @@ func NewCredential(
} }
// buildHandshakeState creates a noise.HandshakeState from this credential. // buildHandshakeState creates a noise.HandshakeState from this credential.
// pskPlacement selects the noise psk modifier; 0 means none. A psk for func (hc *Credential) buildHandshakeState(initiator bool, pattern noise.HandshakePattern) (*noise.HandshakeState, error) {
// placements 2 and up is supplied later with SetPresharedKey.
func (hc *Credential) buildHandshakeState(initiator bool, pattern noise.HandshakePattern, pskPlacement int) (*noise.HandshakeState, error) {
return noise.NewHandshakeState(noise.Config{ return noise.NewHandshakeState(noise.Config{
CipherSuite: hc.cipherSuite, CipherSuite: hc.cipherSuite,
Random: rand.Reader, Random: rand.Reader,
@@ -45,7 +43,7 @@ func (hc *Credential) buildHandshakeState(initiator bool, pattern noise.Handshak
Initiator: initiator, Initiator: initiator,
StaticKeypair: noise.DHKey{Private: hc.privateKey, Public: hc.Cert.PublicKey()}, StaticKeypair: noise.DHKey{Private: hc.privateKey, Public: hc.Cert.PublicKey()},
PresharedKey: []byte{}, PresharedKey: []byte{},
PresharedKeyPlacement: pskPlacement, PresharedKeyPlacement: 0,
}) })
} }
-216
View File
@@ -1,216 +0,0 @@
// Package fec implements the systematic Reed-Solomon erasure code used to carry
// handshake messages that do not fit in a single UDP packet.
//
// A message is split into K equally sized data symbols (the last one zero
// padded). Symbol indexes 0..K-1 are the data symbols themselves; indexes
// K..MaxSymbols-1 are repair symbols. Any K distinct symbols reconstruct the
// message. Repair symbols are generated on demand, so a sender can emit fresh
// ones on every retransmit and a receiver can merge everything it has seen.
package fec
import (
"errors"
"fmt"
)
// MaxSymbols is the number of distinct symbols, data and repair together, one
// message can be encoded into. It is bounded by the size of GF(2^8).
const MaxSymbols = 256
var (
ErrInvalidLayout = errors.New("invalid fec layout")
ErrIndexRange = errors.New("symbol index out of range")
ErrSymbolSize = errors.New("symbol has the wrong size")
ErrNotEnough = errors.New("not enough symbols to decode")
ErrCorrupt = errors.New("decoded message has non-zero padding")
)
// Layout describes how a message is split into symbols. Both ends derive it
// from the message length alone, so it never has to be sent.
type Layout struct {
MsgLen int // length of the message in bytes
K int // number of data symbols
SymbolSize int // size of every symbol, data and repair
}
// NewLayout returns the layout for a message of msgLen bytes carried in symbols
// of at most maxSymbolSize bytes. Symbols are balanced, so padding is always
// less than K bytes.
func NewLayout(msgLen, maxSymbolSize int) (Layout, error) {
if msgLen <= 0 || maxSymbolSize <= 0 {
return Layout{}, ErrInvalidLayout
}
k := (msgLen + maxSymbolSize - 1) / maxSymbolSize
if k >= MaxSymbols {
return Layout{}, fmt.Errorf("%w: %d data symbols", ErrInvalidLayout, k)
}
return Layout{
MsgLen: msgLen,
K: k,
SymbolSize: (msgLen + k - 1) / k,
}, nil
}
// Encoder produces symbols for one message.
type Encoder struct {
layout Layout
data [][]byte
}
// NewEncoder copies msg into padded data symbols.
func NewEncoder(msg []byte, maxSymbolSize int) (*Encoder, error) {
l, err := NewLayout(len(msg), maxSymbolSize)
if err != nil {
return nil, err
}
buf := make([]byte, l.K*l.SymbolSize)
copy(buf, msg)
data := make([][]byte, l.K)
for i := range data {
data[i] = buf[i*l.SymbolSize : (i+1)*l.SymbolSize]
}
return &Encoder{layout: l, data: data}, nil
}
func (e *Encoder) Layout() Layout {
return e.layout
}
// Symbol appends the symbol with the given index to dst and returns the result.
func (e *Encoder) Symbol(dst []byte, index int) ([]byte, error) {
if index < 0 || index >= MaxSymbols {
return nil, ErrIndexRange
}
if index < e.layout.K {
return append(dst, e.data[index]...), nil
}
start := len(dst)
dst = append(dst, make([]byte, e.layout.SymbolSize)...)
out := dst[start:]
for col, d := range e.data {
mulAdd(out, d, coefficient(index, col))
}
return dst, nil
}
// Decoder collects symbols for one message until it can reconstruct it.
// It stores at most K symbols, so its memory is bounded by the layout no matter
// how many symbols arrive.
type Decoder struct {
layout Layout
have [MaxSymbols]bool
syms map[int][]byte
}
// NewDecoder returns a decoder for a message of msgLen bytes that was encoded
// with the same maxSymbolSize.
func NewDecoder(msgLen, maxSymbolSize int) (*Decoder, error) {
l, err := NewLayout(msgLen, maxSymbolSize)
if err != nil {
return nil, err
}
return &Decoder{layout: l, syms: make(map[int][]byte, l.K)}, nil
}
func (d *Decoder) Layout() Layout {
return d.layout
}
// Stored returns the number of symbols held.
func (d *Decoder) Stored() int {
return len(d.syms)
}
// Has reports whether the symbol with the given index is held.
func (d *Decoder) Has(index int) bool {
return index >= 0 && index < MaxSymbols && d.have[index]
}
// Ready reports whether enough symbols have arrived to decode.
func (d *Decoder) Ready() bool {
return len(d.syms) >= d.layout.K
}
// Add stores a copy of the symbol. Duplicates, and anything beyond the first K
// distinct symbols, are ignored. It returns whether the decoder is ready.
func (d *Decoder) Add(index int, sym []byte) (bool, error) {
if index < 0 || index >= MaxSymbols {
return d.Ready(), ErrIndexRange
}
if len(sym) != d.layout.SymbolSize {
return d.Ready(), ErrSymbolSize
}
if d.have[index] || d.Ready() {
return d.Ready(), nil
}
d.have[index] = true
d.syms[index] = append([]byte(nil), sym...)
return d.Ready(), nil
}
// Decode reconstructs the message from the stored symbols.
func (d *Decoder) Decode() ([]byte, error) {
if !d.Ready() {
return nil, ErrNotEnough
}
l := d.layout
out := make([]byte, l.K*l.SymbolSize)
var missing, repair []int
for i := range l.K {
if s, ok := d.syms[i]; ok {
copy(out[i*l.SymbolSize:], s)
} else {
missing = append(missing, i)
}
}
if len(missing) > 0 {
for i := l.K; i < MaxSymbols && len(repair) < len(missing); i++ {
if d.have[i] {
repair = append(repair, i)
}
}
if len(repair) < len(missing) {
return nil, ErrNotEnough
}
// Each repair symbol r satisfies r = sum_col coefficient(r, col) * data[col].
// Move the known data columns to the left hand side, leaving a square
// Cauchy system in the missing columns.
rhs := make([][]byte, len(repair))
m := make([][]byte, len(repair))
for ri, row := range repair {
rhs[ri] = append([]byte(nil), d.syms[row]...)
m[ri] = make([]byte, len(missing))
mi := 0
for col := range l.K {
if mi < len(missing) && missing[mi] == col {
m[ri][mi] = coefficient(row, col)
mi++
continue
}
mulAdd(rhs[ri], out[col*l.SymbolSize:(col+1)*l.SymbolSize], coefficient(row, col))
}
}
if !invert(m) {
// Unreachable for a Cauchy matrix; kept so a broken invariant fails loudly.
return nil, errors.New("fec: singular decode matrix")
}
for mi, col := range missing {
dst := out[col*l.SymbolSize : (col+1)*l.SymbolSize]
for ri := range repair {
mulAdd(dst, rhs[ri], m[mi][ri])
}
}
}
for _, b := range out[l.MsgLen:] {
if b != 0 {
return nil, ErrCorrupt
}
}
return out[:l.MsgLen], nil
}
-318
View File
@@ -1,318 +0,0 @@
package fec
import (
"crypto/sha256"
"encoding/hex"
"math/rand/v2"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// slowMul is carry-less multiplication reduced by gfPoly, used to check the tables.
func slowMul(a, b byte) byte {
var p int
x, y := int(a), int(b)
for y > 0 {
if y&1 != 0 {
p ^= x
}
x <<= 1
if x&0x100 != 0 {
x ^= gfPoly
}
y >>= 1
}
return byte(p)
}
func TestGF(t *testing.T) {
for a := range 256 {
for b := range 256 {
require.Equal(t, slowMul(byte(a), byte(b)), gfMul(byte(a), byte(b)), "%d*%d", a, b)
}
if a != 0 {
require.Equal(t, byte(1), gfMul(byte(a), gfInv(byte(a))), "inv(%d)", a)
}
}
src := make([]byte, 256)
for i := range src {
src[i] = byte(i)
}
for c := range 256 {
dst := make([]byte, 256)
mulAdd(dst, src, byte(c))
for i := range src {
require.Equal(t, gfMul(byte(c), src[i]), dst[i])
}
}
}
func TestLayout(t *testing.T) {
for msgLen := 1; msgLen <= 20000; msgLen++ {
l, err := NewLayout(msgLen, 1176)
require.NoError(t, err)
require.Equal(t, (msgLen+1175)/1176, l.K)
require.LessOrEqual(t, l.SymbolSize, 1176)
require.GreaterOrEqual(t, l.K*l.SymbolSize, msgLen)
require.Less(t, l.K*l.SymbolSize-msgLen, l.K, "padding must be less than one byte per symbol")
}
l, err := NewLayout(65535, 1176)
require.NoError(t, err)
assert.Equal(t, 56, l.K)
_, err = NewLayout(0, 1176)
require.ErrorIs(t, err, ErrInvalidLayout)
_, err = NewLayout(100, 0)
require.ErrorIs(t, err, ErrInvalidLayout)
_, err = NewLayout(256, 1)
require.ErrorIs(t, err, ErrInvalidLayout)
_, err = NewLayout(255, 1)
require.NoError(t, err)
}
func randMsg(r *rand.Rand, n int) []byte {
b := make([]byte, n)
for i := range b {
b[i] = byte(r.Uint32())
}
return b
}
func decodeWith(t *testing.T, enc *Encoder, msgLen, maxSym int, indexes []int) []byte {
t.Helper()
dec, err := NewDecoder(msgLen, maxSym)
require.NoError(t, err)
for _, idx := range indexes {
sym, err := enc.Symbol(nil, idx)
require.NoError(t, err)
_, err = dec.Add(idx, sym)
require.NoError(t, err)
}
require.True(t, dec.Ready())
out, err := dec.Decode()
require.NoError(t, err)
return out
}
// TestExhaustiveSubsets decodes every k-subset of the first k+m symbols for
// small codes, which checks the MDS property directly.
func TestExhaustiveSubsets(t *testing.T) {
r := rand.New(rand.NewPCG(1, 2))
const maxSym = 16
for k := 1; k <= 10; k++ {
for m := 1; m <= 5; m++ {
msgLen := (k-1)*maxSym + 1 + r.IntN(maxSym)
msg := randMsg(r, msgLen)
enc, err := NewEncoder(msg, maxSym)
require.NoError(t, err)
require.Equal(t, k, enc.Layout().K)
n := k + m
for mask := 0; mask < 1<<n; mask++ {
var idx []int
for i := range n {
if mask&(1<<i) != 0 {
idx = append(idx, i)
}
}
if len(idx) != k {
continue
}
require.Equal(t, msg, decodeWith(t, enc, msgLen, maxSym, idx), "k=%d m=%d idx=%v", k, m, idx)
}
}
}
}
// TestRandomHighIndexes draws symbols from the whole index space, the shape
// retransmits produce once fresh repair symbols are in play.
func TestRandomHighIndexes(t *testing.T) {
r := rand.New(rand.NewPCG(3, 4))
for range 2000 {
msgLen := 1 + r.IntN(65535)
msg := randMsg(r, msgLen)
enc, err := NewEncoder(msg, 1176)
require.NoError(t, err)
k := enc.Layout().K
idx := r.Perm(MaxSymbols)[:k]
require.Equal(t, msg, decodeWith(t, enc, msgLen, 1176, idx), "k=%d idx=%v", k, idx)
}
}
func TestDecoderBounds(t *testing.T) {
r := rand.New(rand.NewPCG(5, 6))
msg := randMsg(r, 3000)
enc, err := NewEncoder(msg, 1176)
require.NoError(t, err)
l := enc.Layout()
require.Equal(t, 3, l.K)
require.Equal(t, 1000, l.SymbolSize)
dec, err := NewDecoder(3000, 1176)
require.NoError(t, err)
_, err = dec.Decode()
require.ErrorIs(t, err, ErrNotEnough)
_, err = dec.Add(-1, make([]byte, 1000))
require.ErrorIs(t, err, ErrIndexRange)
_, err = dec.Add(MaxSymbols, make([]byte, 1000))
require.ErrorIs(t, err, ErrIndexRange)
_, err = dec.Add(0, make([]byte, 999))
require.ErrorIs(t, err, ErrSymbolSize)
s5, _ := enc.Symbol(nil, 5)
ready, err := dec.Add(5, s5)
require.NoError(t, err)
require.False(t, ready)
// A duplicate index is ignored, even with different contents.
ready, err = dec.Add(5, make([]byte, 1000))
require.NoError(t, err)
require.False(t, ready)
s0, _ := enc.Symbol(nil, 0)
s9, _ := enc.Symbol(nil, 9)
_, err = dec.Add(0, s0)
require.NoError(t, err)
ready, err = dec.Add(9, s9)
require.NoError(t, err)
require.True(t, ready)
// Symbols beyond K are not stored.
ready, err = dec.Add(1, make([]byte, 1000))
require.NoError(t, err)
require.True(t, ready)
require.Len(t, dec.syms, 3)
out, err := dec.Decode()
require.NoError(t, err)
require.Equal(t, msg, out)
}
func TestCorruptPadding(t *testing.T) {
// 3001 bytes in 3 symbols of 1001 leaves 2 bytes of padding in the last one.
dec, err := NewDecoder(3001, 1176)
require.NoError(t, err)
require.Equal(t, 1001, dec.Layout().SymbolSize)
last := make([]byte, 1001)
last[1000] = 1
for i, s := range [][]byte{make([]byte, 1001), make([]byte, 1001), last} {
_, err = dec.Add(i, s)
require.NoError(t, err)
}
_, err = dec.Decode()
require.ErrorIs(t, err, ErrCorrupt)
}
// TestKnownAnswer pins the encoding. Both ends of a handshake must produce
// identical repair symbols, so any change here is a wire format change.
func TestKnownAnswer(t *testing.T) {
msg := make([]byte, 3000)
for i := range msg {
msg[i] = byte(i*7 + 3)
}
enc, err := NewEncoder(msg, 1176)
require.NoError(t, err)
// Cross-checked against an independent bit-serial GF(2^8) implementation.
want := map[int]string{
3: "52124f58276ec6338d3b3f4eed273bc7cabfe4f0f467928962b046ad1b348f50",
4: "a50928b04738981985a7d4a17ffaf8d7e30014ca64fc53c99800ffb61aec4aa7",
5: "ada4c95a7facbc542aaef584e6f0d081b7b0d0eed69ded03b57d0753a119cfc9",
255: "839066cdf94f963b223fb1c1c1360f698c14f6349b25653ff5b521791a4021b1",
}
for idx, h := range want {
sym, err := enc.Symbol(nil, idx)
require.NoError(t, err)
sum := sha256.Sum256(sym)
assert.Equal(t, h, hex.EncodeToString(sum[:]), "symbol %d", idx)
}
}
func FuzzRoundTrip(f *testing.F) {
f.Add([]byte("hello"), uint16(2), uint64(0))
f.Add(make([]byte, 9000), uint16(1176), uint64(7))
f.Fuzz(func(t *testing.T, msg []byte, maxSym uint16, seed uint64) {
if len(msg) == 0 || maxSym == 0 {
return
}
enc, err := NewEncoder(msg, int(maxSym))
if err != nil {
return
}
r := rand.New(rand.NewPCG(seed, seed))
idx := r.Perm(MaxSymbols)[:enc.Layout().K]
require.Equal(t, msg, decodeWith(t, enc, len(msg), int(maxSym), idx))
})
}
// FuzzDecoder feeds arbitrary symbols to a decoder; it must never panic and
// must either fail or return a message of the declared length.
func FuzzDecoder(f *testing.F) {
f.Add(uint16(3000), []byte{0, 5, 9, 200}, []byte("some symbol bytes"))
f.Fuzz(func(t *testing.T, msgLen uint16, indexes []byte, fill []byte) {
dec, err := NewDecoder(int(msgLen), 1176)
if err != nil {
return
}
size := dec.Layout().SymbolSize
for i, idx := range indexes {
sym := make([]byte, size)
for j := range sym {
if len(fill) > 0 {
sym[j] = fill[(i+j)%len(fill)]
}
}
if _, err := dec.Add(int(idx), sym); err != nil {
t.Fatal(err)
}
}
out, err := dec.Decode()
if err == nil {
require.Len(t, out, int(msgLen))
}
})
}
// BenchmarkRepair and BenchmarkDecode use a v3-sized msg2 (~9.8KB, k=9) with
// three repair symbols, decoding with three data symbols missing.
func BenchmarkRepair(b *testing.B) {
msg := randMsg(rand.New(rand.NewPCG(7, 8)), 9800)
enc, err := NewEncoder(msg, 1176)
require.NoError(b, err)
buf := make([]byte, 0, 1176)
b.SetBytes(int64(len(msg)))
for b.Loop() {
for idx := 9; idx < 12; idx++ {
buf, _ = enc.Symbol(buf[:0], idx)
}
}
}
func BenchmarkDecode(b *testing.B) {
msg := randMsg(rand.New(rand.NewPCG(7, 8)), 9800)
enc, err := NewEncoder(msg, 1176)
require.NoError(b, err)
idx := []int{0, 1, 2, 3, 5, 6, 9, 10, 11}
syms := make([][]byte, len(idx))
for i, x := range idx {
syms[i], _ = enc.Symbol(nil, x)
}
b.SetBytes(int64(len(msg)))
for b.Loop() {
dec, _ := NewDecoder(len(msg), 1176)
for i, x := range idx {
_, _ = dec.Add(x, syms[i])
}
if _, err := dec.Decode(); err != nil {
b.Fatal(err)
}
}
}
-115
View File
@@ -1,115 +0,0 @@
package fec
// Arithmetic over GF(2^8) with the reducing polynomial x^8+x^4+x^3+x^2+1 (0x11d).
// Addition is XOR. Multiplication uses log/exp tables; the exp table is doubled so
// the sum of two logs never needs reducing mod 255.
const gfPoly = 0x11d
var (
gfExp [510]byte
gfLog [256]byte
)
func init() {
x := 1
for i := range 255 {
gfExp[i] = byte(x)
gfExp[i+255] = byte(x)
gfLog[x] = byte(i)
x <<= 1
if x&0x100 != 0 {
x ^= gfPoly
}
}
}
func gfMul(a, b byte) byte {
if a == 0 || b == 0 {
return 0
}
return gfExp[int(gfLog[a])+int(gfLog[b])]
}
// gfInv returns the multiplicative inverse of a. a must be non-zero.
func gfInv(a byte) byte {
return gfExp[255-int(gfLog[a])]
}
// mulAdd computes dst ^= c*src element-wise. dst must be at least as long as src.
func mulAdd(dst, src []byte, c byte) {
switch c {
case 0:
return
case 1:
for i, s := range src {
dst[i] ^= s
}
return
}
var t [256]byte
lc := int(gfLog[c])
for s := 1; s < 256; s++ {
t[s] = gfExp[int(gfLog[s])+lc]
}
for i, s := range src {
dst[i] ^= t[s]
}
}
// coefficient returns the generator matrix entry for repair symbol index row and
// data symbol index col. Repair rows are a Cauchy matrix 1/(x_row + y_col) with
// x_row = row and y_col = col: rows are always >= k and columns < k so the two
// sets are disjoint, which makes every square submatrix invertible and the
// systematic code [I; C] MDS. A row depends only on its index, never on how many
// repair symbols exist, so new repair symbols can be generated at any time.
func coefficient(row, col int) byte {
return gfInv(byte(row) ^ byte(col))
}
// invert inverts the n x n matrix m in place with Gauss-Jordan elimination.
// It returns false if m is singular.
func invert(m [][]byte) bool {
n := len(m)
inv := make([][]byte, n)
for i := range inv {
inv[i] = make([]byte, n)
inv[i][i] = 1
}
for col := range n {
pivot := -1
for r := col; r < n; r++ {
if m[r][col] != 0 {
pivot = r
break
}
}
if pivot < 0 {
return false
}
m[col], m[pivot] = m[pivot], m[col]
inv[col], inv[pivot] = inv[pivot], inv[col]
scale := gfInv(m[col][col])
for c := range n {
m[col][c] = gfMul(m[col][c], scale)
inv[col][c] = gfMul(inv[col][c], scale)
}
for r := 0; r < n; r++ {
if r == col || m[r][col] == 0 {
continue
}
f := m[r][col]
for c := range n {
m[r][c] ^= gfMul(f, m[col][c])
inv[r][c] ^= gfMul(f, inv[col][c])
}
}
}
copy(m, inv)
return true
}
-67
View File
@@ -1,67 +0,0 @@
package fragment
import (
"errors"
"github.com/slackhq/nebula/handshake/fec"
"github.com/slackhq/nebula/header"
)
// ErrMismatch is returned when a fragment disagrees with the message it claims to
// belong to. The caller should discard the whole assembly.
var ErrMismatch = errors.New("fragment does not match its message")
// Assembly collects the fragments of one incoming message. Its memory is bounded
// by the message length: at most K symbols are kept however many arrive.
//
// An Assembly is not safe for concurrent use.
type Assembly struct {
fragID uint32
msgLen uint16
dec *fec.Decoder
received int
}
// NewAssembly starts an assembly for the message described by h. The fragment
// itself still has to be passed to Add.
func NewAssembly(h Header) (*Assembly, error) {
dec, err := fec.NewDecoder(int(h.MsgLen), MaxSymbolSize)
if err != nil {
return nil, err
}
return &Assembly{fragID: h.FragID, msgLen: h.MsgLen, dec: dec}, nil
}
// Add records a fragment parsed by Parse. It returns true once enough symbols
// are held to call Decode.
func (a *Assembly) Add(h Header, sym []byte) (bool, error) {
if h.FragID != a.fragID || h.MsgLen != a.msgLen {
return a.dec.Ready(), ErrMismatch
}
a.received += header.Len + HeaderLen + len(sym)
return a.dec.Add(int(h.Index), sym)
}
// wouldStore returns how many bytes Add would retain for this fragment.
func (a *Assembly) wouldStore(h Header, sym []byte) int {
if h.FragID != a.fragID || h.MsgLen != a.msgLen || a.dec.Ready() || a.dec.Has(int(h.Index)) {
return 0
}
return len(sym)
}
// Decode reconstructs the message. It requires Add to have reported ready.
func (a *Assembly) Decode() ([]byte, error) {
return a.dec.Decode()
}
// Stored returns the number of symbol bytes held.
func (a *Assembly) Stored() int {
return a.dec.Stored() * a.dec.Layout().SymbolSize
}
// Received returns the number of packet bytes received for this message,
// duplicates included. It is the basis for the amplification budget.
func (a *Assembly) Received() int {
return a.received
}
-515
View File
@@ -1,515 +0,0 @@
package fragment
import (
"bytes"
"math/rand/v2"
"sync"
"testing"
"time"
"github.com/slackhq/nebula/header"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const testSubtype = header.MessageSubType(9)
type testKey struct {
src string
fragID uint32
}
func randMsg(r *rand.Rand, n int) []byte {
b := make([]byte, n)
for i := range b {
b[i] = byte(r.Uint32())
}
return b
}
func testConfig() TableConfig {
return TableConfig{
MaxBytes: 1 << 20,
MaxEntries: 64,
MaxPerSource: 4,
MaxMsgLen: 16384,
TTL: 2 * time.Second,
}
}
// checkInvariants verifies the table's bookkeeping against its contents.
func checkInvariants[K, S comparable](t *testing.T, tb *Table[K, S]) {
t.Helper()
tb.mu.Lock()
defer tb.mu.Unlock()
require.Len(t, tb.order, len(tb.entries))
require.LessOrEqual(t, len(tb.entries), tb.cfg.MaxEntries)
bytes := 0
perSource := map[S]int{}
for i, key := range tb.order {
e, ok := tb.entries[key]
require.True(t, ok)
require.Equal(t, i, e.pos)
bytes += e.a.Stored()
perSource[e.source]++
}
require.Equal(t, bytes, tb.bytes)
require.LessOrEqual(t, tb.bytes, tb.cfg.MaxBytes)
require.Equal(t, perSource, tb.perSource)
for _, n := range perSource {
require.LessOrEqual(t, n, tb.cfg.MaxPerSource)
}
}
func parse(t *testing.T, p []byte) (header.H, Header, []byte) {
t.Helper()
var nh header.H
require.NoError(t, nh.Parse(p))
h, sym, err := Parse(p[header.Len:])
require.NoError(t, err)
return nh, h, sym
}
func TestParse(t *testing.T) {
h := Header{FragID: 0xdeadbeef, MsgLen: 3000, Index: 7}
b := h.Encode(nil)
require.Len(t, b, HeaderLen)
b = append(b, make([]byte, 1000)...)
got, sym, err := Parse(b)
require.NoError(t, err)
assert.Equal(t, h, got)
assert.Len(t, sym, 1000)
_, _, err = Parse(b[:HeaderLen-1])
require.ErrorIs(t, err, ErrShortFragment)
_, _, err = Parse(b[:len(b)-1])
require.ErrorIs(t, err, ErrSymbolSize)
_, _, err = Parse(append(b, 0))
require.ErrorIs(t, err, ErrSymbolSize)
bad := Header{FragID: 0, MsgLen: 3000}.Encode(nil)
_, _, err = Parse(append(bad, make([]byte, 1000)...))
require.ErrorIs(t, err, ErrFragID)
bad = Header{FragID: 1, MsgLen: 0}.Encode(nil)
_, _, err = Parse(bad)
require.ErrorIs(t, err, ErrMsgLen)
bad = Header{FragID: 1, MsgLen: 3000, Flags: 0x02}.Encode(nil)
_, _, err = Parse(append(bad, make([]byte, 1000)...))
require.ErrorIs(t, err, ErrFlags)
t.Run("cookie", func(t *testing.T) {
h := Header{FragID: 7, MsgLen: 3000, Index: 4, Flags: FlagCookie}
for i := range h.Cookie {
h.Cookie[i] = byte(i + 1)
}
b := h.Encode(nil)
require.Len(t, b, HeaderLen+CookieLen)
b = append(b, make([]byte, 1000)...)
got, sym, err := Parse(b)
require.NoError(t, err)
assert.Equal(t, h, got)
assert.True(t, got.HasCookie())
assert.Len(t, sym, 1000)
_, _, err = Parse(b[:HeaderLen+CookieLen-1])
require.ErrorIs(t, err, ErrNoCookie)
})
}
func TestDefaultRepair(t *testing.T) {
for k, want := range map[int]int{1: 2, 2: 2, 6: 2, 7: 3, 9: 3, 10: 4, 14: 5} {
assert.Equal(t, want, DefaultRepair(k), "k=%d", k)
}
}
func TestSender(t *testing.T) {
_, err := NewSender(nil, testSubtype, 0, 1, 1)
require.ErrorIs(t, err, ErrMsgLen)
_, err = NewSender(make([]byte, 65536), testSubtype, 0, 1, 1)
require.ErrorIs(t, err, ErrMsgLen)
_, err = NewSender(make([]byte, 10), testSubtype, 0, 1, 0)
require.ErrorIs(t, err, ErrFragID)
msg := randMsg(rand.New(rand.NewPCG(1, 1)), 9800)
s, err := NewSender(msg, testSubtype, 77, 2, 1234)
require.NoError(t, err)
require.Equal(t, 9, s.Layout().K)
indexes := func(ps [][]byte) []int {
var out []int
for _, p := range ps {
require.LessOrEqual(t, len(p), MaxPacketLen)
nh, h, _ := parse(t, p)
assert.Equal(t, header.Handshake, nh.Type)
assert.Equal(t, testSubtype, nh.Subtype)
assert.Equal(t, uint32(77), nh.RemoteIndex)
assert.Equal(t, uint64(2), nh.MessageCounter)
assert.Equal(t, uint32(1234), h.FragID)
assert.Equal(t, uint16(9800), h.MsgLen)
out = append(out, int(h.Index))
}
return out
}
now, later, err := s.Attempt(3)
require.NoError(t, err)
assert.Equal(t, []int{0, 1, 2, 3, 4, 5, 6, 7, 8}, indexes(now))
assert.Equal(t, []int{9, 10, 11}, indexes(later))
// A retry offers only indexes the receiver has not seen.
now, later, err = s.Attempt(3)
require.NoError(t, err)
assert.Equal(t, []int{12, 13, 14, 15, 16, 17, 18, 19, 20}, indexes(now))
assert.Equal(t, []int{21, 22, 23}, indexes(later))
// A sender rebuilt from the same message resumes where it was told to.
s2, err := NewSender(msg, testSubtype, 77, 2, 1234)
require.NoError(t, err)
s2.SetNext(s.Next())
now2, _, err := s2.Attempt(0)
require.NoError(t, err)
assert.Equal(t, []int{24, 25, 26, 27, 28, 29, 30, 31, 32}, indexes(now2))
s2.SetNext(-1)
assert.Equal(t, 255, s2.Next())
// A cookie rides on every later packet, which still fits MaxPacketLen even
// with a full size symbol.
full, err := NewSender(make([]byte, 2*MaxSymbolSize), testSubtype, 0, 1, 9)
require.NoError(t, err)
full.SetCookie(Cookie{1, 2, 3})
pkts, _, err := full.Attempt(1)
require.NoError(t, err)
for _, p := range pkts {
assert.Len(t, p, MaxPacketLen)
_, h, _ := parse(t, p)
assert.True(t, h.HasCookie())
assert.Equal(t, Cookie{1, 2, 3}, h.Cookie)
}
// Indexes wrap once the space is used up.
for range 19 {
_, _, err = s.Attempt(3)
require.NoError(t, err)
}
now, _, err = s.Attempt(3)
require.NoError(t, err)
assert.Equal(t, []int{252, 253, 254, 255, 0, 1, 2, 3, 4}, indexes(now))
}
// TestLossy runs whole messages through heavy loss, reordering and duplication,
// merging across attempts, and checks the table ends empty.
func TestLossy(t *testing.T) {
r := rand.New(rand.NewPCG(2, 2))
tb := NewTable[testKey, string](testConfig())
now := time.Unix(1000, 0)
for i := range 300 {
msg := randMsg(r, 1+r.IntN(16384))
fragID := uint32(i + 1)
s, err := NewSender(msg, testSubtype, 0, 1, fragID)
require.NoError(t, err)
var got *Result
for attempt := 0; got == nil; attempt++ {
require.Less(t, attempt, 20, "message never completed")
a, b, err := s.Attempt(DefaultRepair(s.Layout().K))
require.NoError(t, err)
var pkts [][]byte
for _, p := range append(a, b...) {
if r.Float64() < 0.3 {
continue
}
pkts = append(pkts, p)
if r.Float64() < 0.2 {
pkts = append(pkts, p)
}
}
r.Shuffle(len(pkts), func(i, j int) { pkts[i], pkts[j] = pkts[j], pkts[i] })
for _, p := range pkts {
_, h, sym := parse(t, p)
res, err := tb.Add(testKey{"a", h.FragID}, "a", now, h, sym)
require.NoError(t, err)
if res != nil {
// The table keeps no memory of completed messages, so a late
// duplicate can complete one again; the handshake layer
// recognises the repeat.
require.True(t, bytes.Equal(msg, res.Msg))
got = res
}
checkInvariants(t, tb)
}
now = now.Add(100 * time.Millisecond)
}
require.Positive(t, got.Received)
// Late fragments of a completed message start a fresh entry; drop it.
tb.Expire(now.Add(time.Hour))
}
assert.Zero(t, tb.Len())
assert.Zero(t, tb.Bytes())
}
// TestConcurrent spreads each message's fragments across goroutines, as
// multiple UDP reader routines would.
func TestConcurrent(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
now := time.Unix(1000, 0)
r := rand.New(rand.NewPCG(3, 3))
const msgs = 50
want := make(map[uint32][]byte, msgs)
var pkts [][]byte
for id := uint32(1); id <= msgs; id++ {
msg := randMsg(r, 1+r.IntN(16384))
want[id] = msg
s, err := NewSender(msg, testSubtype, 0, 1, id)
require.NoError(t, err)
a, b, err := s.Attempt(DefaultRepair(s.Layout().K))
require.NoError(t, err)
pkts = append(pkts, a...)
pkts = append(pkts, b...)
}
r.Shuffle(len(pkts), func(i, j int) { pkts[i], pkts[j] = pkts[j], pkts[i] })
var mu sync.Mutex
done := map[uint32]bool{}
var wg sync.WaitGroup
for w := range 8 {
wg.Go(func() {
for i := w; i < len(pkts); i += 8 {
h, sym, err := Parse(pkts[i][16:])
if err != nil {
t.Error(err)
return
}
src := string(rune('a' + h.FragID))
res, err := tb.Add(testKey{src, h.FragID}, src, now, h, sym)
if err != nil {
t.Error(err)
return
}
if res != nil {
if !bytes.Equal(want[h.FragID], res.Msg) {
t.Errorf("message %d decoded wrong", h.FragID)
}
mu.Lock()
done[h.FragID] = true
mu.Unlock()
}
}
})
}
wg.Wait()
assert.Len(t, done, msgs)
checkInvariants(t, tb)
}
func firstFragment(t *testing.T, msgLen int, fragID uint32) (Header, []byte) {
t.Helper()
s, err := NewSender(make([]byte, msgLen), testSubtype, 0, 1, fragID)
require.NoError(t, err)
now, _, err := s.Attempt(0)
require.NoError(t, err)
_, h, sym := parse(t, now[0])
return h, sym
}
func TestTableLimits(t *testing.T) {
now := time.Unix(1000, 0)
t.Run("max message length", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
h, sym := firstFragment(t, 16385, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.ErrorIs(t, err, ErrTooLarge)
assert.Zero(t, tb.Len())
})
t.Run("per source quota", func(t *testing.T) {
cfg := testConfig()
cfg.MaxPerSource = 2
tb := NewTable[testKey, string](cfg)
for id := uint32(1); id <= 3; id++ {
h, sym := firstFragment(t, 5000, id)
_, err := tb.Add(testKey{"a", id}, "a", now, h, sym)
if id <= 2 {
require.NoError(t, err)
} else {
require.ErrorIs(t, err, ErrSourceQuota)
}
}
h, sym := firstFragment(t, 5000, 4)
_, err := tb.Add(testKey{"b", 4}, "b", now, h, sym)
require.NoError(t, err)
assert.Equal(t, 3, tb.Len())
checkInvariants(t, tb)
})
t.Run("max entries evicts", func(t *testing.T) {
cfg := testConfig()
cfg.MaxEntries = 4
tb := NewTable[testKey, string](cfg)
for id := uint32(1); id <= 10; id++ {
h, sym := firstFragment(t, 5000, id)
src := string(rune('a' + id))
_, err := tb.Add(testKey{src, id}, src, now, h, sym)
require.NoError(t, err)
checkInvariants(t, tb)
}
assert.Equal(t, 4, tb.Len())
})
t.Run("max bytes evicts others", func(t *testing.T) {
cfg := testConfig()
cfg.MaxBytes = 3 * 1000
tb := NewTable[testKey, string](cfg)
for id := uint32(1); id <= 10; id++ {
h, sym := firstFragment(t, 5000, id) // 5 symbols of 1000
src := string(rune('a' + id))
_, err := tb.Add(testKey{src, id}, src, now, h, sym)
require.NoError(t, err)
checkInvariants(t, tb)
}
assert.Equal(t, 3000, tb.Bytes())
})
t.Run("no room removes new entry", func(t *testing.T) {
cfg := testConfig()
cfg.MaxBytes = 500
tb := NewTable[testKey, string](cfg)
h, sym := firstFragment(t, 5000, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.ErrorIs(t, err, ErrNoRoom)
assert.Zero(t, tb.Len())
checkInvariants(t, tb)
})
t.Run("duplicates are not charged", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
h, sym := firstFragment(t, 5000, 1)
for range 5 {
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
}
assert.Equal(t, 1000, tb.Bytes())
})
t.Run("mismatch drops the entry", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
h, sym := firstFragment(t, 5000, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
h2, sym2 := firstFragment(t, 6000, 1)
_, err = tb.Add(testKey{"a", 1}, "a", now, h2, sym2)
require.ErrorIs(t, err, ErrMismatch)
assert.Zero(t, tb.Len())
assert.Zero(t, tb.Bytes())
checkInvariants(t, tb)
})
t.Run("load", func(t *testing.T) {
cfg := testConfig()
cfg.MaxEntries = 4
cfg.MaxBytes = 10000
tb := NewTable[testKey, string](cfg)
assert.Zero(t, tb.Load())
h, sym := firstFragment(t, 5000, 1)
_, err := tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
assert.InDelta(t, 0.25, tb.Load(), 0.001, "one of four entries")
h, sym = firstFragment(t, 5000, 2)
_, err = tb.Add(testKey{"b", 2}, "b", now, h, sym)
require.NoError(t, err)
assert.InDelta(t, 0.5, tb.Load(), 0.001)
})
t.Run("ttl", func(t *testing.T) {
tb := NewTable[testKey, string](testConfig())
s, err := NewSender(make([]byte, 5000), testSubtype, 0, 1, 1)
require.NoError(t, err)
pkts, _, err := s.Attempt(0)
require.NoError(t, err)
_, h, sym := parse(t, pkts[0])
_, err = tb.Add(testKey{"a", 1}, "a", now, h, sym)
require.NoError(t, err)
// Activity within the TTL keeps the entry and slides its deadline.
_, h, sym = parse(t, pkts[1])
_, err = tb.Add(testKey{"a", 1}, "a", now.Add(1500*time.Millisecond), h, sym)
require.NoError(t, err)
assert.Equal(t, 2000, tb.Bytes())
tb.Expire(now.Add(3 * time.Second))
assert.Equal(t, 1, tb.Len())
// A stale entry is restarted rather than merged.
_, h, sym = parse(t, pkts[2])
_, err = tb.Add(testKey{"a", 1}, "a", now.Add(10*time.Second), h, sym)
require.NoError(t, err)
assert.Equal(t, 1000, tb.Bytes())
tb.Expire(now.Add(time.Minute))
assert.Zero(t, tb.Len())
checkInvariants(t, tb)
})
}
func FuzzParse(f *testing.F) {
f.Add(append(Header{FragID: 1, MsgLen: 3000}.Encode(nil), make([]byte, 1000)...))
f.Add([]byte{0, 0, 0, 1, 0, 1, 0, 0, 7})
f.Fuzz(func(t *testing.T, b []byte) {
h, sym, err := Parse(b)
if err != nil {
return
}
require.Equal(t, b, append(h.Encode(nil), sym...))
l, err := h.Layout()
require.NoError(t, err)
require.LessOrEqual(t, l.SymbolSize, MaxSymbolSize)
})
}
// FuzzTable drives a small table with arbitrary fragments and checks its
// bookkeeping after every step.
func FuzzTable(f *testing.F) {
f.Add([]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12})
f.Fuzz(func(t *testing.T, ops []byte) {
tb := NewTable[testKey, string](TableConfig{
MaxBytes: 4000,
MaxEntries: 3,
MaxPerSource: 2,
MaxMsgLen: 6000,
TTL: time.Second,
})
now := time.Unix(0, 0)
for len(ops) >= 4 {
src := string(rune('a' + ops[0]%4))
fragID := uint32(ops[1]%5) + 1
msgLen := uint16(ops[2])*32 + 1
index := ops[3]
ops = ops[4:]
h := Header{FragID: fragID, MsgLen: msgLen, Index: index}
l, err := h.Layout()
require.NoError(t, err)
sym := make([]byte, l.SymbolSize)
if index%7 == 0 {
now = now.Add(600 * time.Millisecond)
tb.Expire(now)
}
res, err := tb.Add(testKey{src, fragID}, src, now, h, sym)
if res != nil {
require.NoError(t, err)
require.Len(t, res.Msg, int(msgLen))
}
checkInvariants(t, tb)
}
})
}
-118
View File
@@ -1,118 +0,0 @@
package fragment
import (
"errors"
"math"
"github.com/slackhq/nebula/handshake/fec"
"github.com/slackhq/nebula/header"
)
// DefaultRepair is the starting number of repair symbols per attempt for a
// message of k data symbols. Small messages get proportionally more repair.
func DefaultRepair(k int) int {
return max(2, (k+2)/3)
}
// Sender produces the fragment packets of one outgoing handshake message.
//
// The first attempt sends the K data symbols followed by repair symbols. Every
// later attempt sends symbols with indexes the receiver has not been offered yet,
// so whatever it already holds still counts toward K. Symbols are a deterministic
// function of the message, which keeps retransmits idempotent. Once all
// fec.MaxSymbols indexes are used the sequence wraps and repeats.
//
// A Sender is not safe for concurrent use.
type Sender struct {
enc *fec.Encoder
subtype header.MessageSubType
remoteIndex uint32
counter uint64
fragID uint32
next int
cookie *Cookie
}
// NewSender prepares msg for sending. subtype, remoteIndex and counter fill the
// Nebula header of every fragment; fragID is the sender's local index.
func NewSender(msg []byte, subtype header.MessageSubType, remoteIndex uint32, counter uint64, fragID uint32) (*Sender, error) {
if len(msg) == 0 || len(msg) > math.MaxUint16 {
return nil, ErrMsgLen
}
if fragID == 0 {
return nil, ErrFragID
}
enc, err := fec.NewEncoder(msg, MaxSymbolSize)
if err != nil {
return nil, err
}
return &Sender{
enc: enc,
subtype: subtype,
remoteIndex: remoteIndex,
counter: counter,
fragID: fragID,
}, nil
}
func (s *Sender) Layout() fec.Layout {
return s.enc.Layout()
}
// SetNext sets the index the next attempt starts from, so a sender rebuilt from
// a cached message can carry on offering fresh symbols.
func (s *Sender) SetNext(index int) {
s.next = ((index % fec.MaxSymbols) + fec.MaxSymbols) % fec.MaxSymbols
}
// SetCookie attaches a cookie from a responder under load to every later packet.
func (s *Sender) SetCookie(c Cookie) {
s.cookie = &c
}
// Next returns the index the next attempt starts from.
func (s *Sender) Next() int {
return s.next
}
// Attempt returns the packets for one transmission attempt carrying K+repair
// symbols. now should be sent immediately and later after the repair delay, so
// a tail drop at the end of the first burst does not also take the repair.
func (s *Sender) Attempt(repair int) (now, later [][]byte, err error) {
if repair < 0 {
return nil, nil, errors.New("negative repair count")
}
k := s.enc.Layout().K
now = make([][]byte, 0, k)
later = make([][]byte, 0, repair)
for i := range k + repair {
p, err := s.packet(s.next)
if err != nil {
return nil, nil, err
}
s.next = (s.next + 1) % fec.MaxSymbols
if i < k {
now = append(now, p)
} else {
later = append(later, p)
}
}
return now, later, nil
}
func (s *Sender) packet(index int) ([]byte, error) {
l := s.enc.Layout()
p := make([]byte, header.Len, header.Len+HeaderLen+CookieLen+l.SymbolSize)
header.Encode(p, header.Version, header.Handshake, s.subtype, s.remoteIndex, s.counter)
h := Header{
FragID: s.fragID,
MsgLen: uint16(l.MsgLen),
Index: uint8(index),
}
if s.cookie != nil {
h.Flags |= FlagCookie
h.Cookie = *s.cookie
}
p = h.Encode(p)
return s.enc.Symbol(p, index)
}
-220
View File
@@ -1,220 +0,0 @@
package fragment
import (
"errors"
"math/rand/v2"
"sync"
"time"
)
var (
ErrTooLarge = errors.New("fragmented message exceeds the size limit")
ErrSourceQuota = errors.New("source has too many messages in progress")
ErrNoRoom = errors.New("no room for fragment")
)
// TableConfig bounds a Table.
type TableConfig struct {
MaxBytes int // symbol bytes held across all entries
MaxEntries int // messages in progress across all sources
MaxPerSource int // messages in progress per source
MaxMsgLen int // largest message accepted
TTL time.Duration // an entry idle this long is dropped
}
// Table reassembles messages from senders that have not authenticated yet, the
// responder's view of msg1. Every resource is capped. When a cap is hit a
// random entry is evicted; random eviction is harder to aim than LRU, which a
// flood can use to flush specific entries.
//
// K identifies one message, typically the source plus FragID. S identifies the
// source for quotas, typically the address masked to /32 or /64.
//
// A Table is safe for concurrent use.
type Table[K, S comparable] struct {
mu sync.Mutex
cfg TableConfig
entries map[K]*tableEntry[K, S]
order []K
perSource map[S]int
bytes int
}
type tableEntry[K, S comparable] struct {
a *Assembly
source S
lastSeen time.Time
pos int
}
func NewTable[K, S comparable](cfg TableConfig) *Table[K, S] {
return &Table[K, S]{
cfg: cfg,
entries: make(map[K]*tableEntry[K, S]),
perSource: make(map[S]int),
}
}
// Result describes a completed message.
type Result struct {
Msg []byte // the decoded message
Received int // packet bytes received for it, for the amplification budget
}
// Add records a fragment parsed by Parse. When the message is complete its entry
// is removed and the result returned; otherwise the result is nil. A decode
// error means the message is lost and must be retransmitted.
func (t *Table[K, S]) Add(key K, source S, now time.Time, h Header, sym []byte) (*Result, error) {
if int(h.MsgLen) > t.cfg.MaxMsgLen {
return nil, ErrTooLarge
}
t.mu.Lock()
e, ok := t.entries[key]
if ok && now.Sub(e.lastSeen) > t.cfg.TTL {
t.remove(key, e)
ok = false
}
if !ok {
var err error
e, err = t.insert(key, source, h)
if err != nil {
t.mu.Unlock()
return nil, err
}
}
e.lastSeen = now
if !t.makeRoom(key, e.a.wouldStore(h, sym)) {
if e.a.Stored() == 0 {
t.remove(key, e)
}
t.mu.Unlock()
return nil, ErrNoRoom
}
before := e.a.Stored()
ready, err := e.a.Add(h, sym)
t.bytes += e.a.Stored() - before
if err != nil || !ready {
if errors.Is(err, ErrMismatch) || (err != nil && e.a.Stored() == 0) {
t.remove(key, e)
}
t.mu.Unlock()
return nil, err
}
t.remove(key, e)
t.mu.Unlock()
msg, err := e.a.Decode()
if err != nil {
return nil, err
}
return &Result{Msg: msg, Received: e.a.Received()}, nil
}
// Expire drops every entry idle for longer than the TTL.
func (t *Table[K, S]) Expire(now time.Time) {
t.mu.Lock()
defer t.mu.Unlock()
for i := 0; i < len(t.order); {
key := t.order[i]
e := t.entries[key]
if now.Sub(e.lastSeen) > t.cfg.TTL {
// remove swaps the last entry into position i, so look at i again.
t.remove(key, e)
continue
}
i++
}
}
// Len returns the number of messages in progress.
func (t *Table[K, S]) Len() int {
t.mu.Lock()
defer t.mu.Unlock()
return len(t.entries)
}
// Load returns how full the table is, from 0 to 1, by whichever of its byte and
// entry limits is closer.
func (t *Table[K, S]) Load() float64 {
t.mu.Lock()
defer t.mu.Unlock()
return max(float64(t.bytes)/float64(t.cfg.MaxBytes), float64(len(t.entries))/float64(t.cfg.MaxEntries))
}
// Bytes returns the number of symbol bytes held.
func (t *Table[K, S]) Bytes() int {
t.mu.Lock()
defer t.mu.Unlock()
return t.bytes
}
func (t *Table[K, S]) insert(key K, source S, h Header) (*tableEntry[K, S], error) {
if t.perSource[source] >= t.cfg.MaxPerSource {
return nil, ErrSourceQuota
}
a, err := NewAssembly(h)
if err != nil {
return nil, err
}
for len(t.entries) >= t.cfg.MaxEntries {
if !t.evictRandom(nil) {
return nil, ErrNoRoom
}
}
e := &tableEntry[K, S]{a: a, source: source, pos: len(t.order)}
t.entries[key] = e
t.order = append(t.order, key)
t.perSource[source]++
return e, nil
}
// makeRoom evicts random entries other than keep until n more bytes fit.
func (t *Table[K, S]) makeRoom(keep K, n int) bool {
for t.bytes+n > t.cfg.MaxBytes {
if !t.evictRandom(&keep) {
return false
}
}
return true
}
func (t *Table[K, S]) evictRandom(keep *K) bool {
candidates := len(t.order)
if keep != nil {
if _, ok := t.entries[*keep]; ok {
candidates--
}
}
if candidates <= 0 {
return false
}
for {
key := t.order[rand.IntN(len(t.order))]
if keep != nil && key == *keep {
continue
}
t.remove(key, t.entries[key])
return true
}
}
func (t *Table[K, S]) remove(key K, e *tableEntry[K, S]) {
last := len(t.order) - 1
moved := t.order[last]
t.order[e.pos] = moved
t.entries[moved].pos = e.pos
t.order = t.order[:last]
delete(t.entries, key)
t.bytes -= e.a.Stored()
if t.perSource[e.source] <= 1 {
delete(t.perSource, e.source)
} else {
t.perSource[e.source]--
}
}
-121
View File
@@ -1,121 +0,0 @@
// Package fragment carries handshake messages that do not fit in one UDP
// packet. A message is Reed-Solomon coded into equally sized symbols (see
// handshake/fec) and each symbol travels in its own packet:
//
// Nebula header (16) | Fragment header (8) | [cookie (16)] | symbol
//
// The fragment header is FragID u32 | MsgLen u16 | Index u8 | Flags u8, all big
// endian. Both ends derive the symbol layout from MsgLen, so any MsgLen-derived
// value never has to be sent. FlagCookie marks a cookie a responder under load
// asked for; room for it is always reserved, so a fragment never outgrows
// MaxPacketLen.
package fragment
import (
"encoding/binary"
"errors"
"github.com/slackhq/nebula/handshake/fec"
"github.com/slackhq/nebula/header"
)
const (
// HeaderLen is the size of the fragment header that follows the Nebula header.
HeaderLen = 8
// CookieLen is the size of the cookie a fragment carries when FlagCookie is set.
CookieLen = 16
// FlagCookie marks a fragment that carries a cookie after its header.
FlagCookie uint8 = 0x01
// MaxPacketLen bounds every fragment packet, Nebula header included. A relayed
// fragment gains a 16 byte header and a 16 byte tag, which lands exactly on
// the 1232 byte UDP payload available on an IPv6 minimum MTU path.
MaxPacketLen = 1200
// MaxSymbolSize is the largest symbol that fits in MaxPacketLen alongside a cookie.
MaxSymbolSize = MaxPacketLen - header.Len - HeaderLen - CookieLen
)
var (
ErrShortFragment = errors.New("fragment is too short")
ErrFragID = errors.New("fragment has a zero FragID")
ErrMsgLen = errors.New("fragment has a zero MsgLen")
ErrFlags = errors.New("fragment has reserved flags set")
ErrNoCookie = errors.New("fragment is too short for its cookie")
ErrSymbolSize = errors.New("fragment symbol size does not match its MsgLen")
)
// Cookie proves a sender received a responder's reply at its address.
type Cookie [CookieLen]byte
// Header is the fragment header.
type Header struct {
FragID uint32 // the sender's local index for this handshake, never zero
MsgLen uint16 // length of the whole message
Index uint8 // symbol index, data below K and repair from K
Flags uint8 // FlagCookie, other bits reserved and zero
Cookie Cookie // valid when Flags has FlagCookie
}
func (h Header) HasCookie() bool {
return h.Flags&FlagCookie != 0
}
// Encode appends the header, and its cookie if it has one, to b.
func (h Header) Encode(b []byte) []byte {
b = binary.BigEndian.AppendUint32(b, h.FragID)
b = binary.BigEndian.AppendUint16(b, h.MsgLen)
b = append(b, h.Index, h.Flags)
if h.HasCookie() {
b = append(b, h.Cookie[:]...)
}
return b
}
// Layout returns the symbol layout the header's MsgLen implies.
func (h Header) Layout() (fec.Layout, error) {
return fec.NewLayout(int(h.MsgLen), MaxSymbolSize)
}
// Parse validates the bytes following the Nebula header and splits them into
// the fragment header and the symbol. The symbol aliases b.
func Parse(b []byte) (Header, []byte, error) {
if len(b) < HeaderLen {
return Header{}, nil, ErrShortFragment
}
h := Header{
FragID: binary.BigEndian.Uint32(b[0:4]),
MsgLen: binary.BigEndian.Uint16(b[4:6]),
Index: b[6],
Flags: b[7],
}
if h.FragID == 0 {
return Header{}, nil, ErrFragID
}
if h.MsgLen == 0 {
return Header{}, nil, ErrMsgLen
}
if h.Flags&^FlagCookie != 0 {
return Header{}, nil, ErrFlags
}
b = b[HeaderLen:]
if h.HasCookie() {
if len(b) < CookieLen {
return Header{}, nil, ErrNoCookie
}
copy(h.Cookie[:], b)
b = b[CookieLen:]
}
l, err := h.Layout()
if err != nil {
return Header{}, nil, err
}
sym := b
if len(sym) != l.SymbolSize {
return Header{}, nil, ErrSymbolSize
}
return h, sym, nil
}
-3
View File
@@ -24,9 +24,6 @@ message NebulaHandshakeDetails {
uint64 Cookie = 4 [deprecated = true]; uint64 Cookie = 4 [deprecated = true];
uint64 Time = 5; uint64 Time = 5;
uint32 CertVersion = 8; uint32 CertVersion = 8;
// KEMKey is the initiator's ephemeral ML-KEM-1024 encapsulation key. It is
// only present in msg1 of the ix_psk2_mlkem1024 subtype.
bytes KEMKey = 9;
// reserved for WIP multiport // reserved for WIP multiport
reserved 6, 7; reserved 6, 7;
} }
-68
View File
@@ -1,68 +0,0 @@
package handshake
import (
"crypto/hkdf"
"crypto/mlkem"
"crypto/sha256"
"errors"
"fmt"
)
// kemPSKSalt separates the psk derivation from any other use of the secret.
const kemPSKSalt = "nebula ix_psk2_mlkem1024 v1"
var ErrInvalidKEMKey = errors.New("peer sent an invalid ML-KEM encapsulation key")
// kemPSK derives the noise psk from an ML-KEM shared secret. The ciphertext is
// bound in because, unlike the encapsulation key, it travels outside the noise
// transcript.
func kemPSK(sharedKey, ciphertext []byte) ([]byte, error) {
secret := make([]byte, 0, len(sharedKey)+len(ciphertext))
secret = append(secret, sharedKey...)
secret = append(secret, ciphertext...)
return hkdf.Extract(sha256.New, secret, []byte(kemPSKSalt))
}
// kemEncapsulate is the responder half: encapsulate to the initiator's key,
// install the psk, and remember the ciphertext to prefix the response with.
func (m *Machine) kemEncapsulate(encapsulationKey []byte) error {
ek, err := mlkem.NewEncapsulationKey1024(encapsulationKey)
if err != nil {
return fmt.Errorf("%w: %w", ErrInvalidKEMKey, err)
}
sharedKey, ct := ek.Encapsulate()
psk, err := kemPSK(sharedKey, ct)
if err != nil {
return err
}
if err := m.hs.SetPresharedKey(psk); err != nil {
return err
}
m.kemCiphertext = ct
return nil
}
// kemDecapsulate is the initiator half: recover the shared secret from the
// ciphertext prefixing the response and install the psk. It returns the rest of
// the message, which is the noise message.
func (m *Machine) kemDecapsulate(msg []byte) ([]byte, error) {
if len(msg) < mlkem.CiphertextSize1024 {
return nil, ErrPacketTooShort
}
if m.kemKey == nil {
return nil, errors.New("no ML-KEM decapsulation key for this handshake")
}
ct := msg[:mlkem.CiphertextSize1024]
sharedKey, err := m.kemKey.Decapsulate(ct)
if err != nil {
return nil, err
}
psk, err := kemPSK(sharedKey, ct)
if err != nil {
return nil, err
}
if err := m.hs.SetPresharedKey(psk); err != nil {
return nil, err
}
return msg[mlkem.CiphertextSize1024:], nil
}
-208
View File
@@ -1,208 +0,0 @@
package handshake
import (
"bytes"
"crypto/mlkem"
"net/netip"
"testing"
"time"
"github.com/flynn/noise"
"github.com/slackhq/nebula/cert"
ct "github.com/slackhq/nebula/cert_test"
"github.com/slackhq/nebula/header"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func newKEMMachine(t *testing.T, cs *testCertState, v CertVerifier, initiator bool, index uint32) *Machine {
t.Helper()
m, err := NewMachine(
cs.version, cs.getCredential, v,
func() (uint32, error) { return index, nil },
initiator, header.HandshakeIXPSK2MLKEM1024,
)
require.NoError(t, err)
return m
}
type kemFixture struct {
initCS, respCS *testCertState
v CertVerifier
}
func newKEMFixture(t *testing.T, cipher noise.CipherFunc) kemFixture {
ca, _, caKey, _ := ct.NewTestCaCert(
cert.Version2, cert.Curve_CURVE25519, time.Time{}, time.Time{}, nil, nil, nil,
)
return kemFixture{
initCS: newTestCertStateWithCipher(t, ca, caKey, "initiator", []netip.Prefix{netip.MustParsePrefix("10.0.0.1/24")}, cipher),
respCS: newTestCertStateWithCipher(t, ca, caKey, "responder", []netip.Prefix{netip.MustParsePrefix("10.0.0.2/24")}, cipher),
v: testVerifier(ct.NewTestCAPool(ca)),
}
}
func requireKeysMatch(t *testing.T, a, b *Result) {
t.Helper()
c, err := a.EKey.Encrypt(nil, nil, []byte("hello"))
require.NoError(t, err)
p, err := b.DKey.Decrypt(nil, nil, c)
require.NoError(t, err)
assert.Equal(t, []byte("hello"), p)
c, err = b.EKey.Encrypt(nil, nil, []byte("world"))
require.NoError(t, err)
p, err = a.DKey.Decrypt(nil, nil, c)
require.NoError(t, err)
assert.Equal(t, []byte("world"), p)
}
func TestKEMHandshake(t *testing.T) {
for _, cipher := range []noise.CipherFunc{noise.CipherChaChaPoly, noise.CipherAESGCM} {
t.Run(cipher.CipherName(), func(t *testing.T) {
f := newKEMFixture(t, cipher)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newKEMMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
assert.Equal(t, header.HandshakeIXPSK2MLKEM1024, header.MessageSubType(msg1[1]))
assert.Greater(t, len(msg1), header.Len+mlkem.EncapsulationKeySize1024)
msg2, respR, err := respM.ProcessPacket(nil, msg1)
require.NoError(t, err)
require.NotNil(t, respR)
assert.Greater(t, len(msg2), header.Len+mlkem.CiphertextSize1024)
_, initR, err := initM.ProcessPacket(nil, msg2)
require.NoError(t, err)
require.NotNil(t, initR)
assert.Equal(t, "responder", initR.RemoteCert.Certificate.Name())
assert.Equal(t, "initiator", respR.RemoteCert.Certificate.Name())
assert.Equal(t, uint32(2000), initR.RemoteIndex)
assert.Equal(t, uint32(1000), respR.RemoteIndex)
requireKeysMatch(t, initR, respR)
})
}
}
// TestKEMSecretMatters checks the ML-KEM secret really reaches the transport
// keys: an initiator holding a different decapsulation key cannot complete even
// though every DH input is genuine.
func TestKEMSecretMatters(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newKEMMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
msg2, _, err := respM.ProcessPacket(nil, msg1)
require.NoError(t, err)
other, err := mlkem.GenerateKey1024()
require.NoError(t, err)
initM.kemKey = other
_, res, err := initM.ProcessPacket(nil, msg2)
require.Error(t, err)
assert.Nil(t, res)
}
func TestKEMTamperedCiphertext(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newKEMMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
msg2, respR, err := respM.ProcessPacket(nil, msg1)
require.NoError(t, err)
t.Run("truncated", func(t *testing.T) {
_, res, err := initM.ProcessPacket(nil, msg2[:header.Len+mlkem.CiphertextSize1024-1])
require.ErrorIs(t, err, ErrPacketTooShort)
assert.Nil(t, res)
assert.False(t, initM.Failed())
})
t.Run("flipped ciphertext bit", func(t *testing.T) {
bad := bytes.Clone(msg2)
bad[header.Len+10] ^= 1
_, res, err := initM.ProcessPacket(nil, bad)
require.Error(t, err)
assert.Nil(t, res)
assert.False(t, initM.Failed(), "a forged ciphertext must not kill the handshake")
})
t.Run("genuine response still completes", func(t *testing.T) {
_, initR, err := initM.ProcessPacket(nil, msg2)
require.NoError(t, err)
require.NotNil(t, initR)
requireKeysMatch(t, initR, respR)
})
}
// TestKEMBadMsg1 drives processPayload by hand. With the psk modifier msg1's
// payload is encrypted under a key derived from the public ephemeral, so a
// forged payload would need re-encrypting; the parsing rules are what matter.
func TestKEMBadMsg1(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
msg1Flags := subtypeInfos[header.HandshakeIXPSK2MLKEM1024].msgs[0]
cert := f.initCS.creds[cert.Version2]
payload := func(kemKey []byte) []byte {
return MarshalPayload(nil, Payload{
Cert: cert.Bytes,
CertVersion: uint32(cert.Cert.Version()),
InitiatorIndex: 1000,
Time: 1,
KEMKey: kemKey,
})
}
dk, err := mlkem.GenerateKey1024()
require.NoError(t, err)
t.Run("missing KEM key", func(t *testing.T) {
m := newKEMMachine(t, f.respCS, f.v, false, 2000)
err := m.processPayload(payload(nil), msg1Flags)
require.ErrorIs(t, err, ErrUnexpectedContent)
assert.True(t, m.Failed())
})
t.Run("invalid KEM key", func(t *testing.T) {
m := newKEMMachine(t, f.respCS, f.v, false, 2000)
// Coefficients of 0xfff exceed the modulus, which ML-KEM must reject.
bad := bytes.Repeat([]byte{0xff}, mlkem.EncapsulationKeySize1024)
err := m.kemEncapsulate(bad)
require.ErrorIs(t, err, ErrInvalidKEMKey)
err = m.kemEncapsulate(bad[:100])
require.ErrorIs(t, err, ErrInvalidKEMKey)
})
t.Run("valid KEM key", func(t *testing.T) {
m := newKEMMachine(t, f.respCS, f.v, false, 2000)
require.NoError(t, m.kemEncapsulate(dk.EncapsulationKey().Bytes()))
assert.Len(t, m.kemCiphertext, mlkem.CiphertextSize1024)
})
t.Run("KEM key on the classical subtype", func(t *testing.T) {
m := newTestMachine(t, f.respCS, f.v, false, 2000)
classical := subtypeInfos[header.HandshakeIXPSK0].msgs[0]
err := m.processPayload(payload(dk.EncapsulationKey().Bytes()), classical)
require.ErrorIs(t, err, ErrUnexpectedContent)
assert.True(t, m.Failed())
})
}
// TestKEMDistinctFromClassical makes sure the two subtypes cannot be mixed: a
// classical responder cannot answer a KEM initiator.
func TestKEMDistinctFromClassical(t *testing.T) {
f := newKEMFixture(t, noise.CipherChaChaPoly)
initM := newKEMMachine(t, f.initCS, f.v, true, 1000)
respM := newTestMachine(t, f.respCS, f.v, false, 2000)
msg1, err := initM.Initiate(nil)
require.NoError(t, err)
_, _, err = respM.ProcessPacket(nil, msg1)
require.ErrorIs(t, err, ErrSubtypeMismatch)
}
+5 -54
View File
@@ -2,8 +2,6 @@ package handshake
import ( import (
"bytes" "bytes"
"crypto/mlkem"
"errors"
"fmt" "fmt"
"slices" "slices"
"time" "time"
@@ -69,9 +67,6 @@ type Machine struct {
remoteCertSet bool remoteCertSet bool
payloadSet bool payloadSet bool
failed bool failed bool
kemKey *mlkem.DecapsulationKey1024 // initiator's ephemeral ML-KEM key, until completion
kemCiphertext []byte // responder's ciphertext to prefix its response with
} }
// NewMachine creates a handshake state machine. The subtype determines both // NewMachine creates a handshake state machine. The subtype determines both
@@ -96,7 +91,7 @@ func NewMachine(
return nil, fmt.Errorf("%w: %v", ErrNoCredential, version) return nil, fmt.Errorf("%w: %v", ErrNoCredential, version)
} }
hs, err := cred.buildHandshakeState(initiator, info.pattern, info.pskPlacement) hs, err := cred.buildHandshakeState(initiator, info.pattern)
if err != nil { if err != nil {
return nil, fmt.Errorf("build noise state: %w", err) return nil, fmt.Errorf("build noise state: %w", err)
} }
@@ -142,8 +137,7 @@ func (m *Machine) requireComplete() error {
return nil return nil
} }
// myMsgFlags returns the flags for the current outgoing message, which are // myMsgFlags returns the flags for the current outgoing message.
// also the flags of the next incoming message before it is read.
func (m *Machine) myMsgFlags() msgFlags { func (m *Machine) myMsgFlags() msgFlags {
idx := m.hs.MessageIndex() idx := m.hs.MessageIndex()
if idx < len(m.msgs) { if idx < len(m.msgs) {
@@ -231,18 +225,7 @@ func (m *Machine) ProcessPacket(out, packet []byte) ([]byte, *Result, error) {
// noise returns (cs1, cs2) where cs1 is the initiator->responder cipher. // noise returns (cs1, cs2) where cs1 is the initiator->responder cipher.
// For 3-message patterns where a responder finishes by reading the final // For 3-message patterns where a responder finishes by reading the final
// message, this ordering would be wrong; revisit when XX/pqIX lands. // message, this ordering would be wrong; revisit when XX/pqIX lands.
noiseMsg := packet[header.Len:] msg, eKey, dKey, err := m.hs.ReadMessage(nil, packet[header.Len:])
if m.myMsgFlags().kemCiphertext {
var err error
// Installing a psk derived from a forged ciphertext is harmless: ReadMessage
// then fails without advancing, and the next packet installs its own.
noiseMsg, err = m.kemDecapsulate(noiseMsg)
if err != nil {
return nil, nil, err
}
}
msg, eKey, dKey, err := m.hs.ReadMessage(nil, noiseMsg)
if err != nil { if err != nil {
// Noise ReadMessage failed. The noise library checkpoints and rolls back // Noise ReadMessage failed. The noise library checkpoints and rolls back
// on failure, so the Machine is still alive. The caller can retry with // on failure, so the Machine is still alive. The caller can retry with
@@ -293,8 +276,6 @@ func (m *Machine) ProcessPacket(out, packet []byte) ([]byte, *Result, error) {
} }
func (m *Machine) completed(eKey, dKey *noise.CipherState) *Result { func (m *Machine) completed(eKey, dKey *noise.CipherState) *Result {
m.kemKey = nil
m.kemCiphertext = nil
m.result.EKey = eKey m.result.EKey = eKey
m.result.DKey = dKey m.result.DKey = dKey
m.result.MessageIndex = uint64(m.hs.MessageIndex()) m.result.MessageIndex = uint64(m.hs.MessageIndex())
@@ -303,7 +284,7 @@ func (m *Machine) completed(eKey, dKey *noise.CipherState) *Result {
func (m *Machine) processPayload(msg []byte, flags msgFlags) error { func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
if len(msg) == 0 { if len(msg) == 0 {
if flags.expectsPayload || flags.expectsCert || flags.kemKey { if flags.expectsPayload || flags.expectsCert {
m.failed = true m.failed = true
return ErrMissingContent return ErrMissingContent
} }
@@ -329,12 +310,6 @@ func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
return ErrUnexpectedContent return ErrUnexpectedContent
} }
hasKEMKey := len(payload.KEMKey) > 0
if hasKEMKey != flags.kemKey {
m.failed = true
return ErrUnexpectedContent
}
// Process payload // Process payload
if flags.expectsPayload { if flags.expectsPayload {
var remoteIndex uint32 var remoteIndex uint32
@@ -361,13 +336,6 @@ func (m *Machine) processPayload(msg []byte, flags msgFlags) error {
} }
} }
if flags.kemKey {
if err := m.kemEncapsulate(payload.KEMKey); err != nil {
m.failed = true
return err
}
}
return nil return nil
} }
@@ -412,21 +380,11 @@ func (m *Machine) validateCert(payload Payload) error {
} }
func (m *Machine) marshalOutgoing(flags msgFlags) ([]byte, error) { func (m *Machine) marshalOutgoing(flags msgFlags) ([]byte, error) {
if !flags.expectsPayload && !flags.expectsCert && !flags.kemKey { if !flags.expectsPayload && !flags.expectsCert {
return nil, nil return nil, nil
} }
var p Payload var p Payload
if flags.kemKey {
if m.kemKey == nil {
dk, err := mlkem.GenerateKey1024()
if err != nil {
return nil, fmt.Errorf("generate ML-KEM key: %w", err)
}
m.kemKey = dk
}
p.KEMKey = m.kemKey.EncapsulationKey().Bytes()
}
if flags.expectsPayload { if flags.expectsPayload {
if !m.indexAllocated { if !m.indexAllocated {
index, err := m.allocIndex() index, err := m.allocIndex()
@@ -478,13 +436,6 @@ func (m *Machine) buildResponse(out []byte) ([]byte, *noise.CipherState, *noise.
uint64(m.hs.MessageIndex()+1), uint64(m.hs.MessageIndex()+1),
) )
if flags.kemCiphertext {
if m.kemCiphertext == nil {
return nil, nil, nil, errors.New("no ML-KEM ciphertext to send")
}
out = append(out, m.kemCiphertext...)
}
// noise.WriteMessage appends the encrypted handshake message to out, // noise.WriteMessage appends the encrypted handshake message to out,
// reusing capacity when present. // reusing capacity when present.
// //
+2 -21
View File
@@ -11,16 +11,13 @@ import (
type msgFlags struct { type msgFlags struct {
expectsPayload bool // message carries indexes and time expectsPayload bool // message carries indexes and time
expectsCert bool // message carries the certificate expectsCert bool // message carries the certificate
kemKey bool // payload carries an ML-KEM encapsulation key
kemCiphertext bool // message is prefixed with the ML-KEM ciphertext for the peer's key
} }
// subtypeInfo bundles the noise pattern with the per-message flags for a // subtypeInfo bundles the noise pattern with the per-message flags for a
// given handshake subtype. // given handshake subtype.
type subtypeInfo struct { type subtypeInfo struct {
pattern noise.HandshakePattern pattern noise.HandshakePattern
pskPlacement int // noise psk modifier position, 0 means no psk token msgs []msgFlags
msgs []msgFlags
} }
// subtypeInfos defines the noise pattern and message content layout for each // subtypeInfos defines the noise pattern and message content layout for each
@@ -35,22 +32,6 @@ var subtypeInfos = map[header.MessageSubType]subtypeInfo{
}, },
}, },
// IX with an ephemeral ML-KEM-1024 exchange:
// msg1 (I->R): payload + cert + encapsulation key
// msg2 (R->I): ciphertext || noise(payload + cert)
// The responder encapsulates after reading msg1 and mixes the shared secret in
// through the psk2 token at the end of msg2. The ciphertext travels outside the
// noise message because the initiator needs the secret before it can process
// that token.
header.HandshakeIXPSK2MLKEM1024: {
pattern: noise.HandshakeIX,
pskPlacement: 2,
msgs: []msgFlags{
{expectsPayload: true, expectsCert: true, kemKey: true},
{expectsPayload: true, expectsCert: true, kemCiphertext: true},
},
},
// XX: 3 messages // XX: 3 messages
// msg1 (I->R): payload only // msg1 (I->R): payload only
// msg2 (R->I): payload + cert // msg2 (R->I): payload + cert
-16
View File
@@ -20,7 +20,6 @@ type Payload struct {
ResponderIndex uint32 ResponderIndex uint32
Time uint64 Time uint64
CertVersion uint32 CertVersion uint32
KEMKey []byte // ML-KEM encapsulation key, only in subtypes that carry one
} }
// Proto field numbers for NebulaHandshakeDetails // Proto field numbers for NebulaHandshakeDetails
@@ -30,7 +29,6 @@ const (
fieldResponderIndex = 3 // uint32 fieldResponderIndex = 3 // uint32
fieldTime = 5 // uint64 fieldTime = 5 // uint64
fieldCertVersion = 8 // uint32 fieldCertVersion = 8 // uint32
fieldKEMKey = 9 // bytes
) )
// MarshalPayload encodes a handshake payload in protobuf wire format compatible // MarshalPayload encodes a handshake payload in protobuf wire format compatible
@@ -59,10 +57,6 @@ func MarshalPayload(out []byte, p Payload) []byte {
details = protowire.AppendTag(details, fieldCertVersion, protowire.VarintType) details = protowire.AppendTag(details, fieldCertVersion, protowire.VarintType)
details = protowire.AppendVarint(details, uint64(p.CertVersion)) details = protowire.AppendVarint(details, uint64(p.CertVersion))
} }
if len(p.KEMKey) > 0 {
details = protowire.AppendTag(details, fieldKEMKey, protowire.BytesType)
details = protowire.AppendBytes(details, p.KEMKey)
}
out = protowire.AppendTag(out, 1, protowire.BytesType) out = protowire.AppendTag(out, 1, protowire.BytesType)
out = protowire.AppendBytes(out, details) out = protowire.AppendBytes(out, details)
@@ -167,16 +161,6 @@ func unmarshalPayloadDetails(p *Payload, b []byte) error {
} }
p.CertVersion = uint32(v) p.CertVersion = uint32(v)
b = b[n:] b = b[n:]
case fieldKEMKey:
if typ != protowire.BytesType {
return errInvalidHandshakeDetails
}
v, n := protowire.ConsumeBytes(b)
if n < 0 {
return errInvalidHandshakeDetails
}
p.KEMKey = append([]byte(nil), v...)
b = b[n:]
default: default:
n := protowire.ConsumeFieldValue(num, typ, b) n := protowire.ConsumeFieldValue(num, typ, b)
if n < 0 { if n < 0 {
+46 -151
View File
@@ -10,15 +10,12 @@ import (
"net/netip" "net/netip"
"slices" "slices"
"sync" "sync"
"sync/atomic"
"time" "time"
"github.com/rcrowley/go-metrics" "github.com/rcrowley/go-metrics"
"github.com/slackhq/nebula/cert" "github.com/slackhq/nebula/cert"
"github.com/slackhq/nebula/handshake" "github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header" "github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/ratelimit"
"github.com/slackhq/nebula/udp" "github.com/slackhq/nebula/udp"
) )
@@ -52,7 +49,6 @@ type HandshakeConfig struct {
tryInterval time.Duration tryInterval time.Duration
retries int64 retries int64
triggerBuffer int triggerBuffer int
pq pqHandshakeConfig
messageMetrics *MessageMetrics messageMetrics *MessageMetrics
} }
@@ -78,18 +74,6 @@ type HandshakeManager struct {
// can be used to trigger outbound handshake for the given vpnIp // can be used to trigger outbound handshake for the given vpnIp
trigger chan netip.Addr trigger chan netip.Addr
// pq holds the post-quantum settings, which change on reload.
pq atomic.Pointer[pqHandshakeConfig]
// pqTable reassembles post-quantum msg1s from senders that have not authenticated yet.
pqTable *fragment.Table[pqFragKey, pqFragSource]
// pqPeers holds vpn addresses that completed a post-quantum handshake, for sticky mode.
pqPeers sync.Map
// pqAnswered recognises repeats of post-quantum msg1s already answered.
pqAnswered *pqAnswered
// pqCookies and pqCookieLimiter serve cookies while pqTable is under load.
pqCookies *pqCookies
pqCookieLimiter *ratelimit.Limiter
} }
type HandshakeHostInfo struct { type HandshakeHostInfo struct {
@@ -105,8 +89,6 @@ type HandshakeHostInfo struct {
hostinfo *HostInfo hostinfo *HostInfo
machine *handshake.Machine // The handshake state machine, set during stage 0 (initiator) or beginHandshake (responder multi-message) machine *handshake.Machine // The handshake state machine, set during stage 0 (initiator) or beginHandshake (responder multi-message)
pq *pqAttempt // The initiator's post-quantum attempt, pending alongside the classical one
lastPQ bool // Whether the latest attempt was post-quantum
} }
func (hh *HandshakeHostInfo) cachePacket(l *slog.Logger, t header.MessageType, st header.MessageSubType, packet []byte, f packetCallback, m *cachedPacketMetrics) { func (hh *HandshakeHostInfo) cachePacket(l *slog.Logger, t header.MessageType, st header.MessageSubType, packet []byte, f packetCallback, m *cachedPacketMetrics) {
@@ -135,7 +117,7 @@ func (hh *HandshakeHostInfo) cachePacket(l *slog.Logger, t header.MessageType, s
} }
func NewHandshakeManager(l *slog.Logger, mainHostMap *HostMap, lightHouse *LightHouse, outside udp.Conn, config HandshakeConfig) *HandshakeManager { func NewHandshakeManager(l *slog.Logger, mainHostMap *HostMap, lightHouse *LightHouse, outside udp.Conn, config HandshakeConfig) *HandshakeManager {
hm := &HandshakeManager{ return &HandshakeManager{
vpnIps: map[netip.Addr]*HandshakeHostInfo{}, vpnIps: map[netip.Addr]*HandshakeHostInfo{},
indexes: map[uint32]*HandshakeHostInfo{}, indexes: map[uint32]*HandshakeHostInfo{},
mainHostMap: mainHostMap, mainHostMap: mainHostMap,
@@ -148,16 +130,7 @@ func NewHandshakeManager(l *slog.Logger, mainHostMap *HostMap, lightHouse *Light
metricInitiated: metrics.GetOrRegisterCounter("handshake_manager.initiated", nil), metricInitiated: metrics.GetOrRegisterCounter("handshake_manager.initiated", nil),
metricTimedOut: metrics.GetOrRegisterCounter("handshake_manager.timed_out", nil), metricTimedOut: metrics.GetOrRegisterCounter("handshake_manager.timed_out", nil),
l: l, l: l,
pqTable: fragment.NewTable[pqFragKey, pqFragSource](pqTableConfig),
pqAnswered: newPQAnswered(hsTimeout(config.retries, config.tryInterval)),
} }
pq := config.pq
hm.pq.Store(&pq)
hm.pqCookies = newPQCookies(time.Now())
// Cookie replies are tiny, but a flood of spoofed fragments must not turn
// into a flood of replies.
hm.pqCookieLimiter, _ = ratelimit.New(256, time.Millisecond)
return hm
} }
func (hm *HandshakeManager) Run(ctx context.Context) { func (hm *HandshakeManager) Run(ctx context.Context) {
@@ -172,8 +145,6 @@ func (hm *HandshakeManager) Run(ctx context.Context) {
hm.handleOutbound(vpnIP, true) hm.handleOutbound(vpnIP, true)
case now := <-clockSource.C: case now := <-clockSource.C:
hm.NextOutboundHandshakeTimerTick(now) hm.NextOutboundHandshakeTimerTick(now)
hm.pqTable.Expire(now)
hm.pqAnswered.expire(now)
} }
} }
} }
@@ -183,7 +154,7 @@ func (hm *HandshakeManager) HandleIncoming(via ViaSender, packet []byte, h *head
// don't yet support) are dropped here rather than silently routed through // don't yet support) are dropped here rather than silently routed through
// the IX path. Add a case when introducing a new pattern. // the IX path. Add a case when introducing a new pattern.
switch h.Subtype { switch h.Subtype {
case header.HandshakeIXPSK0, pqSubtype: case header.HandshakeIXPSK0:
// supported // supported
default: default:
hm.l.Debug("dropping handshake with unsupported subtype", hm.l.Debug("dropping handshake with unsupported subtype",
@@ -199,11 +170,6 @@ func (hm *HandshakeManager) HandleIncoming(via ViaSender, packet []byte, h *head
} }
} }
if header.IsFragmented(h.Subtype) {
hm.handleFragment(via, packet, h)
return
}
// First message of a new handshake. The wire format requires RemoteIndex // First message of a new handshake. The wire format requires RemoteIndex
// to be zero here (the initiator has no responder index to fill in yet), // to be zero here (the initiator has no responder index to fill in yet),
// and generateIndex never allocates 0, so any non-zero RemoteIndex on a // and generateIndex never allocates 0, so any non-zero RemoteIndex on a
@@ -215,11 +181,7 @@ func (hm *HandshakeManager) HandleIncoming(via ViaSender, packet []byte, h *head
"from", via, "remoteIndex", h.RemoteIndex) "from", via, "remoteIndex", h.RemoteIndex)
return return
} }
if hm.pqConfig().require { hm.beginHandshake(via, packet, h)
hm.l.Debug("dropping classical handshake, post-quantum handshakes are required", "from", via)
return
}
hm.beginHandshake(via, packet, h, 0)
return return
} }
@@ -268,9 +230,6 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
"style": header.SubTypeName(header.Handshake, hh.machine.Subtype()), "style": header.SubTypeName(header.Handshake, hh.machine.Subtype()),
}) })
} }
if hh.pq != nil {
fields = append(fields, "pqInitiatorIndex", hh.pq.index)
}
hh.hostinfo.logger(hm.l).Info("Handshake timed out", fields...) hh.hostinfo.logger(hm.l).Info("Handshake timed out", fields...)
hm.metricTimedOut.Inc(1) hm.metricTimedOut.Inc(1)
hm.DeleteHostInfo(hostinfo) hm.DeleteHostInfo(hostinfo)
@@ -280,45 +239,25 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
// Increment the counter to increase our delay, linear backoff // Increment the counter to increase our delay, linear backoff
hh.counter++ hh.counter++
// Pick this attempt's style and make sure its first message exists. A PQ // Check if we have a handshake packet to transmit yet
// attempt sends msg1 as fragments: now goes out immediately, later after the if !hh.ready {
// repair delay, and every attempt offers fresh repair symbols. if !hm.buildStage0Packet(hh) {
var now, later [][]byte
var machine *handshake.Machine
hh.lastPQ = hm.usePQ(hh, vpnIp, lighthouseTriggered)
if hh.lastPQ {
if hh.pq == nil && !hm.buildPQAttempt(hh) {
hm.OutboundHandshakeTimer.Add(vpnIp, hm.config.tryInterval*time.Duration(hh.counter)) hm.OutboundHandshakeTimer.Add(vpnIp, hm.config.tryInterval*time.Duration(hh.counter))
return return
} }
var err error
now, later, err = hh.pq.sender.Attempt(fragment.DefaultRepair(hh.pq.sender.Layout().K))
if err != nil {
hostinfo.logger(hm.l).Error("Failed to fragment handshake", "error", err)
hm.OutboundHandshakeTimer.Add(vpnIp, hm.config.tryInterval*time.Duration(hh.counter))
return
}
machine = hh.pq.machine
} else {
if !hh.ready {
if !hm.buildStage0Packet(hh) {
hm.OutboundHandshakeTimer.Add(vpnIp, hm.config.tryInterval*time.Duration(hh.counter))
return
}
}
// TODO: this hardcodes "always retransmit stage 0", which is correct for
// IX (the initiator only ever sends one packet, msg1) but wrong the
// moment a 3+ message pattern lands. The retry loop should resend the
// most recent outgoing message, not always stage 0. That implies
// HandshakeHostInfo tracking a single "currentOutbound" packet (bytes +
// header metadata) that gets replaced as the handshake progresses,
// instead of indexing into HandshakePacket.
now = [][]byte{hostinfo.HandshakePacket[handshakePacketStage0]}
machine = hh.machine
} }
// TODO: this hardcodes "always retransmit stage 0", which is correct for
// IX (the initiator only ever sends one packet, msg1) but wrong the
// moment a 3+ message pattern lands. The retry loop should resend the
// most recent outgoing message, not always stage 0. That implies
// HandshakeHostInfo tracking a single "currentOutbound" packet (bytes +
// header metadata) that gets replaced as the handshake progresses,
// instead of indexing into HandshakePacket.
stage0 := hostinfo.HandshakePacket[handshakePacketStage0]
hsFields := m{ hsFields := m{
"stage": uint64(machine.MessageIndex()), "stage": uint64(hh.machine.MessageIndex()),
"style": header.SubTypeName(header.Handshake, machine.Subtype()), "style": header.SubTypeName(header.Handshake, hh.machine.Subtype()),
} }
remotes := hostinfo.remotes.CopyAddrs(hm.mainHostMap.GetPreferredRanges()) remotes := hostinfo.remotes.CopyAddrs(hm.mainHostMap.GetPreferredRanges())
@@ -346,10 +285,8 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
// Send the handshake to all known ips, stage 2 takes care of assigning the hostinfo.remote based on the first to reply // Send the handshake to all known ips, stage 2 takes care of assigning the hostinfo.remote based on the first to reply
var sentTo []netip.AddrPort var sentTo []netip.AddrPort
hostinfo.remotes.ForEach(hm.mainHostMap.GetPreferredRanges(), func(addr netip.AddrPort, _ bool) { hostinfo.remotes.ForEach(hm.mainHostMap.GetPreferredRanges(), func(addr netip.AddrPort, _ bool) {
hm.messageMetrics.Tx(header.Handshake, machine.Subtype(), int64(len(now)+len(later))) hm.messageMetrics.Tx(header.Handshake, hh.machine.Subtype(), 1)
err := hm.sendFragmented(now, later, func(p []byte) error { err := hm.outside.WriteTo(stage0, addr)
return hm.outside.WriteTo(p, addr)
})
if err != nil { if err != nil {
// These repeat every attempt, so match the success log below and only shout when the remotes changed // These repeat every attempt, so match the success log below and only shout when the remotes changed
level := slog.LevelDebug level := slog.LevelDebug
@@ -385,7 +322,7 @@ func (hm *HandshakeManager) handleOutbound(vpnIp netip.Addr, lighthouseTriggered
) )
} }
hm.f.relayManager.StartRelays(hm.f, vpnIp, hh, now, later) hm.f.relayManager.StartRelays(hm.f, vpnIp, hh, stage0)
// If a lighthouse triggered this attempt then we are still in the timer wheel and do not need to re-add // If a lighthouse triggered this attempt then we are still in the timer wheel and do not need to re-add
if !lighthouseTriggered { if !lighthouseTriggered {
@@ -595,10 +532,6 @@ func (hm *HandshakeManager) DeleteHostInfo(hostinfo *HostInfo) {
func (hm *HandshakeManager) unlockedDeleteHostInfo(hostinfo *HostInfo) { func (hm *HandshakeManager) unlockedDeleteHostInfo(hostinfo *HostInfo) {
for _, addr := range hostinfo.vpnAddrs { for _, addr := range hostinfo.vpnAddrs {
if cur, ok := hm.vpnIps[addr]; ok && cur.hostinfo == hostinfo { if cur, ok := hm.vpnIps[addr]; ok && cur.hostinfo == hostinfo {
// A PQ attempt holds its own index alongside the hostinfo's.
if idx := cur.pqIndex(); idx != 0 && hm.indexes[idx] == cur {
delete(hm.indexes, idx)
}
delete(hm.vpnIps, addr) delete(hm.vpnIps, addr)
} }
} }
@@ -709,8 +642,9 @@ func hsTimeout(tries int64, interval time.Duration) time.Duration {
return time.Duration(tries / 2 * ((2 * int64(interval)) + (tries-1)*int64(interval))) return time.Duration(tries / 2 * ((2 * int64(interval)) + (tries-1)*int64(interval)))
} }
// initiatingVersion picks the certificate version to initiate a handshake with. // buildStage0Packet creates the initial handshake packet for the initiator.
func (hm *HandshakeManager) initiatingVersion(hh *HandshakeHostInfo, cs *CertState) cert.Version { func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
cs := hm.f.pki.getCertState()
v := cs.DefaultVersion() v := cs.DefaultVersion()
if hh.initiatingVersionOverride != cert.VersionPre1 { if hh.initiatingVersionOverride != cert.VersionPre1 {
v = hh.initiatingVersionOverride v = hh.initiatingVersionOverride
@@ -722,13 +656,6 @@ func (hm *HandshakeManager) initiatingVersion(hh *HandshakeHostInfo, cs *CertSta
} }
} }
} }
return v
}
// buildStage0Packet creates the initial handshake packet for the initiator.
func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
cs := hm.f.pki.getCertState()
v := hm.initiatingVersion(hh, cs)
cred := cs.GetCredential(v) cred := cs.GetCredential(v)
if cred == nil { if cred == nil {
@@ -767,10 +694,7 @@ func (hm *HandshakeManager) buildStage0Packet(hh *HandshakeHostInfo) bool {
// beginHandshake handles an incoming handshake packet that doesn't match any // beginHandshake handles an incoming handshake packet that doesn't match any
// existing pending handshake. It creates a new responder Machine and processes // existing pending handshake. It creates a new responder Machine and processes
// the first message. // the first message.
// func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *header.H) {
// received is the number of bytes a fragmented msg1 arrived in, which bounds the
// response sent back before the address is validated; 0 for unfragmented ones.
func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *header.H, received int) {
f := hm.f f := hm.f
cs := f.pki.getCertState() cs := f.pki.getCertState()
@@ -784,7 +708,7 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
machine, err := handshake.NewMachine( machine, err := handshake.NewMachine(
v, cs.GetCredential, v, cs.GetCredential,
hm.certVerifier(), func() (uint32, error) { return generateIndex(f.l) }, hm.certVerifier(), func() (uint32, error) { return generateIndex(f.l) },
false, h.Subtype, false, header.HandshakeIXPSK0,
) )
if err != nil { if err != nil {
f.l.Error("Failed to create handshake machine", "from", via, "error", err) f.l.Error("Failed to create handshake machine", "from", via, "error", err)
@@ -874,15 +798,11 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
existing, err := hm.CheckAndComplete(hostinfo, handshakePacketStage0, f) existing, err := hm.CheckAndComplete(hostinfo, handshakePacketStage0, f)
if err != nil { if err != nil {
hm.handleCheckAndCompleteError(err, existing, hostinfo, via, machine.Subtype(), received) hm.handleCheckAndCompleteError(err, existing, hostinfo, via)
return return
} }
hm.sendHandshakeResponse(via, response, hostinfo, false, received) hm.sendHandshakeResponse(via, response, hostinfo, false)
if header.IsFragmented(machine.Subtype()) {
hm.pqAnswered.add(packet[header.Len:], hostinfo, time.Now())
}
hm.notePQPeer(machine.Subtype(), vpnAddrs)
hostinfo.remotes.RefreshFromHandshake(vpnAddrs) hostinfo.remotes.RefreshFromHandshake(vpnAddrs)
// Don't wait for UpdateWorker // Don't wait for UpdateWorker
@@ -893,32 +813,23 @@ func (hm *HandshakeManager) beginHandshake(via ViaSender, packet []byte, h *head
// continueHandshake feeds an incoming packet to an existing pending handshake Machine. // continueHandshake feeds an incoming packet to an existing pending handshake Machine.
func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostInfo, packet []byte) { func (hm *HandshakeManager) continueHandshake(via ViaSender, hh *HandshakeHostInfo, packet []byte) {
f := hm.f
hh.Lock() hh.Lock()
defer hh.Unlock() defer hh.Unlock()
if !hm.stillPending(hh, hh.hostinfo.localIndexId) { // Re-verify hh is still tracked. Between queryIndex returning and us taking
// hh.Lock, handleOutbound may have timed out and deleted it. Once we hold
// hh.Lock no other deleter can race our index: handleOutbound also takes
// hh.Lock first, and handleRecvError targets a main-hostmap entry with a
// different localIndexId.
hm.RLock()
cur, ok := hm.indexes[hh.hostinfo.localIndexId]
hm.RUnlock()
if !ok || cur != hh {
return return
} }
hm.processResponse(via, hh, hh.machine, packet)
}
// stillPending re-verifies hh is still tracked under index. Between queryIndex
// returning and the caller taking hh.Lock, handleOutbound may have timed out and
// deleted it. Once hh.Lock is held no other deleter can race the index:
// handleOutbound also takes hh.Lock first, and handleRecvError targets a
// main-hostmap entry with a different localIndexId. The caller must hold hh.Lock.
func (hm *HandshakeManager) stillPending(hh *HandshakeHostInfo, index uint32) bool {
hm.RLock()
defer hm.RUnlock()
cur, ok := hm.indexes[index]
return ok && cur == hh
}
// processResponse runs a responder's message through one of the initiator's
// pending attempts and completes the tunnel when the handshake finishes. The
// caller must hold hh.Lock.
func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo, machine *handshake.Machine, packet []byte) {
f := hm.f
hostinfo := hh.hostinfo hostinfo := hh.hostinfo
if !via.IsRelayed { if !via.IsRelayed {
if !f.lightHouse.GetRemoteAllowList().AllowAll(hostinfo.vpnAddrs, via.UdpAddr.Addr()) { if !f.lightHouse.GetRemoteAllowList().AllowAll(hostinfo.vpnAddrs, via.UdpAddr.Addr()) {
@@ -928,6 +839,7 @@ func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo
} }
} }
machine := hh.machine
if machine == nil { if machine == nil {
f.l.Error("No handshake machine available for continuation", f.l.Error("No handshake machine available for continuation",
"vpnAddrs", hostinfo.vpnAddrs, "from", via) "vpnAddrs", hostinfo.vpnAddrs, "from", via)
@@ -950,7 +862,7 @@ func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo
} }
if response != nil { if response != nil {
hm.sendHandshakeResponse(via, response, hostinfo, false, 0) hm.sendHandshakeResponse(via, response, hostinfo, false)
} }
if result == nil { if result == nil {
@@ -1063,10 +975,7 @@ func (hm *HandshakeManager) processResponse(via ViaSender, hh *HandshakeHostInfo
hostinfo.vpnAddrs = vpnAddrs hostinfo.vpnAddrs = vpnAddrs
hostinfo.buildNetworks(f.myVpnNetworksTable, remoteCert.Certificate) hostinfo.buildNetworks(f.myVpnNetworksTable, remoteCert.Certificate)
// The classical and PQ attempts each hold an index; keep the one that won.
hm.settleIndex(hh, result.LocalIndex)
hm.Complete(hostinfo, f) hm.Complete(hostinfo, f)
hm.notePQPeer(machine.Subtype(), vpnAddrs)
if len(hh.packetStore) > 0 { if len(hh.packetStore) > 0 {
if f.l.Enabled(context.Background(), slog.LevelDebug) { if f.l.Enabled(context.Background(), slog.LevelDebug) {
@@ -1145,23 +1054,20 @@ func (hm *HandshakeManager) validatePeerCert(via ViaSender, remoteCert *cert.Cac
// cached is true when msg is a stored response being retransmitted because // cached is true when msg is a stored response being retransmitted because
// the peer's stage-1 retransmit landed (the ErrAlreadySeen path); false on a // the peer's stage-1 retransmit landed (the ErrAlreadySeen path); false on a
// fresh response. // fresh response.
// func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hostinfo *HostInfo, cached bool) {
// received is the number of bytes a fragmented msg1 arrived in; it bounds what a
// fragmented response may send back. It is ignored for unfragmented subtypes.
func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hostinfo *HostInfo, cached bool, received int) {
if msg == nil { if msg == nil {
return return
} }
f := hm.f f := hm.f
subtype := header.MessageSubType(msg[1]) f.messageMetrics.Tx(header.Handshake, header.MessageSubType(msg[1]), 1)
// Common log fields. peerCert may be nil during intermediate // Common log fields. peerCert may be nil during intermediate
// multi-message flows (handshake hasn't completed yet); skip the cert // multi-message flows (handshake hasn't completed yet); skip the cert
// block if so. // block if so.
logFields := []any{ logFields := []any{
"vpnAddrs", hostinfo.vpnAddrs, "vpnAddrs", hostinfo.vpnAddrs,
"handshake", m{"stage": uint64(2), "style": header.SubTypeName(header.Handshake, subtype)}, "handshake", m{"stage": uint64(2), "style": header.SubTypeName(header.Handshake, header.HandshakeIXPSK0)},
"cached", cached, "cached", cached,
"initiatorIndex", hostinfo.remoteIndexId, "initiatorIndex", hostinfo.remoteIndexId,
"responderIndex", hostinfo.localIndexId, "responderIndex", hostinfo.localIndexId,
@@ -1175,17 +1081,6 @@ func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hos
) )
} }
if header.IsFragmented(subtype) {
fields := append(logFields, "from", via)
if err := hm.sendFragmentedResponse(via, msg, hostinfo, cached, received); err != nil {
f.l.Debug("Did not send handshake message", append(fields, "error", err)...)
} else {
f.l.Info("Handshake message sent", fields...)
}
return
}
f.messageMetrics.Tx(header.Handshake, subtype, 1)
if !via.IsRelayed { if !via.IsRelayed {
fields := append(logFields, "from", via) fields := append(logFields, "from", via)
err := f.outside.WriteTo(msg, via.UdpAddr) err := f.outside.WriteTo(msg, via.UdpAddr)
@@ -1212,10 +1107,10 @@ func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hos
// This only fires from the responder-side beginHandshake path, after the // This only fires from the responder-side beginHandshake path, after the
// peer cert has been validated and ConnectionState populated, so peerCert // peer cert has been validated and ConnectionState populated, so peerCert
// is always non-nil for the cases that log it. // is always non-nil for the cases that log it.
func (hm *HandshakeManager) handleCheckAndCompleteError(err error, existing, hostinfo *HostInfo, via ViaSender, subtype header.MessageSubType, received int) { func (hm *HandshakeManager) handleCheckAndCompleteError(err error, existing, hostinfo *HostInfo, via ViaSender) {
f := hm.f f := hm.f
peerCert := hostinfo.ConnectionState.peerCert peerCert := hostinfo.ConnectionState.peerCert
hsFields := m{"stage": uint64(1), "style": header.SubTypeName(header.Handshake, subtype)} hsFields := m{"stage": uint64(1), "style": header.SubTypeName(header.Handshake, header.HandshakeIXPSK0)}
switch err { switch err {
case ErrAlreadySeen: case ErrAlreadySeen:
@@ -1226,7 +1121,7 @@ func (hm *HandshakeManager) handleCheckAndCompleteError(err error, existing, hos
// ephemeral keys; a freshly-built one would have different keys and break // ephemeral keys; a freshly-built one would have different keys and break
// the tunnel even though both sides "completed" the handshake. // the tunnel even though both sides "completed" the handshake.
if msg := existing.HandshakePacket[handshakePacketStage2]; msg != nil { if msg := existing.HandshakePacket[handshakePacketStage2]; msg != nil {
hm.sendHandshakeResponse(via, msg, existing, true, received) hm.sendHandshakeResponse(via, msg, existing, true)
} }
case ErrExistingHostInfo: case ErrExistingHostInfo:
-699
View File
@@ -1,699 +0,0 @@
package nebula
import (
"context"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/binary"
"errors"
"log/slog"
"net/netip"
"sync"
"time"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/handshake"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header"
)
const (
// pqSubtype is the post-quantum handshake this node speaks.
pqSubtype = header.HandshakeIXPSK2MLKEM1024
// DefaultPQRepairDelay is how long repair fragments trail the data fragments.
DefaultPQRepairDelay = 10 * time.Millisecond
// pqAmplification bounds what a responder sends in reply to an unvalidated
// address, as a multiple of what it received from it.
pqAmplification = 3
)
// pqTableConfig bounds the responder's reassembly of msg1 fragments from senders
// that have not authenticated yet.
var pqTableConfig = fragment.TableConfig{
MaxBytes: 4 << 20,
MaxEntries: 1024,
MaxPerSource: 4,
MaxMsgLen: 16384,
TTL: 3 * time.Second,
}
type pqHandshakeConfig struct {
initiate bool // start PQ handshakes
accept bool // answer PQ handshakes
require bool // only PQ handshakes, in either direction
sticky bool // never fall back to classical for a peer that completed PQ
repairDelay time.Duration // delay between data and repair fragments
}
func pqHandshakeConfigFrom(c *config.C) pqHandshakeConfig {
cfg := pqHandshakeConfig{
initiate: c.GetBool("handshakes.pq.initiate", false),
accept: c.GetBool("handshakes.pq.accept", false),
require: c.GetBool("handshakes.pq.require", false),
sticky: c.GetBool("handshakes.pq.sticky", false),
repairDelay: c.GetDuration("handshakes.pq.repair_delay", DefaultPQRepairDelay),
}
if cfg.require {
cfg.initiate = true
cfg.accept = true
}
return cfg
}
// pqConfig returns the post-quantum settings in effect.
func (hm *HandshakeManager) pqConfig() *pqHandshakeConfig {
return hm.pq.Load()
}
// reloadPQ applies changed post-quantum settings. Handshakes already in flight
// keep going; the new settings apply to the next attempt and packet.
func (hm *HandshakeManager) reloadPQ(c *config.C) {
if !c.HasChanged("handshakes.pq") {
return
}
old := hm.pqConfig()
cfg := pqHandshakeConfigFrom(c)
hm.pq.Store(&cfg)
hm.l.Info("Post-quantum handshake settings changed",
"initiate", cfg.initiate, "oldInitiate", old.initiate,
"accept", cfg.accept, "oldAccept", old.accept,
"require", cfg.require, "oldRequire", old.require,
"sticky", cfg.sticky, "oldSticky", old.sticky,
"repairDelay", cfg.repairDelay, "oldRepairDelay", old.repairDelay,
)
}
// pqAttempt is the initiator's post-quantum attempt for a pending handshake. It
// lives alongside the classical attempt in HandshakeHostInfo and has its own
// local index, so whichever answer arrives first wins.
type pqAttempt struct {
machine *handshake.Machine
index uint32
sender *fragment.Sender
msg2 *fragment.Assembly
cookieResendAt time.Time // last resend prompted by a cookie reply
heard bool // a msg2 fragment or cookie reply arrived, so the peer speaks PQ
}
// pqFragKey identifies one msg1 being reassembled.
type pqFragKey struct {
addr netip.AddrPort
relay uint32
fragID uint32
}
// pqFragSource groups msg1 reassemblies for the per-source quota: an IPv4 /32,
// an IPv6 /64, or one relay channel.
type pqFragSource struct {
prefix netip.Prefix
relay uint32
}
func pqFragKeyFor(via ViaSender, fragID uint32) (pqFragKey, pqFragSource) {
if via.IsRelayed {
return pqFragKey{relay: via.relay.LocalIndex, fragID: fragID},
pqFragSource{relay: via.relay.LocalIndex}
}
addr := via.UdpAddr.Addr().Unmap()
bits := 64
if addr.Is4() {
bits = 32
}
prefix, _ := addr.Prefix(bits)
return pqFragKey{addr: via.UdpAddr, fragID: fragID}, pqFragSource{prefix: prefix}
}
// reassembled rebuilds the packet the handshake Machine expects from the header
// of a fragment and the decoded message.
func reassembled(h *header.H, msg []byte) []byte {
p := make([]byte, header.Len, header.Len+len(msg))
header.Encode(p, header.Version, header.Handshake, h.Subtype, h.RemoteIndex, h.MessageCounter)
return append(p, msg...)
}
// usePQ decides whether this outbound attempt is post-quantum. A peer that does
// not speak PQ silently drops it, so timer retries alternate between PQ and
// classical, starting with PQ. An attempt triggered early by a lighthouse reply
// is sending to addresses nothing has been sent to yet, and a new path gets the
// PQ attempt first. With require, or for a peer that completed a PQ handshake
// when sticky is on, every attempt is PQ. Once anything has come back for the PQ
// attempt the peer evidently speaks PQ, and a classical attempt would only race
// it, so classical stops getting turns.
func (hm *HandshakeManager) usePQ(hh *HandshakeHostInfo, vpnIp netip.Addr, triggered bool) bool {
cfg := hm.pqConfig()
switch {
case !cfg.initiate:
return false
case cfg.require:
return true
case hh.pq != nil && hh.pq.heard:
return true
case cfg.sticky:
if _, ok := hm.pqPeers.Load(vpnIp); ok {
return true
}
}
return hh.counter <= 1 || triggered || !hh.lastPQ
}
// relayEstablished pushes the pending handshake with target through a relay that
// just came up, rather than leaving it for the next retry. Nothing has gone over
// the relay yet, so like any new path it gets the PQ attempt first, which then
// counts as the PQ turn: the next retry is classical, as if it had been sent on
// schedule. Without this, relays coming up while the first attempt is in flight
// handed every relayed handshake to the classical retry.
func (hm *HandshakeManager) relayEstablished(target netip.Addr, relayHI *HostInfo, relay *Relay) {
hh := hm.queryVpnIp(target)
if hh == nil {
return
}
hh.Lock()
defer hh.Unlock()
if hm.queryVpnIp(target) != hh {
return
}
var now, later [][]byte
switch {
case hh.pq != nil:
var err error
now, later, err = hh.pq.sender.Attempt(fragment.DefaultRepair(hh.pq.sender.Layout().K))
if err != nil {
return
}
hh.lastPQ = true
case hh.ready:
now = [][]byte{hh.hostinfo.HandshakePacket[handshakePacketStage0]}
default:
return
}
f := hm.f
hh.hostinfo.logger(hm.l).Info("Send handshake via new relay", "relay", relayHI.vpnAddrs[0])
_ = hm.sendFragmented(now, later, func(p []byte) error {
f.SendVia(relayHI, relay, p, make([]byte, 12), make([]byte, mtu), false, 0)
return nil
})
}
// buildPQAttempt creates the initiator's PQ Machine and msg1 sender.
func (hm *HandshakeManager) buildPQAttempt(hh *HandshakeHostInfo) bool {
cs := hm.f.pki.getCertState()
v := hm.initiatingVersion(hh, cs)
if cs.GetCredential(v) == nil {
hm.f.l.Error("Unable to handshake with host because no certificate is available",
"vpnAddrs", hh.hostinfo.vpnAddrs, "certVersion", v)
return false
}
pq := &pqAttempt{}
machine, err := handshake.NewMachine(
v, cs.GetCredential,
hm.certVerifier(), func() (uint32, error) { return hm.allocatePQIndex(hh, pq) },
true, pqSubtype,
)
if err != nil {
hm.f.l.Error("Failed to create handshake machine",
"vpnAddrs", hh.hostinfo.vpnAddrs, "error", err)
return false
}
msg, err := machine.Initiate(nil)
if err != nil {
hm.f.l.Error("Failed to initiate handshake",
"vpnAddrs", hh.hostinfo.vpnAddrs, "error", err)
hm.releasePQIndex(hh, pq)
return false
}
sender, err := fragment.NewSender(msg[header.Len:], pqSubtype, 0, 1, pq.index)
if err != nil {
hm.f.l.Error("Failed to fragment handshake",
"vpnAddrs", hh.hostinfo.vpnAddrs, "error", err)
hm.releasePQIndex(hh, pq)
return false
}
pq.machine = machine
pq.sender = sender
hh.pq = pq
return true
}
// allocatePQIndex registers a local index for the PQ attempt. Unlike
// allocateIndex it leaves hostinfo.localIndexId alone; that is settled when one
// of the attempts completes.
func (hm *HandshakeManager) allocatePQIndex(hh *HandshakeHostInfo, pq *pqAttempt) (uint32, error) {
hm.mainHostMap.RLock()
defer hm.mainHostMap.RUnlock()
hm.Lock()
defer hm.Unlock()
for range 32 {
index, err := generateIndex(hm.l)
if err != nil {
return 0, err
}
_, inPending := hm.indexes[index]
_, inMain := hm.mainHostMap.Indexes[index]
if !inMain && !inPending {
pq.index = index
hm.indexes[index] = hh
return index, nil
}
}
return 0, errors.New("failed to generate unique localIndexId")
}
func (hm *HandshakeManager) releasePQIndex(hh *HandshakeHostInfo, pq *pqAttempt) {
hm.Lock()
defer hm.Unlock()
if pq.index != 0 && hm.indexes[pq.index] == hh {
delete(hm.indexes, pq.index)
}
}
// settleIndex makes winner the hostinfo's local index and releases the index of
// the attempt that lost.
func (hm *HandshakeManager) settleIndex(hh *HandshakeHostInfo, winner uint32) {
hm.Lock()
defer hm.Unlock()
for _, idx := range []uint32{hh.hostinfo.localIndexId, hh.pqIndex()} {
if idx != 0 && idx != winner && hm.indexes[idx] == hh {
delete(hm.indexes, idx)
}
}
hh.hostinfo.localIndexId = winner
}
func (hh *HandshakeHostInfo) pqIndex() uint32 {
if hh.pq == nil {
return 0
}
return hh.pq.index
}
// handleFragment routes one fragment of a PQ handshake message.
func (hm *HandshakeManager) handleFragment(via ViaSender, packet []byte, h *header.H) {
if h.MessageCounter == pqCookieCounter {
hm.handleCookie(via, packet, h)
return
}
fh, sym, err := fragment.Parse(packet[header.Len:])
if err != nil {
hm.l.Debug("dropping malformed handshake fragment", "from", via, "error", err)
return
}
switch h.MessageCounter {
case 1:
if !hm.pqConfig().accept {
hm.l.Debug("dropping post-quantum handshake, not accepting them", "from", via)
return
}
if h.RemoteIndex != 0 {
hm.l.Debug("dropping stage-1 handshake with non-zero RemoteIndex",
"from", via, "remoteIndex", h.RemoteIndex)
return
}
now := time.Now()
key, src := pqFragKeyFor(via, fh.FragID)
if hm.pqTable.Load() >= pqCookieLoad && !(fh.HasCookie() && hm.pqCookies.valid(key, fh.Cookie, now)) {
hm.sendCookie(via, fh, key, len(packet), now)
return
}
res, err := hm.pqTable.Add(key, src, now, fh, sym)
if err != nil {
hm.l.Debug("dropping handshake fragment", "from", via, "error", err)
return
}
if res == nil {
return
}
if existing := hm.pqAnswered.lookup(res.Msg, time.Now()); existing != nil {
// Already answered: resend, rate limited, with fresh symbols.
if msg := existing.HandshakePacket[handshakePacketStage2]; msg != nil {
hm.sendHandshakeResponse(via, msg, existing, true, res.Received)
}
return
}
hm.beginHandshake(via, reassembled(h, res.Msg), h, res.Received)
case 2:
if hh := hm.queryIndex(h.RemoteIndex); hh != nil {
hm.continuePQ(via, hh, h, fh, sym)
}
}
}
// continuePQ collects msg2 fragments for the initiator's PQ attempt.
func (hm *HandshakeManager) continuePQ(via ViaSender, hh *HandshakeHostInfo, h *header.H, fh fragment.Header, sym []byte) {
hh.Lock()
defer hh.Unlock()
pq := hh.pq
if pq == nil || pq.index != h.RemoteIndex || !hm.stillPending(hh, pq.index) {
return
}
pq.heard = true
if pq.msg2 == nil {
a, err := fragment.NewAssembly(fh)
if err != nil {
return
}
pq.msg2 = a
}
ready, err := pq.msg2.Add(fh, sym)
if errors.Is(err, fragment.ErrMismatch) {
// The responder's FragID and MsgLen never change, so this is a stray or a
// forgery. Start over from this fragment rather than wedge.
pq.msg2 = nil
return
}
if err != nil || !ready {
return
}
msg, err := pq.msg2.Decode()
pq.msg2 = nil
if err != nil {
hm.l.Debug("failed to decode handshake fragments", "from", via, "error", err)
return
}
hm.processResponse(via, hh, pq.machine, reassembled(h, msg))
}
// sendFragmented sends packets to a peer: now immediately and later after the
// repair delay. It returns the first error from the immediate sends; errors from
// delayed sends are only logged.
func (hm *HandshakeManager) sendFragmented(now, later [][]byte, send func([]byte) error) error {
var first error
for _, p := range now {
if err := send(p); err != nil && first == nil {
first = err
}
}
sendLater := func() {
for _, p := range later {
if err := send(p); err != nil {
hm.l.Debug("Failed to send handshake repair fragment", "error", err)
}
}
}
delay := hm.pqConfig().repairDelay
switch {
case len(later) == 0:
case delay <= 0:
sendLater()
default:
time.AfterFunc(delay, sendLater)
}
return first
}
// sendFragmentedResponse sends the responder's msg2 as fragments. A resend of a
// cached response (cached) offers fresh symbols and is rate limited to one per
// retry interval. received is how many bytes the triggering msg1 took, which
// bounds what may be sent back before the address is validated.
func (hm *HandshakeManager) sendFragmentedResponse(via ViaSender, msg []byte, hostinfo *HostInfo, cached bool, received int) error {
now := time.Now()
if cached {
last := hostinfo.handshakeResendAt.Load()
if now.UnixNano()-last < int64(hm.config.tryInterval) ||
!hostinfo.handshakeResendAt.CompareAndSwap(last, now.UnixNano()) {
return errors.New("handshake response resend rate limited")
}
} else {
hostinfo.handshakeResendAt.Store(now.UnixNano())
}
var h header.H
if err := h.Parse(msg); err != nil {
return err
}
s, err := fragment.NewSender(msg[header.Len:], h.Subtype, h.RemoteIndex, h.MessageCounter, hostinfo.localIndexId)
if err != nil {
return err
}
l := s.Layout()
repair := fragment.DefaultRepair(l.K)
if received > 0 {
pktLen := header.Len + fragment.HeaderLen + l.SymbolSize
budget := pqAmplification * received / pktLen
if budget < l.K {
return errors.New("handshake response exceeds the amplification budget")
}
repair = min(repair, budget-l.K)
}
n := uint32(l.K + repair)
s.SetNext(int(hostinfo.handshakeNextSymbol.Add(n) - n))
pkts, later, err := s.Attempt(repair)
if err != nil {
return err
}
if via.IsRelayed {
if via.relay == nil {
return errors.New("both addr and via.relay are nil")
}
// We received a valid handshake on this relay, so make sure the relay
// state reflects that, in case it had been marked Disestablished.
hostinfo.relayState.InsertRelayTo(via.relayHI.vpnAddrs[0])
via.relayHI.relayState.UpdateRelayForByIdxState(via.relay.LocalIndex, Established)
}
hm.f.messageMetrics.Tx(header.Handshake, h.Subtype, int64(len(pkts)+len(later)))
return hm.sendFragmented(pkts, later, hm.sendTo(via))
}
// notePQPeer remembers peers that completed a PQ handshake, for sticky mode.
func (hm *HandshakeManager) notePQPeer(subtype header.MessageSubType, vpnAddrs []netip.Addr) {
if !hm.pqConfig().sticky || subtype != pqSubtype {
return
}
for _, a := range vpnAddrs {
hm.pqPeers.Store(a, struct{}{})
}
if hm.l.Enabled(context.Background(), slog.LevelDebug) {
hm.l.Debug("Remembering post-quantum peer", "vpnAddrs", vpnAddrs)
}
}
// pqAnswered remembers the msg1s a responder recently answered, keyed by hash,
// so a repeat is recognised before any noise, KEM or certificate work. Repeats
// are routine: the trailing repair fragments of an attempt rebuild a msg1 that
// its data fragments already completed, and a retrying initiator offers fresh
// symbols for the same msg1. Entries only exist for completed handshakes, which
// need a valid certificate, and live as long as an initiator keeps retrying.
type pqAnswered struct {
sync.Mutex
ttl time.Duration
entries map[[sha256.Size]byte]pqAnsweredEntry
}
type pqAnsweredEntry struct {
hostinfo *HostInfo
at time.Time
}
const pqAnsweredMax = 4096
func newPQAnswered(ttl time.Duration) *pqAnswered {
return &pqAnswered{ttl: ttl, entries: map[[sha256.Size]byte]pqAnsweredEntry{}}
}
func (a *pqAnswered) add(msg1 []byte, hostinfo *HostInfo, now time.Time) {
a.Lock()
defer a.Unlock()
if len(a.entries) >= pqAnsweredMax {
for k := range a.entries {
delete(a.entries, k)
break
}
}
a.entries[sha256.Sum256(msg1)] = pqAnsweredEntry{hostinfo: hostinfo, at: now}
}
func (a *pqAnswered) lookup(msg1 []byte, now time.Time) *HostInfo {
a.Lock()
defer a.Unlock()
e, ok := a.entries[sha256.Sum256(msg1)]
if !ok || now.Sub(e.at) > a.ttl {
return nil
}
return e.hostinfo
}
func (a *pqAnswered) expire(now time.Time) {
a.Lock()
defer a.Unlock()
for k, e := range a.entries {
if now.Sub(e.at) > a.ttl {
delete(a.entries, k)
}
}
}
// A responder whose msg1 table is under load stops keeping state for senders
// that have not shown they receive at their address. It answers a fragment
// without a valid cookie with a cookie reply and keeps nothing; the initiator
// resends with the cookie attached. This is WireGuard's under-load cookie, QUIC's
// Retry and SCTP's state cookie: one extra round trip, only while under attack.
const (
// pqCookieLoad is the table load above which msg1 fragments need a cookie.
pqCookieLoad = 0.5
// pqCookieCounter marks a cookie reply in the header's message counter.
// Handshake messages count from 1.
pqCookieCounter = 0
// pqCookieRotate is how often the cookie secret changes. A cookie stays
// valid for between one and two rotations.
pqCookieRotate = 2 * time.Minute
pqCookieReplyLen = header.Len + fragment.CookieLen
)
// pqCookies mints and checks cookies: an HMAC over the sender's address (or relay
// channel) and FragID under a rotating secret, with the previous secret still
// accepted.
type pqCookies struct {
sync.Mutex
secrets [2][32]byte
rotatedAt time.Time
}
func newPQCookies(now time.Time) *pqCookies {
c := &pqCookies{rotatedAt: now}
_, _ = rand.Read(c.secrets[0][:])
_, _ = rand.Read(c.secrets[1][:])
return c
}
func (c *pqCookies) rotate(now time.Time) {
if now.Sub(c.rotatedAt) < pqCookieRotate {
return
}
c.secrets[1] = c.secrets[0]
_, _ = rand.Read(c.secrets[0][:])
c.rotatedAt = now
}
func cookieMAC(secret []byte, key pqFragKey) fragment.Cookie {
mac := hmac.New(sha256.New, secret)
addr := key.addr.Addr().As16()
mac.Write(addr[:])
var b [10]byte
binary.BigEndian.PutUint16(b[0:], key.addr.Port())
binary.BigEndian.PutUint32(b[2:], key.relay)
binary.BigEndian.PutUint32(b[6:], key.fragID)
mac.Write(b[:])
var c fragment.Cookie
copy(c[:], mac.Sum(nil))
return c
}
func (c *pqCookies) make(key pqFragKey, now time.Time) fragment.Cookie {
c.Lock()
defer c.Unlock()
c.rotate(now)
return cookieMAC(c.secrets[0][:], key)
}
func (c *pqCookies) valid(key pqFragKey, cookie fragment.Cookie, now time.Time) bool {
c.Lock()
defer c.Unlock()
c.rotate(now)
for _, s := range c.secrets {
want := cookieMAC(s[:], key)
if hmac.Equal(want[:], cookie[:]) {
return true
}
}
return false
}
// sendCookie answers a msg1 fragment that arrived without a valid cookie while
// under load. The reply is never larger than what triggered it, and replies are
// rate limited as a whole.
func (hm *HandshakeManager) sendCookie(via ViaSender, fh fragment.Header, key pqFragKey, received int, now time.Time) {
if received < pqCookieReplyLen {
return
}
if _, ok := hm.pqCookieLimiter.Allow(); !ok {
return
}
c := hm.pqCookies.make(key, now)
reply := make([]byte, header.Len, pqCookieReplyLen)
header.Encode(reply, header.Version, header.Handshake, pqSubtype, fh.FragID, pqCookieCounter)
reply = append(reply, c[:]...)
hm.f.messageMetrics.Tx(header.Handshake, pqSubtype, 1)
if err := hm.sendTo(via)(reply); err != nil {
hm.l.Debug("Failed to send handshake cookie", "to", via, "error", err)
}
}
// handleCookie attaches a responder's cookie to the PQ attempt it names and
// resends right away, at most once per retry interval, rather than waiting for
// the next attempt.
func (hm *HandshakeManager) handleCookie(via ViaSender, packet []byte, h *header.H) {
if len(packet) != pqCookieReplyLen {
return
}
hh := hm.queryIndex(h.RemoteIndex)
if hh == nil {
return
}
hh.Lock()
defer hh.Unlock()
pq := hh.pq
if pq == nil || pq.index != h.RemoteIndex || !hm.stillPending(hh, pq.index) {
return
}
var c fragment.Cookie
copy(c[:], packet[header.Len:])
pq.sender.SetCookie(c)
pq.heard = true
now := time.Now()
if now.Sub(pq.cookieResendAt) < hm.config.tryInterval {
return
}
pq.cookieResendAt = now
pkts, later, err := pq.sender.Attempt(fragment.DefaultRepair(pq.sender.Layout().K))
if err != nil {
return
}
hm.messageMetrics.Tx(header.Handshake, pqSubtype, int64(len(pkts)+len(later)))
if err := hm.sendFragmented(pkts, later, hm.sendTo(via)); err != nil {
hm.l.Debug("Failed to resend handshake with cookie", "to", via, "error", err)
}
}
// sendTo returns a function sending a packet back the way via came: directly, or
// through the relay it arrived on.
func (hm *HandshakeManager) sendTo(via ViaSender) func([]byte) error {
f := hm.f
if !via.IsRelayed {
return func(p []byte) error {
return f.outside.WriteTo(p, via.UdpAddr)
}
}
return func(p []byte) error {
if via.relay == nil {
return errors.New("both addr and via.relay are nil")
}
f.SendVia(via.relayHI, via.relay, p, make([]byte, 12), make([]byte, mtu), false, 0)
return nil
}
}
-398
View File
@@ -1,398 +0,0 @@
package nebula
import (
"math/rand/v2"
"net/netip"
"sync"
"testing"
"time"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/handshake/fragment"
"github.com/slackhq/nebula/header"
"github.com/slackhq/nebula/test"
"github.com/slackhq/nebula/udp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// recordingConn captures what is written to it.
type recordingConn struct {
udp.NoopConn
sync.Mutex
sent [][]byte
}
func (c *recordingConn) WriteTo(b []byte, _ netip.AddrPort) error {
c.Lock()
defer c.Unlock()
c.sent = append(c.sent, append([]byte(nil), b...))
return nil
}
func (c *recordingConn) take() [][]byte {
c.Lock()
defer c.Unlock()
s := c.sent
c.sent = nil
return s
}
func newPQTestManager(t *testing.T) (*HandshakeManager, *recordingConn) {
l := test.NewLogger()
conn := &recordingConn{}
cfg := defaultHandshakeConfig
cfg.pq = pqHandshakeConfig{initiate: true, accept: true}
hm := NewHandshakeManager(l, newHostMap(l), newTestLighthouse(), conn, cfg)
hm.f = &Interface{
handshakeManager: hm,
outside: conn,
messageMetrics: newMessageMetricsOnlyRecvError(),
l: l,
}
return hm, conn
}
// fragmentIndexes checks every packet is a fragment of the expected response and
// returns their symbol indexes.
func fragmentIndexes(t *testing.T, pkts [][]byte, fragID, remoteIndex uint32) []int {
t.Helper()
var out []int
for _, p := range pkts {
require.LessOrEqual(t, len(p), fragment.MaxPacketLen)
var h header.H
require.NoError(t, h.Parse(p))
assert.Equal(t, pqSubtype, h.Subtype)
assert.Equal(t, remoteIndex, h.RemoteIndex)
assert.Equal(t, uint64(2), h.MessageCounter)
fh, _, err := fragment.Parse(p[header.Len:])
require.NoError(t, err)
assert.Equal(t, fragID, fh.FragID)
out = append(out, int(fh.Index))
}
return out
}
func TestSendFragmentedResponse(t *testing.T) {
hm, conn := newPQTestManager(t)
via := ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}
hostinfo := &HostInfo{localIndexId: 77}
// A 2000 byte message is 2 symbols of 1000, so packets of 1024 bytes.
msg := make([]byte, header.Len, header.Len+2000)
header.Encode(msg, header.Version, header.Handshake, pqSubtype, 5, 2)
r := rand.New(rand.NewPCG(1, 1))
for range 2000 {
msg = append(msg, byte(r.Uint32()))
}
const pktLen = header.Len + fragment.HeaderLen + 1000
t.Run("first send", func(t *testing.T) {
require.NoError(t, hm.sendFragmentedResponse(via, msg, hostinfo, false, 0))
assert.Equal(t, []int{0, 1, 2, 3}, fragmentIndexes(t, conn.take(), 77, 5))
})
t.Run("resends are rate limited", func(t *testing.T) {
require.Error(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, 10*pktLen))
assert.Empty(t, conn.take())
})
allowResend := func() {
hostinfo.handshakeResendAt.Store(time.Now().Add(-time.Second).UnixNano())
}
t.Run("resend offers fresh symbols", func(t *testing.T) {
allowResend()
require.NoError(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, 10*pktLen))
assert.Equal(t, []int{4, 5, 6, 7}, fragmentIndexes(t, conn.take(), 77, 5))
})
t.Run("amplification budget trims repair", func(t *testing.T) {
allowResend()
// 3x of a little over one packet allows 3 packets: both data and 1 repair.
require.NoError(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, pktLen+10))
assert.Equal(t, []int{8, 9, 10}, fragmentIndexes(t, conn.take(), 77, 5))
})
t.Run("amplification budget below the data refuses", func(t *testing.T) {
allowResend()
require.Error(t, hm.sendFragmentedResponse(via, msg, hostinfo, true, pktLen/2))
assert.Empty(t, conn.take())
})
}
func TestPQAnswered(t *testing.T) {
now := time.Unix(1000, 0)
a := newPQAnswered(5 * time.Second)
hi := &HostInfo{}
assert.Nil(t, a.lookup([]byte("msg1"), now))
a.add([]byte("msg1"), hi, now)
assert.Same(t, hi, a.lookup([]byte("msg1"), now.Add(time.Second)))
assert.Nil(t, a.lookup([]byte("other"), now))
assert.Nil(t, a.lookup([]byte("msg1"), now.Add(6*time.Second)), "stale entries are not answered")
a.expire(now.Add(6 * time.Second))
assert.Empty(t, a.entries)
for i := range pqAnsweredMax + 10 {
a.add([]byte{byte(i), byte(i >> 8)}, hi, now)
}
assert.Len(t, a.entries, pqAnsweredMax)
}
func TestPQFragKeyFor(t *testing.T) {
k1, s1 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}, 9)
k2, s2 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4243")}, 9)
assert.NotEqual(t, k1, k2, "different ports are different messages")
assert.Equal(t, s1, s2, "but share an IPv4 /32 quota")
_, s3 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("[2001:db8::1]:1")}, 9)
_, s4 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("[2001:db8::ffff]:1")}, 9)
assert.Equal(t, s3, s4, "an IPv6 /64 shares a quota")
_, s5 := pqFragKeyFor(ViaSender{UdpAddr: netip.MustParseAddrPort("[::ffff:192.0.2.1]:1")}, 9)
assert.Equal(t, s1, s5, "v4-mapped addresses count as IPv4")
}
func TestPQCookies(t *testing.T) {
now := time.Unix(1000, 0)
c := newPQCookies(now)
key := pqFragKey{addr: netip.MustParseAddrPort("192.0.2.1:4242"), fragID: 9}
cookie := c.make(key, now)
assert.True(t, c.valid(key, cookie, now))
for _, other := range []pqFragKey{
{addr: netip.MustParseAddrPort("192.0.2.2:4242"), fragID: 9},
{addr: netip.MustParseAddrPort("192.0.2.1:4243"), fragID: 9},
{addr: netip.MustParseAddrPort("192.0.2.1:4242"), fragID: 10},
{addr: netip.MustParseAddrPort("192.0.2.1:4242"), relay: 1, fragID: 9},
} {
assert.False(t, c.valid(other, cookie, now), "%+v", other)
}
now = now.Add(pqCookieRotate)
assert.True(t, c.valid(key, cookie, now), "the previous secret is still accepted")
assert.NotEqual(t, cookie, c.make(key, now), "new cookies come from the new secret")
now = now.Add(pqCookieRotate)
assert.False(t, c.valid(key, cookie, now), "two rotations retire a cookie")
}
// pqMsg1Fragment builds the first fragment of a msg1, optionally with a cookie.
func pqMsg1Fragment(t *testing.T, fragID uint32, cookie *fragment.Cookie) ([]byte, *header.H) {
t.Helper()
s, err := fragment.NewSender(make([]byte, 2000), pqSubtype, 0, 1, fragID)
require.NoError(t, err)
if cookie != nil {
s.SetCookie(*cookie)
}
pkts, _, err := s.Attempt(0)
require.NoError(t, err)
var h header.H
require.NoError(t, h.Parse(pkts[0]))
return pkts[0], &h
}
// fillPQTable puts the responder's msg1 table at the cookie threshold.
func fillPQTable(t *testing.T, hm *HandshakeManager) {
t.Helper()
now := time.Now()
for i := 0; hm.pqTable.Load() < pqCookieLoad; i++ {
p, _ := pqMsg1Fragment(t, uint32(i+1), nil)
fh, sym, err := fragment.Parse(p[header.Len:])
require.NoError(t, err)
addr := netip.AddrPortFrom(netip.AddrFrom4([4]byte{198, 18, byte(i >> 8), byte(i)}), 4242)
key, src := pqFragKeyFor(ViaSender{UdpAddr: addr}, fh.FragID)
_, err = hm.pqTable.Add(key, src, now, fh, sym)
require.NoError(t, err)
}
}
func TestPQCookieUnderLoad(t *testing.T) {
hm, conn := newPQTestManager(t)
via := ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}
t.Run("no cookie needed below the threshold", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 5000, nil)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Empty(t, conn.take())
assert.Equal(t, before+1, hm.pqTable.Len())
})
fillPQTable(t, hm)
var cookie fragment.Cookie
t.Run("a fragment without a cookie gets one and leaves no state", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 7, nil)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Equal(t, before, hm.pqTable.Len())
sent := conn.take()
require.Len(t, sent, 1)
reply := sent[0]
require.Len(t, reply, pqCookieReplyLen)
var rh header.H
require.NoError(t, rh.Parse(reply))
assert.Equal(t, header.Handshake, rh.Type)
assert.Equal(t, pqSubtype, rh.Subtype)
assert.Equal(t, uint32(7), rh.RemoteIndex, "the reply names the initiator's FragID")
assert.Equal(t, uint64(pqCookieCounter), rh.MessageCounter)
copy(cookie[:], reply[header.Len:])
})
t.Run("a forged cookie gets a fresh one", func(t *testing.T) {
bad := cookie
bad[0] ^= 1
p, h := pqMsg1Fragment(t, 7, &bad)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Equal(t, before, hm.pqTable.Len())
assert.Len(t, conn.take(), 1)
})
t.Run("the cookie gets the fragment in", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 7, &cookie)
before := hm.pqTable.Len()
hm.handleFragment(via, p, h)
assert.Equal(t, before+1, hm.pqTable.Len())
assert.Empty(t, conn.take())
})
t.Run("a cookie is bound to its address", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 7, &cookie)
hm.handleFragment(ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.99:4242")}, p, h)
assert.Len(t, conn.take(), 1, "another address gets its own cookie instead")
})
t.Run("a reply never outgrows what prompted it", func(t *testing.T) {
p, h := pqMsg1Fragment(t, 8, nil)
hm.sendCookie(via, fragment.Header{FragID: 8}, pqFragKey{}, pqCookieReplyLen-1, time.Now())
assert.Empty(t, conn.take())
hm.handleFragment(via, p, h)
assert.Len(t, conn.take(), 1)
})
}
func TestPQHandleCookie(t *testing.T) {
hm, conn := newPQTestManager(t)
via := ViaSender{UdpAddr: netip.MustParseAddrPort("192.0.2.1:4242")}
s, err := fragment.NewSender(make([]byte, 2000), pqSubtype, 0, 1, 99)
require.NoError(t, err)
_, _, err = s.Attempt(2)
require.NoError(t, err)
hh := &HandshakeHostInfo{hostinfo: &HostInfo{}, pq: &pqAttempt{index: 99, sender: s}}
hm.indexes[99] = hh
reply := make([]byte, header.Len, pqCookieReplyLen)
header.Encode(reply, header.Version, header.Handshake, pqSubtype, 99, pqCookieCounter)
reply = append(reply, bytes16(0xab)...)
var h header.H
require.NoError(t, h.Parse(reply))
hm.handleFragment(via, reply, &h)
assert.True(t, hh.pq.heard, "a cookie reply shows the peer speaks PQ")
sent := conn.take()
require.Len(t, sent, 4, "an immediate resend of fresh symbols")
for i, p := range sent {
fh, _, err := fragment.Parse(p[header.Len:])
require.NoError(t, err)
assert.True(t, fh.HasCookie())
assert.Equal(t, fragment.Cookie(bytes16(0xab)), fh.Cookie)
assert.Equal(t, uint8(4+i), fh.Index)
}
hm.handleFragment(via, reply, &h)
assert.Empty(t, conn.take(), "immediate resends are rate limited")
t.Run("replies for other indexes are ignored", func(t *testing.T) {
other := append([]byte(nil), reply...)
header.Encode(other, header.Version, header.Handshake, pqSubtype, 98, pqCookieCounter)
var oh header.H
require.NoError(t, oh.Parse(other))
hh.pq.cookieResendAt = time.Time{}
hm.handleFragment(via, other, &oh)
assert.Empty(t, conn.take())
})
}
func bytes16(b byte) []byte {
out := make([]byte, 16)
for i := range out {
out[i] = b
}
return out
}
func TestReloadPQ(t *testing.T) {
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString(`handshakes: {pq: {initiate: false}}`))
cfg := defaultHandshakeConfig
cfg.pq = pqHandshakeConfigFrom(c)
hm := NewHandshakeManager(l, newHostMap(l), newTestLighthouse(), &udp.NoopConn{}, cfg)
c.RegisterReloadCallback(hm.reloadPQ)
assert.Equal(t, pqHandshakeConfig{repairDelay: DefaultPQRepairDelay}, *hm.pqConfig())
require.NoError(t, c.ReloadConfigString(`handshakes: {pq: {initiate: true, accept: true, sticky: true, repair_delay: 25ms}}`))
assert.Equal(t, pqHandshakeConfig{initiate: true, accept: true, sticky: true, repairDelay: 25 * time.Millisecond}, *hm.pqConfig())
before := hm.pqConfig()
require.NoError(t, c.ReloadConfigString(`handshakes: {try_interval: 1s, pq: {initiate: true, accept: true, sticky: true, repair_delay: 25ms}}`))
assert.Same(t, before, hm.pqConfig(), "an unrelated change leaves the settings alone")
require.NoError(t, c.ReloadConfigString(`handshakes: {}`))
assert.Equal(t, pqHandshakeConfig{repairDelay: DefaultPQRepairDelay}, *hm.pqConfig(), "removing the section restores the defaults")
require.NoError(t, c.ReloadConfigString(`handshakes: {pq: {require: true}}`))
assert.Equal(t, pqHandshakeConfig{initiate: true, accept: true, require: true, repairDelay: DefaultPQRepairDelay}, *hm.pqConfig(), "require implies initiate and accept")
}
func TestUsePQ(t *testing.T) {
hm, _ := newPQTestManager(t)
peer := netip.MustParseAddr("10.1.0.1")
set := func(cfg pqHandshakeConfig) { hm.pq.Store(&cfg) }
// attempts plays a run of attempts, given whether each was triggered early,
// and returns which were PQ.
attempts := func(triggered ...bool) []bool {
hh := &HandshakeHostInfo{}
var out []bool
for _, tr := range triggered {
hh.counter++
hh.lastPQ = hm.usePQ(hh, peer, tr)
out = append(out, hh.lastPQ)
}
return out
}
set(pqHandshakeConfig{})
assert.Equal(t, []bool{false, false, false}, attempts(false, false, false), "off")
set(pqHandshakeConfig{initiate: true})
assert.Equal(t, []bool{true, false, true, false}, attempts(false, false, false, false), "timer retries alternate, PQ first")
assert.Equal(t, []bool{true, true, false, true, false}, attempts(false, true, false, true, false), "an early trigger is a new path, which gets PQ first")
assert.Equal(t, []bool{true, true, false}, attempts(true, true, false), "a triggered first attempt is PQ")
t.Run("hearing back stops the classical turns", func(t *testing.T) {
hh := &HandshakeHostInfo{pq: &pqAttempt{}}
hh.counter, hh.lastPQ = 1, true
hh.counter++
assert.False(t, hm.usePQ(hh, peer, false), "unheard, the classical attempt gets its turn")
hh.pq.heard = true
assert.True(t, hm.usePQ(hh, peer, false))
})
set(pqHandshakeConfig{initiate: true, accept: true, require: true})
assert.Equal(t, []bool{true, true, true, true}, attempts(false, false, true, false), "require never offers classical")
set(pqHandshakeConfig{initiate: true, sticky: true})
assert.Equal(t, []bool{true, false}, attempts(false, false), "sticky alternates until the peer has done PQ")
hm.notePQPeer(pqSubtype, []netip.Addr{peer})
assert.Equal(t, []bool{true, true, true}, attempts(false, false, false), "then only PQ")
}
+2 -14
View File
@@ -67,19 +67,8 @@ const (
const ( const (
HandshakeIXPSK0 MessageSubType = 0 HandshakeIXPSK0 MessageSubType = 0
HandshakeXXPSK0 MessageSubType = 1 HandshakeXXPSK0 MessageSubType = 1
// HandshakeIXPSK2MLKEM1024 is IX with an ephemeral ML-KEM-1024 exchange mixed
// in through the psk2 slot. Its messages exceed the MTU and travel as
// fragments (see handshake/fragment).
HandshakeIXPSK2MLKEM1024 MessageSubType = 2
) )
// IsFragmented reports whether handshake messages of this subtype travel as
// fragments.
func IsFragmented(st MessageSubType) bool {
return st == HandshakeIXPSK2MLKEM1024
}
var ErrHeaderTooShort = errors.New("header is too short") var ErrHeaderTooShort = errors.New("header is too short")
var subTypeTestMap = map[MessageSubType]string{ var subTypeTestMap = map[MessageSubType]string{
@@ -99,8 +88,7 @@ var subTypeMap = map[MessageType]*map[MessageSubType]string{
Test: &subTypeTestMap, Test: &subTypeTestMap,
CloseTunnel: &subTypeNoneMap, CloseTunnel: &subTypeNoneMap,
Handshake: { Handshake: {
HandshakeIXPSK0: "ix_psk0", HandshakeIXPSK0: "ix_psk0",
HandshakeIXPSK2MLKEM1024: "ix_psk2_mlkem1024",
}, },
Control: &subTypeNoneMap, Control: &subTypeNoneMap,
} }
@@ -211,7 +199,7 @@ func IsValidSubType(t MessageType, s MessageSubType) bool {
case Message: case Message:
return s == MessageNone || s == MessageRelay return s == MessageNone || s == MessageRelay
case Handshake: case Handshake:
return s == HandshakeIXPSK0 || s == HandshakeIXPSK2MLKEM1024 return s == HandshakeIXPSK0
case Test: case Test:
return s == TestReply || s == TestRequest return s == TestReply || s == TestRequest
case Control, CloseTunnel, RecvError, LightHouse: case Control, CloseTunnel, RecvError, LightHouse:
+1 -3
View File
@@ -96,8 +96,7 @@ func TestTypeMap(t *testing.T) {
Test: &subTypeTestMap, Test: &subTypeTestMap,
CloseTunnel: &subTypeNoneMap, CloseTunnel: &subTypeNoneMap,
Handshake: { Handshake: {
HandshakeIXPSK0: "ix_psk0", HandshakeIXPSK0: "ix_psk0",
HandshakeIXPSK2MLKEM1024: "ix_psk2_mlkem1024",
}, },
Control: &subTypeNoneMap, Control: &subTypeNoneMap,
}, subTypeMap) }, subTypeMap)
@@ -123,7 +122,6 @@ func TestIsValidSubType(t *testing.T) {
assert.False(t, IsValidSubType(Message, 2)) assert.False(t, IsValidSubType(Message, 2))
assert.True(t, IsValidSubType(Handshake, HandshakeIXPSK0)) assert.True(t, IsValidSubType(Handshake, HandshakeIXPSK0))
assert.True(t, IsValidSubType(Handshake, HandshakeIXPSK2MLKEM1024))
// HandshakeXXPSK0 is defined but not a wire-valid subtype. // HandshakeXXPSK0 is defined but not a wire-valid subtype.
assert.False(t, IsValidSubType(Handshake, HandshakeXXPSK0)) assert.False(t, IsValidSubType(Handshake, HandshakeXXPSK0))
-5
View File
@@ -260,11 +260,6 @@ type HostInfo struct {
// We need these to avoid replayed handshake packets creating new hostinfos which causes churn // We need these to avoid replayed handshake packets creating new hostinfos which causes churn
HandshakePacket map[uint8][]byte HandshakePacket map[uint8][]byte
// A responder resending a fragmented handshake response offers symbols it has
// not sent before, at most once per retry interval.
handshakeResendAt atomic.Int64 // unix nanos of the last send
handshakeNextSymbol atomic.Uint32 // next fragment symbol index to offer
// nextLHQuery is the earliest we can ask the lighthouse for new information. // nextLHQuery is the earliest we can ask the lighthouse for new information.
// This is used to limit lighthouse re-queries in chatty clients // This is used to limit lighthouse re-queries in chatty clients
nextLHQuery atomic.Int64 nextLHQuery atomic.Int64
-2
View File
@@ -232,11 +232,9 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
retries: int64(c.GetInt("handshakes.retries", DefaultHandshakeRetries)), retries: int64(c.GetInt("handshakes.retries", DefaultHandshakeRetries)),
triggerBuffer: c.GetInt("handshakes.trigger_buffer", DefaultHandshakeTriggerBuffer), triggerBuffer: c.GetInt("handshakes.trigger_buffer", DefaultHandshakeTriggerBuffer),
messageMetrics: messageMetrics, messageMetrics: messageMetrics,
pq: pqHandshakeConfigFrom(c),
} }
handshakeManager := NewHandshakeManager(l, hostMap, lightHouse, udpConns[0], handshakeConfig) handshakeManager := NewHandshakeManager(l, hostMap, lightHouse, udpConns[0], handshakeConfig)
c.RegisterReloadCallback(handshakeManager.reloadPQ)
lightHouse.handshakeTrigger = handshakeManager.trigger lightHouse.handshakeTrigger = handshakeManager.trigger
ds, err := newDnsServerFromConfig(ctx, l, pki, hostMap, c) ds, err := newDnsServerFromConfig(ctx, l, pki, hostMap, c)
+98 -14
View File
@@ -6,8 +6,10 @@ import (
"errors" "errors"
"fmt" "fmt"
"log/slog" "log/slog"
"math/bits"
"net/netip" "net/netip"
"os" "os"
"sync"
"sync/atomic" "sync/atomic"
"syscall" "syscall"
"unsafe" "unsafe"
@@ -28,7 +30,7 @@ type tun struct {
DefaultMTU int DefaultMTU int
Routes atomic.Pointer[[]Route] Routes atomic.Pointer[[]Route]
routeTree atomic.Pointer[bart.Table[routing.Gateways]] routeTree atomic.Pointer[bart.Table[routing.Gateways]]
linkAddr *netroute.LinkAddr linkAddr atomic.Pointer[netroute.LinkAddr]
// hostOwned means the fd arrived from the OS, which has already configured addressing, mtu // hostOwned means the fd arrived from the OS, which has already configured addressing, mtu
// and routes for it. NEPacketTunnelProvider on darwin does this. // and routes for it. NEPacketTunnelProvider on darwin does this.
hostOwned bool hostOwned bool
@@ -243,7 +245,7 @@ func (t *tun) Activate() error {
if linkAddr == nil { if linkAddr == nil {
return fmt.Errorf("unable to discover link_addr for tun interface") return fmt.Errorf("unable to discover link_addr for tun interface")
} }
t.linkAddr = linkAddr t.linkAddr.Store(linkAddr)
for _, network := range t.vpnNetworks { for _, network := range t.vpnNetworks {
if network.Addr().Is4() { if network.Addr().Is4() {
@@ -269,6 +271,7 @@ func (t *tun) Activate() error {
return t.addRoutes(false) return t.addRoutes(false)
} }
// activate4 assigns our vpn address as a /32 and routes the vpn network to the tun explicitly.
func (t *tun) activate4(network netip.Prefix) error { func (t *tun) activate4(network netip.Prefix) error {
s, err := unix.Socket( s, err := unix.Socket(
unix.AF_INET, unix.AF_INET,
@@ -295,22 +298,32 @@ func (t *tun) activate4(network netip.Prefix) error {
MaskAddr: unix.RawSockaddrInet4{ MaskAddr: unix.RawSockaddrInet4{
Len: unix.SizeofSockaddrInet4, Len: unix.SizeofSockaddrInet4,
Family: unix.AF_INET, Family: unix.AF_INET,
Addr: prefixToMask(network).As4(), Addr: [4]byte{255, 255, 255, 255}, //always make the address a /32
}, },
} }
// On Darwin, VPNs don't edit the routing table directly.
// A network extension hands its routes to the system, and configd installs them.
// configd replaces any route already holding the same prefix, including ours.
// As an example, Tailscale with an exit node treats any interface prefix wider than one address as a LAN.
// With LAN access off, it routes each LAN into Tailscale's utun.
// With LAN access on, it excludes each LAN from its tunnel.
// configd routes an excluded prefix like its own routes would, ignoring ours, so it goes out the default route.
// Either way our route is replaced, and traffic for the vpn network no longer reaches nebula.
// A single address isn't a LAN to Tailscale, so our route is left alone.
// We route the vpn network to the tun explicitly instead.
if err := ioctl(uintptr(s), unix.SIOCAIFADDR, uintptr(unsafe.Pointer(&ifr))); err != nil { if err := ioctl(uintptr(s), unix.SIOCAIFADDR, uintptr(unsafe.Pointer(&ifr))); err != nil {
return fmt.Errorf("failed to set tun v4 address: %s", err) return fmt.Errorf("failed to set tun v4 address: %s", err)
} }
err = addRoute(network, t.linkAddr) if err := addRoute(network, t.linkAddr.Load()); err != nil {
if err != nil { return fmt.Errorf("failed to add route for %v: %w", network.Masked(), err)
return err
} }
return nil return nil
} }
// activate6 is activate4 for IPv6
func (t *tun) activate6(network netip.Prefix) error { func (t *tun) activate6(network netip.Prefix) error {
s, err := unix.Socket( s, err := unix.Socket(
unix.AF_INET6, unix.AF_INET6,
@@ -332,20 +345,24 @@ func (t *tun) activate6(network netip.Prefix) error {
PrefixMask: unix.RawSockaddrInet6{ PrefixMask: unix.RawSockaddrInet6{
Len: unix.SizeofSockaddrInet6, Len: unix.SizeofSockaddrInet6,
Family: unix.AF_INET6, Family: unix.AF_INET6,
Addr: prefixToMask(network).As16(), // Always set the netmask to be a /128. See activate4 for why we do this.
Addr: [16]byte{255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255},
}, },
Lifetime: addrLifetime{ Lifetime: addrLifetime{
// never expires // never expires
Vltime: 0xffffffff, Vltime: 0xffffffff,
Pltime: 0xffffffff, Pltime: 0xffffffff,
}, },
Flags: _IN6_IFF_NODAD, Flags: _IN6_IFF_NODAD, // No DAD, a utun has no neighbors.
} }
if err := ioctl(uintptr(s), _SIOCAIFADDR_IN6, uintptr(unsafe.Pointer(&ifr))); err != nil { if err := ioctl(uintptr(s), _SIOCAIFADDR_IN6, uintptr(unsafe.Pointer(&ifr))); err != nil {
return fmt.Errorf("failed to set tun address: %s", err) return fmt.Errorf("failed to set tun address: %s", err)
} }
if err := addRoute(network, t.linkAddr.Load()); err != nil {
return fmt.Errorf("failed to add route for %v: %w", network.Masked(), err)
}
return nil return nil
} }
@@ -394,6 +411,56 @@ func (t *tun) RoutesFor(ip netip.Addr) routing.Gateways {
return routing.Gateways{} return routing.Gateways{}
} }
func fetchRoutes() ([]netroute.Message, error) {
rib, err := netroute.FetchRIB(unix.AF_UNSPEC, unix.NET_RT_DUMP, 0)
if err != nil {
return nil, err
}
return netroute.ParseRIB(unix.NET_RT_DUMP, rib)
}
// routeInterfaceIn returns the index of the interface prefix is routed through, or 0 if msgs has no route for exactly
// that prefix. Routes scoped to an interface are skipped, they don't stop us adding ours.
func routeInterfaceIn(msgs []netroute.Message, prefix netip.Prefix) int {
for _, msg := range msgs {
m, ok := msg.(*netroute.RouteMessage)
if !ok || len(m.Addrs) <= unix.RTAX_NETMASK || m.Flags&unix.RTF_IFSCOPE != 0 {
continue
}
if p, ok := routePrefix(m.Addrs[unix.RTAX_DST], m.Addrs[unix.RTAX_NETMASK]); ok && p == prefix.Masked() {
return m.Index
}
}
return 0
}
// routePrefix returns the prefix a route covers. Host routes have no netmask.
func routePrefix(dst, mask netroute.Addr) (netip.Prefix, bool) {
switch d := dst.(type) {
case *netroute.Inet4Addr:
bits := 32
if m, ok := mask.(*netroute.Inet4Addr); ok {
bits = maskBits(m.IP[:])
}
return netip.PrefixFrom(netip.AddrFrom4(d.IP), bits), true
case *netroute.Inet6Addr:
bits := 128
if m, ok := mask.(*netroute.Inet6Addr); ok {
bits = maskBits(m.IP[:])
}
return netip.PrefixFrom(netip.AddrFrom16(d.IP), bits), true
}
return netip.Prefix{}, false
}
func maskBits(mask []byte) int {
n := 0
for _, b := range mask {
n += bits.OnesCount8(b)
}
return n
}
// Get the LinkAddr for the interface of the given name // Get the LinkAddr for the interface of the given name
// Is there an easier way to fetch this when we create the interface? // Is there an easier way to fetch this when we create the interface?
// Maybe SIOCGIFINDEX? but this doesn't appear to exist in the darwin headers. // Maybe SIOCGIFINDEX? but this doesn't appear to exist in the darwin headers.
@@ -428,7 +495,14 @@ func (t *tun) addRoutes(logErrors bool) error {
return nil return nil
} }
// Activate hasn't run yet, it will install the routes
linkAddr := t.linkAddr.Load()
if linkAddr == nil {
return nil
}
routes := *t.Routes.Load() routes := *t.Routes.Load()
systemRoutes := sync.OnceValues(fetchRoutes)
for _, r := range routes { for _, r := range routes {
if len(r.Via) == 0 || !r.Install { if len(r.Via) == 0 || !r.Install {
@@ -436,10 +510,16 @@ func (t *tun) addRoutes(logErrors bool) error {
continue continue
} }
err := addRoute(r.Cidr, t.linkAddr) err := addRoute(r.Cidr, linkAddr)
if err != nil { if err != nil {
if errors.Is(err, unix.EEXIST) { if errors.Is(err, unix.EEXIST) {
t.l.Warn("unable to add unsafe_route, identical route already exists", "route", r.Cidr) // Every route we added already exists on a reload, only warn about someone else's
msgs, err := systemRoutes()
if err != nil {
t.l.Warn("unable to add unsafe_route, a route already exists but its interface could not be determined", "route", r.Cidr, "error", err)
} else if routeInterfaceIn(msgs, r.Cidr) != linkAddr.Index {
t.l.Warn("unable to add unsafe_route, the destination is already routed through another interface", "route", r.Cidr)
}
} else { } else {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err) retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors { if logErrors {
@@ -450,6 +530,8 @@ func (t *tun) addRoutes(logErrors bool) error {
} }
} else { } else {
t.l.Info("Added route", "route", r) t.l.Info("Added route", "route", r)
// A cidr listed twice would otherwise find our own route missing from an older fetch
systemRoutes = sync.OnceValues(fetchRoutes)
} }
} }
@@ -457,16 +539,18 @@ func (t *tun) addRoutes(logErrors bool) error {
} }
func (t *tun) removeRoutes(routes []Route) error { func (t *tun) removeRoutes(routes []Route) error {
if t.hostOwned { linkAddr := t.linkAddr.Load()
if t.hostOwned || linkAddr == nil {
return nil return nil
} }
for _, r := range routes { for _, r := range routes {
if !r.Install { if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue continue
} }
err := delRoute(r.Cidr, t.linkAddr) err := delRoute(r.Cidr, linkAddr)
if err != nil { if err != nil {
t.l.Error("Failed to remove route", "error", err, "route", r) t.l.Error("Failed to remove route", "error", err, "route", r)
} else { } else {
+103
View File
@@ -0,0 +1,103 @@
//go:build darwin && !ios && !e2e_testing
package overlay
import (
"net"
"net/netip"
"os"
"os/exec"
"strings"
"testing"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/routing"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
netroute "golang.org/x/net/route"
"golang.org/x/sys/unix"
)
// Every mac routes loopback through lo0
func TestRouteInterfaceIn(t *testing.T) {
msgs, err := fetchRoutes()
require.NoError(t, err)
routeInterface := func(s string) int { return routeInterfaceIn(msgs, netip.MustParsePrefix(s)) }
lo0, err := net.InterfaceByName("lo0")
require.NoError(t, err)
assert.Equal(t, lo0.Index, routeInterface("127.0.0.0/8"))
assert.Equal(t, lo0.Index, routeInterface("127.0.0.1/32"), "a host route has no netmask")
assert.Equal(t, lo0.Index, routeInterface("::1/128"))
assert.Equal(t, 0, routeInterface("127.0.0.0/16"), "only an exact prefix counts")
// Most macs also list the default route scoped to their other interfaces
out, err := exec.Command("route", "-n", "get", "default").Output()
require.NoError(t, err)
var name string
for _, line := range strings.Split(string(out), "\n") {
if v, ok := strings.CutPrefix(strings.TrimSpace(line), "interface: "); ok {
name = v
}
}
if name == "" {
t.Skip("no default route")
}
def, err := net.InterfaceByName(name)
require.NoError(t, err)
assert.Equal(t, def.Index, routeInterface("0.0.0.0/0"))
}
func TestRouteInterfaceIn_SkipsScopedCopies(t *testing.T) {
route := func(index, flags int) netroute.Message {
addrs := make([]netroute.Addr, unix.RTAX_MAX)
addrs[unix.RTAX_DST] = &netroute.Inet4Addr{IP: [4]byte{192, 168, 1, 0}}
addrs[unix.RTAX_NETMASK] = &netroute.Inet4Addr{IP: [4]byte{255, 255, 255, 0}}
return &netroute.RouteMessage{Index: index, Flags: flags, Addrs: addrs}
}
p := netip.MustParsePrefix("192.168.1.0/24")
assert.Equal(t, 7, routeInterfaceIn([]netroute.Message{route(4, unix.RTF_UP|unix.RTF_IFSCOPE), route(7, unix.RTF_UP)}, p))
assert.Equal(t, 0, routeInterfaceIn([]netroute.Message{route(4, unix.RTF_UP|unix.RTF_IFSCOPE)}, p))
}
// A reload can happen before Activate
func TestTunRoutesBeforeActivate(t *testing.T) {
tn := &tun{l: test.NewLogger()}
routes := []Route{{
Cidr: netip.MustParsePrefix("10.253.1.0/24"),
Via: routing.Gateways{routing.NewGateway(netip.MustParseAddr("10.250.0.2"), 1)},
Install: true,
}}
tn.Routes.Store(&routes)
assert.NoError(t, tn.addRoutes(true))
assert.NoError(t, tn.removeRoutes(routes))
}
// A tun.routes entry with no via is never installed, so removing it must not delete a route for the same cidr that
// belongs to someone else
func TestTunReloadLeavesRoutesItNeverInstalled(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("creating a utun and adding routes needs root")
}
lo0, err := net.InterfaceByName("lo0")
require.NoError(t, err)
cidr := "10.250.0.128/25"
require.NoError(t, exec.Command("route", "-q", "-n", "add", "-net", cidr, "-interface", "lo0").Run())
t.Cleanup(func() { _ = exec.Command("route", "-q", "-n", "delete", "-net", cidr).Run() })
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("tun:\n routes:\n - route: "+cidr+"\n mtu: 1300\n"))
tn, err := newTun(c, l, []netip.Prefix{netip.MustParsePrefix("10.250.0.1/24")}, false)
require.NoError(t, err)
t.Cleanup(func() { _ = tn.Close() })
require.NoError(t, tn.Activate())
require.NoError(t, c.ReloadConfigString("tun: {}\n"))
msgs, err := fetchRoutes()
require.NoError(t, err)
assert.Equal(t, lo0.Index, routeInterfaceIn(msgs, netip.MustParsePrefix(cidr)))
}
+2 -1
View File
@@ -589,7 +589,8 @@ func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) removeRoutes(routes []Route) error { func (t *tun) removeRoutes(routes []Route) error {
for _, r := range routes { for _, r := range routes {
if !r.Install { if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue continue
} }
+96 -3
View File
@@ -552,13 +552,17 @@ func (t *tun) setDefaultRoute(cidr netip.Prefix) error {
if cidr.Addr().Is6() { if cidr.Addr().Is6() {
nr.Priority = 256 nr.Priority = 256
} }
err := netlink.RouteReplace(&nr) held, err := t.otherLinkRoutes()
if err != nil {
return err
}
err = t.replaceRoute(&nr, held)
if err != nil { if err != nil {
t.l.Warn("Failed to set default route MTU, retrying", "error", err, "cidr", cidr) t.l.Warn("Failed to set default route MTU, retrying", "error", err, "cidr", cidr)
//retry twice more -- on some systems there appears to be a race condition where if we set routes too soon, netlink says `invalid argument` //retry twice more -- on some systems there appears to be a race condition where if we set routes too soon, netlink says `invalid argument`
for range 2 { for range 2 {
time.Sleep(100 * time.Millisecond) time.Sleep(100 * time.Millisecond)
err = netlink.RouteReplace(&nr) err = t.replaceRoute(&nr, held)
if err == nil { if err == nil {
break break
} else { } else {
@@ -578,6 +582,15 @@ func (t *tun) setDefaultRoute(cidr netip.Prefix) error {
} }
func (t *tun) addRoutes(logErrors bool) error { func (t *tun) addRoutes(logErrors bool) error {
held, err := t.otherLinkRoutes()
if err != nil {
if logErrors {
t.l.Error("Failed to add routes", "error", err)
return nil
}
return err
}
// Path routes // Path routes
routes := *t.Routes.Load() routes := *t.Routes.Load()
for _, r := range routes { for _, r := range routes {
@@ -602,7 +615,7 @@ func (t *tun) addRoutes(logErrors bool) error {
nr.Priority = r.Metric nr.Priority = r.Metric
} }
err := netlink.RouteReplace(&nr) err := t.replaceRoute(&nr, held)
if err != nil { if err != nil {
retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err) retErr := util.NewContextualError("Failed to add route", map[string]any{"route": r}, err)
if logErrors { if logErrors {
@@ -618,6 +631,86 @@ func (t *tun) addRoutes(logErrors bool) error {
return nil return nil
} }
// ip6RoutePrioUser is the metric the kernel gives an IPv6 route added without one
const ip6RoutePrioUser = 1024
// routeKey is where the main table files a route. Replacing a route replaces the first one with the same key, whatever
// device it is on
type routeKey struct {
dst netip.Prefix
priority int
}
func routeKeyOf(dst *net.IPNet, family, priority int) (routeKey, bool) {
var p netip.Prefix
if dst == nil {
// A default route
if family == unix.AF_INET6 {
p = netip.PrefixFrom(netip.IPv6Unspecified(), 0)
} else {
p = netip.PrefixFrom(netip.IPv4Unspecified(), 0)
}
} else {
addr, ok := netip.AddrFromSlice(dst.IP)
if !ok {
return routeKey{}, false
}
ones, _ := dst.Mask.Size()
p = netip.PrefixFrom(addr.Unmap(), ones).Masked()
}
if priority == 0 && p.Addr().Is6() {
priority = ip6RoutePrioUser
}
return routeKey{dst: p, priority: priority}, true
}
// otherLinkRoutes returns the keys of the main table routes that are not on our device
func (t *tun) otherLinkRoutes() (map[routeKey]struct{}, error) {
routes, err := netlink.RouteListFiltered(netlink.FAMILY_ALL, &netlink.Route{Table: unix.RT_TABLE_MAIN}, netlink.RT_FILTER_TABLE)
// An interrupted dump may miss a route, which then gets replaced like it always used to be
if err != nil && !errors.Is(err, netlink.ErrDumpInterrupted) {
return nil, fmt.Errorf("failed to list the system routes: %w", err)
}
held := make(map[routeKey]struct{}, len(routes))
for _, r := range routes {
if r.LinkIndex == t.deviceIndex {
continue
}
if k, ok := routeKeyOf(r.Dst, r.Family, r.Priority); ok {
held[k] = struct{}{}
}
}
return held, nil
}
// replaceRoute puts nr in the main table in place of our own route with its key. Replacing would take the route of
// another device holding that key, so ours goes in behind it instead. Theirs keeps routing and the kernel falls back
// to ours once theirs is removed, like when leaving the lan that had it. A different metric decides which one wins.
func (t *tun) replaceRoute(nr *netlink.Route, held map[routeKey]struct{}) error {
family := unix.AF_INET6
if nr.Dst.IP.To4() != nil {
family = unix.AF_INET
}
k, ok := routeKeyOf(nr.Dst, family, nr.Priority)
if _, taken := held[k]; !ok || !taken {
return netlink.RouteReplace(nr)
}
// Appending doesn't update a route that is already ours, it adds a second one
ours := netlink.Route{
LinkIndex: nr.LinkIndex,
Dst: nr.Dst,
Scope: nr.Scope,
Table: nr.Table,
Priority: k.priority,
}
if err := netlink.RouteDel(&ours); err != nil && !errors.Is(err, unix.ESRCH) {
return err
}
t.l.Warn("Another interface already routes this destination at the same metric, our route is used once theirs is removed", "destination", k.dst, "metric", k.priority)
return netlink.RouteAppend(nr)
}
func (t *tun) removeRoutes(routes []Route) { func (t *tun) removeRoutes(routes []Route) {
for _, r := range routes { for _, r := range routes {
if !r.Install { if !r.Install {
+133
View File
@@ -0,0 +1,133 @@
//go:build linux && !android && !e2e_testing
package overlay
import (
"net"
"net/netip"
"os"
"testing"
"github.com/DefinedNet/netlink"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/sys/unix"
)
// An unsafe_route with the same destination and metric as a lan's route must not replace it. Ours waits behind theirs
// and takes over when the lan goes away. Run as root, or unprivileged with `unshare -rn`.
func TestTunRouteLeavesAnotherDevicesRoute(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("creating links and adding routes needs root")
}
lan := &netlink.Veth{LinkAttrs: netlink.LinkAttrs{Name: "nebrt0"}, PeerName: "nebrt1"}
require.NoError(t, netlink.LinkAdd(lan))
t.Cleanup(func() { _ = netlink.LinkDel(lan) })
peer, err := netlink.LinkByName("nebrt1")
require.NoError(t, err)
require.NoError(t, netlink.LinkSetUp(lan))
require.NoError(t, netlink.LinkSetUp(peer))
lanAddrs := []*netlink.Addr{
{IPNet: &net.IPNet{IP: net.ParseIP("198.51.100.5"), Mask: net.CIDRMask(24, 32)}},
{IPNet: &net.IPNet{IP: net.ParseIP("2001:db8:1::5"), Mask: net.CIDRMask(64, 128)}, Flags: unix.IFA_F_NODAD},
}
for _, a := range lanAddrs {
require.NoError(t, netlink.AddrAdd(lan, a))
}
l := test.NewLogger()
c := config.NewC(l)
// The kernel puts a v4 lan route at metric 0, our default, and a v6 one at 256
require.NoError(t, c.LoadString(`
tun:
dev: nebrt2
unsafe_routes:
- route: 198.51.100.0/24
via: 10.250.0.2
- route: 2001:db8:1::/64
via: fd00:250::2
metric: 256
`))
tn, err := newTun(c, l, []netip.Prefix{netip.MustParsePrefix("10.250.0.1/24"), netip.MustParsePrefix("fd00:250::1/64")}, false)
require.NoError(t, err)
t.Cleanup(func() { _ = tn.Close() })
require.NoError(t, tn.Activate())
via := func(dst string) int {
routes, err := netlink.RouteGet(net.ParseIP(dst))
require.NoError(t, err)
require.NotEmpty(t, routes)
return routes[0].LinkIndex
}
devices := func(dst string) []int {
_, n, err := net.ParseCIDR(dst)
require.NoError(t, err)
routes, err := netlink.RouteListFiltered(netlink.FAMILY_ALL, &netlink.Route{Dst: n}, netlink.RT_FILTER_DST)
require.NoError(t, err)
var out []int
for _, r := range routes {
out = append(out, r.LinkIndex)
}
return out
}
for _, dst := range []string{"198.51.100.0/24", "2001:db8:1::/64"} {
assert.ElementsMatch(t, []int{lan.Index, tn.deviceIndex}, devices(dst), dst)
}
assert.Equal(t, lan.Index, via("198.51.100.9"))
assert.Equal(t, lan.Index, via("2001:db8:1::9"))
// Leaving the lan
for _, a := range lanAddrs {
require.NoError(t, netlink.AddrDel(lan, a))
}
assert.Equal(t, tn.deviceIndex, via("198.51.100.9"))
assert.Equal(t, tn.deviceIndex, via("2001:db8:1::9"))
// Dropping the unsafe_routes removes only ours
for _, a := range lanAddrs {
require.NoError(t, netlink.AddrAdd(lan, a))
}
require.NoError(t, c.ReloadConfigString("tun:\n dev: nebrt2\n"))
for _, dst := range []string{"198.51.100.0/24", "2001:db8:1::/64"} {
assert.Equal(t, []int{lan.Index}, devices(dst), dst)
}
}
// The route for our own vpn network is replaced to set its mtu, which must not take a lan's route for the same network
func TestTunVpnNetworkLeavesAnotherDevicesRoute(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("creating links and adding routes needs root")
}
lan := &netlink.Veth{LinkAttrs: netlink.LinkAttrs{Name: "nebrt0"}, PeerName: "nebrt1"}
require.NoError(t, netlink.LinkAdd(lan))
t.Cleanup(func() { _ = netlink.LinkDel(lan) })
require.NoError(t, netlink.LinkSetUp(lan))
require.NoError(t, netlink.AddrAdd(lan, &netlink.Addr{IPNet: &net.IPNet{IP: net.ParseIP("10.250.0.5"), Mask: net.CIDRMask(24, 32)}}))
l := test.NewLogger()
c := config.NewC(l)
require.NoError(t, c.LoadString("tun:\n dev: nebrt2\n"))
tn, err := newTun(c, l, []netip.Prefix{netip.MustParsePrefix("10.250.0.1/24")}, false)
require.NoError(t, err)
t.Cleanup(func() { _ = tn.Close() })
require.NoError(t, tn.Activate())
_, n, err := net.ParseCIDR("10.250.0.0/24")
require.NoError(t, err)
routes, err := netlink.RouteListFiltered(netlink.FAMILY_V4, &netlink.Route{Dst: n}, netlink.RT_FILTER_DST)
require.NoError(t, err)
var devices []int
for _, r := range routes {
devices = append(devices, r.LinkIndex)
if r.LinkIndex == tn.deviceIndex {
assert.Equal(t, tn.DefaultMTU, r.MTU, "ours still carries the mtu")
}
}
assert.ElementsMatch(t, []int{lan.Index, tn.deviceIndex}, devices)
}
+2 -1
View File
@@ -420,7 +420,8 @@ func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) removeRoutes(routes []Route) error { func (t *tun) removeRoutes(routes []Route) error {
for _, r := range routes { for _, r := range routes {
if !r.Install { if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue continue
} }
+2 -1
View File
@@ -399,7 +399,8 @@ func (t *tun) addRoutes(logErrors bool) error {
func (t *tun) removeRoutes(routes []Route) error { func (t *tun) removeRoutes(routes []Route) error {
for _, r := range routes { for _, r := range routes {
if !r.Install { if len(r.Via) == 0 || !r.Install {
// addRoutes never installs these
continue continue
} }
+3 -9
View File
@@ -57,9 +57,8 @@ func (rm *relayManager) GetUseRelays() bool {
// StartRelays drives the relay-establishment side of an outbound handshake attempt. // StartRelays drives the relay-establishment side of an outbound handshake attempt.
// For each candidate relay it either kicks off a handshake to the relay, sends a CreateRelayRequest, retransmits // For each candidate relay it either kicks off a handshake to the relay, sends a CreateRelayRequest, retransmits
// one that may have been lost, or, once the relay is Established, forwards the in-progress // one that may have been lost, or, once the relay is Established, forwards the in-progress
// stage 0 handshake packets for vpnIp through it: now immediately and later after the // stage 0 handshake packet for vpnIp through it.
// handshake repair delay. func (rm *relayManager) StartRelays(f *Interface, vpnIp netip.Addr, hh *HandshakeHostInfo, stage0 []byte) {
func (rm *relayManager) StartRelays(f *Interface, vpnIp netip.Addr, hh *HandshakeHostInfo, now, later [][]byte) {
hostinfo := hh.hostinfo hostinfo := hh.hostinfo
if !rm.GetUseRelays() || len(hostinfo.remotes.relays) == 0 { if !rm.GetUseRelays() || len(hostinfo.remotes.relays) == 0 {
hh.lastRelays = nil hh.lastRelays = nil
@@ -162,10 +161,7 @@ func (rm *relayManager) StartRelays(f *Interface, vpnIp netip.Addr, hh *Handshak
switch existingRelay.State { switch existingRelay.State {
case Established: case Established:
hl.Log(context.Background(), level, "Send handshake via relay", "relay", relay.String()) hl.Log(context.Background(), level, "Send handshake via relay", "relay", relay.String())
_ = f.handshakeManager.sendFragmented(now, later, func(p []byte) error { f.SendVia(relayHostInfo, existingRelay, stage0, make([]byte, 12), make([]byte, mtu), false, 0)
f.SendVia(relayHostInfo, existingRelay, p, make([]byte, 12), make([]byte, mtu), false, 0)
return nil
})
case Disestablished: case Disestablished:
// Mark this relay as 'requested' // Mark this relay as 'requested'
relayHostInfo.relayState.UpdateRelayForByIpState(vpnIp, Requested) relayHostInfo.relayState.UpdateRelayForByIpState(vpnIp, Requested)
@@ -364,8 +360,6 @@ func (rm *relayManager) handleCreateRelayResponse(v cert.Version, h *HostInfo, f
} }
// Do I need to complete the relays now? // Do I need to complete the relays now?
if relay.Type == TerminalType { if relay.Type == TerminalType {
// The relay I asked for is up; a handshake waiting on it can go now.
f.handshakeManager.relayEstablished(relay.PeerAddr, h, relay)
return return
} }
// I'm the middle man. Let the initiator know that the I've established the relay they requested. // I'm the middle man. Let the initiator know that the I've established the relay they requested.
+6 -6
View File
@@ -50,7 +50,7 @@ func TestStartRelaysLogDedupe(t *testing.T) {
var buf bytes.Buffer var buf bytes.Buffer
rm := newRM(&buf) rm := newRM(&buf)
hh := newHH() hh := newHH()
rm.StartRelays(f, vpnIp, hh, nil, nil) rm.StartRelays(f, vpnIp, hh, nil)
assert.Equal(t, []netip.Addr{vpnIp}, hh.lastRelays, "lastRelays should record the relay set we just attempted") assert.Equal(t, []netip.Addr{vpnIp}, hh.lastRelays, "lastRelays should record the relay set we just attempted")
assert.Contains(t, buf.String(), "level=INFO "+msg, "expected Info level on first attempt") assert.Contains(t, buf.String(), "level=INFO "+msg, "expected Info level on first attempt")
}) })
@@ -59,10 +59,10 @@ func TestStartRelaysLogDedupe(t *testing.T) {
var buf bytes.Buffer var buf bytes.Buffer
rm := newRM(&buf) rm := newRM(&buf)
hh := newHH() hh := newHH()
rm.StartRelays(f, vpnIp, hh, nil, nil) rm.StartRelays(f, vpnIp, hh, nil)
first := append([]netip.Addr(nil), hh.lastRelays...) first := append([]netip.Addr(nil), hh.lastRelays...)
buf.Reset() buf.Reset()
rm.StartRelays(f, vpnIp, hh, nil, nil) rm.StartRelays(f, vpnIp, hh, nil)
assert.Equal(t, first, hh.lastRelays) assert.Equal(t, first, hh.lastRelays)
assert.Contains(t, buf.String(), "level=DEBUG "+msg, "expected Debug level on identical retry") assert.Contains(t, buf.String(), "level=DEBUG "+msg, "expected Debug level on identical retry")
assert.NotContains(t, buf.String(), "level=INFO "+msg, "Info should not fire on identical retry") assert.NotContains(t, buf.String(), "level=INFO "+msg, "Info should not fire on identical retry")
@@ -72,13 +72,13 @@ func TestStartRelaysLogDedupe(t *testing.T) {
var buf bytes.Buffer var buf bytes.Buffer
rm := newRM(&buf) rm := newRM(&buf)
hh := newHH() hh := newHH()
rm.StartRelays(f, vpnIp, hh, nil, nil) rm.StartRelays(f, vpnIp, hh, nil)
buf.Reset() buf.Reset()
// The lighthouse handed us a new set this round. // The lighthouse handed us a new set this round.
hh.hostinfo.remotes.relays = []netip.Addr{vpnIp, otherRelay} hh.hostinfo.remotes.relays = []netip.Addr{vpnIp, otherRelay}
rm.StartRelays(f, vpnIp, hh, nil, nil) rm.StartRelays(f, vpnIp, hh, nil)
assert.Equal(t, []netip.Addr{vpnIp, otherRelay}, hh.lastRelays) assert.Equal(t, []netip.Addr{vpnIp, otherRelay}, hh.lastRelays)
assert.Contains(t, buf.String(), "level=INFO "+msg, "expected Info when the relay list changes") assert.Contains(t, buf.String(), "level=INFO "+msg, "expected Info when the relay list changes")
}) })
@@ -90,7 +90,7 @@ func TestStartRelaysLogDedupe(t *testing.T) {
hh := newHH() hh := newHH()
hh.lastRelays = []netip.Addr{vpnIp} hh.lastRelays = []netip.Addr{vpnIp}
rm.StartRelays(f, vpnIp, hh, nil, nil) rm.StartRelays(f, vpnIp, hh, nil)
assert.Nil(t, hh.lastRelays, "with relays disabled lastRelays should be cleared") assert.Nil(t, hh.lastRelays, "with relays disabled lastRelays should be cleared")
assert.NotContains(t, buf.String(), msg, "should not log when we shortcut out") assert.NotContains(t, buf.String(), msg, "should not log when we shortcut out")
}) })
+33 -13
View File
@@ -60,6 +60,20 @@ type sshDeviceInfoFlags struct {
func wireSSHReload(l *slog.Logger, ssh *sshd.SSHServer, c *config.C) { func wireSSHReload(l *slog.Logger, ssh *sshd.SSHServer, c *config.C) {
c.RegisterReloadCallback(func(c *config.C) { c.RegisterReloadCallback(func(c *config.C) {
if c.GetBool("sshd.enabled", false) { if c.GetBool("sshd.enabled", false) {
// Restarting the server drops every open session, the one that asked for this reload included. Only restart
// when it isn't running or its config changed. The host key file can change under the same config, new
// handshakes use the new key without a restart
if ssh.Running() && !c.HasChanged("sshd") {
key, err := loadSSHHostKey(c)
if err == nil {
err = ssh.SetHostKey(key)
}
if err != nil {
l.Error("Failed to reconfigure the sshd", "error", err)
ssh.Stop()
}
return
}
sshRun, err := configSSH(l, ssh, c) sshRun, err := configSSH(l, ssh, c)
if err != nil { if err != nil {
l.Error("Failed to reconfigure the sshd", "error", err) l.Error("Failed to reconfigure the sshd", "error", err)
@@ -74,6 +88,22 @@ func wireSSHReload(l *slog.Logger, ssh *sshd.SSHServer, c *config.C) {
}) })
} }
// loadSSHHostKey returns sshd.host_key, read from its file when it names one.
func loadSSHHostKey(c *config.C) ([]byte, error) {
hostKeyPathOrKey := c.GetString("sshd.host_key", "")
if hostKeyPathOrKey == "" {
return nil, fmt.Errorf("sshd.host_key must be provided")
}
if strings.Contains(hostKeyPathOrKey, "-----BEGIN") {
return []byte(hostKeyPathOrKey), nil
}
b, err := os.ReadFile(hostKeyPathOrKey)
if err != nil {
return nil, fmt.Errorf("error while loading sshd.host_key file: %s", err)
}
return b, nil
}
// configSSH reads the ssh info out of the passed-in Config and // configSSH reads the ssh info out of the passed-in Config and
// updates the passed-in SSHServer. On success, it returns a function // updates the passed-in SSHServer. On success, it returns a function
// that callers may invoke to run the configured ssh server. On // that callers may invoke to run the configured ssh server. On
@@ -92,19 +122,9 @@ func configSSH(l *slog.Logger, ssh *sshd.SSHServer, c *config.C) (func(), error)
return nil, fmt.Errorf("sshd.listen can not use port 22") return nil, fmt.Errorf("sshd.listen can not use port 22")
} }
hostKeyPathOrKey := c.GetString("sshd.host_key", "") hostKeyBytes, err := loadSSHHostKey(c)
if hostKeyPathOrKey == "" { if err != nil {
return nil, fmt.Errorf("sshd.host_key must be provided") return nil, err
}
var hostKeyBytes []byte
if strings.Contains(hostKeyPathOrKey, "-----BEGIN") {
hostKeyBytes = []byte(hostKeyPathOrKey)
} else {
hostKeyBytes, err = os.ReadFile(hostKeyPathOrKey)
if err != nil {
return nil, fmt.Errorf("error while loading sshd.host_key file: %s", err)
}
} }
err = ssh.SetHostKey(hostKeyBytes) err = ssh.SetHostKey(hostKeyBytes)
+48 -13
View File
@@ -8,14 +8,18 @@ import (
"log/slog" "log/slog"
"net" "net"
"sync" "sync"
"sync/atomic"
"github.com/armon/go-radix" "github.com/armon/go-radix"
"golang.org/x/crypto/ssh" "golang.org/x/crypto/ssh"
) )
type SSHServer struct { type SSHServer struct {
config *ssh.ServerConfig // config is replaced whole when the host key changes, ssh.ServerConfig.AddHostKey only replaces a key of the same
l *slog.Logger // algorithm and appends any other
config atomic.Pointer[ssh.ServerConfig]
authenticate func(ssh.ConnMetadata, ssh.PublicKey) (*ssh.Permissions, error)
l *slog.Logger
certChecker *ssh.CertChecker certChecker *ssh.CertChecker
@@ -28,7 +32,12 @@ type SSHServer struct {
// List of available commands // List of available commands
helpCommand *Command helpCommand *Command
commands *radix.Tree commands *radix.Tree
listener net.Listener listener atomic.Pointer[net.Listener]
// runs counts the Runs that are listening. A reload's Run can start before the one it replaced has returned
runs atomic.Int32
// addr is where the newest Run listens, nil when none is
addr atomic.Pointer[net.Addr]
// ctx parents per-Run contexts. Cancelling it (e.g. via Control.Stop) tears the server down even // ctx parents per-Run contexts. Cancelling it (e.g. via Control.Stop) tears the server down even
// across reloads, since each Run derives a fresh child rather than reusing this one directly. // across reloads, since each Run derives a fresh child rather than reusing this one directly.
@@ -85,10 +94,8 @@ func NewSSHServer(ctx context.Context, l *slog.Logger) (*SSHServer, error) {
}, },
} }
s.config = &ssh.ServerConfig{ s.authenticate = cc.Authenticate
PublicKeyCallback: cc.Authenticate, s.config.Store(s.newConfig())
ServerVersion: fmt.Sprintf("SSH-2.0-Nebula???"),
}
s.RegisterCommand(&Command{ s.RegisterCommand(&Command{
Name: "help", Name: "help",
@@ -107,10 +114,32 @@ func (s *SSHServer) SetHostKey(hostPrivateKey []byte) error {
return fmt.Errorf("failed to parse private key: %s", err) return fmt.Errorf("failed to parse private key: %s", err)
} }
s.config.AddHostKey(private) config := s.newConfig()
config.AddHostKey(private)
s.config.Store(config)
return nil return nil
} }
func (s *SSHServer) newConfig() *ssh.ServerConfig {
return &ssh.ServerConfig{
PublicKeyCallback: s.authenticate,
ServerVersion: fmt.Sprintf("SSH-2.0-Nebula???"),
}
}
// Addr is the address the newest Run listens on, nil when none is.
func (s *SSHServer) Addr() net.Addr {
if a := s.addr.Load(); a != nil {
return *a
}
return nil
}
// Running is true while a Run is listening.
func (s *SSHServer) Running() bool {
return s.runs.Load() > 0
}
func (s *SSHServer) ClearTrustedCAs() { func (s *SSHServer) ClearTrustedCAs() {
s.authLock.Lock() s.authLock.Lock()
s.trustedCAs = []ssh.PublicKey{} s.trustedCAs = []ssh.PublicKey{}
@@ -181,10 +210,16 @@ func (s *SSHServer) Run(addr string) error {
// this run owns. They start equal but a fast reload may overwrite s.listener with the next run's // this run owns. They start equal but a fast reload may overwrite s.listener with the next run's
// listener before this run's watcher fires, so each run must close its own listener via the local // listener before this run's watcher fires, so each run must close its own listener via the local
// reference. // reference.
s.listener = listener s.listener.Store(&listener)
s.runs.Add(1)
bound := listener.Addr()
s.addr.Store(&bound)
runCtx, cancel := context.WithCancel(s.ctx) runCtx, cancel := context.WithCancel(s.ctx)
defer cancel() defer cancel()
// Deferred after cancel so they run first, a run on its way out no longer counts as listening
defer s.runs.Add(-1)
defer s.addr.CompareAndSwap(&bound, nil)
// Close the listener when this run's context is cancelled. That can come from the parent // Close the listener when this run's context is cancelled. That can come from the parent
// (Control.Stop), from Run returning normally (defer cancel above), or transitively when a sibling // (Control.Stop), from Run returning normally (defer cancel above), or transitively when a sibling
@@ -197,7 +232,7 @@ func (s *SSHServer) Run(addr string) error {
} }
}() }()
s.l.Info("SSH server is listening", "sshListener", addr) s.l.Info("SSH server is listening", "sshListener", bound)
// Run loops until there is an error // Run loops until there is an error
s.run(runCtx, listener) s.run(runCtx, listener)
@@ -226,7 +261,7 @@ func (s *SSHServer) run(ctx context.Context, listener net.Listener) {
<-sessionContext.Done() <-sessionContext.Done()
c.Close() c.Close()
}() }()
conn, chans, reqs, err := ssh.NewServerConn(c, s.config) conn, chans, reqs, err := ssh.NewServerConn(c, s.config.Load())
fp := "" fp := ""
if conn != nil { if conn != nil {
fp = conn.Permissions.Extensions["fp"] fp = conn.Permissions.Extensions["fp"]
@@ -264,8 +299,8 @@ func (s *SSHServer) run(ctx context.Context, listener net.Listener) {
} }
func (s *SSHServer) Stop() { func (s *SSHServer) Stop() {
if s.listener != nil { if l := s.listener.Load(); l != nil {
if err := s.listener.Close(); err != nil { if err := (*l).Close(); err != nil && !errors.Is(err, net.ErrClosed) {
s.l.Warn("Failed to close the sshd listener", "error", err) s.l.Warn("Failed to close the sshd listener", "error", err)
} }
} }
+196
View File
@@ -0,0 +1,196 @@
package nebula
import (
"crypto"
"crypto/ed25519"
"crypto/rand"
"crypto/rsa"
"encoding/pem"
"fmt"
"net"
"os"
"path/filepath"
"testing"
"time"
"github.com/slackhq/nebula/config"
"github.com/slackhq/nebula/sshd"
"github.com/slackhq/nebula/test"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"golang.org/x/crypto/ssh"
)
type sshdReloadHarness struct {
c *config.C
server *sshd.SSHServer
keyFile string
signer ssh.Signer
extra string
}
func newSSHDReloadHarness(t *testing.T) *sshdReloadHarness {
t.Helper()
l := test.NewLogger()
h := &sshdReloadHarness{keyFile: filepath.Join(t.TempDir(), "host_key")}
h.writeHostKey(t, newEd25519Key(t))
_, userKey, err := ed25519.GenerateKey(rand.Reader)
require.NoError(t, err)
h.signer, err = ssh.NewSignerFromKey(userKey)
require.NoError(t, err)
h.c = config.NewC(l)
require.NoError(t, h.c.LoadString(h.config()))
h.server, err = sshd.NewSSHServer(t.Context(), l)
require.NoError(t, err)
wireSSHReload(l, h.server, h.c)
run, err := configSSH(l, h.server, h.c)
require.NoError(t, err)
go run()
t.Cleanup(h.server.Stop)
require.Eventually(t, h.server.Running, 5*time.Second, 10*time.Millisecond)
return h
}
func newEd25519Key(t *testing.T) crypto.Signer {
_, key, err := ed25519.GenerateKey(rand.Reader)
require.NoError(t, err)
return key
}
// writeHostKey replaces the host key file and returns the public key a client should now see
func (h *sshdReloadHarness) writeHostKey(t *testing.T, key crypto.Signer) ssh.PublicKey {
t.Helper()
block, err := ssh.MarshalPrivateKey(key, "")
require.NoError(t, err)
require.NoError(t, os.WriteFile(h.keyFile, pem.EncodeToMemory(block), 0o600))
pub, err := ssh.NewPublicKey(key.Public())
require.NoError(t, err)
return pub
}
func (h *sshdReloadHarness) config() string {
return fmt.Sprintf("sshd:\n enabled: true\n listen: 127.0.0.1:0\n host_key: %s\n authorized_users:\n - user: test\n keys: [%q]\n%s",
h.keyFile, string(ssh.MarshalAuthorizedKey(h.signer.PublicKey())), h.extra)
}
// dial connects to wherever the server listens now, a restart on :0 moves it. hostKeyAlgorithms limits which host keys
// the client accepts, none means any
func (h *sshdReloadHarness) dial(hostKeyAlgorithms ...string) (*ssh.Client, ssh.PublicKey, error) {
addr := h.server.Addr()
if addr == nil {
return nil, nil, fmt.Errorf("not listening")
}
var hostKey ssh.PublicKey
c, err := ssh.Dial("tcp", addr.String(), &ssh.ClientConfig{
User: "test",
Auth: []ssh.AuthMethod{ssh.PublicKeys(h.signer)},
HostKeyCallback: func(_ string, _ net.Addr, key ssh.PublicKey) error {
hostKey = key
return nil
},
HostKeyAlgorithms: hostKeyAlgorithms,
Timeout: time.Second,
})
return c, hostKey, err
}
func (h *sshdReloadHarness) connect(t *testing.T) (*ssh.Client, ssh.PublicKey) {
t.Helper()
var client *ssh.Client
var hostKey ssh.PublicKey
require.Eventually(t, func() bool {
var err error
client, hostKey, err = h.dial()
return err == nil
}, 5*time.Second, 20*time.Millisecond)
t.Cleanup(func() { _ = client.Close() })
return client, hostKey
}
// dropped is true when the server closed the client's connection within wait
func dropped(client *ssh.Client, wait time.Duration) bool {
done := make(chan struct{})
go func() {
_ = client.Wait()
close(done)
}()
select {
case <-done:
return true
case <-time.After(wait):
return false
}
}
// Restarting sshd drops every open session, the one that asked for the reload included. A reload that touches nothing
// sshd reads leaves it alone
func TestSSHDReload_UnchangedKeepsSessions(t *testing.T) {
h := newSSHDReloadHarness(t)
client, _ := h.connect(t)
require.NoError(t, h.c.ReloadConfigString(h.config()+"punchy:\n punch: true\n"))
assert.False(t, dropped(client, 500*time.Millisecond), "an unchanged sshd was restarted")
}
func TestSSHDReload_ChangedConfigRestarts(t *testing.T) {
h := newSSHDReloadHarness(t)
client, _ := h.connect(t)
h.extra = " trusted_cas: []\n"
require.NoError(t, h.c.ReloadConfigString(h.config()))
assert.True(t, dropped(client, 5*time.Second), "a changed sshd config was not applied")
require.Eventually(t, h.server.Running, 5*time.Second, 10*time.Millisecond)
h.connect(t)
}
// The config can stay the same while the host key file it names changes. New connections get the new key, open
// sessions are left alone
func TestSSHDReload_ChangedHostKeyFile(t *testing.T) {
h := newSSHDReloadHarness(t)
client, _ := h.connect(t)
want := h.writeHostKey(t, newEd25519Key(t))
require.NoError(t, h.c.ReloadConfigString(h.config()))
_, got := h.connect(t)
assert.Equal(t, want.Marshal(), got.Marshal(), "a new host key was not applied")
assert.False(t, dropped(client, 500*time.Millisecond), "a new host key restarted sshd")
}
// A host key of another algorithm replaces the old one rather than being offered next to it
func TestSSHDReload_HostKeyAlgorithmChange(t *testing.T) {
h := newSSHDReloadHarness(t)
h.connect(t)
rsaKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
want := h.writeHostKey(t, rsaKey)
require.NoError(t, h.c.ReloadConfigString(h.config()))
_, got := h.connect(t)
assert.Equal(t, want.Marshal(), got.Marshal())
_, _, err = h.dial(ssh.KeyAlgoED25519)
assert.Error(t, err, "the old ed25519 host key is still offered")
}
// A server that isn't running, say its port was busy, is tried again on every reload as before
func TestSSHDReload_NotRunningRestarts(t *testing.T) {
h := newSSHDReloadHarness(t)
h.server.Stop()
require.Eventually(t, func() bool { return !h.server.Running() }, 5*time.Second, 10*time.Millisecond)
require.NoError(t, h.c.ReloadConfigString(h.config()))
require.Eventually(t, h.server.Running, 5*time.Second, 10*time.Millisecond)
h.connect(t)
}
// A host key that no longer loads stops sshd, the same as a config that doesn't
func TestSSHDReload_BadHostKeyFileStops(t *testing.T) {
h := newSSHDReloadHarness(t)
h.connect(t)
require.NoError(t, os.WriteFile(h.keyFile, []byte("not a key"), 0o600))
require.NoError(t, h.c.ReloadConfigString(h.config()))
require.Eventually(t, func() bool { return !h.server.Running() }, 5*time.Second, 10*time.Millisecond)
}