Every multiport log line so far was a reason it turned itself off, so a node
running it looked identical to one that never tried. Add the other half:
- an Info at startup with the lane count and port range
- the negotiated lanes on both "Handshake message received" lines, which
required allocating the lane set before the log rather than after; it still
lands before CheckAndComplete/Complete, which was the ordering requirement
- multiport.lanes.up and multiport.lanes.tunnels gauges
The lane field is a slog.Attr so it vanishes on a node without multiport, and
reads zeros when we offered lanes and the peer had none - the case an operator
is actually looking for.
The gauges walk the hostmap instead of keeping a counter at promote/demote: a
tunnel torn down while its lanes are up never demotes them, so a counter would
drift upward forever. Only a multiport node pays for the walk.
How many lanes a tunnel has is partly the peer's call: it advertises how many it
sends on and we have to be able to receive all of them, up to the header's 256.
Deriving them all when the handshake completes meant a peer advertising a large
count cost us a replay window and two cipher states per lane, per tunnel, for
lanes it may never send on.
Derive each session on the first packet that needs it instead. The TX side asks
through laneSet.session, which installs on the spot — we only ask for lanes we
chose to send on. The RX side can't do that: anyone who can spoof a tunnel's
local index can name any lane, and installing on sight would hand them the same
allocation for free. So laneSession hands back a session without publishing it
and reports that it did; outside.go installs it only once the packet has
decrypted, which is the first moment the lane is known to be real. A spoofer
gets an HKDF per packet and nothing retained.
Only the session table is sized by the peer's advert now. txAddr, demand and
probe are sized by the lanes we will actually send on, so the peer can no longer
size our per-lane tx state either.
Also refuse relayed lane packets before the session lookup rather than after,
so a junk relay packet can't reach the derivation path at all.
Lanes used to be separate HostInfos, each with its own handshake, its own
half-established states, its own lifetime and its own slot bookkeeping. That
bought nothing: a lane is the same tunnel over a different underlay 5-tuple.
Derive lane sessions instead. Noise leaves us with A.eKey == B.dKey, so both
sides HKDF-expand the same two keys with the same per-lane label and land on a
matched pair without exchanging anything. Lanes now cost no handshake, have no
half-established state, and die exactly when their base tunnel does. Which lane
a packet belongs to rides the low byte of the nebula header's Reserved field,
inside the AEAD's associated data.
Receiving on a lane needs no permission, since the session exists the moment the
base handshake completes. Sending on one needs proof the new 5-tuple works, so a
lane stays down until a probe on it is acked and falls back to the base tunnel
the moment it stops being acked. Probing is demand-driven off the connection
manager's per-tunnel traffic tick: a peer we exchange a trickle with never costs
more than its base tunnel, however many lanes are configured. The ack rides the
base session on purpose, so a broken reverse lane can't fail a working one.
Because the data now rides lane counters, the rehandshake, exhaustion and
swap-primary checks take the max counter across the base session and its lanes;
otherwise the base counter would sit near zero while a lane ran its keys past
the nonce ceiling.
Removes OutboundLaneTimer, EnsureLanes, startLaneHandshake, completeLane,
completeLaneResponder, makeLaneTrafficDecision and the lane fields on HostInfo.
Handshake payload field 3 (the per-lane handshake index) is permanently
reserved; peers advertise a TxLanes count instead.
Lanes were established for every slot as soon as the base tunnel came up,
so a host with `routines: N` paid N-1 extra Noise handshakes, sessions and
keepalives for every peer it talked to, including peers it exchanges a
trickle with. Only routines that actually carry traffic to a peer need a
lane.
Add a per-slot demand flag to laneState. sendInsideMessage already loads
txLanes[laneSlot] on every direct-path packet; that load missing is now
the signal, and EnsureLanes only starts slots the data plane asked for.
The miss path is unchanged otherwise: the packet rides the base tunnel,
the same fallback used while a lane is down.
noteLaneDemand load-guards its store so repeated misses on a lane-less
slot are plain reads rather than a cache line ping-ponging between the
routines sharing it. The demand check is EnsureLanes' last condition, so
a slot that is pending or inside its backoff keeps the flag for the tick
that can act on it. Consuming the flag also stops a lane whose routine
went quiet from being rebuilt forever after it dies.
The wire format, lane negotiation and port-offset scheme are untouched,
so a lazy node interoperates with an eager one in both directions, and
inbound lane handshakes from a busy peer are still accepted regardless of
our own demand.
handleOutsideRelayPacket filled ViaSender.remoteIdx with relay.RemoteIndex,
an index from the relay peer's index space, but the rescue in
sendHandshakeResponse looks that value up in relayForByIdx, which is keyed
by local index. The lookup could never hit, so a terminal relay entry left
Disestablished by a one-sided teardown stayed Disestablished even after a
valid handshake arrived over it. The responder's first transmit then failed
to find an Established relay, deleted its only relay entry, and every
subsequent send was silently dropped until dead-tunnel detection forced a
re-handshake.
* firewall can distinguish if the host connecting has an overlapping network, is a VPN peer without an overlapping network, or is a unsafe network
* Cross stack subnet stuff (#1512)
* experiment with not filtering out non-common addresses in hostinfo.networks
* allow handshakes without overlaps
* unsafe network test
* change HostInfo.buildNetworks argument to reference the cert
* try to make certificate addition/removal reloadable in some cases
* very spicy change to respond to handshakes with cert versions we cannot match with a cert that we can indeed match
* even spicier change to rehandshake if we detect our cert is lower-version than our peer, and we have a newer-version cert available
* make tryRehandshake easier to understand
A local index collision happens when two tunnels attempt to use the same
random int32 index ID. This is a rare chance, and we have code to deal
with it, but we have a panic because we return the wrong thing in this
case. This change should fix the panic.
We had a rare deadlock in GetOrHandshake because we kept the hostmap
lock when we do the call to StartHandshake. StartHandshake can block
while sending to the lighthouse query worker channel, and that worker
needs to be able to grab the hostmap lock to do its work. Other calls
for StartHandshake don't hold the hostmap lock so we should be able to
drop it here.
This lock was originally added with: https://github.com/slackhq/nebula/pull/954
* avoid deadlock in lighthouse queryWorker
If the lighthouse queryWorker tries to grab to call StartHandshake on
a lighthouse vpnIp, we can deadlock on the handshake_manager lock. This
change drops the handshake_manager lock before we send on the lighthouse
queryChan (which could block), and also avoids sending to the channel if
this is a lighthouse IP itself.
* need to hold lock during cacheCb
* add calculated_remotes
This setting allows us to "guess" what the remote might be for a host
while we wait for the lighthouse response. For networks that hard
designed with in mind, it can help speed up handshake performance, as well as
improve resiliency in the case that all lighthouses are down.
Example:
lighthouse:
# ...
calculated_remotes:
# For any Nebula IPs in 10.0.10.0/24, this will apply the mask and add
# the calculated IP as an initial remote (while we wait for the response
# from the lighthouse). Both CIDRs must have the same mask size.
# For example, Nebula IP 10.0.10.123 will have a calculated remote of
# 192.168.1.123
10.0.10.0/24:
- mask: 192.168.1.0/24
port: 4242
* figure out what is up with this test
* add test
* better logic for sending handshakes
Keep track of the last light of hosts we sent handshakes to. Only log
handshake sent messages if the list has changed.
Remove the test Test_NewHandshakeManagerTrigger because it is faulty and
makes no sense. It relys on the fact that no handshake packets actually
get sent, but with these changes we would send packets now (which it
should!)
* use atomic.Pointer
* cleanup to make it clearer
* fix typo in example
Wade Simmons