Set tun.staged.workers to a positive count on Linux, with tun.use_offloads
enabled, to move encryption off each tun reader. The reader filters and
reserves counters for a whole read, hands the batch to that many encrypt
workers, and an ordered sender flushes the batches in read order. Eight slots
per routine bound the batches in flight; a full pipeline blocks the reader.
Zero workers, the default, keeps today's inline path, as do tun queues that
cannot hand off their read buffer and builds whose AES-GCM needs nonce-ordered
encryption (BoringCrypto, FIPS).
IMPORTANT: staged senders need peers with the 8192-entry replay window of
master. Released v1.11.x peers use 1024 and will drop reordered batches. The
window is not negotiated.
Readers keep today's pinning. The sender is only pinned with an explicit
tun.cpu_affinity: sender i uses entry i+1, unless that is its reader's CPU,
so a one-entry list leaves the sender unpinned rather than sharing it: two
busy transmit threads on one CPU serialise (eight TCP streams over 100 GbE
fell from 25.9 to 22.8 Gbit/s). Without a list the sender is not pinned. Placing it by queue index on a
hybrid CPU put it on an efficiency core, and one TCP stream over 100 GbE
reached 18.6-19.6 Gbit/s pinned against 20.2-20.6 unpinned.
New API, each needed across a package boundary:
- tio.Offload.SwapRxBuf hands the buffer of the last Read to the reader and
reads into the one given. It is a method on the concrete queue, found through
an unexported interface in nebula, so tio.Queue and its other
implementations are unchanged.
- batch.SendBatch.Reset drops queued packets without sending, for a sender
whose sockets are already closed; Flush now uses it.
Metrics, registered only for active pipelines: flow.tx.batches counts reads
handed to the sender, flow.stage.{tx_read,tx_encrypt,tx_send}.q<N>.busy_ns the
elapsed busy time of each stage, to find the limiting one.
Tests cover ordering and counters under backpressure with both ciphers (AES
skipped where nonce order is required), read errors with batches in flight,
shutdown, exhaustion, relays, fallback, firewall parity, and the actual reader
and send thread affinity for every placement, with pinning on and off: the
sender is pinned only to an explicit entry other than its reader's CPU.
The goroutines Main starts stop only when its context does, and only a caller
holding the Control can arrange that. Cancelling solely on error left a config
test, which returns no Control and no error, with the lighthouse query worker
and a hostname resolver per dns named static host running for the life of the
process. Every embedder that validates a config in process rather than by
exec'ing pays for that on each config load, dnclient and the apple clients
included.
Co-authored-by: Nate Brown <nbrown.us@gmail.com>
* populate default Build version if missing
Use the Go module information built into the binary if the Build var
wasn't set during the build.
This means if you install via a specific tag, you get:
go install github.com/slackhq/nebula/cmd/nebula@v1.9.5
$ nebula -version
Version: 1.9.5
And if you install master, you get:
go install github.com/slackhq/nebula/cmd/nebula@master
$ nebula -version
Version: 1.9.5-0.20250408154034-18279ed17b10
* also default in the library
* cleanup
* upgrade to yaml.v3
The main nice fix here is that maps unmarshal into `map[string]any`
instead of `map[any]any`, so it cleans things up a bit.
* add config.AsBool
Since yaml.v3 doesn't automatically convert yes to bool now, for
backwards compat
* use type aliases for m
* more cleanup
* more cleanup
* more cleanup
* go mod cleanup
This used to work correctly because when the multiple routines work was
first added in #382, but an important part to discover the listen port
before opening the other listeners on the same socket was lost in this
PR: #653.
This change should fix the regression and allow multiple routines to
work correctly when listen.port is set to `0`.
Thanks to @rawdigits for tracking down and discovering this regression.
* Added firewall.rules.hash metric
Added a FNV-1 hash of the firewall rules as a Prometheus value.
* Switch FNV has to int64, include both hashes in log messages
* Use a uint32 for the FNV hash
Let go-metrics cast the uint32 to a int64, so it won't be lossy
when it eventually emits a float64 Prometheus metric.
By default, Nebula replies to packets it has no tunnel for with a `recv_error` packet. This packet helps speed up re-connection
in the case that Nebula on either side did not shut down cleanly. This response can be abused as a way to discover if Nebula is running
on a host though. This option lets you configure if you want to send `recv_error` packets always, never, or only to private network remotes.
valid values: always, never, private
This setting is reloadable with SIGHUP.
Ryan Huber