Wade SimmonsandJohn Maguire
ae688d038d
Backport from #1835 to release 1.11 ## What Since the bump to netlink v1.3.0, `netlink.AddrList` returns `ErrDumpInterrupted` ("results may be incomplete or inconsistent") when the kernel sets `NLM_F_DUMP_INTR` on an address dump. `addIPs` treats any error from `AddrList` as fatal, so a transient interrupted dump aborts startup: ``` failed to start nebula: failed to get tun address list: results may be incomplete or inconsistent ``` Before v1.3.0, the pinned netlink release had no `NLM_F_DUMP_INTR` handling at all and silently returned partial results, so this condition was previously invisible. ## Root cause The dump is racing nebula's own setup. `RTM_GETADDR` dumps addresses for the whole system, and the kernel flags the dump as interrupted if any address changes between recvmsg batches. When the cert contains an IPv6 network, the `AddrReplace` a few lines above returns while the new address is still tentative; the kernel's DAD worker flips it to preferred asynchronously ~100-400us later, and that flip lands inside the immediately-following dump. The race is entirely self-contained - no concurrent restart or interface churn is required, and the hit rate scales with how many addresses are on the host (more dump batches = wider window). Reproduced with a standalone tool replaying this exact netlink sequence on a host with ~80 addresses: an IPv6 /80 alone interrupts 10/200 cycles and v4+v6 together 16/200, while IPv4-only runs 0/800. `ip -ts monitor addr` shows the tentative add and the async flag-clear as two events during the dump window. Hosts with DAD disabled (`net.ipv6.conf.all.accept_dad=0`, e.g. LXC containers) never reproduce, confirming the flip is the trigger. Observed in the wild as dnclient package upgrades reliably failing their post-install restart (systemd start timeout, package left half-configured). ## How Treat `ErrDumpInterrupted` as success with a warning. netlink still returns the partial result set alongside the error, and this call site only uses the list to prune addresses that are not in the certs; a missed stale address survives until the next reload, which is strictly better than failing startup. All cert addresses were already applied via `AddrReplace` above. Co-authored-by: John Maguire <john@defined.net>
What is Nebula?
Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, Windows, iOS, and Android. It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.
Nebula incorporates a number of existing concepts like encryption, security groups, certificates, and tunneling. What makes Nebula different to existing offerings is that it brings all of these ideas together, resulting in a sum that is greater than its individual parts.
Further documentation can be found here.
You can read more about Nebula here.
You can also join the NebulaOSS Slack group here.
Supported Platforms
Desktop and Server
Check the releases page for downloads or see the Distribution Packages section.
- Linux - 64 and 32 bit, arm, and others
- Windows
- MacOS
- Freebsd
Distribution Packages
-
sudo pacman -S nebula -
sudo dnf install nebula -
sudo apt install nebula -
sudo apk add nebula -
brew install nebula -
docker pull nebulaoss/nebula
Mobile (source code)
Technical Overview
Nebula is a mutually authenticated peer-to-peer software-defined network based on the Noise Protocol Framework. Nebula uses certificates to assert a node's IP address, name, and membership within user-defined groups. Nebula's user-defined groups allow for provider agnostic traffic filtering between nodes. Discovery nodes (aka lighthouses) allow individual peers to find each other and optionally use UDP hole punching to establish connections from behind most firewalls or NATs. Users can move data between nodes in any number of cloud service providers, datacenters, and endpoints, without needing to maintain a particular addressing scheme.
Nebula uses Elliptic-curve Diffie-Hellman (ECDH) key exchange and AES-256-GCM in its default configuration.
Nebula was created to provide a mechanism for groups of hosts to communicate securely, even across the internet, while enabling expressive firewall definitions similar in style to cloud security groups.
Getting started (quickly)
Don't want to manage your own PKI and lighthouses? Managed Nebula from Defined Networking handles all of this for you.
To set up a Nebula network, you'll need:
1. The Nebula binaries or Distribution Packages for your specific platform. Specifically you'll need nebula-cert and the specific nebula binary for each platform you use.
2. (Optional, but you really should..) At least one discovery node with a routable IP address, which we call a lighthouse.
Nebula lighthouses allow nodes to find each other, anywhere in the world. A lighthouse is the only node in a Nebula network whose IP should not change. Running a lighthouse requires very few compute resources, and you can easily use the least expensive option from a cloud hosting provider. If you're not sure which provider to use, a number of us have used $6/mo DigitalOcean droplets as lighthouses.
Once you have launched an instance, ensure that Nebula udp traffic (default port udp/4242) can reach it over the internet.
3. A Nebula certificate authority, which will be the root of trust for a particular Nebula network.
./nebula-cert ca -name "Myorganization, Inc"
This will create files named ca.key and ca.cert in the current directory. The ca.key file is the most sensitive file you'll create, because it is the key used to sign the certificates for individual nebula nodes/hosts. Please store this file somewhere safe, preferably with strong encryption.
Be aware! By default, certificate authorities have a 1-year lifetime before expiration. See this guide for details on rotating a CA.
4. Nebula host keys and certificates generated from that certificate authority
This assumes you have four nodes, named lighthouse1, laptop, server1, host3. You can name the nodes any way you'd like, including FQDN. You'll also need to choose IP addresses and the associated subnet. In this example, we are creating a nebula network that will use 192.168.100.x/24 as its network range. This example also demonstrates nebula groups, which can later be used to define traffic rules in a nebula network.
./nebula-cert sign -name "lighthouse1" -ip "192.168.100.1/24"
./nebula-cert sign -name "laptop" -ip "192.168.100.2/24" -groups "laptop,home,ssh"
./nebula-cert sign -name "server1" -ip "192.168.100.9/24" -groups "servers"
./nebula-cert sign -name "host3" -ip "192.168.100.10/24"
By default, host certificates will expire 1 second before the CA expires. Use the -duration flag to specify a shorter lifetime.
5. Configuration files for each host
Download a copy of the nebula example configuration.
-
On the lighthouse node, you'll need to ensure
am_lighthouse: trueis set. -
On the individual hosts, ensure the lighthouse is defined properly in the
static_host_mapsection, and is added to the lighthousehostssection.
6. Copy nebula credentials, configuration, and binaries to each host
For each host, copy the nebula binary to the host, along with config.yml from step 5, and the files ca.crt, {host}.crt, and {host}.key from step 4.
DO NOT COPY ca.key TO INDIVIDUAL NODES.
7. Run nebula on each host
./nebula -config /path/to/config.yml
For more detailed instructions, find the full documentation here.
Building Nebula from source
Make sure you have go installed and clone this repo. Change to the nebula directory.
To build nebula for all platforms:
make all
To build nebula for a specific platform (ex, Windows):
make bin-windows
See the Makefile for more details on build targets
Curve P256 and BoringCrypto
The default curve used for cryptographic handshakes and signatures is Curve25519. This is the recommended setting for most users. If your deployment has certain compliance requirements, you have the option of creating your CA using nebula-cert ca -curve P256 to use NIST Curve P256. The CA will then sign certificates using ECDSA P256, and any hosts using these certificates will use P256 for ECDH handshakes.
In addition, Nebula can be built using the BoringCrypto GOEXPERIMENT by running either of the following make targets:
make bin-boringcrypto
make release-boringcrypto
This is not the recommended default deployment, but may be useful based on your compliance requirements.
Credits
Nebula was created at Slack Technologies, Inc by Nate Brown and Ryan Huber, with contributions from Oliver Fross, Alan Lam, Wade Simmons, and Lining Wang.