7c38aa7e6b
rawWrite passed the iovec pointer to syscall.Syscall as a uintptr, so
the Go compiler's escape analysis could not keep the underlying
[]unix.Iovec (or the payload slices its Base pointers reach) rooted
across the syscall. Under heavy sustained write load, GC could
collect or move these before tun_chr_write_iter finished reading
them, at which point the kernel read freed memory.
Observed on a UniFi UXG-Pro (Annapurna Labs Alpine V2, arm64, Linux
4.19.152) forwarding 1 Gbps iperf3 -R between LAN and a remote
Nebula peer, as two paired kernel warnings in the same second:
refcount_t: underflow; use-after-free
sock_wfree -> skb_release_head_state -> kfree_skb
-> skb_release_data -> __kfree_skb -> tcp_recvmsg ...
refcount_t: addition on 0; use-after-free
skb_set_owner_w -> sock_alloc_send_pskb
-> tun_get_user -> tun_chr_write_iter -> do_iter_write
-> vfs_writev -> do_writev -> __arm64_sys_writev
The Annapurna watchdog then soft-rebooted the device. No crash or
kernel WARN after patching; box ran sustained 1 Gbps iperf3 -R
without issue.
Fix: add a variadic `keepAlive ...interface{}` parameter to
rawWrite, and call runtime.KeepAlive on the iovec plus every
supplied root after the syscall returns. writeWithScratch now
passes its buffer + iovec; WriteGSO passes the iovec array, the
header buffer, and the payload fragment slice.
runtime.KeepAlive is a compiler directive, not a runtime barrier,
so the cost is effectively zero: it just forces the compiler's
liveness analysis to treat the object as used at that point.
What is Nebula?
Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, Windows, iOS, and Android. It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.
Nebula incorporates a number of existing concepts like encryption, security groups, certificates, and tunneling. What makes Nebula different to existing offerings is that it brings all of these ideas together, resulting in a sum that is greater than its individual parts.
Further documentation can be found here.
You can read more about Nebula here.
You can also join the NebulaOSS Slack group here.
Supported Platforms
Desktop and Server
Check the releases page for downloads or see the Distribution Packages section.
- Linux - 64 and 32 bit, arm, and others
- Windows
- MacOS
- Freebsd
Distribution Packages
-
sudo pacman -S nebula -
sudo dnf install nebula -
sudo apt install nebula -
sudo apk add nebula -
brew install nebula -
docker pull nebulaoss/nebula
Mobile (source code)
Technical Overview
Nebula is a mutually authenticated peer-to-peer software-defined network based on the Noise Protocol Framework. Nebula uses certificates to assert a node's IP address, name, and membership within user-defined groups. Nebula's user-defined groups allow for provider agnostic traffic filtering between nodes. Discovery nodes (aka lighthouses) allow individual peers to find each other and optionally use UDP hole punching to establish connections from behind most firewalls or NATs. Users can move data between nodes in any number of cloud service providers, datacenters, and endpoints, without needing to maintain a particular addressing scheme.
Nebula uses Elliptic-curve Diffie-Hellman (ECDH) key exchange and AES-256-GCM in its default configuration.
Nebula was created to provide a mechanism for groups of hosts to communicate securely, even across the internet, while enabling expressive firewall definitions similar in style to cloud security groups.
Getting started (quickly)
Don't want to manage your own PKI and lighthouses? Managed Nebula from Defined Networking handles all of this for you.
To set up a Nebula network, you'll need:
1. The Nebula binaries or Distribution Packages for your specific platform. Specifically you'll need nebula-cert and the specific nebula binary for each platform you use.
2. (Optional, but you really should..) At least one discovery node with a routable IP address, which we call a lighthouse.
Nebula lighthouses allow nodes to find each other, anywhere in the world. A lighthouse is the only node in a Nebula network whose IP should not change. Running a lighthouse requires very few compute resources, and you can easily use the least expensive option from a cloud hosting provider. If you're not sure which provider to use, a number of us have used $6/mo DigitalOcean droplets as lighthouses.
Once you have launched an instance, ensure that Nebula udp traffic (default port udp/4242) can reach it over the internet.
3. A Nebula certificate authority, which will be the root of trust for a particular Nebula network.
./nebula-cert ca -name "Myorganization, Inc"
This will create files named ca.key and ca.cert in the current directory. The ca.key file is the most sensitive file you'll create, because it is the key used to sign the certificates for individual nebula nodes/hosts. Please store this file somewhere safe, preferably with strong encryption.
Be aware! By default, certificate authorities have a 1-year lifetime before expiration. See this guide for details on rotating a CA.
4. Nebula host keys and certificates generated from that certificate authority
This assumes you have four nodes, named lighthouse1, laptop, server1, host3. You can name the nodes any way you'd like, including FQDN. You'll also need to choose IP addresses and the associated subnet. In this example, we are creating a nebula network that will use 192.168.100.x/24 as its network range. This example also demonstrates nebula groups, which can later be used to define traffic rules in a nebula network.
./nebula-cert sign -name "lighthouse1" -ip "192.168.100.1/24"
./nebula-cert sign -name "laptop" -ip "192.168.100.2/24" -groups "laptop,home,ssh"
./nebula-cert sign -name "server1" -ip "192.168.100.9/24" -groups "servers"
./nebula-cert sign -name "host3" -ip "192.168.100.10/24"
By default, host certificates will expire 1 second before the CA expires. Use the -duration flag to specify a shorter lifetime.
5. Configuration files for each host
Download a copy of the nebula example configuration.
-
On the lighthouse node, you'll need to ensure
am_lighthouse: trueis set. -
On the individual hosts, ensure the lighthouse is defined properly in the
static_host_mapsection, and is added to the lighthousehostssection.
6. Copy nebula credentials, configuration, and binaries to each host
For each host, copy the nebula binary to the host, along with config.yml from step 5, and the files ca.crt, {host}.crt, and {host}.key from step 4.
DO NOT COPY ca.key TO INDIVIDUAL NODES.
7. Run nebula on each host
./nebula -config /path/to/config.yml
For more detailed instructions, find the full documentation here.
Building Nebula from source
Make sure you have go installed and clone this repo. Change to the nebula directory.
To build nebula for all platforms:
make all
To build nebula for a specific platform (ex, Windows):
make bin-windows
See the Makefile for more details on build targets
Curve P256 and BoringCrypto
The default curve used for cryptographic handshakes and signatures is Curve25519. This is the recommended setting for most users. If your deployment has certain compliance requirements, you have the option of creating your CA using nebula-cert ca -curve P256 to use NIST Curve P256. The CA will then sign certificates using ECDSA P256, and any hosts using these certificates will use P256 for ECDH handshakes.
In addition, Nebula can be built using the BoringCrypto GOEXPERIMENT by running either of the following make targets:
make bin-boringcrypto
make release-boringcrypto
This is not the recommended default deployment, but may be useful based on your compliance requirements.
Credits
Nebula was created at Slack Technologies, Inc by Nate Brown and Ryan Huber, with contributions from Oliver Fross, Alan Lam, Wade Simmons, and Lining Wang.