Caleb JasikandClaude Opus 5.5
61947ff1c2
Passthrough.Flush hands its whole lane to a new tio.BatchWriter in one WriteBatch call when the queue implements it, and NewSingleQueue exposes WriteBatch when its source does. On darwin, tun.WriteBatch writes up to 64 packets per sendmsg_x on the utun control socket, which is a connected datagram socket, so xnu takes the batched sendit_x path rather than one sendit per message. utun registers no send_list handler, so the kernel still passes each packet to utun on its own: this saves syscalls and fd locking, not per-packet kernel work. Error handling follows what sendmsg_x reports: - A short count with no error (a packet past the send buffer mid-call, or a later send failing with ENOBUFS): the remainder goes out in the next sendmsg_x. - EMSGSIZE: only returned when the first packet is too large, before the kernel takes any, so that packet alone goes through Write, which reports it, and the rest are batched again. - ENOSYS, EPERM, EOPNOTSUPP: sendmsg_x is private API, so WriteBatch logs once and falls back to writev per packet for good. - Anything else: sendmsg_x returns the error without the count it took, so the call is reported rather than retried, which could deliver packets twice. One loss mode has no userspace fix. sendit_x counts each packet as sent once it is copied into an mbuf; if a later allocation fails, it frees the copied prefix unsent, returns its length, and sendmsg_x drops the ENOBUFS because the count is nonzero. That looks exactly like a real short count, so under mbuf exhaustion WriteBatch silently drops packets that writev would have delivered. The iOS tun path (tun_ios.go) is unchanged. Inside a macOS Network Extension, download goes from 640 to 832 Mbps against the recvmsg_x change alone in three paired rounds, at 15% less CPU per Mbps; the likely cause is that its utun charges more per write than a raw one. Raw dnclient already runs download near line rate, and there the CPU difference is within noise. TestTunWriteBatch* round-trip through a connected AF_UNIX datagram socketpair, which takes sendmsg_x the same way, over both the batched and the fallback path: order, AF prefixes, skipped invalid packets, and an oversize packet mid-call and first-in-call. Dropping Namelen from msghdrX fails them. Mutation testing with gremlins leaves only equivalent survivors in WriteBatch, plus which error is reported first when several fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What is Nebula?
Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, Windows, iOS, and Android. It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.
Nebula incorporates a number of existing concepts like encryption, security groups, certificates, and tunneling. What makes Nebula different to existing offerings is that it brings all of these ideas together, resulting in a sum that is greater than its individual parts.
Further documentation can be found here.
You can read more about Nebula here.
You can also join the NebulaOSS Slack group here.
Supported Platforms
Desktop and Server
Check the releases page for downloads or see the Distribution Packages section.
- Linux - 64 and 32 bit, arm, and others
- Windows
- MacOS
- Freebsd
Distribution Packages
-
sudo pacman -S nebula -
sudo dnf install nebula -
sudo apt install nebula -
sudo apk add nebula -
brew install nebula -
docker pull nebulaoss/nebula
Mobile (source code)
Technical Overview
Nebula is a mutually authenticated peer-to-peer software-defined network based on the Noise Protocol Framework. Nebula uses certificates to assert a node's IP address, name, and membership within user-defined groups. Nebula's user-defined groups allow for provider agnostic traffic filtering between nodes. Discovery nodes (aka lighthouses) allow individual peers to find each other and optionally use UDP hole punching to establish connections from behind most firewalls or NATs. Users can move data between nodes in any number of cloud service providers, datacenters, and endpoints, without needing to maintain a particular addressing scheme.
Nebula uses Elliptic-curve Diffie-Hellman (ECDH) key exchange and AES-256-GCM in its default configuration.
Nebula was created to provide a mechanism for groups of hosts to communicate securely, even across the internet, while enabling expressive firewall definitions similar in style to cloud security groups.
Getting started (quickly)
Don't want to manage your own PKI and lighthouses? Managed Nebula from Defined Networking handles all of this for you.
To set up a Nebula network, you'll need:
1. The Nebula binaries or Distribution Packages for your specific platform. Specifically you'll need nebula-cert and the specific nebula binary for each platform you use.
2. (Optional, but you really should..) At least one discovery node with a routable IP address, which we call a lighthouse.
Nebula lighthouses allow nodes to find each other, anywhere in the world. A lighthouse is the only node in a Nebula network whose IP should not change. Running a lighthouse requires very few compute resources, and you can easily use the least expensive option from a cloud hosting provider. If you're not sure which provider to use, a number of us have used $6/mo DigitalOcean droplets as lighthouses.
Once you have launched an instance, ensure that Nebula udp traffic (default port udp/4242) can reach it over the internet.
3. A Nebula certificate authority, which will be the root of trust for a particular Nebula network.
./nebula-cert ca -name "Myorganization, Inc"
This will create files named ca.key and ca.cert in the current directory. The ca.key file is the most sensitive file you'll create, because it is the key used to sign the certificates for individual nebula nodes/hosts. Please store this file somewhere safe, preferably with strong encryption.
Be aware! By default, certificate authorities have a 1-year lifetime before expiration. See this guide for details on rotating a CA.
4. Nebula host keys and certificates generated from that certificate authority
This assumes you have four nodes, named lighthouse1, laptop, server1, host3. You can name the nodes any way you'd like, including FQDN. You'll also need to choose IP addresses and the associated subnet. In this example, we are creating a nebula network that will use 192.168.100.x/24 as its network range. This example also demonstrates nebula groups, which can later be used to define traffic rules in a nebula network.
./nebula-cert sign -name "lighthouse1" -ip "192.168.100.1/24"
./nebula-cert sign -name "laptop" -ip "192.168.100.2/24" -groups "laptop,home,ssh"
./nebula-cert sign -name "server1" -ip "192.168.100.9/24" -groups "servers"
./nebula-cert sign -name "host3" -ip "192.168.100.10/24"
By default, host certificates will expire 1 second before the CA expires. Use the -duration flag to specify a shorter lifetime.
5. Configuration files for each host
Download a copy of the nebula example configuration.
-
On the lighthouse node, you'll need to ensure
am_lighthouse: trueis set. -
On the individual hosts, ensure the lighthouse is defined properly in the
static_host_mapsection, and is added to the lighthousehostssection.
6. Copy nebula credentials, configuration, and binaries to each host
For each host, copy the nebula binary to the host, along with config.yml from step 5, and the files ca.crt, {host}.crt, and {host}.key from step 4.
DO NOT COPY ca.key TO INDIVIDUAL NODES.
7. Run nebula on each host
./nebula -config /path/to/config.yml
For more detailed instructions, find the full documentation here.
Building Nebula from source
Make sure you have go installed and clone this repo. Change to the nebula directory.
To build nebula for all platforms:
make all
To build nebula for a specific platform (ex, Windows):
make bin-windows
See the Makefile for more details on build targets
Curve P256 and FIPS 140-3 mode
The default curve used for cryptographic handshakes and signatures is Curve25519. This is the recommended setting for most users. If your deployment has certain compliance requirements, you have the option of creating your CA using nebula-cert ca -curve P256 to use NIST Curve P256. The CA will then sign certificates using ECDSA P256, and any hosts using these certificates will use P256 for ECDH handshakes.
Nebula can be built to support the FIPS 140-3 mode of Go by running either of the following make targets. (This sets GOFIPS140=v1.0.0, which must be done at compile time so that the correct AES-GCM can be used for FIPS 140-3 enforcement mode).
make fips140
make fips140 test
make release-fips140
Nebula can also be built using the BoringCrypto GOEXPERIMENT by running either of the following make targets.
make bin-boringcrypto
make release-boringcrypto
NOTE: boringcrypto support is deprecated and will be removed in the next release. Users should migrate to the native FIPS 140-3 mode described above.
This is not the recommended default deployment, but may be useful based on your compliance requirements.
Credits
Nebula was created at Slack Technologies, Inc by Nate Brown and Ryan Huber, with contributions from Oliver Fross, Alan Lam, Wade Simmons, and Lining Wang.