Compare commits

..

5 Commits

Author SHA1 Message Date
Jay Wren c618b96fef add stats.pprof flag to expose pprof on stats listener 2026-02-03 15:40:35 -05:00
Jack Doan 42bee7cf17 Report if Nebula start fails because of tun device name (#1588)
gofmt / Run gofmt (push) Failing after 2s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 2s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
* specifically report if nebula start fails because of tun device name

* close all routines when closing the tun
2026-01-28 10:03:36 -06:00
Caleb Jasik 02d8bcac68 Remove lighthouse goroutine leaks in lighthouse_test.go (#1589)
gofmt / Run gofmt (push) Failing after 3s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 2s
Build and test / Build all and test on ubuntu-linux (push) Failing after 3s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
Using <https://go.dev/doc/go1.26#goroutineleak-profiles> + Claude, I was able to run nebula's unit tests and e2e tests with the leak detector enabled.

Added a TestMain that queries pprof to see if there are any reported goroutine leaks.
I'd love to get some form of this in CI whenever go 1.26 comes out, though I'd also like to prove this is properly useful past the just five detections it got here.

<details>
<summary>TestMain</summary>


```go
package nebula

import (
    "fmt"
    "os"
    "runtime/pprof"
    "strings"
    "testing"
)

// TestMain runs after all tests and checks for goroutine leaks
func TestMain(m *testing.M) {
    // Run all tests
    exitCode := m.Run()

    // Check for goroutine leaks after all tests complete
    prof := pprof.Lookup("goroutineleak")
    if prof != nil {
        var sb strings.Builder
        if err := prof.WriteTo(&sb, 2); err != nil {
            fmt.Fprintf(os.Stderr, "Failed to write goroutineleak profile: %v\n", err)
            os.Exit(1)
        }

        content := sb.String()
        leakedCount := strings.Count(content, "(leaked)")

        if leakedCount > 0 {
            fmt.Fprintf(os.Stderr, "\n=== GOROUTINE LEAK DETECTED ===\n")
            fmt.Fprintf(os.Stderr, "Found %d leaked goroutine(s) in package nebula\n\n", leakedCount)

            goros := strings.Split(content, "\n\n")
            for _, goro := range goros {
                if strings.Contains(goro, "(leaked)") {
                    fmt.Fprintln(os.Stderr, goro)
                    fmt.Fprintln(os.Stderr)
                }
            }
            os.Exit(1)
        } else {
            fmt.Println("✓ No goroutine leaks detected in package nebula")
        }
    }

    os.Exit(exitCode)
}
```

</details>

Also had to install go1.26rc2 and update the makefile to use that go binary + set ex:

```makefile
test-goroutineleak:
	GOEXPERIMENT=goroutineleakprofile go1.26rc2 test -v ./...
```
2026-01-27 23:44:43 -06:00
Wade Simmons 0b02d982b2 v1.10.2 (#1584)
gofmt / Run gofmt (push) Failing after 3s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 3s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
Update CHANGELOG for Nebula v1.10.2
2026-01-21 12:42:34 -05:00
Wade Simmons e1e92f017c initialize routesFromSystem (#1580)
gofmt / Run gofmt (push) Failing after 3s
smoke-extra / Run extra smoke tests (push) Failing after 2s
smoke / Run multi node smoke test (push) Failing after 3s
Build and test / Build all and test on ubuntu-linux (push) Failing after 2s
Build and test / Build and test on linux with boringcrypto (push) Failing after 2s
Build and test / Build and test on linux with pkcs11 (push) Failing after 2s
Build and test / Build and test on macos-latest (push) Has been cancelled
Build and test / Build and test on windows-latest (push) Has been cancelled
This is a regression introduced by #1573. We need to initialize this
map.

Fixes: #1579
2026-01-20 11:15:20 -05:00
10 changed files with 73 additions and 14 deletions
+13 -1
View File
@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [1.10.2] - 2026-01-21
### Fixed
- Fix panic when using `use_system_route_table` that was introduced in v1.10.1. (#1580)
### Changed
- Fix some typos in comments. (#1582)
- Dependency updates. (#1581)
## [1.10.1] - 2026-01-16
See the [v1.10.1](https://github.com/slackhq/nebula/milestone/26?closed=1) milestone for a complete list of changes.
@@ -764,7 +775,8 @@ created.)
- Initial public release.
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.10.1...HEAD
[Unreleased]: https://github.com/slackhq/nebula/compare/v1.10.2...HEAD
[1.10.2]: https://github.com/slackhq/nebula/releases/tag/v1.10.2
[1.10.1]: https://github.com/slackhq/nebula/releases/tag/v1.10.1
[1.10.0]: https://github.com/slackhq/nebula/releases/tag/v1.10.0
[1.9.7]: https://github.com/slackhq/nebula/releases/tag/v1.9.7
+6
View File
@@ -396,6 +396,7 @@ func (c *certificateV2) validate() error {
return NewErrInvalidCertificateProperties("non-CA certificate must contain at least 1 network")
}
hasV4Networks := false
hasV6Networks := false
for _, network := range c.details.networks {
if !network.IsValid() || !network.Addr().IsValid() {
@@ -414,6 +415,7 @@ func (c *certificateV2) validate() error {
return NewErrInvalidCertificateProperties("4in6 networks are not allowed: %s", network)
}
hasV4Networks = hasV4Networks || network.Addr().Is4()
hasV6Networks = hasV6Networks || network.Addr().Is6()
}
@@ -437,6 +439,10 @@ func (c *certificateV2) validate() error {
if !hasV6Networks {
return NewErrInvalidCertificateProperties("IPv6 unsafe networks require an IPv6 address assignment: %s", network)
}
} else if network.Addr().Is4() {
if !hasV4Networks {
return NewErrInvalidCertificateProperties("IPv4 unsafe networks require an IPv4 address assignment: %s", network)
}
}
}
}
+5
View File
@@ -318,6 +318,11 @@ logging:
#subsystem: nebula
#interval: 10s
# enables pprof endpoints on the stats listener
# endpoints: /debug/pprof/, /debug/pprof/cmdline, /debug/pprof/profile,
# /debug/pprof/symbol, /debug/pprof/trace
#pprof: false
# enables counter metrics for meta packets
# e.g.: `messages.tx.handshake`
# NOTE: `message.{tx,rx}.recv_error` is always emitted
+8
View File
@@ -490,6 +490,14 @@ func (f *Interface) Close() error {
f.l.WithError(err).Error("Error while closing udp socket")
}
}
for i, r := range f.readers {
if i == 0 {
continue // f.readers[0] is f.inside, which we want to save for last
}
if err := r.Close(); err != nil {
f.l.WithError(err).Error("Error while closing tun reader")
}
}
// Release the tun device
return f.inside.Close()
+8 -9
View File
@@ -1,7 +1,6 @@
package nebula
import (
"context"
"encoding/binary"
"fmt"
"net/netip"
@@ -42,14 +41,14 @@ func Test_lhStaticMapping(t *testing.T) {
c := config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"hosts": []any{lh1}}
c.Settings["static_host_map"] = map[string]any{lh1: []any{"1.1.1.1:4242"}}
_, err := NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
_, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh2 := "10.128.0.3"
c = config.NewC(l)
c.Settings["lighthouse"] = map[string]any{"hosts": []any{lh1, lh2}}
c.Settings["static_host_map"] = map[string]any{lh1: []any{"100.1.1.1:4242"}}
_, err = NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
_, err = NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.EqualError(t, err, "lighthouse 10.128.0.3 does not have a static_host_map entry")
}
@@ -71,7 +70,7 @@ func TestReloadLighthouseInterval(t *testing.T) {
}
c.Settings["static_host_map"] = map[string]any{lh1: []any{"1.1.1.1:4242"}}
lh, err := NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
@@ -99,7 +98,7 @@ func BenchmarkLighthouseHandleRequest(b *testing.B) {
}
c := config.NewC(l)
lh, err := NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
lh, err := NewLightHouseFromConfig(b.Context(), l, c, cs, nil, nil)
require.NoError(b, err)
hAddr := netip.MustParseAddrPort("4.5.6.7:12345")
@@ -202,7 +201,7 @@ func TestLighthouse_Memory(t *testing.T) {
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
lh, err := NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
lh.ifce = &mockEncWriter{}
require.NoError(t, err)
lhh := lh.NewRequestHandler()
@@ -288,7 +287,7 @@ func TestLighthouse_reload(t *testing.T) {
myVpnNetworksTable: nt,
}
lh, err := NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
nc := map[string]any{
@@ -523,7 +522,7 @@ func TestLighthouse_Dont_Delete_Static_Hosts(t *testing.T) {
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
lh, err := NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
@@ -589,7 +588,7 @@ func TestLighthouse_DeletesWork(t *testing.T) {
myVpnNetworks: []netip.Prefix{myVpnNet},
myVpnNetworksTable: nt,
}
lh, err := NewLightHouseFromConfig(context.Background(), l, c, cs, nil, nil)
lh, err := NewLightHouseFromConfig(t.Context(), l, c, cs, nil, nil)
require.NoError(t, err)
lh.ifce = &mockEncWriter{}
+9
View File
@@ -12,6 +12,15 @@ import (
const DefaultMTU = 1300
type NameError struct {
Name string
Underlying error
}
func (e *NameError) Error() string {
return fmt.Sprintf("could not set tun device name: %s because %s", e.Name, e.Underlying)
}
// TODO: We may be able to remove routines
type DeviceFactory func(c *config.C, l *logrus.Logger, vpnNetworks []netip.Prefix, routines int) (Device, error)
+1 -1
View File
@@ -266,7 +266,7 @@ func newTun(c *config.C, l *logrus.Logger, vpnNetworks []netip.Prefix, _ bool) (
}
// Set the device name
ioctl(fd, syscall.SIOCSIFNAME, uintptr(unsafe.Pointer(&ifrr)))
_ = ioctl(fd, syscall.SIOCSIFNAME, uintptr(unsafe.Pointer(&ifrr)))
}
t := &tun{
+8 -2
View File
@@ -112,9 +112,13 @@ func newTun(c *config.C, l *logrus.Logger, vpnNetworks []netip.Prefix, multiqueu
if multiqueue {
req.Flags |= unix.IFF_MULTI_QUEUE
}
copy(req.Name[:], c.GetString("tun.dev", ""))
nameStr := c.GetString("tun.dev", "")
copy(req.Name[:], nameStr)
if err = ioctl(uintptr(fd), uintptr(unix.TUNSETIFF), uintptr(unsafe.Pointer(&req))); err != nil {
return nil, err
return nil, &NameError{
Name: nameStr,
Underlying: err,
}
}
name := strings.Trim(string(req.Name[:]), "\x00")
@@ -137,6 +141,7 @@ func newTunGeneric(c *config.C, l *logrus.Logger, file *os.File, vpnNetworks []n
TXQueueLen: c.GetInt("tun.tx_queue", 500),
useSystemRoutes: c.GetBool("tun.use_system_route_table", false),
useSystemRoutesBufferSize: c.GetInt("tun.use_system_route_table_buffer_size", 0),
routesFromSystem: map[netip.Prefix]routing.Gateways{},
l: l,
}
@@ -712,6 +717,7 @@ func (t *tun) Close() error {
if t.ioctlFd > 0 {
_ = os.NewFile(t.ioctlFd, "ioctlFd").Close()
t.ioctlFd = 0
}
return nil
+4 -1
View File
@@ -74,7 +74,10 @@ func newTun(c *config.C, l *logrus.Logger, vpnNetworks []netip.Prefix, _ bool) (
l.WithError(err).Debug("Failed to create wintun device, retrying")
tunDevice, err = wintun.CreateTUNWithRequestedGUID(deviceName, guid, t.MTU)
if err != nil {
return nil, fmt.Errorf("create TUN device failed: %w", err)
return nil, &NameError{
Name: deviceName,
Underlying: fmt.Errorf("create TUN device failed: %w", err),
}
}
}
t.tun = tunDevice.(*wintun.NativeTun)
+11
View File
@@ -6,6 +6,7 @@ import (
"log"
"net"
"net/http"
"net/http/pprof"
"runtime"
"strconv"
"time"
@@ -93,6 +94,8 @@ func startPrometheusStats(l *logrus.Logger, i time.Duration, c *config.C, buildV
return nil, fmt.Errorf("stats.path should not be empty")
}
enablePprof := c.GetBool("stats.pprof", false)
pr := prometheus.NewRegistry()
pClient := mp.NewPrometheusProvider(metrics.DefaultRegistry, namespace, subsystem, pr, i)
if !configTest {
@@ -119,6 +122,14 @@ func startPrometheusStats(l *logrus.Logger, i time.Duration, c *config.C, buildV
startFn = func() {
l.Infof("Prometheus stats listening on %s at %s", listen, path)
http.Handle(path, promhttp.HandlerFor(pr, promhttp.HandlerOpts{ErrorLog: l}))
if enablePprof {
l.Info("pprof endpoints enabled on stats listener")
http.HandleFunc("GET /debug/pprof/", pprof.Index)
http.HandleFunc("GET /debug/pprof/cmdline", pprof.Cmdline)
http.HandleFunc("GET /debug/pprof/profile", pprof.Profile)
http.HandleFunc("GET /debug/pprof/symbol", pprof.Symbol)
http.HandleFunc("GET /debug/pprof/trace", pprof.Trace)
}
log.Fatal(http.ListenAndServe(listen, nil))
}
}