Headscale — Testing#

Test metrics#

• Test files: 98
• Source files (non-test): 139
• Ratio (test files / source files): ~0.71 — high coverage density
• Test frameworks: testify (assert + require), stdlib testing, go-cmp (deep equality), zombiezen.com/go/postgrestest (in-process Postgres), ory/dockertest (Docker E2E)
• Table-driven test instances: 553 occurrences of testCases/tt./tc.name/t.Run patterns across test files

Test organization#

Placement#

Both same-package (white-box) and external _test package tests are used:

• Unit and DB tests use internal packages (e.g., package db in db_test.go)
• The servertest tier uses the external package servertest_test convention, treating the harness as a public API
• Policy compat tests are internal, since they need access to private helpers

Three-tier test pyramid#

Headscale has an unusually well-structured three-tier strategy:

Tier 1 — Unit tests (hscontrol/{mapper,policy,state,types,util,db}/)

• Fast, focused, run with go test -race ./...
• Cover individual functions, edge cases, error paths
• Use hand-rolled mocks (no gomock/mockery) and table-driven cases

Tier 2 — In-process integration (hscontrol/servertest/)

• A custom harness that wires a real Headscale server to real tailscale.com/control/controlclient.Direct instances over net/http/httptest
• No Docker, no network isolation — runs entirely in-process and fast
• Covers the full control protocol (noise handshake, MapRequest/MapResponse loop, policy changes, ephemeral nodes, route propagation, races)
• Files: server.go, client.go, harness.go, assertions.go (infra) + 12 test files

Tier 3 — Docker E2E (integration/)

• Uses ory/dockertest to spin up headscale and multiple Tailscale client containers with real network isolation
• Tests the entire stack against actual Tailscale binaries
• Covers: ACL policies, auth flows (preauth key, OIDC, web), CLI operations, DNS, DERP, SSH, route management, tags
• Version matrix: tests against head, unstable, and historical Tailscale versions via capver.TailscaleLatestMajorMinor

Helper packages#

Fixtures / testdata#

• hscontrol/types/testdata/: YAML config fixtures for config_test.go
• hscontrol/policy/v2/testdata/acl_results/, grant_results/, routes_results/, ssh_results/: JSON golden files for ACL/grant/route/SSH compat tests (hundreds of files from Tailscale SaaS captures)
• hscontrol/db/testdata/: DB migration SQL fixtures

Test patterns#

Table-driven tests#

• Prevalence: Heavy — 553 occurrences. Almost all logic tests use this style.
• Style: Anonymous struct slices with named fields. Executed with t.Run(tc.name, ...).
• Example: hscontrol/mapper/batcher_unit_test.go — each test case configures a mockNodeConnection setup and asserts on sent MapResponse values
• Policy tests: hscontrol/policy/policy_test.go drives hundreds of ACL evaluation cases via []struct{name, policy, nodes, peers, want}

Mocking approach#

• Strategy: Hand-written interface mocks — no gomock, no mockery.
• Example: batcher_unit_test.go defines mockNodeConnection implementing the nodeConnection interface:

  type mockNodeConnection struct {
      id     types.NodeID
      sendFn func(*tailcfg.MapResponse) error
      sent   []*tailcfg.MapResponse
      mu     sync.Mutex
      peers  *xsync.Map[tailcfg.NodeID, struct{}]
  }

  withSendError(err) configures failure injection. This is minimal, purposeful mocking — only the batcher’s nodeConnection interface is mocked; everything else uses real implementations.

In-process integration: the servertest harness#

The most distinctive testing pattern in the codebase. servertest.NewServer(tb, opts...):

1. Creates a real types.Config pointing to a tmpDir SQLite database
2. Calls hscontrol.NewHeadscale(&cfg) — full initialization
3. Starts app.StartBatcherForTest and app.StartEphemeralGCForTest
4. Wraps in httptest.NewServer(app.HTTPHandler())
5. Returns a TestServer from which callers create TestClient instances using real controlclient.NewDirect with the test server’s URL

TestClient.startPoll runs controlclient.Direct.PollNetMap in a goroutine, storing each NetworkMap update in a history slice and signalling a buffered updates channel. WaitForPeers, WaitForCondition, and WaitForUpdate block on this channel up to a deadline.

TestHarness composes multiple clients: NewHarness(t, 5) creates 5 clients in a shared user and calls WaitForMeshComplete before returning. Test body starts from a known converged state.

assertions.go provides reusable assertions: AssertMeshComplete, AssertSymmetricVisibility, AssertPeerOnline/Offline/Gone, AssertPeerHasAllowedIPs, EventuallyAssertMeshComplete.

servertest options (functional options pattern): WithBatchDelay, WithBufferedChanSize, WithEphemeralTimeout allow test-specific tuning of the control plane without affecting source code.

Golden file / data-driven tests#

hscontrol/policy/v2/tailscale_acl_data_compat_test.go and sibling files load every testdata/acl_results/ACL-*.json file at test time, run headscale’s policy engine against the embedded policy and nodes, then compare against the golden expected packet filter rules. This technique captures hundreds of compatibility edge cases while keeping test code minimal. The golden files themselves were converted from Tailscale SaaS API captures.

Integration tests#

• Present: Yes — integration/ directory with 16 test files
• How: ory/dockertest manages containers. hsic.New(pool, ...) starts a headscale container; tsic.New(pool, version, ...) starts Tailscale clients at specific versions. All containers share a Docker network.
• Separation: All integration tests are in integration/ package, invoked via go run ./cmd/hi run "TestName". No build tags.
• EventuallyWithT: All external (network/container) calls are wrapped in assert.EventuallyWithT blocks to handle distributed system eventual consistency. This is codified as mandatory in AGENTS.md.
• PostgreSQL: DB tests support both SQLite (default) and PostgreSQL (via zombiezen.com/go/postgrestest for in-process; --postgres flag for Docker E2E).

Concurrency and race tests#

hscontrol/mapper/batcher_concurrency_test.go (1,836 lines) is a dedicated race-condition test suite. It exercises concurrent AddToBatch, AddNode, RemoveNode, and Close calls using goroutines with sync.WaitGroup and checks that no panics, deadlocks, or incorrect peer states occur. Run with -race flag.

hscontrol/servertest/race_test.go and poll_race_test.go test concurrent connect/disconnect under the full server stack (not just the batcher in isolation).

Benchmarks#

hscontrol/mapper/batcher_bench_test.go (18 benchmarks) and batcher_scale_bench_test.go (additional scale benchmarks) exercise the hot path:

• BenchmarkFullPipeline — end-to-end change → map response → send
• BenchmarkBroadcastToN — fan-out to N concurrent nodes
• BenchmarkConnectionChurn — add/remove nodes under load
• BenchmarkScale_* — parameterised from 10 to 10,000 nodes

These benchmarks are the primary guard against performance regressions in the most write-intensive part of the system.

CI configuration#

Unit tests use gotestsum (not plain go test) for structured output and retry support. Nix is used to guarantee reproducible toolchain (Go version, linter, protoc).

Test quality observations#

What’s done well#

• Three-tier pyramid is explicit and maintained. Unit, in-process integration, and Docker E2E have clearly defined scopes. The servertest tier is the standout — real Tailscale SDK clients against a real Headscale server running in-process gives high fidelity without Docker overhead, enabling t.Parallel() across dozens of sub-tests.
• Benchmarks on the hot path. The batcher benchmark suite is thorough and granular enough to catch regressions in individual operations (peer diff, channel send, concurrent churn).
• Golden file tests for protocol compatibility. Loading hundreds of JSON fixtures from testdata/acl_results/ catches Tailscale ACL parity regressions without inflating test code. The fixtures are derivable from real SaaS captures, making them trustworthy ground truth.
• Both database backends tested. zombiezen.com/go/postgrestest starts a real in-process Postgres for DB unit tests — not a mock, not SQLite approximating Postgres behavior.
• Race detection mandated. go test -race ./... is listed in AGENTS.md as the required test command, and the dedicated concurrency test files make races detectable.
• AGENTS.md codifies testing rules for AI agents. The mandatory EventuallyWithT pattern, separation of external call blocks, and integration test agent instructions mean the testing discipline is encoded for automation.

What could improve#

• No visible coverage metric targets. No cover flags in CI workflows or coverage thresholds enforced.
• stress_test.go uses real time.After rather than testing/synctest (which would make timing deterministic). The comment in client.go mentions synctest compatibility as future work.
• Integration tests require Docker daemon — hi doctor must pass, adding infrastructure friction that can mask code failures as environment failures.
• No fuzz tests visible (go test -fuzz), despite the policy engine parsing untrusted HuJSON — a natural fuzz target.

Patterns worth emulating#

1. The servertest harness design: Using httptest.Server + real SDK clients is transferable to any project whose clients have a usable test constructor. The TestHarness abstraction (converged multi-node mesh in one call) makes multi-node tests trivially writable.
2. Functional options on the test server: WithBatchDelay(50ms) lets individual tests probe timing-sensitive behavior without patching globals or mutating shared state.
3. Dedicated assertions.go package: Domain-specific assertions (AssertMeshComplete, AssertSymmetricVisibility) keep test bodies readable and error messages contextual. Worth extracting alongside any harness.
4. Golden file test loader pattern: Dynamically loading testdata/**/*.json and running one sub-test per file keeps the test file small while scaling the fixture set indefinitely. New compatibility cases are added by dropping a JSON file, not editing Go code.