Headscale — Patterns#

Concurrency patterns#

Worker pool (Batcher fan-out)#

  • Usage: mapper.Batcher maintains a configurable pool of worker goroutines (default via cfg.Tuning.BatcherWorkers) that drain a single buffered work channel (workCh with capacity workers*200). A supervisor goroutine (doWork) ticks on a batch interval and spawns per-worker goroutines.
  • Example: hscontrol/mapper/batcher.go:385-411 — doWork() spawns b.workers goroutines each calling b.worker(i), which selects on workCh and done.
  • Assessment: Well-structured. Pool size and batch delay are operator-tunable (cfg.Tuning). sync.WaitGroup ensures Close() blocks until all workers drain before tearing down node connections, preventing send-on-closed-channel races.

Fan-out via per-node buffered channels#

  • Usage: Every connected Tailscale node has a chan *tailcfg.MapResponse in the Batcher’s multiChannelNodeConn (multiple connections per node are supported for rapid reconnect). Workers fan out MapResponse values to all affected node channels; each channel is drained by the node’s long-polling mapSession.serve() loop.
  • Example: hscontrol/mapper/batcher.go:222-302 (AddNode); hscontrol/poll.go (mapSession loop that selects on ch, keepAlive timer, and ctx).
  • Assessment: Classic producer-consumer fan-out. Buffered channels decouple generation latency from HTTP delivery. The multiChannelNodeConn design (supporting multiple concurrent channels per node) elegantly handles the rapid-reconnect window without dropping updates.

Context cancellation#

  • Usage: Context is pervasive — 347 context.Context appearances. Every HTTP handler, gRPC call, and long-lived goroutine accepts a context for cancellation. mapSession.serve() selects on ctx.Done() as its shutdown path.
  • Example: hscontrol/poll.go — the map session loop has case <-ctx.Done(): return nil as one of its three select arms.
  • Assessment: Idiomatic. Context cancellation is the primary mechanism for stopping long-poll sessions on client disconnect or server shutdown.

Graceful shutdown#

  • Usage: Batcher.Close() closes a done channel via sync.Once, waits for wg.Wait(), then closes all node connection channels. The application uses signal.Notify for SIGTERM/SIGINT (hscontrol/app.go:822) and an errgroup for coordinated server exit.
  • Example: hscontrol/mapper/batcher.go:355-383 — doneOnce.Do(func() { close(b.done) }) followed by b.wg.Wait().
  • Assessment: Pattern: close a done channel (never close a work channel that senders may still write to), then join via WaitGroup. The comment in Close() explicitly explains why workCh is left open — a useful teaching example of correct channel lifecycle reasoning.

Idempotent start / stop with atomics#

  • Usage: Batcher.Start() uses b.started.CompareAndSwap(false, true) to ensure it is called exactly once. Batcher.doneOnce sync.Once ensures the done channel is closed exactly once even if Close() is called concurrently.
  • Example: hscontrol/mapper/batcher.go:345-353 and 355-383.
  • Assessment: Correct dual pattern: atomic.Bool + CAS for “start once”, sync.Once for “close once”. The two tools are used appropriately for their respective semantics.

Lock-free concurrent map (xsync)#

  • Usage: mapper.Batcher.nodes is *xsync.Map[types.NodeID, *multiChannelNodeConn] from puzpuzpuz/xsync/v4. The Compute API (with CancelOp/DeleteOp return values) enables atomic read-modify-write without an external lock.
  • Example: hscontrol/mapper/batcher.go:630-647 — Compute callback returns xsync.CancelOp if the node is already being tracked, xsync.DeleteOp to remove it.
  • Assessment: Well-chosen for the hot path. sync.Map would work but xsync.Map is generics-typed and provides the Compute primitive needed for CAS-style updates. The design comment in node_conn.go:64 explains the choice explicitly.

errgroup for parallel server startup#

  • Usage: app.Serve() uses golang.org/x/sync/errgroup to start multiple servers (HTTP, Unix-socket gRPC, TCP gRPC, grpc-gateway, debug HTTP) concurrently. errorGroup.Wait() blocks the application lifecycle.
  • Example: hscontrol/app.go:615.
  • Assessment: Standard idiom for starting multiple independent listeners that should all survive together. Any single failure tears down the whole errgroup.

Scheduled background tasks#

  • Usage: app.scheduledTasks() (hscontrol/app.go:984) runs a single goroutine with a select over multiple time.Ticker channels for node expiry checks, DERP map refresh, and DNS record updates. All tickers are co-located.
  • Assessment: Simple and readable — one goroutine per concern group. Avoids goroutine proliferation for periodic work.

Error handling#

  • Style: Mixed — sentinel errors combined with fmt.Errorf("%w", ...) wrapping. No pkg/errors.
  • Error types defined:
    • Package-level var Err* = errors.New(...) sentinels in each package (ErrInvalidNodeID, ErrMapperNil, ErrNodeConnectionNil, ErrNodeNotFoundMapper in mapper; ErrNodeNotInNodeStore in state; etc.).
    • One string-based custom error type: type Error string in cmd/headscale/cli/mockoidc.go:22 with func (e Error) Error() string.
    • Domain errors in hscontrol/types (e.g., types.ErrCannotRemoveAllTags).
  • Wrapping approach: fmt.Errorf("context: %w", err) throughout. Consistent use of %w rather than format strings — allows errors.Is/errors.As unwrapping. Example: fmt.Errorf("generating initial map for node %d: %w", id, err) (batcher.go:270).
  • Sentinel chaining: fmt.Errorf("%w: %d", ErrInvalidNodeID, nodeID) — wraps a sentinel with context data, preserving errors.Is semantics.
  • Examples:
    • hscontrol/mapper/batcher.go:22-27 — package-level error sentinels
    • cmd/headscale/cli/utils.go:64 — fmt.Errorf("creating new headscale: %w", err)
    • hscontrol/state/state.go:447 — fmt.Errorf("%w: %d", ErrNodeNotInNodeStore, node.ID())

Configuration pattern#

  • Approach: Config struct with sub-structs. No functional options for the application core.
  • Main config: hscontrol/types/config.go — Config struct with nested DatabaseConfig, DERPConfig, OIDCConfig, DNSConfig, TLSConfig, Tuning (performance knobs).
  • Tuning sub-struct: cfg.Tuning exposes NodeStoreBatchSize, NodeStoreBatchTimeout, BatcherWorkers, RegisterCacheExpiration, NodeMapSessionBufferedChanSize. This is a deliberate separation: functional config vs. performance tuning.
  • Functional options — integration layer only: type Option = func(c *Container) in integration/tsic, integration/dsic, integration/hsic, and hscontrol/servertest. Used exclusively for test scaffolding.
  • Environment variable overrides: tailscale.com/envknob for debug/diagnostic flags (HEADSCALE_DEBUG_DEADLOCK, etc.) — these bypass Viper entirely and are for developer use only.
  • Example: NewBatcherAndMapper(cfg *types.Config, state *state.State) takes the whole config and reads cfg.Tuning.BatchChangeDelay and cfg.Tuning.BatcherWorkers — flat field access, no indirection.

Dependency injection#

  • Approach: Manual wiring. No framework (no Wire, no dig, no fx).
  • Evidence: NewHeadscale(cfg) constructs every subsystem explicitly: reads or creates private keys, calls state.NewState(cfg) (which chains db.NewHeadscaleDatabase → db.NewIPAllocator → policy.NewPolicyManager → NewNodeStore), then auth.NewAuthProviderWeb, optionally auth.NewAuthProviderOIDC, derp.NewDERPServer. All subsystems are stored as fields on type Headscale struct.
  • Composition root: Headscale struct in hscontrol/app.go is the composition root. All wiring is visible in NewHeadscale and Serve() — no magic.
  • Closure injection: Where full struct injection is too heavy, closures are used. NewNodeStore receives a peersFunc func(types.NodeID) views.Slice[types.NodeView] closure from PolicyManager. NewEphemeralGarbageCollector receives a deleteFunc func(types.NodeID). This is a lightweight form of interface-free DI for single-method dependencies.

Other notable patterns#

Value-type change descriptor (Change struct)#

The change.Change type in hscontrol/types/change/change.go is a first-class value type that describes what changed without prescribing what to do. It has:

  • Named constructor functions (NodeAdded, PolicyChange, DERPMap, NodeOnlineFor, etc.) that set the right combination of boolean flags.
  • A Merge(other Change) Change method for combining multiple in-flight changes before sending.
  • A Type() string method returning a bounded label string for Prometheus (distinct from free-form Reason for logging).
  • IsEmpty(), IsFull(), IsSelfOnly(), ShouldSendToNode() query methods.

This is a deliberate data-oriented design: the mapper is told what changed; it decides how to respond. Adding a new change category requires only a new constructor function and a new case in the mapper — no new interfaces, no callbacks.

Immutable view types (tailscale viewer pattern)#

hscontrol/types/types_view.go contains generated NodeView, UserView, and PreAuthKeyView wrappers (generated by tailscale.com/cmd/viewer). Each view wraps a pointer to the mutable struct in a field named ж (a Cyrillic letter chosen to look like a dangerous pointer). Views expose read-only accessor methods and AsStruct() (which clones). views.Slice[types.NodeView] (from tailscale.com/types/views) is used throughout the hot path to pass node lists without risk of accidental mutation.

This is adopted directly from Tailscale’s codebase and propagates into headscale’s internal API surface. It prevents the class of bugs where a background goroutine mutates a node that the mapper is concurrently serialising.

Generic DB transaction helpers#

hscontrol/db/db.go:1112 and 1137 define generic functions:

func Read[T any](db *gorm.DB, fn func(rx *gorm.DB) (T, error)) (T, error)
func Write[T any](db *gorm.DB, fn func(tx *gorm.DB) (T, error)) (T, error)

These wrap GORM transactions in a begin/defer-rollback/commit envelope, returning typed values. The zero-value of T (var no T) is returned on error. This is an elegant use of Go 1.18 generics to eliminate the repetitive transaction boilerplate that plagued pre-generic GORM code.

Builder pattern (MapResponseBuilder)#

hscontrol/mapper/builder.go uses a fluent builder for tailcfg.MapResponse construction:

m.NewMapResponseBuilder(nodeID).
    WithCapabilityVersion(v).
    WithSelfNode().
    WithPeers(peers).
    WithDERPMap().
    Build()

Errors are accumulated in the builder (b.errs []error) and surfaced in Build() via multierr. This avoids passing error return values through every With* call. Particularly appropriate here because the number of optional sections in a MapResponse varies significantly by change type.

Table-driven tests#

  • Prevalence: Heavy use — 985 occurrences of testCase/tt./tc./[]struct patterns across test files.
  • Style: Anonymous struct slices with named fields. Run with t.Run(tc.name, ...).
  • Example: hscontrol/mapper/batcher_unit_test.go and hscontrol/servertest/ contain particularly comprehensive table-driven suites covering node visibility, policy evaluation edge cases, and concurrent access correctness.

Prometheus metrics with promauto#

All performance-critical subsystems register metrics at package init via promauto.NewCounterVec / promauto.NewHistogramVec / promauto.NewGauge. The promauto package auto-registers with the default registry on first use, eliminating manual prometheus.MustRegister calls.

  • hscontrol/mapper/batcher.go:33-37 — mapResponseGenerated counter vec (labelled by response type).
  • hscontrol/state/node_store.go:27-67 — 7 metrics covering batch sizes, operation durations, snapshot build time, peer calculation duration, and queue depth. This is unusually thorough instrumentation for an in-memory cache.

Structured logging (zerolog with incremental builder)#

Zerolog is used exclusively. The AGENTS.md documents the project’s preferred pattern: build log.Event incrementally with logEvent = logEvent.Str(...) rather than chaining (to allow conditional fields). This is enforced by linting. ~405 log calls in non-test, non-generated source.

Typed ID newtypes#

types.NodeID, types.UserID, types.PreAuthKeyID etc. are defined as typed integers (e.g., type NodeID uint64). This prevents accidental mixing of IDs across entity types and enables type-safe map keys. NodeID.NodeID() converts to tailcfg.NodeID (Tailscale’s wire type) at the boundary layer.

Generics in utility functions#

Beyond the DB transaction helpers, generics appear in:

  • hscontrol/noise.go:275: func urlParam[T any](req *http.Request, key string) (T, error) — generic URL parameter extraction.
  • integration/cli_test.go:26: func executeAndUnmarshal[T any](...) error — test helper for gRPC + JSON unmarshal.
  • integration/cli_test.go:46: func sortWithID[T GRPCSortable](a, b T) int — generic sort comparator for protobuf nodes with an ID field.

These are small but well-targeted — generics used where type parameterisation genuinely reduces duplication without abstracting away semantics.

sync.Once for one-shot operations#

Used in two distinct contexts:

  1. Batcher.doneOnce sync.Once — ensures close(done) is called at most once regardless of how many callers invoke Close().
  2. Generated protobuf code uses sync.Once for lazy descriptor registration (standard pattern). The started atomic.Bool + CompareAndSwap in Batcher.Start() is distinct: it’s a “run-once but panic-free” guard rather than a “do exactly once” semantic.
Backward Headscale — Architecture Headscale — Interfaces Forward