Headscale — Architecture#

Architectural style#

Layered monolith with an event-driven fan-out path.

Headscale is a single-binary server with clear vertical layers (CLI → application → state → persistence) and a horizontal fan-out tier (the mapper.Batcher) that pushes change notifications to all connected Tailscale clients in real time. It is not a microservice: everything runs in-process. The event-driven character is narrow and purposeful — it applies only to the hot path of broadcasting network-map changes to long-polling HTTP streams, not to the entire application.

Evidence: Headscale.Change() (app.go:1069) enqueues change.Change values into mapper.Batcher.AddWork(); workers fan them out to per-node channels; each channel feeds a long-lived HTTP response stream opened by a Tailscale client (poll.go). The rest of the application — authentication, admin API, DERP relay — is fully synchronous RPC.

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────────┐
│ Binary: headscale                                                   │
│                                                                     │
│  CLI (Cobra)                                                        │
│  cmd/headscale/cli/                                                 │
│    serve  users  nodes  apikeys  preauthkeys  policy  routes ...    │
│       │                      │                                      │
│       │ newHeadscaleServer()  │ gRPC client (Unix socket)           │
│       ▼                      ▼                                      │
│  ┌─────────────────────────────────────────────┐                   │
│  │  Headscale (app.go)                         │                   │
│  │  ─────────────────────────────────────────  │                   │
│  │  cfg  state  noisePrivateKey  DERPServer     │                   │
│  │  authProvider  mapBatcher  ephemeralGC       │                   │
│  │                                             │                   │
│  │  Serve()  ──►  errgroup of listeners:       │                   │
│  │    • HTTP (chi)    ← Noise/MapRequest        │                   │
│  │    • gRPC/Unix     ← admin API (no-auth)     │                   │
│  │    • gRPC/TCP      ← admin API (TLS+auth)    │                   │
│  │    • grpc-gateway  ← REST bridge over Unix   │                   │
│  │    • debugHTTP     ← pprof / tailsql         │                   │
│  └─────────────────────────────────────────────┘                   │
│          │                     │                                    │
│          │ state.*             │ mapBatcher.AddWork(change)         │
│          ▼                     ▼                                    │
│  ┌────────────────┐   ┌─────────────────────────────────────────┐  │
│  │ state.State    │   │ mapper.Batcher                          │  │
│  │ ─────────────  │   │ ──────────────────────────────────────  │  │
│  │ NodeStore (CoW)│   │ per-node channel map                   │  │
│  │ PolicyManager  │   │ worker pool                            │  │
│  │ IPAllocator    │   │ mapper (state → tailcfg.MapResponse)   │  │
│  │ derpMap(atomic)│   │                                        │  │
│  │ authCache      │   │ AddNode / RemoveNode                   │  │
│  │ primaryRoutes  │   │ AddWork → fan-out                      │  │
│  └────────────────┘   └─────────────────────────────────────────┘  │
│          │                     │ channels                           │
│          │                     ▼                                    │
│  ┌──────────────┐   ┌─────────────────────────────────────────┐    │
│  │ hscontrol/db │   │  mapSession (poll.go)                   │    │
│  │ GORM + mig.  │   │  long-polling HTTP response writer      │    │
│  │ SQLite / PG  │   │  keepAlive ticker                       │    │
│  └──────────────┘   └─────────────────────────────────────────┘    │
│          │                     │ HTTP stream to Tailscale client    │
│  ┌──────────────┐              ▼                                    │
│  │ policy/      │   Tailscale client (external)                    │
│  │ PolicyManager│                                                   │
│  │ ACL engine   │                                                   │
│  └──────────────┘                                                   │
│                                                                     │
│  Cross-cutting: derp/  dns/  routes/  types/  capver/  util/       │
└─────────────────────────────────────────────────────────────────────┘

Core components#

Headscale (application shell)#

  • Package: hscontrol (file: app.go)
  • Responsibility: Top-level wiring. Holds all subsystems, constructs the HTTP and gRPC server mux, drives the application lifecycle, and owns the Change() entry point for broadcasting state mutations.
  • Key types: type Headscale struct — cfg, state, noisePrivateKey, DERPServer, authProvider, mapBatcher, ephemeralGC
  • Dependencies: All subsystems (state, mapper, db, derp, dns, auth)

state.State (central coordinator)#

  • Package: hscontrol/state
  • Responsibility: Thread-safe coordinator for all runtime state. Holds the NodeStore (in-memory node cache), the PolicyManager (ACL evaluation), IPAllocator, derpMap (atomic pointer), authentication cache, and primary route assignments. All write paths go through State.
  • Key types: State — db, ipAlloc, nodeStore, polMan, derpMap (atomic.Pointer), authCache, primaryRoutes, connectGen (sync.Map), sshCheckAuth
  • Dependencies: hscontrol/db, hscontrol/policy, hscontrol/routes, hscontrol/types

NodeStore (CoW in-memory cache)#

  • Package: hscontrol/state
  • Responsibility: Copy-on-write in-memory node snapshot used by the hot poll path. Decouples database I/O from the high-frequency MapRequest processing (every 15–60 seconds per client). Writes are batched by count (defaultNodeStoreBatchSize = 100) or timeout (500ms) before the snapshot is atomically replaced. The peersFunc closure (provided by PolicyManager) decides which nodes each node can see.
  • Key types: NodeStore — batched writes, snapshot rebuild, deadlock-detected RWMutex
  • Dependencies: hscontrol/policy (via peersFunc closure), hscontrol/types

mapper.Batcher (change fan-out)#

  • Package: hscontrol/mapper
  • Responsibility: Receives typed change.Change values (from h.Change()), resolves which nodes are affected, generates tailcfg.MapResponse objects, and delivers them over per-node buffered channels to the long-polling HTTP sessions. Maintains a lock-free map from NodeID to multiChannelNodeConn (supporting multiple simultaneous connections per node for rapid reconnect).
  • Key types: Batcher — workCh, done, nodes (xsync.Map), mapper, worker pool; multiChannelNodeConn — connectionEntry list; mapper — translates State → tailcfg.MapResponse
  • Dependencies: hscontrol/state, hscontrol/types/change, tailscale.com/tailcfg

mapSession (poll protocol handler)#

  • Package: hscontrol (file: poll.go)
  • Responsibility: Handles a single Tailscale MapRequest from a connected node. Creates a buffered channel, registers it with the Batcher, then pumps responses from the channel to the HTTP response writer, interleaving keepAlive frames. Handles both streaming (long-poll) and one-shot requests.
  • Key types: mapSession — Headscale ref, req, ctx, ch (chan *tailcfg.MapResponse), cancelCh, keepAliveTicker, node, w
  • Dependencies: hscontrol (Headscale), mapper.Batcher, hscontrol/types

DB layer (hscontrol/db)#

  • Package: hscontrol/db
  • Responsibility: Persistent storage via GORM ORM. Supports SQLite (pure-Go modernc driver, WAL mode) and PostgreSQL. Manages schema migrations with an immutable ordered migration list. Provides CRUD for nodes, users, pre-auth keys, API keys, routes, policy bytes. IPAllocator is co-located here.
  • Key types: HSDatabase, IPAllocator, EphemeralGarbageCollector
  • Dependencies: gorm.io/gorm, gorm.io/driver/sqlite, gorm.io/driver/postgres, hscontrol/types

policy.PolicyManager (ACL engine)#

  • Package: hscontrol/policy
  • Responsibility: Parses and evaluates HuJSON access-control policy. Determines peer visibility (BuildPeerMap), route auto-approval, and SSH check-auth rules. The v2/ subdirectory contains a next-generation policy system actively replacing policy/. The matcher/ sub-package implements rule matching.
  • Key types: PolicyManager interface, concrete implementation in policy.go and v2/
  • Dependencies: hscontrol/types, tailscale.com/tailcfg

gRPC management API (hscontrol/grpcv1.go + gen/)#

  • Package: hscontrol, generated stubs in gen/go/headscale/v1
  • Responsibility: Admin API defined in 8 .proto files, served simultaneously as gRPC (over Unix socket and optionally TCP) and REST (via grpc-gateway bridge). Operations: node CRUD, user management, pre-auth keys, API keys, policy, routes.
  • Key types: HeadscaleServiceServer (generated interface), headscaleV1APIServer (implementation)
  • Dependencies: hscontrol/state, gen/go/headscale/v1

Auth providers (auth.go, oidc.go)#

  • Package: hscontrol
  • Responsibility: Implements the AuthProvider interface with two concrete providers: AuthProviderWeb (browser-based node registration flow) and AuthProviderOIDC (OpenID Connect SSO). Selected at startup based on configuration.
  • Key types: AuthProvider interface, AuthProviderWeb, AuthProviderOIDC
  • Dependencies: hscontrol/state, coreos/go-oidc

DERP relay (hscontrol/derp/)#

  • Package: hscontrol/derp, hscontrol/derp/server
  • Responsibility: Fetches and maintains the DERP map (list of relay regions) from Tailscale’s public list or custom URLs. Optionally runs an embedded DERP server in the same process. DERP provides NAT traversal fallback when direct WireGuard connections fail.
  • Key types: DERPServer, DERP map fetcher
  • Dependencies: tailscale.com/derp, hscontrol/types

Data flow#

1. Node registration (first connection)#

Tailscale client
  → HTTPS /key            (public Noise key exchange)
  → HTTPS /ts2021         (Noise upgrade, noise.go)
  → noise.go HandshakeServer()
  → auth.go HandleRegisterRequest()
      → state.GetRegistrationByKey() / authCache lookup
      → authProvider.HandleAuthCallback() (web or OIDC)
      → state.RegisterNode()
          → db.RegisterNode()  (IP allocation, persist)
          → nodeStore.Write()  (CoW snapshot update)
  → h.Change(change.NodeAdded(nodeID))
      → mapBatcher.AddWork()
          → worker fan-out → affected nodes' channels

2. Steady-state polling (every 15–60 s)#

Tailscale client
  → POST /machine/map     (Noise-encrypted MapRequest, handlers.go)
  → poll.go PollNetMapHandler()
      → h.newMapSession()
      → mapSession.serve()
          → mapBatcher.AddNode(nodeID, ch, version, stopFn)
              → mapper.MapResponseFromChange(change.FullSelf)
              → ch ← initialMapResponse
          → loop:
              select {
                ch    → write MapResponse to HTTP stream (zstd-compressed)
                timer → write keepAlive frame
                ctx   → graceful shutdown
              }

3. Change broadcast (e.g., node endpoint update)#

Tailscale client  →  MapRequest with Hostinfo/Endpoints
  → poll.go processes update
  → state.SaveNodeEndpoints() / state.UpdateNode()
      → db.WriteNode()
      → nodeStore.Write(change)
  → h.Change(change.NodeChanged(nodeID))
      → mapBatcher.AddWork(change)
          → doWork goroutine
              → for each affected node in Batcher.nodes:
                  → mapper.MapResponseFromChange(nodeID, change)
                  → nodeConn.ch ← mapResponse
              → each mapSession.serve() loop delivers to client

4. Admin API call (e.g., headscale nodes list)#

headscale CLI (admin subcommand)
  → gRPC dial Unix socket (no TLS, no auth)
  → grpcv1.go ListNodes()
      → state.ListNodes()
          → nodeStore.Snapshot()  (read-only CoW view)
      → protobuf marshal → gRPC response

Initialization / Bootstrap#

Sequence (in order):

  1. main() — sets up zerolog, detects color support, calls cli.Execute()
  2. Cobra root → serve subcommand (cli/serve.go)
  3. newHeadscaleServerWithConfig() — loads YAML config via viper, calls NewHeadscale(cfg)
  4. NewHeadscale(cfg): a. readOrCreatePrivateKey() — Noise private key (persisted to disk) b. state.NewState(cfg):
    • hsdb.NewHeadscaleDatabase() — GORM setup, migration runner, schema validation
    • hsdb.NewIPAllocator() — IP range initialisation, bitmap of allocated IPs
    • db.ListNodes() + db.ListUsers() — bulk load
    • policy.NewPolicyManager() — parse ACL, build initial peer map
    • NewNodeStore(nodes, peersFunc, batchSize, batchTimeout) — CoW cache start c. db.NewEphemeralGarbageCollector() — GC for ephemeral node inactivity d. NewAuthProviderWeb() → optionally NewAuthProviderOIDC() (OIDC discovery with 30 s timeout) e. derpServer.NewDERPServer() if embedded DERP is configured
  5. app.Serve(): a. mapper.NewBatcherAndMapper(cfg, state) + .Start() — worker goroutines running b. DERPServer.ServeSTUN() goroutine (if embedded) c. derp.GetDERPMap() — fetch initial DERP map (with exponential backoff retry) d. state.SetDERPMap() — atomic store e. ephemeralGC.Start() — schedules existing ephemeral nodes f. dns.NewExtraRecordsManager() goroutine (if configured) g. scheduledTasks() goroutine — node expiry, DERP map refresh, DNS extra records h. Unix socket gRPC server (no auth, grpc-gateway connects here) i. grpc-gateway HTTP mux (RegisterHeadscaleServiceHandler) j. Remote gRPC server on TCP (TLS + auth interceptor, if configured) k. HTTP server (chi router) — Noise endpoint, MapRequest, admin REST bridge l. debug HTTP server (pprof, optional tailsql) m. errgroup.Wait() — blocks until all servers exit or any returns error n. SIGTERM/SIGINT signal handler closes context, triggering graceful shutdown

Dependency injection pattern: Manual wiring. No framework (no Wire, no dig, no fx). NewHeadscale constructs all subsystems explicitly and passes them as fields. The Headscale struct is the composition root.

Configuration#

  • Primary mechanism: YAML file (default ~/.headscale/config.yaml, override via --config flag)
  • Library: Viper (loaded in CLI layer before NewHeadscale)
  • Type: hscontrol/types/config.go — Config struct with sub-structs for Database, DERP, OIDC, DNS, TLS, Tuning, etc.
  • Debug env knobs: tailscale.com/envknob used for HEADSCALE_DEBUG_DEADLOCK, HEADSCALE_DEBUG_PROFILING_ENABLED, HEADSCALE_DEBUG_TAILSQL_*, HEADSCALE_DEBUG_DUMP_CONFIG — these bypass viper entirely and are intended for developer diagnostics.
  • Tuning sub-config: cfg.Tuning exposes NodeStoreBatchSize, NodeStoreBatchTimeout, RegisterCacheExpiration, NodeMapSessionBufferedChanSize — allowing operators to tune performance-critical paths without code changes.
  • Policy: Loaded either from a file path (mode: file) or from the database (mode: db); hot-reloadable via state.ReloadPolicy().

Key design decisions#

1. CoW NodeStore as the hot-path cache#

The NodeStore (hscontrol/state/node_store.go) maintains an atomically-swapped in-memory snapshot of all nodes. MapRequest processing reads from this snapshot without any database round-trips. Writes are batched (100 ops or 500 ms) before triggering a snapshot rebuild, amortising the cost of recomputing the policy-filtered peer map. This is the single most architecturally significant performance investment in the project.

2. Typed change bus decouples mutation from fan-out#

Every state mutation (node registered, endpoint changed, policy updated, DERP map refreshed) is expressed as a typed change.Change value pushed through h.Change() → mapBatcher.AddWork(). The Batcher decides which nodes are affected and generates MapResponses. This separates what changed from who needs to know, making it straightforward to add new change types without touching the fan-out logic.

3. Dual-socket gRPC for security without friction#

The admin gRPC server runs on two sockets simultaneously: a Unix socket (no auth, no TLS) that only the local grpc-gateway and CLI can reach, and an optional TCP socket with API key authentication and TLS. The grpc-gateway bridges REST calls over the Unix socket, avoiding a second authentication layer for local requests. This avoids the complexity of a separate REST server while still providing a clean JSON API.

4. Direct consumption of tailscale.com libraries#

Headscale imports tailscale.com v1.94.1 as a Go dependency and uses its types (tailcfg.MapRequest, tailcfg.MapResponse, tailcfg.DERPMap, key.MachinePrivate, etc.) directly. This eliminates manual serialisation and keeps headscale structurally compatible with Tailscale client updates. The cost is tracking upstream API changes; capver/ manages capability version negotiation to gracefully reject clients that are too old.

5. Tags-as-identity as a core invariant#

Tagged nodes and user-owned nodes are mutually exclusive ownership models enforced at every layer: registration (auth.go), storage (db/node.go), wire protocol (mapper/tail.go always sends TaggedDevices.ID for tagged nodes), and admin API (grpcv1.go rejects SetTags on user-owned nodes). This is a non-trivial architectural constraint that permeates the codebase and is explicitly validated in tests.

Backward Headscale — Dependencies Headscale — Patterns Forward