Consul — Architecture#

Architectural style#

Layered Monolith with Role-Based Dual Personality

Consul is a single binary (consul) that can operate in two fundamentally different roles — server and client agent — within the same process architecture. The overarching style is a layered monolith: all subsystems are statically linked into one binary, sharing memory, and wired together at startup via explicit manual dependency injection.

Within server mode, Consul exhibits characteristics of an event-driven, state-machine-driven system: Raft drives state changes, Serf gossip provides membership events, and leadership transitions trigger cascading subsystem activation. Within client agent mode, it operates as a local proxy and cache node that syncs state to servers and pushes configuration to data-plane proxies (Envoy).

An emerging secondary architecture (“v2”) follows a Kubernetes controller pattern: generic resource CRUD over gRPC + asynchronous reconciliation loops, layered on top of the existing v1 Raft machinery. Both coexist in the running binary.

Evidence: The delegate interface in agent/agent.go:153 and agent.Start() at line 666 show the explicit fork based on ServerMode. The internal/controller and internal/resource packages are the v2 layer; agent/consul/ with its FSM and state store is v1.

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────┐
│                         consul binary                            │
│                                                                  │
│  main.go → mitchellh/cli → command/registry → command/agent     │
│                                    │                             │
│                         ┌──────────▼───────────┐                │
│                         │   agent.BaseDeps      │                │
│                         │  (manual DI root)     │                │
│                         │  - Config             │                │
│                         │  - TLSConfigurator    │                │
│                         │  - Cache              │                │
│                         │  - ConnPool (net/rpc) │                │
│                         │  - GRPCConnPool       │                │
│                         │  - Router             │                │
│                         │  - LeafCertManager    │                │
│                         │  - EventPublisher     │                │
│                         │  - AutoConfig         │                │
│                         └──────────┬───────────┘                │
│                                    │                             │
│                         ┌──────────▼───────────┐                │
│                         │     agent.Agent       │                │
│                         │  (runtime coordinator)│                │
│                         │  - local.State        │                │
│                         │  - ae.StateSyncer     │                │
│                         │  - checks/* (runners) │                │
│                         │  - proxycfg           │                │
│                         │  - xds server         │                │
│                         │  - apiServers (HTTP)  │                │
│                         │  - DNS server         │                │
│                         │  - delegate ──────────┼─┐             │
│                         └──────────────────────┘  │             │
│                                                    │             │
│              ┌─────────────────────────────────────┘             │
│              │ delegate (interface)                              │
│              │                                                   │
│    ┌─────────▼──────────┐      ┌───────────────────┐            │
│    │   consul.Server    │  OR  │   consul.Client   │            │
│    │ (server mode)      │      │  (client mode)    │            │
│    │ - raft.Raft        │      │  - ConnPool RPC   │            │
│    │ - FSM + StateStore │      │  - Serf LAN only  │            │
│    │ - Serf LAN + WAN   │      └───────────────────┘            │
│    │ - net/rpc Server   │                                        │
│    │ - gRPC handler     │                                        │
│    │ - ACL resolver     │                                        │
│    │ - CAManager        │                                        │
│    │ - ResourceService  │                                        │
│    │ - Controllers (v2) │                                        │
│    └────────────────────┘                                        │
│                                                                  │
│  External APIs                                                   │
│  ┌─────────────────┐  ┌──────────┐  ┌────────────────────────┐ │
│  │ HTTP REST :8500 │  │ DNS :8600│  │ gRPC :8502 (external)  │ │
│  └─────────────────┘  └──────────┘  └────────────────────────┘ │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │ Multiplexed RPC :8300 (net/rpc | gRPC-internal | Raft)   │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Core components#

Agent (Runtime Coordinator)#

  • Package: agent/
  • Responsibility: Coordinates all subsystems for both server and client modes. Manages the lifecycle of health checks, local state, anti-entropy, HTTP/DNS APIs, and the xDS server. Holds a delegate (either consul.Server or consul.Client) that it dispatches RPC calls through.
  • Key types: Agent (agent/agent.go:234), delegate interface (agent/agent.go:153), BaseDeps (agent/setup.go:55)
  • Dependencies: Virtually everything — it is the composition root.

BaseDeps (Manual DI Root)#

  • Package: agent/
  • Responsibility: The explicit dependency injection container. NewBaseDeps() constructs and wires together every shared infrastructure object before Agent.New() is called. Returns a value type (not a pointer) passed into Agent.New().
  • Key types: BaseDeps struct (agent/setup.go:55) which embeds consul.Deps
  • Dependencies: Config, TLS, logging, telemetry, cache, connection pools, router, leaf cert manager, event publisher.

consul.Server (Server-mode Core)#

  • Package: agent/consul/
  • Responsibility: The distributed systems engine for server nodes. Manages Raft consensus, Serf gossip (LAN and WAN), the in-memory state store (via go-memdb), all net/rpc RPC endpoints, gRPC handlers, ACL resolution, CA management, and leader election logic.
  • Key types: Server (agent/consul/server.go:169), FSM (agent/consul/fsm/), Store (agent/consul/state/)
  • Dependencies: hashicorp/raft, hashicorp/serf, hashicorp/go-memdb, tlsutil, acl, token, pool, router

State Store#

  • Package: agent/consul/state/
  • Responsibility: In-memory database (backed by hashicorp/go-memdb) holding the authoritative cluster state: nodes, services, health checks, KV entries, intentions, config entries, ACL tokens. All reads go here; all writes go through Raft → FSM → state store.
  • Key types: Store, table schemas in schema.go, custom indexers in indexer.go
  • Dependencies: hashicorp/go-memdb; pure in-memory, no disk I/O (Raft handles persistence via BoltDB/WAL).

Raft FSM#

  • Package: agent/consul/fsm/
  • Responsibility: The finite-state machine that Raft calls to apply committed log entries. Routes log entries by type to the appropriate state store write operations. Implements raft.FSM interface: Apply(), Snapshot(), Restore().
  • Key types: FSM struct, command dispatch table
  • Dependencies: agent/consul/state, hashicorp/raft

Local State + Anti-Entropy#

  • Package: agent/local/, agent/ae/
  • Responsibility: local.State maintains the agent’s view of services and checks registered locally. ae.StateSyncer periodically pushes local state to the server (full sync) and reacts to server changes (partial sync). This is Consul’s anti-entropy mechanism.
  • Key types: State (agent/local/state.go), StateSyncer (agent/ae/state.go)
  • Dependencies: consul.Server (via delegate RPC), local config, tokens

Cache Layer#

  • Package: agent/cache/
  • Responsibility: Client-side in-memory cache for data fetched from servers via blocking queries. Cache types (registered at startup) define how each data type is fetched and how staleness is managed. Used by proxycfg, health checks, and the API.
  • Key types: Cache, Type interface (how to fetch a resource), blocking query infrastructure
  • Dependencies: net/rpc via delegate, submatview for streaming-based types

ProxyCfg (Envoy Configuration Watcher)#

  • Package: agent/proxycfg/, agent/proxycfg-glue/, agent/proxycfg-sources/
  • Responsibility: Generates configuration snapshots for Envoy sidecar proxies. Maintains internal watches on catalog, config entries, intentions, and leaf certs. On client agents, these watches go through the cache layer; on servers, directly to the state store. Produces ConfigSnapshot structs consumed by the xDS server.
  • Key types: Manager, ConfigSnapshot
  • Dependencies: cache, leafcert, state store (via glue), config entries

xDS Server#

  • Package: agent/xds/
  • Responsibility: Implements Envoy’s Aggregated Discovery Service (ADS) gRPC protocol. Receives ConfigSnapshot from proxycfg and converts them to Envoy-native Clusters, Listeners, Routes, and Endpoints. Uses the Incremental ADS (delta xDS) variant for efficiency.
  • Key types: Server, delta.go (core of delta xDS streaming)
  • Dependencies: proxycfg, envoyextensions, proto-public

Health Check Runners#

  • Package: agent/checks/
  • Responsibility: Runs each health check type concurrently: HTTP, TCP, UDP, gRPC, Docker exec, Script/TTL, Alias, OS service. Each check type is a struct with a goroutine loop. Results are written back to local.State.
  • Key types: CheckHTTP, CheckTCP, CheckGRPC, CheckMonitor, CheckTTL, etc.
  • Dependencies: local.State (to write results), TLS, config

ACL Engine#

  • Package: acl/, agent/consul/ (ACLResolver)
  • Responsibility: Policy parsing, token → policy resolution, and authorization decision making. The acl/ package is pure policy evaluation. ACLResolver in agent/consul/ handles token lookup (local cache + fallback to server) and policy resolution. Supports legacy and current ACL systems.
  • Key types: Authorizer interface, Policy, ACLResolver
  • Dependencies: state store (token/policy lookup), cache (client-side token caching)

V2 Resource System#

  • Package: internal/resource/, internal/controller/, internal/storage/
  • Responsibility: Generic resource CRUD over a gRPC ResourceService. Teams register resource types in a TypeRegistry with validation hooks. Controllers subscribe to resource changes and run reconciliation loops. Storage backends are swappable (Raft-backed in-memory for production, pure in-memory for tests).
  • Key types: ResourceService (gRPC), TypeRegistry, Controller interface, storage.Backend interface
  • Dependencies: internal/storage (Raft backend), agent/consul (Raft handle for log application), proto-public/pbresource

Data flow#

Service Registration (Client Agent)#

HTTP PUT /v1/agent/services/register
  → agent HTTP handler (agent/agent_endpoint.go)
  → agent.addServiceLocked()
  → local.State.AddService()         [local state updated immediately]
  → ae.StateSyncer.SyncFull.Trigger  [notify anti-entropy]
  → StateSyncer.syncServices()
  → delegate.RPC("Catalog.Register") [net/rpc to server]
  → consul.Server RPC handler
  → Raft.Apply(msgpack-encoded log)
  → FSM.Apply() → state.Store.EnsureService()
  → go-memdb write

Service Discovery (DNS)#

DNS query "web.service.consul" (port 8600)
  → agent/dns.DNSServer.handleQuery()
  → agent.RPC("Health.ServiceNodes") or "Catalog.ServiceNodes"
  → (client mode) delegate.RPC() → net/rpc → consul.Server
  → (server mode) directly to state.Store.ServiceNodes()
  → go-memdb query → results
  → DNS response (A/SRV records)

Envoy Proxy Configuration#

Envoy sidecar connects via xDS gRPC (ADS) to agent port 8502
  → agent/xds.Server.DeltaAggregatedResources()
  → proxycfg.Manager.Watch(proxyID) → ConfigSnapshot stream
    ↖ proxycfg watches catalog + config entries + intentions + leaf certs
      → (client agent) agent/cache → blocking RPC to server
      → (server) direct state.Store reads + event subscription
  → xds/delta.go generates Clusters/Listeners/Routes/Endpoints diff
  → streamed back to Envoy over gRPC

KV Write (Server Mode)#

HTTP PUT /v1/kv/mykey
  → agent HTTP handler
  → delegate.RPC("KV.Apply")
  → consul.Server.KV.Apply() (net/rpc handler)
  → Raft.Apply(KVSSet command)
  → FSM.Apply() → state.Store.KVSSet()
  → go-memdb write + event published to stream.EventPublisher
  → clients watching via blocking queries unblocked

Initialization / Bootstrap#

Sequence:

  1. main() → command.RegisteredCommands() builds the command map (all ~35 CLI commands registered in command/registry.go)
  2. mitchellh/cli.CLI.Run() dispatches to command/agent.cmd.Run()
  3. agent.NewBaseDeps(configLoader, logOut, nil) — the explicit DI phase:
    • Loads and validates config (agent/config.Load())
    • Sets up structured logging (hclog), metrics (Prometheus/statsd)
    • Creates TLS configurator
    • Creates client-side cache (agent/cache)
    • Creates connection pools: pool.ConnPool (net/rpc), grpcInt.ClientConnPool (gRPC)
    • Creates gRPC resolver + balancer (consul-specific server resolver)
    • Creates router.Router for datacenter-aware routing
    • Creates leafcert.Manager (depends on cache + NetRPC)
    • Creates autoconf.AutoConfig (JWT-based bootstrap)
    • Creates stream.EventPublisher (10s TTL window)
    • Creates consul.TypeRegistry (v2 resource types)
  4. agent.New(bd) — creates Agent, registers cache types, creates gRPC service clients
  5. agent.Start(ctx):
    • Runs AutoConfig.InitialConfiguration() (may rewrite config via server-signed TLS)
    • Creates local.State + ae.StateSyncer
    • Branches on ServerMode:
      • Server: creates external.NewServer (gRPC), then consul.NewServer() which sets up Raft, Serf LAN/WAN, net/rpc server, registers all RPC endpoints, starts leader loop
      • Client: creates consul.NewClient() which sets up Serf LAN only, no Raft
    • Sets a.delegate to the server or client
    • Starts HTTP API servers, DNS server, xDS server, proxycfg manager
    • Starts anti-entropy syncer
    • Starts leaf cert manager, grpc external server

No DI framework is used. Everything is manually wired. BaseDeps and the consul.Deps struct it embeds serve as explicit parameter objects passed down through constructors.

Configuration#

Configuration is loaded via agent/config.Load() which:

  • Merges multiple HCL/JSON config files (from -config-file and -config-dir flags)
  • Merges CLI flags as an override layer
  • Merges environment variables (e.g., CONSUL_HTTP_TOKEN)
  • Produces a RuntimeConfig struct (the canonical in-memory config representation)

There is no Viper. Consul predates Viper’s dominance and uses mitchellh/mapstructure for HCL decoding. The agent/config/ package contains its own multi-source merge logic.

Auto-config: In agent/auto-config/, Consul supports auto_config blocks that allow client agents to bootstrap their TLS certificates and ACL tokens by contacting a server with a signed JWT, eliminating the need to pre-distribute config. This overwrites parts of RuntimeConfig at startup.

Hot reload: SIGHUP triggers agent.ReloadConfig() → delegate.ReloadConfig(), which updates TLS certs, log levels, and some check configs without restart. Raft-persisted config entries are separate (updated via API).

Key design decisions#

1. Protocol multiplexing on port 8300#

The single “server” port multiplexes four protocols: net/rpc, gRPC (internal), Raft, and TLS ALPN for WAN federation. The first byte of an incoming connection determines routing in handleConn (agent/consul/rpc.go). This reduces firewall rules but complicates debugging — hence the tools/internal-grpc-proxy debugging tool.

Rationale: Minimizing port count for operational simplicity in enterprise environments with strict firewall policies.

2. delegate interface isolating Server vs. Client#

The Agent struct never directly references consul.Server or consul.Client — it only knows the delegate interface. This allows Agent to implement all check management, HTTP serving, and proxycfg logic once, regardless of role.

Rationale: Code deduplication between roles while supporting fundamentally different underlying implementations.

3. Anti-entropy architecture#

Rather than push-based health monitoring (central server polling), Consul agents run checks locally and use edge-triggered updates via ae.StateSyncer. The gossip layer provides failure detection at the node level independently of health check results. This combination scales to large clusters without O(n²) polling.

Rationale: Decentralized health checking scales linearly; centralized polling does not.

4. Cache with blocking queries (long-poll)#

agent/cache implements a client-side cache backed by blocking queries (index-based long-polling against the server). Each cache type registers a Fetch function that returns a result with an Index value; the cache re-fetches when the index changes. This gives clients near-real-time updates without persistent connections.

Rationale: Enables consistent reads with minimal server load; clients absorb the connection cost.

5. Emerging v2 controller architecture (Kubernetes-inspired)#

internal/resource + internal/controller introduce a resource-oriented API where teams register types and write controllers (reconciliation loops). This shifts feature development from “modify every layer” (HTTP handler → RPC handler → MemDB table → Raft command → CLI) to “define a resource type + controller.” The v2 ResourceService gRPC API coexists with v1 on the same running server, sharing the same Raft log via a type-prefix scheme in the FSM.

Rationale: The v1 approach required full-stack knowledge for every feature; v2 enables autonomous team ownership. The coexistence strategy avoids a big-bang migration.

Backward Consul — Dependencies Consul — Patterns Forward