etcd — Architecture#

Architectural style#

Layered Distributed System with Consensus at the Core

etcd is a classic layered architecture where every write flows through a Raft consensus log before being applied to state, guaranteeing linearizability. Layers from top to bottom:

  1. API layer — gRPC services (KV, Watch, Lease, Cluster, Auth, Maintenance) with a REST/JSON grpc-gateway shim
  2. Server coordination layer — EtcdServer, the central orchestrator: accepts requests, proposes to Raft, waits for commitment, applies results
  3. Consensus layer — go.etcd.io/raft/v3 (external module), a pure state machine library that orders writes across the cluster
  4. Apply layer — apply.UberApplier + applierV3 decorator chain: translates committed Raft log entries into state machine mutations
  5. Storage layer — MVCC KV store (B-tree index + bbolt backend) + Write-Ahead Log
  6. Transport layer — rafthttp.Transporter for peer-to-peer Raft messages; cmux multiplexer for client gRPC on the same port

Evidence: server/etcdserver/server.go (EtcdServer struct with r raftNode, kv mvcc.WatchableKV, lessor lease.Lessor, uberApply apply.UberApplier), server/embed/etcd.go (bootstrap sequence), server/etcdserver/apply/uber_applier.go (applier chain).

Component diagram (textual)#

 ┌─────────────────────────────────────────────────────────────────────┐
 │                       CLIENT LAYER                                  │
 │  etcdctl / embed / client/v3 (gRPC + grpc-gateway REST)             │
 └───────────────────────────────┬─────────────────────────────────────┘
                                 │ gRPC (KV/Watch/Lease/Auth/Cluster)
 ┌───────────────────────────────▼─────────────────────────────────────┐
 │                 v3rpc LAYER (server/etcdserver/api/v3rpc)            │
 │  kvServer  watchServer  leaseServer  clusterServer  authServer       │
 │  Interceptors: logging → prometheus metrics → auth/quota             │
 └───────────────────────────────┬─────────────────────────────────────┘
                                 │ calls RaftKV / Lessor / etc.
 ┌───────────────────────────────▼─────────────────────────────────────┐
 │                   EtcdServer (server/etcdserver)                     │
 │  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐  ┌──────────┐  │
 │  │  raftNode   │  │ UberApplier  │  │   Lessor    │  │AuthStore │  │
 │  │ (wraps      │  │ (applierV3   │  │ (lease TTL  │  │ (RBAC,   │  │
 │  │  raft.Node) │  │  chain)      │  │  management)│  │  JWT)    │  │
 │  └──────┬──────┘  └──────┬───────┘  └──────┬──────┘  └──────────┘  │
 │         │                │                  │                        │
 └─────────┼────────────────┼──────────────────┼────────────────────────┘
           │                │                  │
 ┌─────────▼──────┐  ┌──────▼──────────────────▼────────────────────┐
 │ go.etcd.io/    │  │        STORAGE LAYER                         │
 │ raft/v3        │  │  ┌──────────────┐  ┌──────────────────────┐  │
 │ (consensus     │  │  │  mvcc.store  │  │  WAL                  │  │
 │  state machine)│  │  │  (B-tree     │  │  (write-ahead log,   │  │
 └─────────┬──────┘  │  │  index +     │  │   encode/decode/     │  │
           │         │  │  backend.KV) │  │   repair)             │  │
           │         │  └──────┬───────┘  └──────────────────────┘  │
           │         │         │                                      │
           │         │  ┌──────▼───────────────────────┐             │
           │         │  │  backend.Backend (bbolt)      │             │
           │         │  │  (batching, hooks, snapshots) │             │
           │         │  └──────────────────────────────┘             │
           │         └─────────────────────────────────────────────-─┘
           │
 ┌─────────▼──────────────────────────────────────────┐
 │  rafthttp.Transporter (peer-to-peer Raft HTTP)      │
 │  cmux multiplexer: gRPC + Raft HTTP on same port    │
 └─────────────────────────────────────────────────────┘

Core components#

embed.Etcd#

  • Package: server/embed
  • Responsibility: Public embedding API and top-level lifecycle coordinator. StartEtcd(cfg) is the single entry point for all users (standalone binary + embedded use cases). Creates network listeners, translates embed.Config → config.ServerConfig, creates EtcdServer, starts peer/client/metrics servers. The Etcd struct holds references to all listeners and the EtcdServer.
  • Key types: Etcd (struct with Server *etcdserver.EtcdServer, Peers, Clients, sctxs), Config (all user-facing options)
  • Dependencies: etcdserver, rafthttp, backend, verify, cmux

EtcdServer#

  • Package: server/etcdserver
  • Responsibility: The central orchestrator. Accepts client requests (via RaftKV, Lessor, Authenticator interfaces), serializes them as InternalRaftRequest protobuf messages, proposes them to the Raft node, and waits for commitment via pkg/wait.Wait (ID-keyed WaitGroup). Also drives the apply loop (consuming raftNode.applyc channel) and the compaction loop.
  • Key types: EtcdServer (massive struct ~50 fields), Server interface, ServerV3 interface, RaftKV interface
  • Dependencies: raftNode, mvcc.WatchableKV, lease.Lessor, auth.AuthStore, apply.UberApplier, v3alarm.AlarmStore, backend.Backend, membership.RaftCluster

raftNode#

  • Package: server/etcdserver (unexported type in raft.go)
  • Responsibility: Wraps go.etcd.io/raft/v3 Node interface. Drives the Raft tick loop (heartbeat timer), processes raft.Ready structs: persists hard state and entries to WAL, sends messages via rafthttp.Transporter, sends toApply structs to applyc channel for the server’s apply goroutine.
  • Key types: raftNode, raftNodeConfig, toApply
  • Dependencies: raft.Node (from external module), serverstorage.Storage, rafthttp.Transporter

apply.UberApplier + applierV3 chain#

  • Package: server/etcdserver/apply
  • Responsibility: Translates committed InternalRaftRequest entries into mutations of the state machine (MVCC KV, lease store, auth store, cluster membership). Implements a decorator chain pattern:
    • applierV3Backend — base: calls mvcc.KV, lease.Lessor, auth.AuthStore directly
    • quotaApplierV3 — wraps base: enforces storage quota (raises NOSPACE alarm if exceeded)
    • authApplierV3 — wraps quota: checks RBAC permissions before any mutation
    • applyV3Capped — swapped in when NOSPACE alarm is active: rejects all writes
    • applyV3Corrupt — swapped in when CORRUPT alarm is active: rejects all requests
    • UberApplier — outer shell: handles alarm state transitions, swaps the inner chain
  • Key types: UberApplier (interface), applierV3 (internal interface, ~30 methods), Result
  • Dependencies: mvcc.KV, lease.Lessor, auth.AuthStore, v3alarm.AlarmStore, etcdserver/txn

mvcc.store (WatchableKV)#

  • Package: server/storage/mvcc
  • Responsibility: Multi-version concurrency control key-value store. Maintains a B-tree in-memory index (kvindex: maps key → slice of revision{main, sub}) and delegates actual byte storage to backend.Backend (bbolt). Supports range queries at arbitrary historical revisions (key for Watch), compaction (deletes superseded revisions), and watch streams (notifies watchers of mutations).
  • Key types: KV (interface), WatchableKV (interface), TxnRead / TxnWrite (transaction interfaces), store (implementation), WatchStream, ReadView, WriteView
  • Dependencies: backend.Backend, lease.Lessor (for lease attachment), pkg/schedule

backend.Backend (bbolt wrapper)#

  • Package: server/storage/backend
  • Responsibility: Wraps go.etcd.io/bbolt with write batching (coalesces writes for 100ms or 10,000 ops) and concurrent read transactions (ConcurrentReadTx copies the write buffer so reads don’t block writes). Provides snapshot capability for Raft log truncation.
  • Key types: Backend (interface), BatchTx (buffered write tx), ReadTx, ConcurrentReadTx
  • Dependencies: go.etcd.io/bbolt

storage/wal#

  • Package: server/storage/wal
  • Responsibility: Append-only write-ahead log. Each WAL record is a CRC-checked protobuf entry (walpb.Record). Supports segment rotation, repair (truncates at last valid CRC), and log replay during bootstrap. WAL persistence happens before bbolt commit — ensuring durability even if bbolt batch hasn’t flushed.
  • Key types: WAL, walpb.Record, walpb.Snapshot
  • Dependencies: client/pkg/fileutil, pbutil

lease.Lessor#

  • Package: server/lease
  • Responsibility: Manages TTL leases. Leases are created/revoked via Raft (go through consensus), but renewal (LeaseRenew) is handled locally on the leader (then checkpointed periodically). Expired leases are revoked asynchronously with rate limiting. Keys attach to leases; revocation deletes all attached keys.
  • Key types: Lessor (interface), lessor (impl), Lease, LeaseID
  • Dependencies: backend.Backend, schema

auth.AuthStore#

  • Package: server/auth
  • Responsibility: RBAC: users, roles, key-range permissions. Supports two token types — simple (random token with in-memory TTL) and jwt (stateless JWT with golang-jwt/jwt). Auth check happens in the authApplierV3 decorator.
  • Key types: AuthStore (interface), authStore (impl), TokenProvider (interface with simpleTokenProvider / jwtTokenProvider)
  • Dependencies: backend.Backend, api/authpb, golang-jwt/jwt, bcrypt

rafthttp.Transporter#

  • Package: server/etcdserver/api/rafthttp
  • Responsibility: Peer-to-peer transport for Raft messages using HTTP/2 (long-polling pipeline + snapshot streams). Implements the Transporter interface; uses Raft interface (on EtcdServer) to deliver incoming messages. Includes probing (health checks) of remote peers.
  • Key types: Transporter (interface), Transport (impl), Raft (interface back to EtcdServer), peer
  • Dependencies: raft/v3, client/pkg/transport (TLS), xiang90/probing

Data flow#

Write path (Put request)#

1. Client (etcdctl/client/v3) → gRPC Put RPC
2. v3rpc.kvServer.Put()
   - Interceptors: log → prometheus → auth/quota check
3. EtcdServer.Put() [v3_server.go]
   - Serializes to InternalRaftRequest{Put: ...}
   - Calls processInternalRaftRequestOnce()
   - Generates unique request ID via idutil.Generator
   - Registers wait on w.Register(id)
   - Calls r.Propose(ctx, data)   ← submits to raft.Node
4. raft.Node (go.etcd.io/raft/v3)
   - Leader: broadcasts AppendEntries to quorum
   - Followers: persist to WAL
   - Leader: once quorum acks, marks entry committed
5. raftNode.run() loop
   - Receives raft.Ready from r.Node.Ready() channel
   - Persists hard state + entries to WAL via storage.Save()
   - Sends Ready.Messages to peers via transport.Send()
   - Sends toApply{entries} to applyc channel
6. EtcdServer.applyAll() goroutine
   - Receives toApply from applyc
   - Calls apply.Apply(entry, uberApplier, w, shouldApplyV3)
7. apply.UberApplier.Apply()
   - Dispatches to applierV3 chain (auth → quota → backend)
8. applierV3Backend.Put()
   - Calls txn.Put() → mvcc.KV.Write().Put(key, value, leaseID)
9. mvcc.store.Put()
   - Updates B-tree index (new revision)
   - Writes to backend.BatchTx (buffered bbolt write)
   - Broadcasts to watching WatchStreams
10. w.Trigger(id, result) → unblocks EtcdServer.Put() goroutine
11. Response returned up the call stack → gRPC response to client

Read path (linearizable Range)#

1. Client → gRPC Range RPC
2. v3rpc.kvServer.Range() → EtcdServer.Range()
3. EtcdServer issues ReadIndex request to raft.Node
   - Leader sends heartbeat to confirm quorum, gets commit index
   - readStateC receives ReadState{Index, RequestCtx}
4. Wait until appliedIndex >= ReadIndex.Index
   (server.applyWait.Wait(readIndex))
5. Execute range on mvcc.store (reads at current applied revision)
6. Return results — guaranteed linearizable (no stale reads)

Watch path#

1. Client opens Watch gRPC stream
2. v3rpc.watchServer creates serverWatchStream
3. On each Put/Delete/Txn committed through the apply path,
   mvcc.store notifies registered WatchStreams
4. WatchStream sends Events (KeyValue + op type) upstream
5. v3rpc.watchServer relays events to client gRPC stream

Initialization / Bootstrap#

The bootstrap sequence is orchestrated across three files:

os.Args
  └─ etcdmain.Main()
       └─ startEtcdOrProxyV2()           [etcdmain/etcd.go]
            ├─ parse flags → embed.Config
            ├─ startEtcd(&cfg.ec)
            │    └─ embed.StartEtcd(cfg)  [embed/etcd.go]
            │         ├─ inCfg.Validate()
            │         ├─ configurePeerListeners()  → net.Listener per peer URL
            │         ├─ configureClientListeners() → net.Listener per client URL (cmux)
            │         ├─ embed.Config → config.ServerConfig (field-by-field copy)
            │         ├─ etcdserver.NewServer(srvcfg)
            │         │    └─ bootstrap(cfg)       [etcdserver/bootstrap.go]
            │         │         ├─ bootstrapSnapshot()
            │         │         ├─ bootstrapBackend()   → opens/creates bbolt db
            │         │         ├─ bootstrapWALFromSnapshot() (if existing member)
            │         │         ├─ bootstrapCluster()   → builds RaftCluster
            │         │         │    (new: DNS/token discovery; existing: read from WAL)
            │         │         └─ bootstrapStorage()   → creates mvcc.store, Lessor, AuthStore
            │         │              └─ returns *bootstrappedServer
            │         │    └─ NewServer builds EtcdServer from bootstrappedServer
            │         │    └─ creates UberApplier, CorruptionChecker
            │         ├─ e.Server.CorruptionChecker().InitialCheck() (if re-starting member)
            │         ├─ e.Server.Start()    → starts run() goroutine, raft tick loop
            │         ├─ e.servePeers()      → starts rafthttp handlers
            │         ├─ e.serveClients()    → starts gRPC server + grpc-gateway
            │         └─ e.serveMetrics()    → starts Prometheus /metrics endpoint
            ├─ wait for e.Server.ReadyNotify()
            └─ notifySystemd()

Dependency injection pattern: Manual constructor wiring. No DI framework (no wire/dig/fx). bootstrap() creates each subsystem in dependency order and returns a bootstrappedServer struct. NewServer() assembles EtcdServer from that struct. The ApplierOptions struct acts as a parameter object for wiring the applier chain.

Configuration#

  • Source: Command-line flags (via pflag) parsed in etcdmain, mapped to embed.Config. Environment variables are also supported (every flag has a corresponding ETCD_* env var).
  • Structure: Two config structs: embed.Config (user-facing, with URL strings, human-readable durations) and config.ServerConfig (server-internal, with parsed values). The conversion in embed.StartEtcd() is an explicit field-by-field copy (~60 fields).
  • Feature gates: pkg/featuregate provides a Kubernetes-style feature gate system. Server features are defined in server/features/. Features can be GA, Beta, or Alpha (unsafe). Enabled/disabled via --feature-gates=FeatureName=true.
  • No Viper: Configuration uses pflag directly. A YAML config file is not supported for the server (only etcdctl has --config-file via a YAML client config).

Key design decisions#

1. Raft as the universal serialization point#

Every mutation — KV writes, lease grants, auth changes, cluster membership changes — is serialized through the Raft log. This single architectural decision provides linearizability without explicit locking between subsystems. Even lease renewals on the leader eventually checkpoint through Raft. Evidence: v3_server.go processInternalRaftRequest is called by all write operations.

2. Decorator chain for the apply path#

The applierV3 interface (~30 methods) is wrapped by thin decorators (authApplierV3, quotaApplierV3, applyV3Capped, applyV3Corrupt), each adding a single cross-cutting concern. When an alarm fires, UberApplier.restoreAlarms() replaces the inner applyV3 reference with the appropriate wrapper — no conditional branches scattered across apply logic. Evidence: apply/uber_applier.go:NewUberApplier() and restoreAlarms().

3. MVCC enables watch semantics#

The B-tree index stores a complete revision history per key. When a client opens a watch at revision R, the store can send all events after R without keeping additional state on the server side. Compaction is an explicit, rate-limited operation that deletes old revisions. This design trades disk space for elegant watch semantics — a fundamental differentiator from simpler KV stores. Evidence: storage/mvcc/kv.go WatchableKV, storage/mvcc/index.go.

4. WAL before bbolt: durability by ordering#

The WAL is flushed (via fsync) before the bbolt batch is committed. This ordering guarantees that even if the process crashes after WAL write but before bbolt commit, the node can replay from WAL on restart. bbolt’s ACID guarantees handle the inverse case. Evidence: storage/wal/wal.go, raftNode.run() in etcdserver/raft.go (storage.Save precedes advancing the commit).

5. cmux for port unification#

etcd serves gRPC (for clients), Raft HTTP (for peers), grpc-gateway REST, and /metrics / /health HTTP all on configurable ports. The soheilhy/cmux library multiplexes gRPC and HTTP/1.x on the same TCP port by inspecting the first bytes of each connection (HTTP/2 preface vs. HTTP/1.x). This simplifies firewall rules and Kubernetes Service configuration. Evidence: server/embed/serve.go, server/embed/etcd.go (configureClientListeners).

6. Separate Raft module (go.etcd.io/raft/v3)#

The Raft consensus algorithm is cleanly separated into an independent Go module. The module is a pure state machine — it produces raft.Ready structs describing what to persist and send, but never performs I/O itself. etcd’s raftNode drives it by feeding Ready back into the WAL and transport. This separation allows other projects to use the same battle-tested Raft implementation, and contrib/raftexample demonstrates the standalone API.

Backward etcd — Dependencies etcd — Patterns Forward