etcd — Patterns#

Concurrency patterns#

Sharded ID-keyed Channel Map (pkg/wait.Wait)#

• Usage: Central to every write operation — used to bridge the proposing goroutine and the applying goroutine across the Raft consensus loop.
• Example: pkg/wait/wait.go — Register(id) returns a buffered channel; Trigger(id, result) sends the apply result and closes the channel. Used in server/etcdserver/v3_server.go:973 (ch := s.w.Register(id)) and triggered at server/etcdserver/apply/uber_applier.go.
• Implementation detail: Uses 64 shards (buckets by id % 64), each with its own sync.RWMutex and map[uint64]chan any, to avoid a single global lock under high request throughput.
• Assessment: Elegant solution to a hard problem — a goroutine proposes to Raft (async), Raft commits asynchronously across the cluster, the apply loop picks up the committed entry and must notify the original caller. The sharded map avoids contention that a single mutex would create at high QPS. This is a foundational primitive worth emulating.

Logical Deadline Wait (pkg/wait.WaitTime)#

• Usage: Used for linearizable reads — EtcdServer must wait until appliedIndex >= readIndex before returning a read result.
• Example: pkg/wait/wait_time.go — Wait(deadline uint64) <-chan struct{} returns a channel that is closed when Trigger(deadline) is called with a value >= the registered deadline. Uses a pre-closed closec channel as a fast path when the condition is already met.
• Assessment: Clever use of a pre-allocated closed channel for the “already done” case eliminates a branch in the hot path. The logical deadline (commit index, not wall clock) avoids time-of-day dependencies.

Event Loop with Ticker — Raft Drive Loop (raftNode.start())#

• Usage: The Raft heartbeat/election tick loop. Drives the external Raft state machine.
• Example: server/etcdserver/raft.go:181-335. A single goroutine selects over:
  • r.ticker.C → calls r.Tick() to advance Raft timer
  • r.Ready() → receives committed entries, WAL-persists them, forwards to apply channel
  • r.stopped → graceful shutdown
• Assessment: Canonical Go event loop pattern — one goroutine owns the state machine; other goroutines communicate exclusively through channels. The ticker drives the Raft heartbeat/election clock, keeping it fully decoupled from wall-clock jitter.

FIFO Scheduler with sync.Cond (pkg/schedule.FIFOScheduler)#

• Usage: Serializes jobs that must run in order (e.g., compaction tasks, lease expiry).
• Example: pkg/schedule/schedule.go — NewFIFOScheduler starts a single goroutine. Jobs are appended to pendings []Job under a mutex; a resume chan struct{} (capacity 1) unblocks the runner when new work arrives. WaitFinish(n) blocks via sync.Cond.Wait() until n jobs have completed.
• Assessment: sync.Cond is underused in Go codebases but ideal here — the caller needs “wait until some threshold of completions” which channels can’t express cleanly. The resume channel avoids spinning when the queue empties.

Channel-based Graceful Shutdown (stopc/donec pairs)#

• Usage: Pervasive — at least 20+ components use the stopc chan struct{} / donec chan struct{} pattern.
• Example: server/auth/simple_token.go:48-95 — token expiry loop; server/proxy/tcpproxy/userspace.go:65-228 — TCP proxy; raftNode.stopped / raftNode.done. The idiom: Stop() sends on stopc, the goroutine reads <-stopc, cleans up, then closes donec. Caller waits with <-donec.
• Assessment: Idiomatic Go shutdown. Consistently applied. Notable: some components use a signal channel (send struct{}{}), others use a context.CancelFunc. The heterogeneity is minor — the pattern is clear everywhere it appears.

Watch Fan-out via Broadcast Coalescing (grpc proxy)#

• Usage: The gRPC proxy coalesces many client watch streams on the same key into a single upstream watch.
• Example: server/proxy/grpcproxy/watch_broadcast.go — watchBroadcast maintains a map[*watcher]struct{} of subscriber client-side watchers. One goroutine streams from the upstream wch := wp.cw.Watch(...) channel; bcast(wr) iterates receivers under sync.RWMutex and sends to each.
• Assessment: Classic fan-out pattern. The coalescing is important for Kubernetes use cases where thousands of pods watch the same key prefix. The sync.RWMutex allows concurrent reads (finding receivers) while only write-locking for add/remove.

Context Cancellation Discipline#

• Usage: 1,490 uses of context.Context across the codebase — passed as first arg to every RPC, every storage read/write, every long-running operation.
• Example: server/etcdserver/v3_server.go:925 — processInternalRaftRequestOnce(ctx context.Context, ...). Context timeout causes <-ctx.Done() to fire before <-ch (the wait channel), returning ctx.Err().
• Assessment: Disciplined. Context is the primary cancellation mechanism throughout. No raw goroutine leaks observable from pattern analysis.

Rate Limiting (dial retrial in rafthttp)#

• Usage: server/etcdserver/api/rafthttp/stream.go:367 — rl *rate.Limiter throttles reconnection attempts to unavailable peers.
• Assessment: Uses golang.org/x/time/rate (token bucket). Scope is narrow — only for peer reconnection. Not a general backpressure mechanism for client requests.

Error handling#

• Style: Mixed — gRPC status errors dominate the server/client boundary; fmt.Errorf %w wrapping dominates internal code; sentinel errors (var Err* = errors.New(...)) define leaf error conditions.

• gRPC error taxonomy (api/v3rpc/rpctypes): etcd defines a two-layer error system:

  1. Server-side sentinels (ErrGRPC*): var ErrGRPCEmptyKey = status.Error(codes.InvalidArgument, "...") — ~40 typed gRPC status errors covering every API error condition. Mapped to gRPC codes.Code semantically (e.g., ErrGRPCCompacted → codes.OutOfRange, ErrGRPCNoSpace → codes.ResourceExhausted).
  2. Client-side mirrors (Err*): A parallel set (e.g., ErrCompacted, ErrNoSpace) for client code. The function rpctypes.Error(err) converts a raw gRPC status error to its typed EtcdError equivalent, enabling errors.Is checks on the client.
  • Example: api/v3rpc/rpctypes/error.go:241 — type EtcdError struct { code codes.Code; desc string }. Client retry logic in client/v3/retry_interceptor.go:158 does errors.Is(rpctypes.Error(err), rpctypes.ErrUserEmpty).
  • Assessment: Extremely thorough. Every gRPC error is a first-class typed value with its HTTP status equivalent mapped. The bidirectional translation (grpc status ↔ EtcdError) enables both server-side enumeration and client-side programmatic handling.

• Error types defined:

  • rpctypes.EtcdError — wraps gRPC code + description; implements error
  • concurrency.stmError — panic carrier for STM (see STM section)
  • fileutil.ErrLocked — sentinel for file lock conflicts
  • concurrency.ErrLocked, ErrSessionExpired, ErrLockReleased — distributed mutex errors

• Wrapping approach: fmt.Errorf("failed to open wal: %w", err) throughout the server. errors.Is / errors.As used for unwrapping. No github.com/pkg/errors dependency observed in server code.

• STM panic-based error propagation: client/v3/concurrency/stm.go:60 — type stmError struct{ err error }. Within an STM transaction closure, calling Get() on an error panics with stmError{err}. The outer NewSTM loop recovers this panic and returns the error. Unusual but intentional: it allows transaction code to be written without explicit error returns, matching database-style callback APIs.

Configuration pattern#

• Approach: Two distinct layers:

  1. Server configuration: Direct pflag flag binding → embed.Config (user-facing string/duration fields) → config.ServerConfig (parsed internal fields). No functional options; explicit field-by-field translation.
  2. Client library: Functional options — type Option func(*Client) in client/v3/client.go:111, with constructors like WithZapLogger(lg). Clean Go library idiom.
  3. Transport/TLS: type ListenerOption func(*ListenerOptions) in client/pkg/transport/listener_opts.go — WithTimeout, WithTLSInfo, WithSocketOpts, etc.
  4. Concurrency utilities: type stmOption func(*stmOptions) in concurrency/stm.go — WithIsolation(lvl), WithAbortContext(ctx), WithPrefetch(keys...).

• Feature gates (pkg/featuregate): Copied verbatim from k8s.io/component-base to avoid circular dependency. Full Kubernetes feature gate lifecycle: Alpha (off by default), Beta (on by default), GA (locked on). Runtime-configurable via --feature-gates=Name=true. Features defined in server/features/. Master toggles AllAlpha, AllBeta allow bulk enable/disable for testing.

• Example (client):

  cli, err := clientv3.New(clientv3.Config{
      Endpoints: []string{"localhost:2379"},
      DialTimeout: 5 * time.Second,
  })
  // OR with options:
  cli, err := clientv3.NewCtxClient(ctx, clientv3.WithZapLogger(logger))

Dependency injection#

• Approach: Manual constructor wiring. No DI framework.
• Evidence: server/etcdserver/bootstrap.go — bootstrap(cfg) builds each subsystem in dependency order:
  1. Backend (bbolt) → bootstrapBackend()
  2. WAL + snapshot → bootstrapWALFromSnapshot()
  3. Cluster membership → bootstrapCluster()
  4. Storage (mvcc, lease, auth) → bootstrapStorage() Returns a bootstrappedServer struct carrying all components.
• ApplierOptions parameter struct: server/etcdserver/apply/interface.go:79 — groups the ~8 dependencies needed to build the applier chain into one struct, avoiding a constructor with 8 positional arguments.
• Assessment: Manual wiring is appropriate for etcd’s complexity — the bootstrap sequence is itself a critical correctness concern (WAL before backend, snapshot before WAL, etc.). A DI framework would obscure this ordering.

Other notable patterns#

Decorator Chain (Apply Path)#

• Pattern: applierV3 interface (~30 methods) is wrapped by thin single-responsibility decorators.
• Example: server/etcdserver/apply/ — authApplierV3 wraps quotaApplierV3 wraps applierV3Backend. UberApplier.restoreAlarms() swaps in applyV3Capped or applyV3Corrupt when alarms fire.
• Assessment: Textbook decorator / chain-of-responsibility. Adding a new cross-cutting concern (e.g., rate limiting) requires only a new thin struct that embeds applierV3. The UberApplier outer shell handles alarm-triggered chain mutation without conditional branches in the base applier. Worth highlighting in the book as an elegant alternative to scattered if alarm { ... } guards.

Software Transactional Memory (STM)#

• Pattern: Optimistic concurrency control at the client library layer — read, compute, compare-and-swap in a retry loop.
• Example: client/v3/concurrency/stm.go:89 — NewSTM(c, func(stm STM) error { ... }) executes the closure, builds a conditional transaction from the read set, submits it via client.Txn(...).If(...).Then(...).Commit(), and retries if the CAS fails (reads were stale).
• Four isolation levels: SerializableSnapshot, Serializable, RepeatableReads, ReadCommitted.
• Assessment: Sophisticated and rare in Go client libraries. Leverages etcd’s native Txn API (If/Then/Else with Compare predicates) to implement multi-key atomic operations with configurable isolation. The panic-based error propagation within the closure (via stmError) is unusual but keeps closure code clean.

Distributed Mutex via Lease + Watch#

• Pattern: Distributed locking built on etcd primitives: Session (lease-backed keepalive), Mutex (prefix-keyed create-revision ordering).
• Example: client/v3/concurrency/mutex.go — Lock() does:
  1. Put a key pfx/<leaseID> if not exists (atomic via client.Txn().If(CreateRevision==0).Then(Put))
  2. Get all keys with prefix pfx/, ordered by CreateRevision
  3. If my key has the lowest revision, I hold the lock; otherwise waitDeletes(ctx, pfx, myRev-1) watches for keys with earlier revisions to be deleted.
• Assessment: Elegant use of etcd’s linearizable compare-and-swap + watch semantics. The revision ordering guarantees FIFO lock acquisition — no starvation. The lockerMutex adapter implements sync.Locker for drop-in use. The Election pattern in election.go follows the same structure.

gRPC Interceptor Chain (Server + Client)#

• Server-side: server/etcdserver/api/v3rpc/grpc.go builds a chain: newLogUnaryInterceptor → newUnaryInterceptor (auth + quota + leader check). grpc.ChainUnaryInterceptor(...) composes them.
• Client-side: client/v3/retry_interceptor.go — custom retry interceptor with write-at-most-once semantics. Distinguishes retryable errors (network, leader change) from non-retryable (bad request) before retrying writes.
• Assessment: The client retry interceptor is particularly noteworthy — it was adapted from go-grpc-middleware/retry but modified for etcd’s stricter idempotency requirements (writes must not be retried unless the server confirms no-op).

Environment-controlled Assertions (verify package)#

• Pattern: Production-safe assertions gated by ETCD_VERIFY environment variable.
• Example: client/pkg/verify/verify.go — Assert(condition, msg, ...) always panics; Verify(msg, func() (bool, map[string]any)) only runs when ETCD_VERIFY=assert|all. Used for invariant checks that would be too expensive in production (e.g., B-tree consistency checks after mutations).
• Assessment: Clever two-tier assertion system. Assert for cheap mandatory invariants; Verify for expensive optional checks enabled in CI/debugging. Avoids the all-or-nothing choice between always-on panics and stripped-out checks.

Compact ID Generation (pkg/idutil.Generator)#

• Pattern: Compact 8-byte IDs encoding memberID + timestamp + counter, using atomic increment.
• Example: pkg/idutil/id.go — Layout: [2B: memberID][5B: timestamp ms][1B: counter]. Next() does atomic.AddUint64(&g.suffix, 1) — counter overflow intentionally bleeds into the timestamp field, extending the window to 2^48 events per millisecond.
• Assessment: Solves the “unique across restarts” and “unique across members” problems with a single 64-bit integer. The intentional counter→timestamp overflow is well-documented and clever. Avoids UUID overhead for a hot path (every Raft proposal gets an ID).

Interface Embedding for sync.Locker Compatibility#

• Pattern: Wrapping a domain-specific type in an adapter struct that implements a stdlib interface.
• Example: client/v3/concurrency/mutex.go — type lockerMutex struct{ *Mutex } with Lock() / Unlock() wrappers that call Mutex.Lock(client.Ctx()) and panic on error. NewLocker(s, pfx) returns sync.Locker.
• Assessment: Idiomatic Go adapter pattern. Allows etcd distributed mutexes to be passed anywhere a sync.Locker is accepted, including sync.RWMutex-style patterns. The panic-on-error tradeoff is documented and matches sync.Mutex semantics.

Generics for Type-safe Collections (cache.ringBuffer[T])#

• Usage: Limited but targeted.
• Example: cache/ringbuffer.go — type ringBuffer[T any] with RevisionOf[T any] func(T) int64 and IterFunc[T any] type parameters. Used for caching watch events by revision.
• Other usages: tests/robustness/random/random.go — PickRandom[T any]; server/storage/wal/wal.go:241 — createNewWALFile[T *os.File | *fileutil.LockedFile] (union type constraint to share WAL creation logic for locked vs unlocked files).
• Assessment: Conservative generics adoption. Used only where type erasure would require repeated implementations or unsafe casts. The LockedFile | *os.File union constraint in WAL is particularly practical — eliminates a near-duplicate function.

Table-driven Configuration via Builder Struct#

• Pattern: embed.Config is a large flat struct (~60 fields) initialized to defaults by NewConfig(), then fields are set via pflag binding or direct assignment.
• Assessment: Not functional options — etcd’s server config predates that idiom. The explicit field-by-field copy from embed.Config to config.ServerConfig in embed.StartEtcd() is verbose but transparent; mismatches would be caught at compile time.

Pattern summary table#