etcd vs CockroachDB: Two Go Database Systems#

Summary#

etcd and CockroachDB are the two most architecturally significant Go database projects in the corpus — one a focused distributed key-value store that powers every Kubernetes cluster, the other a full-scale distributed SQL database competing with Oracle and Spanner. Both place Raft consensus at their core, but they represent opposite ends of the design spectrum: etcd pursues radical simplicity with ~1,100 Go files; CockroachDB accepts radical complexity with ~9,000+. Comparing them reveals how the same foundational algorithm (Raft), the same language (Go), and the same constraint (strong consistency) produce architectures that are structurally similar yet philosophically divergent.

Comparison dimensions#

Raft integration strategy#

Narrative: etcd made a principled architectural bet: extract Raft into a standalone module that does no I/O, then drive it from a single raftNode goroutine that owns the entire WAL-persist → Apply pipeline. CockroachDB started from the same etcd/raft codebase but forked it because the needs of a multi-range database (thousands of Raft groups per process, pipelined proposals, pre-vote, joint consensus, storage interface abstraction) diverged enough to make upstream tracking impractical. The fork now implements LogStorage/Storage interfaces that Replica fills in, whereas etcd’s module returns pure Ready structs with no interface boundary on storage. Both designs are correct; the choice is about how much coupling you accept between the consensus algorithm and your specific storage subsystem.

Storage engine#

Narrative: The storage divergence reflects fundamentally different workloads. etcd targets small datasets (the Kubernetes etcd recommendation is 8GB max) where the entire B-tree index fits in memory, enabling O(log n) key lookups at in-memory speed. bbolt’s memory-mapping means reads never go through the kernel for cached pages. CockroachDB targets petabyte-scale datasets where nothing fits in memory; Pebble’s LSM design amortizes write cost through compaction and handles range scans efficiently with bloom filters. Both implement MVCC, but etcd’s MVCC is a Go struct built on top of bbolt, while CockroachDB’s MVCC is encoded directly into Pebble’s key space — a lower-level but more flexible approach that enables range tombstones and point-in-time reads across arbitrary time windows (for AS OF SYSTEM TIME queries).

The WAL strategies reflect the same tradeoff. etcd maintains an independent WAL that it flushes before committing the bbolt batch — two durable writes per Raft entry, but with a clear recovery path if the process crashes between them. CockroachDB relies on Pebble’s built-in WAL, which is tightly integrated with LSM compaction. etcd’s explicit WAL-before-bbolt ordering is visible code (in raftNode.run()) and documentable; CockroachDB’s ordering guarantee is implicit in Pebble’s design.

API model#

Narrative: This is the sharpest divergence. etcd’s API surface is deliberately minimal: eight gRPC services (KV, Watch, Lease, Cluster, Auth, Maintenance, Election, Lock). Every operation is a typed protobuf RPC. CockroachDB’s API surface is PostgreSQL — the most widely used database wire protocol — meaning it gets decades of tooling for free. The grpc-gateway REST shim in etcd is a convenience for HTTP clients but not the primary interface. CockroachDB’s inter-node gRPC (for BatchRequest RPCs between Node instances) is an implementation detail, not an API.

The consequence for Go programmers: etcd’s client/v3 is a Go-idiomatic library with functional options and context-threading. CockroachDB’s “client” is any SQL driver. etcd treats its Go client as a first-class artifact (separate module, semantic versioning, backward compatibility testing); CockroachDB’s Go client is pgx or database/sql — third-party software.

Consistency and isolation model#

Narrative: etcd provides linearizability — the strongest single-object consistency model — for individual keys. Its Txn primitive (If/Then/Else with Compare predicates) enables conditional multi-key atomicity but is not a general-purpose transaction: there is no retry loop, no isolation level choice, and no cross-shard atomicity beyond what the client implements using the STM library. CockroachDB provides full serializable isolation across arbitrary SQL statements, including cross-shard writes, using HLC timestamps and a sophisticated retry protocol built into TxnCoordSender.

etcd’s design choice is elegant: by keeping consistency simple (one Raft log, one revision counter), it can make strong guarantees without a timestamp oracle or clock synchronization protocol. CockroachDB pays for its broader consistency model with the HLC, clock skew enforcement (nodes crash on excessive skew), and the txnSpanRefresher that re-validates all read spans before commit.

Concurrency and goroutine management#

Narrative: Both projects take goroutine lifecycle seriously, but they solve it differently at different scales. etcd’s stopc/donec pattern is decentralized: each component owns its shutdown protocol by reading from a stopc channel and signaling completion via donec. It is idiomatic Go and easy to audit in any component. CockroachDB’s Stopper is a centralized registry: every goroutine registers with the stopper and receives a cancellable context. The stopper can enumerate all active tasks (useful for debugging via /debug/stopper), enforce shutdown ordering, and throttle new goroutine creation. At etcd’s scale (hundreds of goroutines), either approach works. At CockroachDB’s scale (one Replica goroutine per Raft range, potentially thousands), the centralized approach prevents goroutine leaks that would be hard to track down.

The ctxgroup wrapper deserves attention. CockroachDB discovered through production incidents that errgroup.WithContext has a subtle bug surface: the returned context is cancel-on-first-error, which interacts badly with deferred cleanup in large codebases. ctxgroup.WithContext does not return a new context, forcing callers to receive ctx explicitly in each goroutine. This is a small API change that prevents a class of bugs — the kind of lesson that only emerges at scale.

Error handling#

Narrative: etcd built a bespoke but contained error taxonomy: ~40 ErrGRPC* typed status errors on the server side, mirrored as EtcdError values on the client side. The translation function rpctypes.Error(err) makes server errors programmatically matchable via errors.Is. This is thorough and self-contained within the etcd ecosystem.

CockroachDB’s error strategy is far more ambitious because it serves SQL clients who expect PostgreSQL error semantics. Every error that surfaces to a SQL connection must carry a SQLSTATE code (e.g., 23505 for unique violation). The cockroachdb/errors library extends Go’s error model with structured annotations (hint, detail, issue link), stack capture, and protobuf encoding — the last being essential for transmitting typed errors across gRPC boundaries and reconstructing them on the receiving node. The 4,949 errors.AssertionFailedf usages represent a philosophical choice: internal invariant violations are errors (not panics), logged with a stack trace, and surfaced as SQL internal error responses rather than crashing the node. This trades crash-on-corruption for operational continuity, appropriate for a multi-tenant database serving paying customers.

Configuration patterns#

Narrative: The cluster settings pattern is CockroachDB’s most transferable invention. Instead of deploying a new binary to change a tuning parameter, an operator runs SET CLUSTER SETTING kv.rangefeed.enabled = true in SQL and the change propagates to all nodes within seconds via gossip. Each setting is a typed package-level variable; adding a new setting requires one registration call and zero changes to call sites. etcd’s feature gates approximate this for binary behavior (Alpha/Beta/GA feature lifecycle), but they require a restart and are not operator-accessible via SQL. For long-lived server processes, the cluster settings pattern vastly reduces operational friction.

Dependency injection and extensibility#

Narrative: The CCL hook pattern is CockroachDB’s architectural answer to the OSS/enterprise split. The OSS binary defines function-variable hooks initialized to nil; the commercial binary activates them with a single blank import that triggers CCL init() chains. No #ifdef, no build tags in core code, no conditional compilation. The tradeoff is implicit initialization ordering: CCL init() runs before main(), so activation is guaranteed but non-auditable from main(). The TestingHook / HookGlobal utilities make these injection points testable without framework overhead.

etcd’s extensibility model is different: it is designed to be embedded (via embed.StartEtcd(cfg)) but not extended. The decorator chain pattern in the apply path (auth wraps quota wraps backend) is elegant for internal concerns but is not an extension point for external users. The contrib/ directory has examples (raftexample, auth provider implementations) but these are copy-and-modify patterns, not plugin registration points.

Testing strategy#

Narrative: etcd’s robustness testing suite is unusual in open-source Go: it models the entire cluster as a distributed system and checks that all observed histories are linearizable using the Porcupine checker. This reflects etcd’s correctness-over-features philosophy — the project has dedicated CI for chaos scenarios. CockroachDB’s testing infrastructure is more pragmatic: 55 TestingKnobs fields for fine-grained fault injection, datadriven golden tests for SQL output verification, and the syncutil.Mutex swap for deadlock detection in CI. Both strategies are appropriate for their complexity levels; CockroachDB’s TestingKnobs pattern scales better to a large team contributing isolated features.

Common patterns#

Both projects share:

1. Manual dependency injection — Neither uses wire, dig, or fx. etcd’s bootstrap() and CockroachDB’s NewServer() are explicit wiring functions. Both treat the bootstrap sequence as a correctness concern, not boilerplate.

2. Raft Ready loop — The same fundamental pattern: a goroutine reads Ready structs from the Raft module, persists entries to durable storage, sends messages to peers, and signals commit. The loop structure is nearly identical at the conceptual level.

3. Decorator/interceptor chain for cross-cutting concerns — etcd’s applierV3 decorator chain and CockroachDB’s txnInterceptor chain are structural equivalents: both use a common interface, compose thin single-purpose wrappers, and allow swapping implementations at runtime (etcd swaps in applyV3Capped on alarm; CockroachDB can add interceptors to the chain without touching others).

4. Context threading — 1,490 usages in etcd; 26,543 in CockroachDB. Both use context.Context as the primary cancellation and deadline mechanism throughout, with no exceptions.

5. No Viper for server configuration — Both bind directly to flag libraries (pflag/cobra) for static configuration.

6. MVCC for time-travel and watch semantics — Both implement multi-version concurrency control, though at different layers and for different reasons. etcd’s MVCC enables Watch; CockroachDB’s enables AS OF SYSTEM TIME and serializable isolation.

Divergent choices#

Scope: minimal vs maximal#

The most fundamental divergence. etcd chose to solve one problem — reliable distributed KV storage — and solve it completely. CockroachDB chose to solve the full relational database problem on top of distributed storage. This shapes everything else: etcd has 13 modules and ~1,100 Go files; CockroachDB has ~9,000+. etcd has 8 gRPC services; CockroachDB has an entire SQL optimizer (Cascades, code-generated from optgen DSL) and a vectorized execution engine.

Error philosophy: operational continuity vs. crash-on-corruption#

etcd’s verify package uses Assert() (always panics) for cheap mandatory invariants and Verify() (env-gated) for expensive optional checks. The implicit philosophy: data corruption is unrecoverable; crashing is preferable to serving stale data. CockroachDB’s errors.AssertionFailedf converts invariant violations to errors, logs them with stacks, and surfaces them as SQL internal errors. The implicit philosophy: in a multi-tenant service, one tenant’s corruption should not crash all tenants.

Raft scope: one log vs. many groups#

etcd uses a single Raft log for the entire cluster state. This is correct for a coordination store (total ordering of all mutations) but limits throughput and scale. CockroachDB uses one Raft group per key range (default 512MB), with potentially thousands of Raft groups per node. This enables horizontal scaling but requires the raftScheduler sharded priority worker pool to multiplex thousands of Raft groups onto a bounded thread pool — a complexity etcd never faces.

Time model: revision-based vs. HLC#

etcd uses a global revision counter (monotonically incrementing integer) for MVCC versioning. This is simple, cheap, and correct for a single-cluster store. CockroachDB uses a Hybrid Logical Clock (HLC) for MVCC timestamps, enabling cross-cluster causality, AS OF SYSTEM TIME queries, and serializable isolation across arbitrary distributed writes. The HLC requires clock synchronization within 500ms (enforced at RPC boundaries) and causes node crashes on excessive skew — operational complexity etcd avoids entirely.

Recommendations for practitioners#

Choose etcd’s patterns when:

• Building distributed coordination infrastructure (leader election, configuration distribution, distributed locks)
• Strong consistency for small datasets is the requirement
• You need a reusable, embeddable component (embed.StartEtcd)
• Simplicity and auditability matter more than feature richness
• Patterns to steal: pkg/wait.Wait (sharded channel map for async request-response), pkg/wait.WaitTime (logical deadline wait), decorator chain for apply path, bidirectional error translation between gRPC status and typed client errors

Choose CockroachDB’s patterns when:

• Building a server process at scale where operational flexibility is critical
• Patterns to steal: Stopper for goroutine lifecycle (superior to manual stopc/donec at scale), ctxgroup wrapper to enforce errgroup discipline, cluster settings registry for zero-restart runtime configuration, TestingKnobs for structured test injection, syncutil.Mutex build-tag variants for deadlock detection in CI, init()-hook injection for OSS/enterprise feature split

For both: The Raft Ready loop pattern (goroutine owns state machine, communicates results via channels, drives I/O externally) is the canonical correct structure for integrating a consensus library. Study both implementations before building your own.

Book angle#

The comparison frames a core tension in distributed systems engineering: how much problem do you solve?

etcd proves that solving a narrow problem completely — with exceptional correctness guarantees, a minimal API, and a reusable Raft module — creates outsized ecosystem value. The entire Kubernetes ecosystem runs on etcd; its 8 gRPC services have become infrastructure primitives.

CockroachDB proves that the Raft building block can scale all the way to a full-featured relational database if you are willing to pay the complexity cost. The Sender interface threading the entire KV stack, the HLC time model, the CCL injection pattern, and the cluster settings registry are each architectural innovations worth studying independently.

The deepest lesson is about pattern evolution. Both projects use a decorator/interceptor chain for cross-cutting concerns. etcd’s chain (authApplierV3 → quotaApplierV3 → applierV3Backend) is a small, auditable composition of ~30-method interfaces. CockroachDB’s txnInterceptor chain is 7 interceptors in a single interceptorAlloc struct, with stack-allocation optimization to avoid heap fragmentation. Same pattern, different scales, different implementation details. The pattern is language-agnostic; the implementation is Go-specific. That distinction — pattern vs. implementation — is the book’s central thesis made concrete.

The comparison also illustrates the compounding cost of scope. etcd’s codebase is readable by a single engineer in a week. CockroachDB’s is not. Both are necessary; neither is wrong. The choice of scope is the most consequential architectural decision a distributed system project makes, and it is almost never revisited.