CockroachDB — Architecture#

Architectural style#

Layered Monolith with Microkernel-style CCL Injection

CockroachDB is a layered monolith: every CockroachDB node runs the same single binary, with responsibilities organized into strict layers that depend only downward. The layers span from PostgreSQL wire protocol at the top to Pebble LSM storage at the bottom, with a transactional distributed KV store in between.

The “microkernel” aspect appears in how enterprise features (CCL) are wired in. The OSS core defines hook points — function variables, init()-registered callbacks, and interface slots — that the commercial pkg/ccl packages fill in via Go’s init() mechanism when imported with a blank import in main.go. This means the core binary compiles and runs with or without CCL; it just lacks commercial capabilities without it.

A more recent architectural concern is multi-tenancy: the serverController manages on-demand instantiation of SQLServerWrapper tenants that share the same KV storage cluster but run isolated SQL processes. This is the foundation for CockroachDB Serverless.

Evidence from code:

  • pkg/cmd/cockroach/main.go: 5 lines — imports pkg/ccl (blank) and calls cli.Main()
  • pkg/server/server.go: topLevelServer struct with ~40 field dependencies, all manually wired
  • pkg/kv/sender.go: Sender interface implemented by kv.Txn, TxnCoordSender, Node, Store, Replica — a single interface threading the entire KV call stack
  • pkg/server/server_controller.go: serverController manages dynamic tenant server fleet

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────────────────┐
│  CLIENT APPLICATION (PostgreSQL-compatible driver)                          │
└───────────────────────────────────┬─────────────────────────────────────────┘
                                    │ PostgreSQL wire protocol (TCP)
┌───────────────────────────────────▼─────────────────────────────────────────┐
│  pkg/sql/pgwire  —  pgwire.conn                                              │
│  Read/parse/execute loop: net.Conn → parser → stmtBuffer → connExecutor     │
└───────────────────────────────────┬─────────────────────────────────────────┘
                                    │ parsed tree.Statement
┌───────────────────────────────────▼─────────────────────────────────────────┐
│  pkg/sql  —  connExecutor (FSM) + sql.planner                                │
│  Transaction state machine → makeOptimizerPlan() → dispatchToExecutionEngine│
│    ┌──────────────┐    ┌───────────────────────┐    ┌──────────────────┐    │
│    │ pkg/sql/opt  │    │ pkg/sql/distsql        │    │ pkg/sql/colexec  │    │
│    │ Cascades CBO │    │ DistSQLPlanner         │    │ Vectorized engine│    │
│    │ memo, xform  │    │ PlanAndRun → FlowSpec  │    │ columnar ops     │    │
│    └──────────────┘    └───────────────────────┘    └──────────────────┘    │
└───────────────────────────────────┬─────────────────────────────────────────┘
                                    │ kv.Txn (implements kv.Sender)
┌───────────────────────────────────▼─────────────────────────────────────────┐
│  pkg/kv/kvclient/kvcoord  —  TxnCoordSenderFactory                          │
│  TxnCoordSender: interceptor chain for retry, intent tracking, heartbeating  │
│    → DistSender: routes BatchRequest to owning range via RangeCache          │
└─────────────┬───────────────────────────────────────────────┬───────────────┘
              │ gRPC/DRPC (to remote nodes)                   │ (loopback if local)
┌─────────────▼─────────────────────────────────────────────  │  ─────────────┐
│  pkg/server  —  Node (implements kvpb.InternalServer)        │               │
│  BatchInternal(): fan-out to local Stores                     │               │
└─────────────────────────────────────────────────────────────▼───────────────┘
                                    │
┌───────────────────────────────────▼─────────────────────────────────────────┐
│  pkg/kv/kvserver  —  Store + Replica (both implement kv.Sender)              │
│  Store: 40+ subsystems (allocator, queues, intent resolver, raft transport)  │
│  Replica: per-range FSM, leaseholder logic, batch evaluation                 │
│    ┌──────────────────────────────────────────────────────────┐              │
│    │  pkg/raft  —  RawNode                                    │              │
│    │  Custom Raft fork: ProposeConfChange, Step, Ready loop   │              │
│    └──────────────────────────────────────────────────────────┘              │
└───────────────────────────────────┬─────────────────────────────────────────┘
                                    │ MVCC reads/writes via Engine interface
┌───────────────────────────────────▼─────────────────────────────────────────┐
│  pkg/storage  —  Pebble engine                                               │
│  Engine interface: Reader, Writer, MVCCIterator hierarchy                    │
│  MVCC key encoding, range tombstones, point key versioning                   │
└─────────────────────────────────────────────────────────────────────────────┘

Cross-cutting infrastructure (used by all layers):

  • pkg/util/hlc — Hybrid Logical Clock for causality tracking
  • pkg/util/tracing — OpenTelemetry-compatible distributed tracing
  • pkg/util/stop — Stopper for coordinated goroutine lifecycle
  • pkg/settings/cluster — cluster.Settings for runtime-reconfigurable cluster settings
  • pkg/gossip — cluster membership and system config propagation
  • pkg/util/admission — multi-resource admission control (CPU, storage I/O)

Core components#

pgwire (PostgreSQL Wire Protocol Handler)#

  • Package: pkg/sql/pgwire
  • Responsibility: Accepts TCP connections from PostgreSQL-compatible clients. Implements the full PostgreSQL wire protocol: startup/auth handshake, message framing, query parsing, result serialization, cancellation. Each connection is a goroutine running conn.serveImpl().
  • Key types: pgwire.conn (wraps net.Conn), PreServeConnHandler (pre-auth routing for multi-tenant), Server (listener + conn pool management)
  • Dependencies: pkg/sql (for connExecutor), pkg/security (TLS/auth), pkg/util/stop (lifecycle)

connExecutor (SQL Session State Machine)#

  • Package: pkg/sql
  • Responsibility: The stateful per-connection SQL execution engine. Maintains transaction state as a finite state machine (Open, Aborted, RestartWait, NoTxn). Reads from the statement buffer pushed by pgwire.conn, executes statements, dispatches to the planner and execution engine.
  • Key types: connExecutor (5000+ line struct), connExecutor.execStmtInOpenState(), dispatchToExecutionEngine()
  • Dependencies: pkg/sql/opt (optimizer), pkg/sql/distsql (physical planner), pkg/kv (transactions), pkg/sql/pgwire (result writer via ClientComm)

Cascades Cost-Based Optimizer#

  • Package: pkg/sql/opt, pkg/sql/opt/optbuilder, pkg/sql/opt/xform, pkg/sql/opt/memo
  • Responsibility: Converts SQL AST into an optimized logical query plan using the Cascades framework. Builds a memo (space-efficient representation of all equivalent plan variants), applies normalization and exploration transformation rules (generated via optgen), then selects the lowest-cost plan.
  • Key types: memo.Memo, optbuilder.Builder, xform.Optimizer, opt.RelExpr, opt.ScalarExpr
  • Dependencies: pkg/sql/sem/tree (AST), pkg/sql/catalog (schema metadata), statistics (table row counts, histograms)

DistSQLPlanner (Distributed Physical Planner)#

  • Package: pkg/sql (distsql_physical_planner.go)
  • Responsibility: Converts the logical plan from the optimizer into a distributed physical plan (PhysicalPlan). Determines which nodes own which ranges and assigns plan fragments (FlowSpec) to them. Coordinates execution across nodes via gRPC SetupFlow calls. Also manages the local vectorized execution engine path.
  • Key types: DistSQLPlanner, PhysicalPlan, execinfrapb.FlowSpec, execinfrapb.ProcessorSpec
  • Dependencies: pkg/kv/kvclient/rangecache (for range → node mapping), pkg/sql/execinfra (Processor interface), pkg/sql/colexec (vectorized operators)

TxnCoordSender (Transaction Coordinator)#

  • Package: pkg/kv/kvclient/kvcoord
  • Responsibility: Implements kv.TxnSender. Sits between kv.Txn (SQL-visible) and DistSender. Manages per-transaction state: tracks write intents, issues heartbeats for long-running transactions, handles automatic retry on serialization conflicts, and aggregates leaf transaction state in DistSQL flows. Uses an interceptor chain pattern for layered concerns.
  • Key types: TxnCoordSender, TxnCoordSenderFactory, txnInterceptor (chain: txnHeartbeater, txnSeqNumAllocator, txnPipeliner, txnSpanRefresher, txnCommitter, txnMetricRecorder, txnLockGatekeeper)
  • Dependencies: DistSender (downstream), pkg/kv (Txn, Sender interfaces)

DistSender (Range Router)#

  • Package: pkg/kv/kvclient/kvcoord
  • Responsibility: Routes BatchRequest operations to the correct ranges by consulting the range descriptor cache. Splits batches that span multiple ranges, sends each part to the range’s leaseholder, and merges responses. Handles retries for range splits/merges and replica unavailability.
  • Key types: DistSender, DistSenderConfig, RangeCache (descriptor + lease cache)
  • Dependencies: pkg/rpc/nodedialer (gRPC/DRPC transport), pkg/kv/kvclient/rangecache

Node (KV Request Dispatcher)#

  • Package: pkg/server
  • Responsibility: Implements kvpb.InternalServer. Receives incoming BatchRequest RPCs from other nodes (via gRPC/DRPC) and from the local DistSender (via loopback). Fans out to the appropriate local Store based on the range’s store ID. Also serves as the gossip hub for node-level cluster state.
  • Key types: Node, Stores (collection of local stores), perReplicaServer (per-replica RPC handler)
  • Dependencies: pkg/kv/kvserver (Store), pkg/gossip, pkg/kv/kvserver/liveness

Store (KV Store Manager)#

  • Package: pkg/kv/kvserver
  • Responsibility: Manages all Raft replicas (ranges) stored on a single physical disk. Owns all background maintenance work: range splitting/merging decisions, replica placement via the allocator, GC, Raft log truncation, snapshot sending/receiving, and the timestamp cache. The Store is a god-struct with ~40 embedded queues and subsystems.
  • Key types: Store (40+ fields including allocator, splitQueue, replicateQueue, intentResolver, raftTransport), StoreConfig
  • Dependencies: pkg/raft (RawNode per-replica), pkg/storage (Engine), pkg/kv/kvserver/concurrency (lock table), pkg/kv/kvserver/batcheval (command evaluation)

Replica (Per-Range Raft Participant)#

  • Package: pkg/kv/kvserver
  • Responsibility: Represents one replica of a Raft consensus group (a key range). Manages the Raft state machine for its range via pkg/raft.RawNode. For the leaseholder replica, evaluates KV batches, applies the Raft log, and serves reads from MVCC storage. Circuit breaker protects against liveness failures.
  • Key types: Replica, ReplicaID, replicaMu (large mutex-guarded state including state.Desc, raft state, lease)
  • Dependencies: pkg/raft (RawNode), pkg/storage (Engine/MVCC), pkg/kv/kvserver/batcheval, pkg/kv/kvserver/concurrency

Storage Engine (Pebble MVCC Layer)#

  • Package: pkg/storage
  • Responsibility: Wraps the Pebble LSM tree with CockroachDB’s MVCC key encoding. Provides the Engine interface (composing Reader + Writer) for all KV I/O. Handles MVCC point keys (versioned by HLC timestamp) and range tombstones. All reads/writes below the Raft log go through this layer.
  • Key types: Engine interface, Pebble (concrete implementation), MVCCIterator, Batch (atomic write groups)
  • Dependencies: github.com/cockroachdb/pebble (external LSM library), pkg/util/hlc (timestamp encoding)

serverController (Multi-Tenant Fleet Manager)#

  • Package: pkg/server
  • Responsibility: Manages on-demand instantiation of secondary SQL tenant servers (SQLServerWrapper) within a single KV node process. Each tenant gets its own SQL server process equivalent, sharing the KV storage layer but isolated in SQL processing. Handles HTTP routing between tenants and graceful drain.
  • Key types: serverController, onDemandServer interface, SQLServerWrapper (tenant SQL server), tenantServerCreator
  • Dependencies: pkg/sql/pgwire (SQL protocol for each tenant), pkg/multitenant (tenant capabilities), pkg/kv (shared KV client)

Raft Consensus Module#

  • Package: pkg/raft
  • Responsibility: CockroachDB’s own Raft implementation (forked from etcd/raft with deep modifications). Provides the RawNode API: application proposes entries and config changes, Raft produces Ready structs with entries to persist and messages to send. The application loop is owned by Replica.
  • Key types: RawNode, Config, LogStorage / Storage interface (implemented by kvserver), raftpb.Entry, raftpb.Message
  • Dependencies: None outside the package (pure algorithm implementation); raftpb for wire types

Data flow#

A simple SELECT query (local range, single-node)#

1. Client → pgwire.conn.serveImpl():
   TCP packet → ReadMsg() → parser.Parse() → stmtBuffer.Push(ExecStmt)

2. connExecutor.execCmd():
   Dequeues ExecStmt → execStmtInOpenState() → opens kv.Txn (implicit txn)
   → dispatchToExecutionEngine()

3. sql.planner.makeOptimizerPlan():
   AST → optbuilder.Builder.Build() → memo.Memo
   → xform.Optimizer.Optimize() → explore transformation rules, cost each plan
   → select lowest-cost RelExpr as logical plan

4. DistSQLPlanner.PlanAndRun():
   Logical plan → PhysicalPlan (assign each operator to a node based on range locality)
   For a local single-node query: construct a local flow (no SetupFlow RPCs)
   → instantiate Processors (row-by-row) or colexec operators (vectorized)

5. Processor.Run():
   TableReader processor → kv.Txn.Scan() or Get()

6. kv.Txn → TxnCoordSender.Send():
   Wraps batch in interceptor chain (sequence numbers, intent tracking, span refresh)
   → DistSender.Send()

7. DistSender.Send():
   Look up range descriptor + leaseholder in RangeCache
   → send BatchRequest via nodedialer (loopback for local node)

8. Node.BatchInternal():
   Fan-out to Store.Send() for matching range

9. Store → Replica.Send():
   Check lease validity → execute read against MVCC storage (no Raft needed for reads)
   → storage.MVCCGet/Scan() on Pebble engine
   → return BatchResponse up the chain

10. Results flow back through Processor → DistSQL flow → connExecutor
    → pgwire serializes rows → sends PostgreSQL DataRow messages to client

A write that goes through Raft consensus#

Steps 1–7 same as above, then:

8. Replica.Send() for a write:
   Leaseholder replica evaluates the write batch (batcheval package)
   → Replica.propose() → raft.RawNode.Propose(encoded command)

9. Raft consensus:
   Leader appends entry to Raft log → sends AppendEntries to follower replicas
   → followers acknowledge → quorum reached

10. Raft Ready loop (raftScheduler goroutine):
    RawNode.Ready() returns committed entries
    → Replica.handleCommittedEntriesRaftMuLocked()
    → apply entries to MVCC storage via storage.Batch.Commit()
    → wake up waiting proposer goroutine

11. Proposer returns BatchResponse up the call chain back to client

Distributed query (multi-node DistSQL)#

DistSQLPlanner detects ranges live on multiple nodes:
  → Generate FlowSpec for each remote node (TableReader on remote, join/aggregation on gateway)
  → Send SetupFlow RPC to each remote node
  → Remote nodes instantiate their Processor chains and start streaming
  → Gateway aggregates results from remote RowSources via network streams
  → connExecutor returns merged results to client

Initialization / Bootstrap#

CockroachDB uses manual dependency injection (no DI framework). NewServer() in pkg/server/server.go is ~1200 lines of explicit construction:

cli.Main() → runStartJoin() → NewServer(cfg, stopper)
  1. clock = newClockFromConfig()             // HLC clock
  2. nodeRegistry/appRegistry/sysRegistry = metric.NewRegistry() × 3
  3. engines = cfg.CreateEngines()            // Pebble instances per --store flag
  4. g = gossip.New()                         // gossip node
  5. rpcContext = rpc.NewContext()            // gRPC + DRPC context
  6. grpcServer, drpcServer = newGRPCServer() // register gossip, init RPCs
  7. kvNodeDialer = nodedialer.New()          // range-aware dialer
  8. distSender = kvcoord.NewDistSender()     // range router
  9. tcsFactory = kvcoord.NewTxnCoordSenderFactory()
  10. db = kv.NewDB(tcsFactory, distSender)   // kv.DB wraps the sender stack
  11. node = NewNode(db, stores, ...)         // registers as InternalServer
  12. sqlServer = newSQLServer(cfg)           // SQL engine, pgwire, optimizer
  13. serverController = newServerController()// multi-tenant fleet manager
  ...

server.PreStart():
  - Start HTTP listener, RPC server
  - Initialize gossip, node liveness
  - Bootstrap or join cluster (via initServer)
  - Start Stores (opens Pebble, recovers Raft state)
  - Start background work (GC, allocator, rebalancing)

server.AcceptClients():
  - Start accepting pgwire SQL connections
  - Start accepting HTTP/admin API connections

Stopper pattern: Every long-lived goroutine is launched via stopper.RunAsyncTask() or stopper.RunWorker(). On stopper.Stop(), all tasks receive context cancellation and drain gracefully. This is the universal shutdown coordination mechanism.

CCL activation: The main.go blank import _ "github.com/cockroachdb/cockroach/pkg/ccl" causes pkg/ccl/ccl_init.go to run, which transitively imports all CCL packages. Each CCL package’s init() registers hooks into core package function variables (e.g., jobs.MakeChangefeedMetricsHook), replacing nil function vars with real implementations. The OSS binary (cockroach-short) omits this import, so hooks stay nil and CCL features are unavailable.

Configuration#

CockroachDB has a two-tier configuration system:

Static configuration (startup flags / env vars)#

  • Package: pkg/cli/cliflagcfg, pkg/base, pkg/server.Config
  • Mechanism: Cobra persistent flags (--join, --listen-addr, --store, --max-sql-memory, etc.) bound to base.Config and server.Config structs
  • Env var fallback: pkg/util/envutil provides EnvOrDefaultXxx() for overriding flag defaults via COCKROACH_* environment variables
  • Example: COCKROACH_BLOCK_PROFILE_RATE=100 overrides block profiling rate

Dynamic configuration (cluster settings)#

  • Package: pkg/settings, pkg/settings/cluster
  • Mechanism: Each setting is a typed global variable declared with settings.RegisterXxxSetting(). Values are stored in the settings.Values container (part of cluster.Settings). Operators change settings with SET CLUSTER SETTING SQL, which propagates via gossip + internal KV writes to all nodes.
  • Key type: cluster.Settings — a single shared instance per node, passed everywhere. Contains SV settings.Values, Version clusterversion.Handle, Manual atomic.Value.
  • Example: kv.rangefeed.enabled is a cluster setting controlling RangeFeed availability; changing it takes effect without restart.

Key design decisions#

1. kv.Sender as the universal KV interface#

The single-method Sender interface — Send(context.Context, *BatchRequest) (*BatchResponse, *Error) — is implemented by every component in the KV stack: kv.Txn, TxnCoordSender, DistSender, Node, Store, Replica. This creates a uniform call chain from SQL down to storage. The simplicity of the interface hides enormous complexity (the code comment in sender.go candidly notes it is “now considered regrettable because it’s too narrow and at times leaky”), but it enables a clean interceptor chain at the TxnCoordSender level.

2. Cascades cost-based optimizer with optgen code generation#

The query optimizer uses the Cascades framework, which explores a space of equivalent query plans using transformation rules. CockroachDB generates these rules from a DSL (optgen) into Go code. This is unusual: most Go databases use hand-written or yacc-based approaches. The generated memo and rule system allows the optimizer to be extended by adding rule files without modifying core data structures. The SQL parser itself is generated by goyacc from a PostgreSQL-derived grammar.

3. Multi-tenancy via process-in-process SQL servers#

The serverController runs multiple SQL server instances (SQLServerWrapper) inside a single KV node process. Each tenant has full SQL isolation (separate connExecutor pool, separate schema catalog, separate admission control) but shares the underlying KV storage. Tenant routing happens at the pgwire.PreServeConnHandler level — connections are dispatched to the correct tenant’s SQL server based on SNI or connection parameters before authentication. This is how CockroachDB Serverless achieves tenant isolation without a separate process per tenant.

4. Distributed SQL execution with leaf transaction state#

DistSQL flows fan out SQL execution across multiple nodes, each running Processor chains. Since a single SQL kv.Txn cannot safely be used concurrently from multiple goroutines, CockroachDB uses RootTxn / LeafTxn split: the gateway node owns the root transaction; remote nodes run leaf transactions that accumulate write intents and return them to the root for consolidation at commit. This allows distributed parallel reads and writes within a single ACID transaction.

5. Admission control as a first-class subsystem#

pkg/util/admission implements multi-resource admission control for CPU and storage I/O (LSM compaction backpressure). Work items have a WorkPriority (e.g., KVWork, SQLKVResponseWork, SQLSQLResponseWork). When a resource is saturated, lower-priority work is queued in admission queues rather than consuming goroutine scheduler slots. This prevents priority inversion and maintains throughput under load — a concern unique to databases, not typical Go services.

6. Hybrid Logical Clock (HLC) for distributed causality#

CockroachDB does not use physical wall-clock time for MVCC timestamps. Instead, pkg/util/hlc provides an HLC: a (walltime, logical) pair that advances monotonically across the cluster. Every node’s clock is kept within a maxOffset (default 500ms) of true time, enforced at RPC boundaries. This enables snapshot reads at a point in time (for AS OF SYSTEM TIME queries) and serializable isolation without a centralized timestamp oracle. Clock skew violations cause node crashes to preserve correctness — a dramatic but necessary safety guarantee.

7. init()-hook CCL injection for OSS/enterprise split#

The open-source binary defines all feature interfaces (empty function variables, nil hooks). The commercial binary activates them through Go’s init() execution chain triggered by a single blank import. This is architecturally elegant — the OSS codebase compiles and ships independently, and no #ifdef-style build tags pepper the core code — but it means that the CCL packages’ init() functions run before main(), making startup ordering implicit rather than explicit. The pattern scales to the ~15 CCL feature packages currently using it.

Backward CockroachDB — Dependencies CockroachDB — Patterns Forward