CockroachDB — Patterns#

Sampling note#

CockroachDB is an XL-tier project (~9,000+ Go files). Pattern detection was performed with grep across the full repository, then key files were read for depth. Packages sampled in depth: pkg/util/stop, pkg/kv/kvclient/kvcoord (TxnCoordSender interceptors), pkg/kv/kvserver/scheduler.go, pkg/util/ctxgroup, pkg/base/testing_knobs.go, pkg/util/syncutil, pkg/sql/conn_executor.go (FSM), pkg/testutils/hook.go. Counts are from full-repo grep, excluding vendor/.

Concurrency patterns#

Stopper — Coordinated Goroutine Lifecycle#

• Usage: Universal across the entire codebase. 303 calls to stopper.RunAsyncTask / stopper.RunWorker. Every long-lived goroutine is managed through this mechanism.
• Example: pkg/util/stop/stopper.go — Stopper wraps a quotapool-based semaphore for async task throttling and a sync.WaitGroup tracking all active tasks.
• Pattern: Callers invoke stopper.RunAsyncTask(ctx, "task-name", func(ctx context.Context) {...}). The stopper provides a derived context that is cancelled when shutdown begins. On stopper.Stop(), it cancels all contexts and waits for all tasks to finish. stopper.WithCancelOnQuiesce(ctx) lets non-task code also honour drain. The ErrUnavailable sentinel is returned if a new task is submitted after quiesce begins.
• Assessment: Highly idiomatic and effective for a server process with complex shutdown ordering. Significantly more structured than bare sync.WaitGroup + context.WithCancel patterns. The task-name argument enables debugging via stopper.HandleDebug HTTP endpoint. The semaphore throttling (ErrThrottled) adds back-pressure without goroutine explosion.

raftScheduler — Sharded Priority Worker Pool#

• Usage: pkg/kv/kvserver/scheduler.go. One pool per Store; processes all Raft events for all ranges hosted on that store.
• Example:

  type raftScheduler struct {
      processor   raftProcessor
      shards      []*raftSchedulerShard // shard 0 = priority, rest = round-robin by RangeID
      priorityIDs syncutil.Set[roachpb.RangeID]
      done        sync.WaitGroup
  }
  type raftSchedulerShard struct {
      syncutil.Mutex
      cond       *sync.Cond
      queue      rangeIDQueue[queuedRangeID]
      state      map[roachpb.RangeID]raftScheduleState
      numWorkers int
  }

• Pattern: Ranges are assigned to shards by RangeID % (numShards - 1) (shard 0 is reserved for priority ranges, such as liveness range). Workers block on sync.Cond.Wait() within their shard. The deduplication state map (state map[RangeID]raftScheduleState) prevents redundant Raft ticks from queuing multiple times for the same range — a key optimization when thousands of ranges exist on one store.
• Assessment: A sophisticated, production-grade worker pool pattern. The sharding approach trades memory for lock contention reduction at high worker counts. Priority routing via a dedicated shard is uncommon in open-source Go and worth studying for latency-sensitive background work.

ctxgroup — errgroup with Explicit Context Discipline#

• Usage: pkg/util/ctxgroup/ctxgroup.go. Used throughout for fan-out concurrent operations.
• Example:

  g := ctxgroup.WithContext(ctx)
  g.GoCtx(func(ctx context.Context) error {
      // ctx is explicitly passed — no shadowed variable risk
      return api.Call(ctx, val)
  })
  return g.Wait()
  // GroupWorkers convenience:
  return ctxgroup.GroupWorkers(ctx, numWorkers, func(ctx context.Context, workerID int) error {...})

• Pattern: ctxgroup.WithContext does NOT return a new context (unlike stdlib errgroup.WithContext), forcing callers to receive ctx explicitly in GoCtx. The package documentation explains the specific bugs this prevents: context variable shadowing causing use-after-cancel in a large codebase. GroupWorkers further reduces boilerplate for homogeneous worker pools.
• Assessment: An excellent example of wrapping a stdlib/x package to enforce a discipline. The bug examples in the package doc are taken from real CockroachDB incidents. Projects with large teams benefit significantly from this approach.

Root/Leaf Transaction Split (Distributed Fan-out)#

• Usage: pkg/kv/sender.go, pkg/kv/kvclient/kvcoord. Used during DistSQL query execution to distribute a single ACID transaction across multiple nodes.
• Pattern:

  // pkg/kv/sender.go
  type TxnType int
  const (
      RootTxn  TxnType = iota // owns intent tracking, commit
      LeafTxn                 // accumulates intents, returns to Root at merge
  )
  type TxnSender interface {
      GetLeafTxnInputState(ctx context.Context, ...) (*roachpb.LeafTxnInputState, error)
      GetLeafTxnFinalState(ctx context.Context) (*roachpb.LeafTxnFinalState, error)
      UpdateRootWithLeafFinalState(ctx context.Context, tfs *roachpb.LeafTxnFinalState)
      ...
  }

• Mechanism: The gateway node creates a RootTxn. Each remote DistSQL worker receives a LeafTxnInputState snapshot and operates on a LeafTxn. On completion, the leaf returns LeafTxnFinalState (accumulated write intents, span refreshes) to the root, which merges it before commit. This enables parallel reads and writes across nodes within a single serializable transaction without sharing a *kv.Txn object (which is not goroutine-safe).
• Assessment: Unique to CockroachDB in open-source Go. The Root/Leaf split is the correct solution to distributed transaction fan-out and avoids the naive (and incorrect) alternative of shared mutable transaction state.

Channel Fan-out with select (2437 channels, 1716 selects)#

• Usage: Pervasive throughout the codebase for async event delivery, pipeline stages, and coordination.
• Example: pkg/kv/kvserver/store_raft.go — incoming Raft messages arrive on a channel; the raftScheduler uses sync.Cond-based notification internally to avoid the overhead of a select per range.
• Assessment: Standard Go concurrency; CockroachDB applies it at scale. The sync.Cond preference in the scheduler over channels reflects a performance trade-off for high-frequency events.

Context Propagation (26,543 usages)#

• Usage: Every function that does I/O, RPC, or can block accepts context.Context as its first parameter — a strict discipline enforced project-wide.
• Assessment: Exemplary context discipline. CockroachDB was an early adopter and the codebase demonstrates correct context threading even in deeply nested call chains. The ctxgroup pattern above reinforces this discipline.

Graceful Drain (Stopper + serverController)#

• Usage: pkg/server/drain.go, pkg/server/server_controller.go. Multi-tenant aware drain that calls gracefulDrain() on each active server (system + tenant) in sequence.
• Pattern: stopper.WithCancelOnQuiesce(ctx) creates contexts that cancel when the server enters quiesce mode. SQL connections are politely drained: active transactions are given a window to commit before connections are forcibly closed. sqlServer.gracefulDrainComplete is an atomic bool signalling completion.
• Assessment: Correct multi-phase drain (stop accepting new work → drain in-flight → hard stop) implemented consistently.

Error handling#

• Style: Custom error library (github.com/cockroachdb/errors) with rich structured annotations and PostgreSQL SQLSTATE codes. 3,380 imports of the library across the codebase.
• Error categories:
  1. Assertion failures (errors.AssertionFailedf) — 4,949 usages. Used for internal invariant violations (the equivalent of panic but recoverable and logged with full stack). These are programming errors, not expected runtime errors.
  2. Structured user-facing errors — errors.WithHint, errors.WithDetail, errors.WithIssueLink (547 usages combined). Errors surfaced to SQL clients carry a hint (what to do), a detail (what went wrong), and optionally a link to a GitHub issue. This maps to PostgreSQL’s HINT and DETAIL error fields.
  3. PostgreSQL SQLSTATE codes — pgerror.WithCandidateCode(err, pgcode.X) (3,188 usages). Every SQL error must carry a SQLSTATE code for PostgreSQL wire compatibility. The WithCandidateCode function attaches a “candidate” code that can be overridden further up the stack.
  4. Standard wrapping — errors.Wrap, errors.Wrapf, fmt.Errorf with %w for adding context during propagation.
• Error types defined: kvpb.NodeUnavailableError, roachpb.RangeNotFoundError, roachpb.WriteIntentError, roachpb.TransactionRetryError, roachpb.AmbiguousResultError, and many more — defined as protobuf messages so they can be transmitted over the wire and reconstructed on the client side. The errors.Is/errors.As chain works through these protobuf-transmitted errors via custom Mark helpers.
• Circuit breaker errors: pkg/kv/kvserver/replica_circuit_breaker.go wraps cockroachdb/circuitbreaker. When a replica’s liveness probe fails, the breaker trips and requests immediately return circuit.ErrBreakerOpen instead of hanging indefinitely.
• Examples:
  • pkg/cli/clierror/formatted_error.go: extracts pgcode, hint, and detail from any error and formats it for the CLI user — demonstrates the full structured error extraction path.
  • pkg/kv/kvpb/batch.go:408: errors.AssertionFailedf for invariant violation in batch processing.

Configuration pattern#

Approach: Two-tier — Static flags + Dynamic cluster settings#

• Static (startup): Cobra persistent flags bound to base.Config and server.Config structs, with pkg/util/envutil providing COCKROACH_* environment variable overrides for each flag default.
• Dynamic (cluster settings): pkg/settings registry with 1,056 RegisterXxxSetting calls (bool, int, float, string, duration, byte-size). Settings are declared as typed package-level variables and initialized at program start. Operators change them via SET CLUSTER SETTING SQL, which propagates to all nodes via gossip + KV writes. The cluster.Settings struct carries a settings.Values container; every component that needs a setting receives *cluster.Settings (not individual settings values), so new settings can be added without changing call sites.
• Example:

  // pkg/rpc/settings.go
  var enableRPCCircuitBreakers = settings.RegisterBoolSetting(
      settings.SystemOnly,
      "rpc.circuit_breaker.enabled",
      "enable circuit breakers for RPC connections",
      true,
  )
  // Usage: enableRPCCircuitBreakers.Get(&settings.SV)

• Assessment: The cluster settings pattern is one of CockroachDB’s most important and most reusable patterns. It separates “what the setting is” (declaration) from “how to change it” (SQL DDL) and “how to read it” (typed accessor), with zero-restart runtime propagation. It is far superior to env-var or flag reloading for long-lived processes.

TestingKnobs — Structured Test Injection#

• Approach: base.TestingKnobs (55+ fields) carries per-module ModuleTestingKnobs interfaces. Components receive a *base.TestingKnobs at construction and use nil-checks to activate test behaviour.
• Example:

  // pkg/base/testing_knobs.go
  type TestingKnobs struct {
      Store                       ModuleTestingKnobs
      KVClient                    ModuleTestingKnobs
      SQLExecutor                 ModuleTestingKnobs
      // ... 50+ more fields
  }
  type ModuleTestingKnobs interface { ModuleTestingKnobs() } // marker

• Pattern: Each package defines its own testing_knobs.go with a concrete TestingKnobs struct that has function-typed fields for injecting faults, interceptors, and timing hooks. Production code checks if cfg.TestingKnobs.Store != nil { knobs := cfg.TestingKnobs.Store.(*kvserver.StoreTestingKnobs) }. The //go:build !test tag is never used — these knobs are in production code but optimized away by the compiler when nil.
• Assessment: This pattern enables fine-grained fault injection without reflection or build tags. The central base.TestingKnobs struct with typed slots prevents knobs from proliferating as ad-hoc global variables. The 1,155 usages outside test files demonstrate how deeply testing concerns are woven into the production path.

Dependency injection#

• Approach: Manual wiring. No DI framework (no wire, dig, or fx).
• Evidence: pkg/server/server.go NewServer() is ~1,200 lines of explicit construction. Every dependency is wired by hand, in dependency order, with constructor functions returning both a value and an error.
• CCL injection via init() hooks: The most distinctive DI pattern. Core packages (e.g., pkg/jobs/metrics.go) declare function-variable hooks initialized to nil:

  // pkg/jobs/metrics.go
  var MakeChangefeedMetricsHook func(time.Duration, *cidr.Lookup) metric.Struct
  var MakeBackupMetricsHook     func(time.Duration) metric.Struct

  CCL packages override these in their init() functions:

  // pkg/ccl/changefeedccl/metrics.go (approximate)
  func init() {
      jobs.MakeChangefeedMetricsHook = makeChangefeedMetrics
  }

  The commercial binary’s main.go has a single blank import _ "github.com/cockroachdb/cockroach/pkg/ccl" which triggers the entire CCL init() chain. The OSS binary omits this import; hooks remain nil; features are absent.
• TestingHook utility:

  // pkg/testutils/hook.go
  func TestingHook(ptr, val interface{}) func() {
      global := reflect.ValueOf(ptr).Elem()
      orig := reflect.New(global.Type()).Elem()
      orig.Set(global)
      global.Set(reflect.ValueOf(val))
      return func() { global.Set(orig) }
  }
  // Go 1.18+ generic version:
  func HookGlobal[T any](ptr *T, val T) func() { ... }

  Used in tests to temporarily replace package-level function variables (e.g., defer testutils.TestingHook(&getCurrentTime, func() time.Time {...})()). This makes the CCL hook pattern testable.
• Assessment: The init()-hook pattern is architecturally elegant for an OSS/enterprise split but has implicit ordering: CCL init() runs before main(), so hooks are always set before use. The TestingHook utility makes this injection point testable and is generalized for any package-level variable replacement. The lack of a DI framework is deliberate — manual wiring is explicit, and in a large codebase, the explicitness outweighs the boilerplate.

Other notable patterns#

txnInterceptor Chain (Stack-allocated Middleware)#

• File: pkg/kv/kvclient/kvcoord/txn_coord_sender.go
• Pattern: TxnCoordSender maintains a chain of 7 txnInterceptor implementations for cross-cutting transaction concerns. All 7 are embedded in a single interceptorAlloc struct to avoid separate heap allocations:

  interceptorAlloc struct {
      arr [7]txnInterceptor      // the chain, ordered from outermost to innermost
      txnHeartbeater             // keeps transaction alive
      txnSeqNumAllocator         // assigns sequence numbers to requests
      txnWriteBuffer             // buffers writes until commit (reduces round-trips)
      txnPipeliner               // async write pipelining through Raft
      txnCommitter               // handles commit protocol details
      txnSpanRefresher           // refreshes read spans on serialization failure
      txnMetricRecorder          // records transaction metrics
      txnLockGatekeeper          // not in chain array; manages lock table interaction
  }

  Each interceptor implements lockedSender (SendLocked(context.Context, *BatchRequest) (*BatchResponse, *Error)) and setWrapped(lockedSender) to form a chain. A request traverses all interceptors before reaching DistSender.
• Assessment: An elegant combination of the middleware/chain-of-responsibility pattern with Go’s struct embedding to avoid heap fragmentation. The embedding within interceptorAlloc means a TxnCoordSender allocation brings all 7 interceptors into the same memory block. Adding new cross-cutting concerns (e.g., the write buffer was added later) requires only adding a new struct and inserting it into the chain, with no changes to the other interceptors.

optgen — DSL-driven Code Generation for Optimizer Rules#

• Files: pkg/sql/opt/norm/rules/*.opt, pkg/sql/opt/xform/rules/*.opt, pkg/sql/optgen/
• Pattern: CockroachDB’s query optimizer transformation rules are written in a custom DSL called optgen (optimizer generator). Rules look like:

  # pkg/sql/opt/norm/rules/agg.opt
  [EliminateAggDistinct, Normalize]
  (AggDistinct $input:(Min | Max | BoolAnd | BoolOr))
  =>
  $input

  The optgen compiler reads .opt files and generates Go code (*_gen.go files) implementing the memo.Memo data structures, the Cascades exploration engine, and the rule dispatch tables. This keeps rule logic concise (pattern matching syntax) while generating efficient Go code.
• Assessment: Rare in open-source Go databases. Most use hand-written rule tables or imperative optimizer code. The optgen approach allows non-Go programmers to contribute optimizer rules and keeps the rule count manageable (~500 rules). The tradeoff is a custom build step and a non-standard toolchain.

Explicit FSM for SQL Transaction State (connExecutor)#

• File: pkg/sql/conn_executor.go (5000+ lines), pkg/sql/txn_state.go
• Pattern: Each SQL connection’s transaction lifecycle is managed by an explicit finite state machine using the pkg/util/fsm package. States include stateNoTxn, stateOpen, stateAborted, stateRestartWait, stateCommitWait. Transitions are triggered by SQL events (eventTxnStart, eventTxnCommit, eventTxnRollback, eventNonRetriableErr, eventRetriableErr, etc.). Type switches on ex.machine.CurState() implement state-specific behaviour:

  case stateOpen:
      return ex.execStmtInOpenState(ctx, ast, res)
  case stateAborted:
      return ex.execStmtInAbortedState(ctx, ast, res)

• Assessment: Using an explicit, typed FSM for protocol state (rather than boolean flags) makes the transaction lifecycle verifiable and auditable. The fsm package ensures only valid transitions are taken; TransitionNotFoundError makes illegal state transitions visible at runtime. This pattern is appropriate for any complex protocol that must handle partial failures correctly.

syncutil — Instrumented Mutex with Build-tag Variants#

• Package: pkg/util/syncutil/
• Pattern: CockroachDB replaces sync.Mutex with syncutil.Mutex throughout the codebase. Three implementations are compiled via build tags:
  1. mutex_sync.go — thin wrapper around sync.Mutex (default, zero overhead)
  2. mutex_deadlock.go (//go:build deadlock) — wraps github.com/sasha-s/go-deadlock with 5-minute timeout for deadlock detection in CI
  3. mutex_tracing.go — adds TracedLock(ctx) that emits a trace event if lock acquisition is slow, and TimedLock() returning duration
• Assessment: A mature multi-implementation pattern that allows development/production tradeoffs without #ifdef. The AssertHeld() method (on some variants) enables documentation-as-enforcement of “must be called under X lock” invariants. The naming convention *MuLocked (1,384 usages in kvserver) for functions that assert the caller holds a lock is a disciplined commenting convention that AssertHeld() can eventually enforce.

baseQueue — Template for Raft Maintenance Background Work#

• Package: pkg/kv/kvserver, files: *_queue.go
• Pattern: CockroachDB has ~10 background queues (mvccGCQueue, mergeQueue, splitQueue, replicateQueue, raftLogQueue, etc.) that process replicas periodically. Each embeds baseQueue which provides: priority queue, rate limiting, metrics, and the dispatch loop. Each concrete queue implements:

  shouldQueue(ctx, now, replica, confReader) (shouldQueue bool, priority float64)
  process(ctx, replica, confReader) (processed bool, err error)

• Assessment: A clean template method pattern in Go via interface embedding. The shouldQueue / process split is elegant: the framework handles scheduling priority and rate limiting, the concrete queue handles only domain logic. Adding a new maintenance concern requires only implementing two methods.

Registry Pattern for Jobs and Cloud Backends#

• Examples:
  • pkg/jobs — job types register a Resumer factory by jobspb.Type enum
  • pkg/cloud/externalconn/impl_registry.go — external connection types register by FactoryType
  • pkg/sql/sem/builtins/ — SQL built-in functions registered in maps
• Pattern:

  // pkg/cloud/externalconn/impl_registry.go
  var factoryFactories = map[FactoryType]func(connectionpb.ConnectionProvider) connectionParserFactory{}
  func RegisterConnectionDetailsFromURIFactory(t FactoryType, f func(...) connectionParserFactory) {
      factoryFactories[t] = f
  }

• Assessment: Standard registry pattern for extensible dispatch. CockroachDB uses it in preference to large switch statements for runtime extensibility. CCL packages register additional job types and cloud backends into these registries during init().

Functional Options (selective usage)#

• Usage: Not universal — used for test utilities, CLI options, and some client packages. About 50 func With* patterns found.
• Example: pkg/backup/backuptestutils/testutils.go — WithParams, WithBank, WithInitFunc, WithTempDir etc. for composable test cluster setup. pkg/roachprod/promhelperclient/client.go — WithIAPTokenSource, WithCustomURL.
• Assessment: CockroachDB uses functional options where appropriate (optional parameters, test configuration) but does not apply them universally. The majority of core construction uses explicit structs (StoreConfig, DistSenderConfig) with zero values as defaults — a style preference that trades functional-options elegance for explicit visibility of all options.

datadriven Testing#

• Usage: 718 datadriven. usages in test files. pkg/testutils/datadriven (or the external github.com/cockroachdb/datadriven) implements file-based golden tests.
• Pattern: Test files contain commands and expected output. The test function reads the file, executes each command, and diffs actual vs. expected output. Updating expected output is done by re-running with -rewrite.
• Assessment: Highly effective for SQL query testing, optimizer rule testing, and any system with human-readable output. Reduces test verbosity for large output comparisons and makes diff-based review natural. Used throughout the optimizer and SQL packages.

Pattern summary#