CockroachDB — Interfaces#

Sampling note#

CockroachDB contains 1,481 interface definitions across non-vendor, non-test, non-generated .go files. A full catalog is infeasible in a single analysis session. This document focuses on the ~15 interfaces that are most architecturally significant: those that define the system’s primary extension points, cross-layer contracts, and core execution models. Interfaces in asim/ (allocator simulator), generated proto helpers, and narrow internal helpers are excluded.

Interface catalog#

kv.Sender#

  • Package: pkg/kv
  • File: pkg/kv/sender.go:53
  • Methods:
    Send(context.Context, *kvpb.BatchRequest) (*kvpb.BatchResponse, *kvpb.Error)
  • Purpose: The universal routing interface for the entire KV call stack. Every component from SQL’s transaction handle down to a physical Raft replica implements this single-method interface.
  • Implementations: kv.Txn (user-facing transaction), kvcoord.TxnCoordSender (per-txn coordinator), kvcoord.DistSender (range router), server.Node (cross-node dispatch), kvserver.Store (per-store dispatch), kvserver.Replica (per-range evaluation)
  • Design quality: The interface is famously minimal — exactly one method — enabling a clean interceptor chain (at TxnCoordSender) and uniform composition across layers. The file’s own comment acknowledges it is “now considered regrettable because it’s too narrow and at times leaky.” In practice the BatchRequest carries so much semantic richness (txn metadata, routing keys, multi-verb batches) that it functions as a wide interface in disguise. Still, the ISP adherence enables testing: any layer can be replaced by a mock Sender with one method.

kv.TxnSender#

  • Package: pkg/kv
  • File: pkg/kv/sender.go:95
  • Methods: Extends Sender with ~20 additional methods:
    GetLeafTxnInputState(context.Context, interval.Tree) (*roachpb.LeafTxnInputState, error)
GetLeafTxnFinalState(context.Context) (*roachpb.LeafTxnFinalState, error)
UpdateRootWithLeafFinalState(context.Context, *roachpb.LeafTxnFinalState) error
SetIsoLevel(isolation.Level) error
IsoLevel() isolation.Level
SetUserPriority(roachpb.UserPriority) error
SetDebugName(name string)
BufferedWritesEnabled() bool
SetBufferedWritesEnabled(bool)
// ... ~12 more
  • Purpose: Per-transaction contract for managing the root/leaf transaction split in DistSQL flows, transaction metadata, isolation levels, and commit coordination.
  • Implementations: kvcoord.TxnCoordSender (sole production implementation), kvcoord.LeafTxnCoordSender
  • Design quality: Well-segregated relative to the base Sender. The leaf/root split methods encode a non-trivial distributed systems concept: DistSQL nodes run leaf transactions whose accumulated write intents must be reconciled with the root before commit. The interface makes this contract explicit and testable.

storage.Engine#

  • Package: pkg/storage
  • File: pkg/storage/engine.go:919
  • Methods: Composes Reader + Writer, plus engine-level operations:
    Reader + Writer (embedded)
Attrs() roachpb.Attributes
Capacity() (roachpb.StoreCapacity, error)
Compact(ctx context.Context) error
Flush() error
GetMetrics() Metrics
NewBatch() Batch
NewReader(durability DurabilityRequirement) Reader
NewReadOnly(durability DurabilityRequirement) ReadWriter
NewSnapshot() Reader
// ... ~20 more lifecycle/inspection methods
  • Purpose: The storage abstraction. Everything above the storage layer interacts with Pebble through this interface, enabling the theoretical swap of storage engines.
  • Implementations: storage.Pebble (production), storage.InMem (testing, in-memory Pebble), storage.intentInterleavingIterator wraps at iterator level
  • Design quality: Excellent interface composition — Reader, Writer, ReadWriter, Engine, and Batch form a clean capability lattice. Reader.NewMVCCIterator() returns an MVCCIterator (not the raw Pebble iterator), keeping MVCC concerns inside the storage layer. One concern: the interface is necessarily broad (engine management + read + write), but this is an inherent property of a storage engine abstraction.

storage.Reader and storage.Writer#

  • Package: pkg/storage
  • File: pkg/storage/engine.go:523 (Reader), pkg/storage/engine.go:616 (Writer)
  • Key methods (Reader):
    NewMVCCIterator(ctx, iterKind MVCCIterKind, opts IterOptions) (MVCCIterator, error)
NewEngineIterator(ctx, opts IterOptions) (EngineIterator, error)
MVCCIterate(ctx, start, end roachpb.Key, ..., f func(MVCCKeyValue, MVCCRangeKeyStack) error) error
ConsistentIterators() bool
PinEngineStateForIterators(readCategory) error
  • Key methods (Writer):
    ClearMVCC(key MVCCKey, opts ClearOptions) error
PutMVCC(key MVCCKey, value MVCCValue) error
PutRawMVCC(key MVCCKey, value []byte) error
ApplyBatchRepr(repr []byte, sync bool) error
// ... versioned clear/merge/range operations
  • Purpose: MVCC-aware read and write contracts. All I/O above Raft uses these interfaces — they are the boundary between CockroachDB’s semantics and Pebble’s raw key-value interface.
  • Design quality: The Writer interface is wide (15+ methods) because MVCC has intrinsically many write modes (versioned point key, unversioned key, range tombstone, range key, intent, etc.). The separation of Reader/Writer from Engine enables passing read-only views to code that must not write, improving safety.

storage.SimpleMVCCIterator and storage.MVCCIterator#

  • Package: pkg/storage
  • File: pkg/storage/engine.go:118 (Simple), pkg/storage/engine.go:234 (MVCC)
  • Methods (SimpleMVCCIterator, 12 methods):
    SeekGE(key MVCCKey)
Valid() (bool, error)
Next()
NextKey()
UnsafeKey() MVCCKey
UnsafeValue() ([]byte, error)
HasPointAndRange() (bool, bool)
RangeBounds() roachpb.Span
RangeKeys() MVCCRangeKeyStack
// ...
  • Purpose: Bi-directional MVCC iteration with range tombstone support. MVCCIterator extends SimpleMVCCIterator with bidirectional seek, stats, and advanced options. EngineIterator provides raw (non-MVCC) access for internal uses.
  • Implementations: pebbleIterator (wraps Pebble), intentInterleavingIter (injects intent visibility), rocksDBIterator (legacy, removed)
  • Design quality: SimpleMVCCIterator follows ISP well — consumers that only need forward iteration get a narrower contract. The range key methods (HasPointAndRange(), RangeKeys()) represent a late addition for MVCC range tombstones — a clean extension that avoided breaking the simpler contract.

execinfra.Processor#

  • Package: pkg/sql/execinfra
  • File: pkg/sql/execinfra/processorsbase.go:37
  • Methods:
    OutputTypes() []*types.T
MustBeStreaming() bool
Run(context.Context, RowReceiver)
Resume(output RowReceiver)
Close(context.Context)
  • Purpose: The base interface for all DistSQL row-by-row processors. Every operator in the row execution engine — TableReader, HashJoiner, SortedAggregator, etc. — implements this.
  • Implementations: 50+ processors in pkg/sql/rowexec/, including TableReader, JoinReader, HashJoiner, SortedAggregator, Windower, ZigzagJoiner, ChangeAggregator
  • Design quality: Clean separation of lifecycle (Run, Resume, Close) from schema (OutputTypes). The Resume method was added for pausable portals, showing how the interface evolved without breaking existing implementations. The MustBeStreaming() marker enables the runtime to decide whether to buffer or stream output.

execinfra.RowSource and execinfra.RowReceiver#

  • Package: pkg/sql/execinfra
  • File: pkg/sql/execinfra/base.go:115 (RowSource), pkg/sql/execinfra/base.go:67 (RowReceiver)
  • Methods (RowSource):
    OutputTypes() []*types.T
Start(context.Context)
Next() (rowenc.EncDatumRow, *execinfrapb.ProducerMetadata)
ConsumerDone()
ConsumerClosed()
  • Methods (RowReceiver):
    Push(row rowenc.EncDatumRow, meta *execinfrapb.ProducerMetadata) ConsumerStatus
ProducerDone()
  • Purpose: Producer/consumer interfaces for the row-based DistSQL data flow. RowSource.Next() is a pull model; RowReceiver.Push() is a push model. Processors implement RowSource; routers and output buffers implement RowReceiver.
  • Implementations: RowBuffer, RowChannel, DistSQLReceiver, routerBase, various sync adapters
  • Design quality: The ConsumerStatus return value (NeedMoreRows, DrainRequested, ConsumerClosed) is an elegant backpressure mechanism — the consumer communicates its state to the producer without a separate channel, enabling early termination and drain propagation.

colexecop.Operator#

  • Package: pkg/sql/colexecop
  • File: pkg/sql/colexecop/operator.go:22
  • Methods:
    Init(ctx context.Context)
Next() (coldata.Batch, *execinfrapb.ProducerMetadata)
execopnode.OpNode  // embedded
  • Purpose: The vectorized (columnar) execution engine operator interface. Where Processor works row-by-row, Operator works on coldata.Batch — columnar batches of up to 1024 values per column. All operators in pkg/sql/colexec/ implement this.
  • Implementations: 100+ vectorized operators: projConst*Op, sel*Op, hashAgg*Op, mergeJoinBase, colSorter, etc.
  • Design quality: Deliberately minimal — two lifecycle methods — which is achievable because the pull model (Next() returns a batch) is inherently simpler than the row engine’s push/pull hybrid. The OpNode embedding enables tracing and introspection. The Batch return enables zero-copy columnar access. The interface is a clean ISP design; specializations like ClosableOperator, ResettableOperator, BufferingInMemoryOperator are opt-in via separate embedding.

catalog.Descriptor#

  • Package: pkg/sql/catalog
  • File: pkg/sql/catalog/descriptor.go:211
  • Methods:
    NameEntry  // embedded: GetName(), GetParentID(), GetParentSchemaID(), GetID()
LeasableDescriptor  // IsUncommittedVersion(), GetVersion(), GetModificationTime()
privilege.Object  // embedded
GetPrivileges() *catpb.PrivilegeDescriptor
DescriptorType() DescriptorType
GetAuditMode() descpb.TableDescriptor_AuditMode
Public() bool
Adding() bool
Dropped() bool
Offline() bool
// ... schema/catalog introspection methods
  • Purpose: The base interface for all schema objects: tables, databases, schemas, types, functions. The catalog leasing system works exclusively through this interface, ensuring version tracking and modification time are always available.
  • Implementations: tabledesc.immutable, dbdesc.immutable, typedesc.immutable, schemadesc.immutable, funcdesc.immutable — all with corresponding mutable variants
  • Design quality: The descriptor hierarchy (Descriptor → TableDescriptor, DatabaseDescriptor, TypeDescriptor, etc.) is well-segregated. TableDescriptor is itself enormous (~600 methods) because tables have the most metadata, but this is an inherent domain complexity. The immutable/mutable split (via MutableTableDescriptor extending TableDescriptor) is clean and prevents accidental mutation of leased descriptors.

concurrency.Manager#

  • Package: pkg/kv/kvserver/concurrency
  • File: pkg/kv/kvserver/concurrency/concurrency_control.go:146
  • Methods: Composed via embedding:
    RequestSequencer    // SequenceReq(), PoisonReq(), FinishReq()
ContentionHandler   // HandleLockConflictError(), HandleTransactionPushError()
LockManager         // AcquireLock(), UpdateLocks()
TransactionManager  // OnTransactionUpdated(), GetDependents()
RangeStateListener  // OnDescriptorUpdated(), OnLeaseUpdated(), OnClosedTimestampUpdated()
MetricExporter      // LatchMetrics(), LockTableMetrics()
TestingAccessor     // GetTestingAccessor() (for tests only)
  • Purpose: The per-range concurrency control subsystem. Manages latch acquisition (exclusive/shared key ranges), lock tables (for pessimistic transactions), and transaction conflict resolution. Every BatchRequest passing through a Replica must be sequenced through this manager.
  • Implementations: concurrency.managerImpl (sole production implementation)
  • Design quality: Excellent decomposition via interface embedding. Each sub-interface (RequestSequencer, ContentionHandler, LockManager, etc.) has a single, well-defined concern. TestingAccessor is isolated in its own embedding — an unusual but pragmatic design that makes the testing surface explicit without polluting the production interface.

kvserver/apply.StateMachine#

  • Package: pkg/kv/kvserver/apply
  • File: pkg/kv/kvserver/apply/task.go:25
  • Methods:
    NewEphemeralBatch() EphemeralBatch
NewBatch() Batch
ApplySideEffects(context.Context, CheckedCommand) (AppliedCommand, error)
  • Purpose: The abstraction for applying committed Raft log entries to a Replica’s state. The apply package provides a task runner that decodes Raft entries, checks commands (via EphemeralBatch.Stage()), applies persistent state transitions (via Batch.ApplyToStateMachine()), then triggers in-memory side effects (via ApplySideEffects()).
  • Implementations: kvserver.replicaStateMachine (the Replica as a state machine)
  • Design quality: Beautifully typed. The three-phase commit pipeline — check → apply-persistent → apply-side-effects — is encoded as distinct types (EphemeralBatch, Batch, AppliedCommand) that cannot be confused. ErrRemoved is a sentinel error from ApplySideEffects indicating replica removal — an elegant way to signal lifecycle termination through the interface.

batcheval.EvalContext#

  • Package: pkg/kv/kvserver/batcheval
  • File: pkg/kv/kvserver/batcheval/eval_context.go:46
  • Methods: ~35+ methods providing batch evaluation commands with access to Replica state:
    ClusterSettings() *cluster.Settings
Clock() *hlc.Clock
AbortSpan() *abortspan.AbortSpan
GetConcurrencyManager() concurrency.Manager
NodeID() roachpb.NodeID
Desc() *roachpb.RangeDescriptor
GetMVCCStats() enginepb.MVCCStats
GetLease() (roachpb.Lease, roachpb.Lease)
CanCreateTxnRecord(ctx, txnID, txnKey, txnMinTS) (bool, reason)
RevokeLease(context.Context, roachpb.LeaseSequence)
// ... 25+ more
  • Purpose: Provides each KV command evaluator (Get, Put, Scan, EndTxn, etc.) with a stable view of the Replica’s runtime state, without exposing the full Replica struct. Each KV verb in pkg/kv/kvserver/batcheval/cmd_*.go receives an EvalContext.
  • Implementations: kvserver.Replica (the sole production implementation)
  • Design quality: An intentionally wide interface — this is a “context object” anti-pattern elevated to pragmatic necessity. The comment on the interface acknowledges this. The benefit is that each command evaluator is decoupled from Replica’s implementation details, can be unit-tested with a mock EvalContext, and the compiler enforces that Replica implements the full contract. The width is bounded by the Replica capabilities that commands legitimately need.

jobs.Resumer#

  • Package: pkg/jobs
  • File: pkg/jobs/registry.go:1358
  • Methods:
    Resume(ctx context.Context, execCtx interface{}) error
OnFailOrCancel(ctx context.Context, execCtx interface{}, jobErr error) error
CollectProfile(ctx context.Context, execCtx interface{}) error
  • Purpose: The extension interface for implementing background jobs. Every long-running distributed job — BACKUP, RESTORE, CREATE INDEX, IMPORT, CDC, schema changes, migrations — implements Resumer. The jobs.Registry manages lifecycle (heartbeating, failure detection, distributed coordination) and calls these three methods.
  • Implementations: backupResumer, restoreResumer, changefeedResumer, createIndexResumer, migrationResumer, autoStatsResumer, ~20+ more
  • Design quality: Intentionally small (3 methods). The execCtx interface{} parameter is a concession to circular imports — the actual type is sql.JobExecContext, which cannot be declared in pkg/jobs without a cycle. This is a deliberate trade-off, documented in the code. The CollectProfile method was added later for observability, showing the interface’s evolution.

cloud.ExternalStorage#

  • Package: pkg/cloud
  • File: pkg/cloud/external_storage.go:41
  • Methods:
    io.Closer  // embedded
Conf() cloudpb.ExternalStorage
ReadFile(ctx, basename string, opts ReadOptions) (ioctx.ReadCloserCtx, fileSize int64, error)
Writer(ctx, basename string) (io.WriteCloser, error)
List(ctx, prefix string, opts ListOptions, fn ListingFn) error
Delete(ctx, basename string) error
Size(ctx, basename string) (int64, error)
RequiresExternalIOAccounting() bool
Settings() *cluster.Settings
  • Purpose: Abstraction over external blob storage systems (S3, GCS, Azure Blob, HTTP, nodelocal, userfile). Used by BACKUP/RESTORE, IMPORT/EXPORT, and log archiving.
  • Implementations: s3Storage, gcsStorage, azureStorage, httpStorage, localFileStorage, userFileStorage, nodelocal.Storage
  • Design quality: Clean file system metaphor (read, write, list, delete, size). The List method uses a callback (fn ListingFn) rather than returning a slice — appropriate for potentially millions of objects. Builds on io.Closer rather than duplicating lifecycle methods.

server.onDemandServer#

  • Package: pkg/server
  • File: pkg/server/server_controller.go:40
  • Methods:
    orchestratedServer  // embedded: notifyDraining(), notifyStopped(), getTenantID()
getHTTPHandlerFn() http.HandlerFunc
handleCancel(ctx, cancelKey pgwirecancel.BackendKeyData)
serveConn(ctx, conn net.Conn, status pgwire.PreServeStatus) error
getSQLAddr() string
getRPCAddr() string
  • Purpose: The per-tenant SQL server abstraction managed by serverController. Enables the controller to manage heterogeneous server types (system tenant server vs. secondary tenant SQL server) through a uniform interface.
  • Implementations: server.topLevelServer (system tenant), server.SQLServerWrapper (secondary tenant)
  • Design quality: Well-scoped to controller responsibilities. The separation of orchestratedServer (lifecycle) from onDemandServer (routing) reflects the two concerns of multi-tenancy: managing server lifetime and routing connections. The interface was introduced when multi-tenancy was added, retrofitting the existing topLevelServer without modifying its internals.

Interface patterns#

Size distribution#

  • Very small (1–3 methods): kv.Sender (1), apply.StateMachine (3), colexecop.Operator (2), jobs.Resumer (3), cloud.ExternalStorage.ReadFile variant
  • Medium (4–10 methods): execinfra.Processor (5), execinfra.RowReceiver (2+1), cloud.ExternalStorage (7), server.onDemandServer (6)
  • Large (11–30 methods): storage.Engine (~30), storage.Reader (~8), kv.TxnSender (~20)
  • Very large (31+ methods): batcheval.EvalContext (~35), catalog.TableDescriptor (~600 for the schema hierarchy)

The median interface in CockroachDB has 4–6 methods. The very large interfaces (EvalContext, TableDescriptor) exist at layer boundaries where one side has rich state and the other has rich operations — they are “context” or “descriptor” objects, not service interfaces.

Embedding for composition#

CockroachDB makes heavy use of interface embedding for composition:

  • storage.ReadWriter = Reader + Writer
  • storage.Engine = Reader + Writer + engine ops
  • concurrency.Manager = 7 embedded interfaces
  • kv.TxnSender extends kv.Sender
  • colexecop.DrainableClosableOperator = ClosableOperator + MetadataSource
  • catalog.Descriptor embeds NameEntry + LeasableDescriptor + privilege.Object

This is idiomatic Go at scale: capabilities are composed from smaller contracts rather than inherited.

Implicit satisfaction (consumer-defined vs. provider-defined)#

CockroachDB uses both patterns:

  • Provider-defined (conventional): jobs.Resumer, cloud.ExternalStorage, colexecop.Operator — the provider framework defines what implementors must satisfy.
  • Consumer-defined (Go idiom): batcheval.EvalContext, concurrency.Manager’s sub-interfaces, kvserver/apply.StateMachine — the consumer package defines the interface, keeping coupling unidirectional. The Replica satisfies EvalContext without batcheval importing kvserver.

The consumer-defined pattern is used systematically to prevent import cycles across the large layered monolith. It appears at nearly every major layer boundary.

stdlib interfaces used#

  • io.Closer — embedded in storage.Batch, cloud.ExternalStorage
  • fmt.Stringer — embedded in batcheval.EvalContext, kvpb.Request
  • context.Context — used as a parameter in virtually every interface method (not embedded but ubiquitous)
  • io.Reader/io.Writer — used in cloud.ExternalStorage.ReadFile/Writer return types

Standard library interfaces are used as building blocks (embedding) rather than as full implementations — CockroachDB’s domain is too specific for stdlib interfaces to be primary contracts.

Key abstractions#

1. kv.Sender — The vertical integration interface#

The single most architecturally significant interface. It threads the entire KV stack from SQL transactions to Raft replicas. Its minimalism (one method) enables a clean interceptor chain at TxnCoordSender and makes testing trivially simple. The cost is that BatchRequest carries enormous semantic richness — the “too narrow” comment is accurate but the narrowness is what makes the chain composable.

2. storage.Engine / Reader / Writer hierarchy — The storage capability lattice#

A well-designed four-level hierarchy: Reader, Writer, ReadWriter, Engine. Each level adds capabilities, and code that doesn’t need write access receives a Reader. The MVCCIterator interface (not Engine) is what most MVCC code interacts with, keeping the MVCC layer thin above the raw storage interface.

3. execinfra.Processor + RowSource/RowReceiver + colexecop.Operator — The dual execution engine#

The existence of two parallel interfaces — one for row-by-row (Processor/RowSource) and one for vectorized (Operator) — reflects the dual execution engine architecture. The vectorized path is simpler (pull model, batch return) but handles a narrower subset of operations. Bridging adapters allow vectorized and row operators to be mixed in a single flow.

4. concurrency.Manager — The access control composition#

The seven-interface composition is a textbook example of the Interface Segregation Principle applied to a complex subsystem. Each concern (sequencing, contention, locks, transactions, range state, metrics, testing) is separately testable and independently understandable. The sole production implementation (managerImpl) satisfies all seven.

5. jobs.Resumer — The background work extension point#

The smallest “big” interface: 3 methods, 20+ implementations. It is the primary plugin point for adding new distributed background operations to CockroachDB. The narrow interface hides the complexity of distributed coordination — resumability, failure recovery, heartbeating — which the Registry handles uniformly for all Resumer implementations.

Interface-driven extensibility#

CockroachDB uses interfaces for extensibility in five distinct patterns:

1. Storage engine swap (storage.Engine)#

The Engine interface was designed to allow storage engine replacement. In early versions, a RocksDB engine existed; today only Pebble exists, but the interface remains. The interface boundary ensures future engines can be introduced without touching the KV layer.

2. Cloud backend plugins (cloud.ExternalStorage)#

The ExternalStorage interface is CockroachDB’s primary plugin point for cloud integration. New storage backends (S3, GCS, Azure, etc.) implement this interface and register themselves via cloud.impl_registry.go. This enables seamless addition of cloud providers without modifying BACKUP/RESTORE logic.

3. Background job types (jobs.Resumer)#

New distributed background operations (like a new index backfill algorithm or a new migration job) implement Resumer and register a constructor with jobs.RegisterConstructor(). The registry provides distributed coordination, heartbeating, and lifecycle management without the job implementor needing to understand distributed state machines.

4. CCL feature injection (function variables + init())#

Not interface-based strictly, but functionally equivalent: OSS code defines nil function variables where enterprise features would hook in. CCL packages replace these via init() registration. This achieves open/closed extensibility without interfaces — the OSS code is “closed” to modification while CCL is “open” for extension.

5. Multi-tenant server fleet (server.onDemandServer)#

The serverController manages heterogeneous server types through the onDemandServer interface. Adding a new type of tenant server (e.g., a future proxy server) requires implementing this interface rather than modifying the controller. The interface was designed specifically to enable the multi-tenancy extension while keeping the system tenant’s server unchanged.

Backward CockroachDB — Patterns CockroachDB — API Surface Forward