CockroachDB — Testing#

Sampling note#

CockroachDB is an XL-tier project (~9,000+ Go files). Testing-specific exploration was performed with grep across the full repository, followed by deep dives into key test infrastructure: pkg/testutils/, pkg/util/leaktest/, pkg/sql/logictest/, pkg/kv/kvnemesis/, pkg/storage/metamorphic/, pkg/kv/kvserver/asim/, pkg/cmd/roachtest/, and pkg/testutils/lint/. Sample test files were read from pkg/kv/kvserver/split/, pkg/testutils/sqlutils/, and pkg/testutils/echotest/.

Test metrics#

• Test files: 3,067 (*_test.go, excluding vendor/)
• Source files (non-test): 6,118 (excluding vendor/)
• Ratio (test / source): ~0.50 — roughly one test file per two source files
• Benchmark test files: 316 files containing func Benchmark*
• Test frameworks: stdlib testing (universal) + testify/require + testify/assert (2,200 files) + custom assertion helpers
• leaktest.AfterTest usage: 16,363 call sites — used in essentially every test function
• Table-driven tests: 8,730 matches for t.Run/testCases/tc.name patterns
• datadriven golden tests: 718 usages in test files, 274 imports of cockroachdb/datadriven

Test organization#

Placement#

CockroachDB uses both package foo (white-box) and package foo_test (black-box) placements, often in the same directory. The convention is:

• Same-package tests for testing internal types and unexported methods
• _test package suffix for integration-style tests that only use the public API
• main_test.go in most packages registers the server factory shim (avoiding circular imports)

Helper packages#

CockroachDB has built one of the most sophisticated testing helper ecosystems in open-source Go. The pkg/testutils/ directory contains 30+ sub-packages:

Fixtures#

• testdata/ directories: 170+ directories across the codebase containing:
  • datadriven-format text files (SQL, Raft log, optimizer rule expected outputs)
  • Golden-output files used by echotest
• Generated fixtures: Schema change corpus, optimizer memo snapshots
• Reusable fixtures on disk: testfixtures.ReuseOrGenerate caches expensive benchmark setup (e.g., large Pebble databases) in ~/.cache/crdb-test-fixtures/

Test patterns#

Table-driven tests#

• Prevalence: Very heavy — 8,730 matches. Default pattern for any test with multiple inputs.
• Style: Primarily anonymous struct slices:

  testCases := []struct {
      name     string
      input    someType
      expected otherType
  }{
      {name: "empty", input: ..., expected: ...},
      {name: "overflow", input: ..., expected: ...},
  }
  for _, tc := range testCases {
      t.Run(tc.name, func(t *testing.T) {
          // ...
      })
  }

• Example: pkg/kv/kvserver/split/weighted_finder_test.go — tests split point selection with named cases for empty reservoir, uniform distribution, non-uniform load.

datadriven golden tests#

• Prevalence: 718 usages in test files; the dominant pattern for SQL, optimizer, and parser testing.
• Style:

  # pkg/sql/logictest/testdata/logic_test/aggregate
  statement ok
  CREATE TABLE kv (k INT PRIMARY KEY, v INT, ...)
  
  query IIIIRR
  SELECT min(v), max(v), count(v)...
  FROM kv
  ----
  NULL NULL 0 ...

  The test runner reads the file, executes the command (SQL statement or optimizer rule application), and diffs actual vs. expected output after the ---- line. Running go test -run TestFoo -rewrite regenerates expected output.
• Example: pkg/sql/opt/norm/testdata/rules/agg.opt — normalization rule application; pkg/sql/logictest/testdata/logic_test/aggregate — 493 SQL logic test files covering the full SQL test suite.

SucceedsSoon — async condition polling#

• Prevalence: Pervasive in distributed and storage tests.
• Style:

  testutils.SucceedsSoon(t, func() error {
      n, err := sqlDB.QueryRow("SELECT count(*) FROM ...").Scan(&count)
      if err != nil { return err }
      if count != expectedCount { return errors.Errorf("expected %d, got %d", ...) }
      return nil
  })

• Implementation: Exponential backoff from 1ns to 45s (225s under race detector). On timeout, writes a goroutine dump to a file alongside the test failure for debugging. Adapts timeout to syncutil.DeadlockEnabled as well.
• Assessment: Essential for testing eventual consistency, gossip propagation, and async background work. Far better than time.Sleep — tests pass faster in the common case and provide rich failure info.

leaktest.AfterTest — goroutine leak detection#

• Prevalence: 16,363 call sites — near-universal.
• Usage: defer leaktest.AfterTest(t)() as the first line of almost every test.
• Implementation: Takes a stack snapshot of “interesting” goroutines before the test, polls for leaked goroutines after the test, fails if new goroutines remain after a timeout. Excludes known-acceptable persistent goroutines (log flusher, sentry transport, goschedstats, etc.) from the leak check.
• Assessment: Production-grade goroutine hygiene enforcement. Catches tests that start a goroutine without stopping it — a common source of test flakiness when leaked goroutines from test N interfere with test N+1. The maintenance of the exclusion list is a small ongoing cost.

Mocking approach#

• Primary strategy: TestingKnobs structured injection (see patterns analysis). Rather than mocking at the interface level, CockroachDB uses 55+ typed testing hook slots in base.TestingKnobs. Each component’s _testing_knobs.go file defines function-typed fields:

  // pkg/kv/kvserver/store_test_helpers.go (approximate)
  type StoreTestingKnobs struct {
      TestingRequestFilter   kvserverbase.ReplicaRequestFilter
      TestingApplyCalledTwiceFilter func(args apply.CommandList) bool
      ...
  }

  Production code checks if knobs != nil { knobs.SomeHook(...) } — zero overhead when nil, which is the production case.
• testutils.TestingHook / testutils.HookGlobal: For replacing package-level function variables temporarily:

  defer testutils.TestingHook(&timeutil.Now, func() time.Time { return fakeNow })()
  // or (generics):
  defer testutils.HookGlobal(&someVar, fakeValue)()

• No gomock/mockery: CockroachDB does not use code-generated mock frameworks. Where interfaces need faking, hand-written fakes are in testutils/ sub-packages (e.g., testutils/sqlutils, testutils/gossiputils, roachtest’s cluster/mock/).
• Assessment: The TestingKnobs approach scales better than generated mocks for a large codebase because it keeps test concerns co-located with production code and avoids the maintenance burden of regenerating mocks. The downside is that production binaries include nil-check overhead for knobs.

Integration tests#

• Present: Yes — both file-naming convention (*_integration_test.go) and dedicated testing framework (roachtest).
• In-process integration: pkg/testutils/testcluster spins up a real multi-node CockroachDB cluster in a single process for integration tests. Used heavily in:
  • pkg/kv/kvserver/txn_recovery_integration_test.go
  • pkg/kv/kvserver/flow_control_integration_test.go
  • pkg/jobs/ash_integration_test.go These use real storage (in-memory Pebble), real SQL processing, real Raft, and real network (TCP loopback). No Docker or containers required.
• TestCluster interface:

  tc := testcluster.StartTestCluster(t, 3, base.TestClusterArgs{
      ServerArgs: base.TestServerArgs{
          Knobs: base.TestingKnobs{
              Store: &kvserver.StoreTestingKnobs{
                  DisableMergeQueue: true,
              },
          },
      },
  })
  defer tc.Stopper().Stop(ctx)

• Remote integration: roachtest deploys real CockroachDB binaries on actual cloud VMs (via roachprod) for multi-machine tests that cannot be run in-process.
• Separation: Integration tests are separated by:
  • *_integration_test.go naming convention
  • skip.UnderRace(t, "slow under race") guards
  • Build tags (bazel, linux) for platform-specific tests

Unique testing frameworks#

1. logictest — SQL Logic Test Engine#

• Location: pkg/sql/logictest/
• Scale: 493 test data files; logic.go is ~6000 lines
• Origin: Inspired by SQLite’s sqllogictest; extended for CockroachDB-specific features (multi-tenant, cluster settings, retry handling).
• Format:

  # LogicTest: !local-prepared
  
  statement ok
  CREATE TABLE kv (k INT PRIMARY KEY, v INT)
  
  query II rowsort
  SELECT k, v FROM kv WHERE v > 10
  ----
  100  200
  101  201

  Directives: statement ok, statement error <regexp>, query <type_codes> [options], subtest <name>, # LogicTest: <config filter>.
• Capabilities: Runs the same test file against multiple configurations (local/vectorized/distsql/multi-tenant), validates results, handles transaction retries, checks SQLSTATE codes. The ---- separator specifies expected output with column types encoded as single-char codes (I=int, T=string, R=real, B=bool).
• Assessment: The most complete SQL integration testing system in open-source Go databases. The ability to run 493 test files across 8+ execution engine configurations from a single corpus is a significant competitive advantage.

2. kvnemesis — Serializable Correctness Checker#

• Location: pkg/kv/kvnemesis/
• What it does: Generates random concurrent KV operations (puts, deletes, scans, transactions, splits, merges) against a real CockroachDB cluster, then validates that the observed results are consistent with serializable isolation using the MVCC history as ground truth.
• Mechanism:
  1. Workers concurrently execute random Operations (each mutation tagged with a unique kvnemesisutil.Seq stored in the value)
  2. A RangeFeed watcher ingests the MVCC history in real-time
  3. After all workers complete, Validate() checks that every read saw a value consistent with serializability — made tractable by the unique seq tags (O(n) rather than NP-hard)
• Assessment: One of the most sophisticated correctness testing tools in any open-source database. The Elle-based approach (borrowed from Jepsen’s Elle tool) provides formal correctness guarantees. The crdb_test build tag enables sequence number embedding in production MVCC values without exposing it in release builds.

3. storage/metamorphic — Storage Engine Metamorphic Testing#

• Location: pkg/storage/metamorphic/
• What it does: Generates random sequences of MVCC operations (10,000 by default) against different storage engine configurations, then verifies that the results are identical across configurations and consistent across restarts.
• Mechanism:
  • Generator produces a random but deterministic (seeded) operation sequence
  • Multiple engineSequence configurations are tested (e.g., different Pebble sstable formats, block cache sizes, WAL settings)
  • compare-files mode: re-run a specific sequence and check output equality
  • -keep flag: retain temp directories on failure for debugging
• Assessment: Metamorphic testing is a powerful technique for storage engines where the “expected output” isn’t known a priori — the invariant is that equivalent configurations must produce the same results. CockroachDB uses it to detect bugs when storage engine parameters are varied.

4. asim — Allocator Simulator#

• Location: pkg/kv/kvserver/asim/
• What it does: Simulates an entire CockroachDB cluster’s replica placement and rebalancing decisions without running actual Raft or SQL, enabling fast testing of allocator policy changes.
• Components: state (simulated cluster state), workload (synthetic load generation), queue (replica queue simulation), storerebalancer (simulated rebalancer), assertion (policy assertion checking), history (metrics recording)
• Usage: Used in data-driven tests in pkg/kv/kvserver/asim/tests/testdata/ to assert that the allocator converges to expected distributions given various initial conditions.
• Assessment: Enables rapid iteration on rebalancing algorithms that would otherwise require multi-hour roachtest runs. The simulator reuses production allocator code, so bugs found in the simulator translate directly to real behavior.

5. roachtest — Distributed End-to-End Testing#

• Location: pkg/cmd/roachtest/
• What it does: Framework for running multi-machine integration tests against real CockroachDB deployments on cloud VMs, orchestrated via roachprod.
• Test structure:

  func registerMyTest(r registry.Registry) {
      r.Add(registry.TestSpec{
          Name:             "my-test/3node",
          Cluster:          r.MakeClusterSpec(3),
          Timeout:          20 * time.Minute,
          Run: func(ctx context.Context, t test.Test, c cluster.Cluster) {
              c.Put(ctx, cockroach, "./cockroach")
              c.Start(ctx, t, c.All())
              db := c.Conn(ctx, t, 1)
              // ... test logic
          },
      })
  }

• Capabilities: Node add/remove, rolling upgrades, network partition simulation, clock skew injection, mixed-version testing. Tests are categorized by Tag (e.g., weekly, aws, nightly) and selected by roachtest list/run.
• Assessment: The most production-realistic testing layer. The roachstress.sh script can run a roachtest repeatedly to detect flaky tests. Mixed-version tests (mixedversion/) test upgrade compatibility — critical for a database that must support rolling upgrades.

Custom static analysis (pkg/testutils/lint/passes/)#

CockroachDB has 20+ custom go/analysis passes enforcing project-specific rules:

These passes run as part of the Bazel build and CI, making them enforcement rather than suggestion.

Build tags and test configuration#

crdb_test build tag — metamorphic builds#

• Purpose: When crdb_test is set (all Bazel test targets), production constants are randomized using metamorphic.ConstantWithTestRange:

  // pkg/kv/kvserver/store.go (approximate)
  var replicaScanMinIdleness = metamorphic.ConstantWithTestRange(
      "replica-scan-min-idleness", /* production */ 50*time.Millisecond,
      /* test range: */ time.Millisecond, 100*time.Millisecond,
  )

  With 80% probability on a metamorphic build, the constant takes a random value within the test range. This tests boundary conditions (e.g., very small batch sizes, very short timeouts) that are never exercised in production.
• Assessment: A highly effective way to get combinatorial coverage without writing explicit tests for every boundary condition. Found numerous bugs in practice.

Race detector (race build tag)#

• SucceedsSoon extends its timeout 5x under -race
• Many tests use skip.UnderRace(t, "slow under race") to skip very slow tests
• syncutil.Mutex has a mutex_sync_race_test.go variant testing race-specific behavior
• The deadlock detector (deadlock build tag) further extends SucceedsSoon timeouts

bazel build tag#

• Some tests are Bazel-specific (e.g., pkg/util/grunning/enabled_test.go)
• Used to distinguish Bazel-managed test execution from go test

Test quality observations#

What’s done well#

1. Goroutine hygiene is world-class. leaktest.AfterTest in 16,363 test functions with a comprehensive exclusion list represents a genuine commitment to goroutine hygiene that most Go projects lack. Tests that start goroutines are forced to clean them up.

2. The logictest corpus is unmatched. 493 test files covering the full SQL surface, runnable across 8+ engine configurations from a single test binary. This is the primary reason CockroachDB’s SQL compatibility is so high.

3. Correctness testing goes beyond functional. kvnemesis (serializability checker) and storage/metamorphic (metamorphic storage testing) test properties that cannot be verified with unit tests — they test mathematical invariants under random concurrent load. Very few databases have equivalents.

4. TestingKnobs makes fault injection systematic. Rather than scattering ad-hoc global test booleans, the base.TestingKnobs registry with 55+ typed slots gives every module a well-defined place to register test hooks. The zerofields package even catches when a knobs struct is initialized without setting expected fields.

5. Metamorphic constants stress boundary conditions automatically. The 80% probability of using random small values for constants like batch sizes, timeout durations, and queue capacities on crdb_test builds means every test run implicitly tests edge cases.

6. Custom linters enforce test discipline. The leaktestcall linter ensures no test file imports leaktest without calling it. The errcmp linter prevents err == someError comparisons that break with wrapped errors. These catches bugs before code review.

7. The SucceedsSoon + goroutine dump pattern provides rich failure diagnostics. When an async condition fails to become true within 45 seconds, the test output includes a full goroutine dump showing exactly what all goroutines were doing.

8. TestCluster in-process multi-node testing allows integration-level coverage without Docker or external services. A test can spin up a 5-node cluster in-process in milliseconds using in-memory Pebble.

What could improve#

1. Test file ratio (~0.50) is lower than expected for a project of this complexity. Many complex packages (e.g., pkg/sql/conn_executor.go at 5000+ lines) have proportionally fewer direct unit tests, relying instead on the logictest corpus for coverage.

2. roachtest tests require cloud access — they cannot be run by external contributors without cloud VMs. The README acknowledges this but it limits the contribution surface for distributed integration tests.

3. TestingKnobs pattern increases production binary size. The 1,155 usages of testing knobs in production code paths (verified from patterns analysis) mean that nil-check overhead is present in production binaries. This is a deliberate tradeoff but worth documenting.

4. datadriven test files can become large and hard to review. The SQL logic test directory has 493 files totaling thousands of lines. Adding a test requires understanding the file format and the -rewrite workflow, which is a learning curve.

Patterns worth emulating#

1. leaktest.AfterTest pattern — every Go project with goroutines should have goroutine leak detection. CockroachDB’s implementation is production-grade and easily extractable.

2. SucceedsSoon with goroutine dump on timeout — the combination of exponential backoff polling with automatic goroutine dumps on failure is significantly better than any time.Sleep-based approach.

3. TestingKnobs registry — projects that find themselves scattering test booleans across packages should centralize them in a struct with typed slots, using the CockroachDB pattern.

4. Metamorphic constants — the crdb_test build tag + ConstantWithTestRange pattern automatically exercises boundary conditions without test maintenance.

5. datadriven golden tests for human-readable output — any system with SQL, query plans, or structured text output benefits enormously from datadriven golden tests with -rewrite support.

6. Custom go/analysis passes for project-specific rules — deferloop, errcmp, leaktestcall are immediately applicable to any large Go codebase and catch bugs that generic linters miss.

Testing pyramid summary#