etcd — Testing#

Test metrics#

• Test files: 405 *_test.go files
• Total Go files: 1,099 (excluding vendor)
• Ratio (test files / source files): ~37% — substantial, reflecting a project where correctness is existential
• Test frameworks: stdlib testing is the foundation; github.com/stretchr/testify (require + assert) is used pervasively; no ginkgo, gomock, gocheck, or goconvey

Test organization#

Placement#

Hybrid: 222 test files co-located with source packages; 263 files in the dedicated tests/ top-level directory. The majority of complex integration and e2e tests live in tests/.

Helper packages#

Fixtures / testdata#

• tests/integration/snapshot/testdata/ — snapshot fixtures
• tests/robustness/testdata/ — saved linearizability traces for regression
• server/storage/wal/testdata/ — WAL segment fixtures
• tests/fixtures/ — TLS cert fixtures used across test suites

Test patterns#

Table-driven tests#

• Prevalence: Heavy. 641 occurrences of t.Run / tt.Run / tc.name patterns in test files.
• Style: Anonymous struct slices are standard:

  for _, tc := range []struct { name string; ... }{ ... } {
      t.Run(tc.name, func(t *testing.T) { ... })
  }

• Example: tests/common/kv_test.go:TestKVPut — iterates clusterTestCases() which returns different cluster configs (TLS, auth, proxy), running the same test body against each backend.

The multi-backend test framework (tests/framework/)#

This is etcd’s most architecturally distinctive testing feature. The framework defines a TestRunner interface in tests/framework/interfaces/interface.go:

type TestRunner interface {
    TestMain(m *testing.M)
    BeforeTest(testing.TB)
    NewCluster(context.Context, testing.TB, ...config.ClusterOption) Cluster
}

The Cluster and Client interfaces abstract over both in-process and out-of-process cluster implementations. Tests in tests/common/ are written once against these interfaces, then compiled with different build tags to run against:

• //go:build integration → framework.IntegrationTestRunner (in-process etcd cluster, fast, no subprocess)
• //go:build e2e → framework.E2eTestRunner (spawns real etcd binary processes, slower, higher fidelity)
• //go:build !(e2e || integration) (unit default)

This means a single test like TestKVPut validates both the in-process library and the actual binary behavior without code duplication. The cluster_proxy build tag further toggles the cluster through a gRPC proxy, testing that code path at zero additional test-writing cost.

Mocking approach#

• Strategy: Hand-written fakes using the testutil.Recorder infrastructure — no code generation.
• Recorder pattern: client/pkg/testutil/recorder.go defines a Recorder interface with Record(Action) and Wait(n). Fakes embed RecorderBuffered or RecorderStream and record every call. Tests then assert recorder.Action() returns the expected sequence of Action{Name, Params} values.
• Example: server/mock/mockstorage/storage_recorder.go implements raft.Storage by recording Save, Append, SetHardState calls. Unit tests for the Raft integration verify that specific storage operations were called in the right order, without running a real backend.
• Client mock: client/v3/mock/mockserver/ provides an in-process gRPC server implementing etcdserverpb.KVServer — used for client interceptor and retry logic tests without needing a real cluster.

Integration tests#

• Present: Yes, extensive — tests/integration/ and the in-process framework.
• How: The tests/framework/integration package embeds etcd in-process using embed.StartEtcd(). A ClusterConfig controls member count, TLS, auth, proxy. Tests create real clusters, issue real client operations, and verify real behavior — all within a single go test invocation.
• Separation: Build tag //go:build integration gates all integration-specific init files. The test binary itself contains no build-tag-guarded test functions — only the init/setup code is gated, so the same test functions compile for all backends.
• E2e tests: tests/e2e/ spawns the etcd binary as a subprocess. The e2e.EtcdProcessCluster type manages process lifecycle, stdout/stderr capture, and gRPC endpoint detection. Used for testing upgrade paths, TLS, downgrade, and CLI tooling (etcdctl).

Robustness tests (tests/robustness/)#

etcd’s most sophisticated testing layer — a property-based fault-injection framework:

• Goal: Verify linearizability of the key-value store under arbitrary failures.
• Mechanism:
  1. scenarios.Exploratory(t) / scenarios.Regression(t) generate test scenarios (cluster size, failpoints).
  2. failpoint.PickRandom(c, profile) injects a random fault (kill leader, network partition, disk stall, etc.).
  3. traffic.SimulateTraffic(...) runs concurrent clients issuing random puts, gets, txns, and watches.
  4. validate.ValidateAndReturnVisualize(...) runs the collected operations through porcupine — a linearizability checker.
• Porcupine: github.com/anishathalye/porcupine is used to verify that the observed history of client operations is consistent with a single linearizable key-value store. If any operation history is non-linearizable, the test fails with a visualization artifact.
• Failpoints: Uses go.etcd.io/gofail — runtime-injectable failure points compiled into the binary under a build tag. Allows pausing, panicking, or returning errors from specific code paths without modifying source.
• Regression testdata: tests/robustness/testdata/ stores saved operation histories from past bugs, replayed as regression tests.
• Antithesis integration: tests/antithesis/ provides configuration for the Antithesis platform (autonomous fault injection at hypervisor level) — etcd is one of the few Go projects with this level of formal testing investment.

Goroutine leak detection#

client/pkg/testutil/leak.go provides CheckLeakedGoroutine() and RegisterLeakDetection(t). After each test, it inspects goroutine stacks via runtime.Stack and filters known-safe goroutines. A comment notes “TODO: Replace with https://github.com/uber-go/goleak" — the implementation is custom but functionally equivalent. Used in TestMain of many packages via testutil.MustTestMainWithLeakDetection(m).

Test quality observations#

What’s done well#

• The multi-backend framework is exemplary. Writing tests once against an interface and compiling them for both in-process and external-process backends is a powerful technique that maximizes coverage without duplication. The //go:build integration / e2e + init-file pattern is a clean Go idiom for this.
• Robustness testing goes beyond unit/integration. Using porcupine for linearizability checking is production-grade correctness assurance — rare in open source. Combined with failpoint injection and traffic simulation, this is closer to what Jepsen tests provide but integrated into CI.
• Goroutine leak detection is built-in. Every test package that spawns goroutines guards against leaks. This catches a class of bugs (goroutine leaks after test failures) that most projects miss entirely.
• Recorder-based fakes are honest. Rather than mocking return values, fakes record the sequence of calls. Tests assert on the call sequence, which verifies behavior rather than just outputs.
• Table-driven tests are pervasive and consistent — 641 t.Run invocations give good coverage variance across configs with minimal boilerplate.

What could improve#

• The goroutine leak detector is a home-grown reimplementation of goleak with a TODO noting the same. This is technical debt — goleak is more maintained and handles edge cases better.
• No generated mocks. The Recorder infrastructure is powerful but verbose to extend — adding a new mock requires manually implementing every interface method. gomock or mockery would reduce this burden.
• Build tag proliferation. Three build tags (integration, e2e, cluster_proxy) create a combinatorial test matrix that is hard to reason about locally. The Makefile targets help, but new contributors frequently miss running the right combination.

Patterns worth emulating#

1. Multi-backend framework with interface-driven test runners — write once, run against in-process and real binary. Applicable to any project with both library and CLI layers.
2. Robustness testing with linearizability checking — the porcupine-based model + traffic simulation pattern is directly reusable for any distributed storage system.
3. Recorder-based fakes — recording call sequences rather than just stubbing return values makes tests more behavioral and catches ordering bugs.
4. Build-tag-gated init files — separating the test setup from the test logic via build tags (rather than testing.Short() or flag checks) is a clean, explicit way to manage test tiers.
5. Goroutine leak detection as a first-class test primitive — wrapping TestMain with leak detection costs almost nothing but catches a real class of concurrency bug.