Vault — Testing#

Test metrics#

  • Test files: 758
  • Source files (total Go): ~2,212
  • Ratio (test files / source files): ~34% (roughly 1:3)
  • Test frameworks: stdlib testing (primary), testify/assert + testify/require (widespread), go-test/deep (deep equality diffs in core), sasha-s/go-deadlock (in-process deadlock detection in core tests)

Test organization#

  • Placement: Both same-package tests (package vault in vault/core_test.go) and external _test packages (package api in vault/external_tests/api/). The dominant style is same-package, matching Go idiom for whitebox testing.
  • Helper packages: A rich ecosystem in helper/testhelpers/ containing per-service helpers:
    • corehelpers/RetryUntil, MakeTestPluginDir, logger factories; designed to avoid import cycles with the vault package
    • logical/ — in-process request helpers for SDK backend testing
    • minimal/ — minimal Vault core configurations for lightweight tests
    • pki/, certhelpers/ — TLS and PKI fixture generation
    • postgresql/, mysql/, mongodb/, mssql/, cassandra/ — database service setup helpers (likely Docker-based)
    • consul/, etcd/, minio/, azurite/, fakegcsserver/ — storage backend helpers
    • seal/ — seal/unseal key helpers
    • replication/ — enterprise HA test helpers
    • sdk/database/dbplugin/v5/testing/ — protocol-level database plugin test helpers (AssertInitialize, AssertNewUser, VerifyInitialize) with configurable request timeouts via VAULT_TEST_DATABASE_REQUEST_TIMEOUT
    • sdk/plugin/mock/ — complete mock logical.Backend implementation for SDK plugin tests
    • command/agentproxyshared/sink/mock/mock_sink.go — manual mock for agent sink interface
  • Fixtures: testdata/ directories in api/cliconfig/, vault/, sdk/framework/, physical/raft/, builtin/logical/pkiext/pkiext_binary/, and code-checker packages. Primarily certificate files, JSON responses, and HCL config fixtures.

Test patterns#

Table-driven tests#

  • Prevalence: Extremely heavy — 850 grep matches for testCases/tt.Run/tc.name patterns in *_test.go files, and 964 occurrences noted in the patterns analysis. This is the default choice for any function with multiple scenarios.
  • Style: Two styles are both used: []struct{} slice for ordered cases, and map[string]struct{} for unordered. The map-keyed style is prominent in vault/core_test.go (tests := map[string]struct{}{...}) — parallel sub-tests where ordering is irrelevant. The slice style dominates in command tests where order matters.
  • Example: vault/core_test.goTestNewCore_configureAuditBackends uses map[string]struct{} with t.Parallel() inside each subtest, demonstrating the recommended pattern for truly independent cases.

Parallel testing#

  • Usage: 1,436 occurrences of t.Parallel() in test files — vault commits heavily to parallel test execution. Both top-level tests and sub-tests are parallelized wherever safe.
  • Pattern: Test files frequently open with t.Parallel() at the top-level test function AND repeat it inside each table-driven subtest, following the name := name; tc := tc capture idiom.

Mocking approach#

  • Strategy: Interface-based manual fakes — no gomock or mockery code generation is used. Dependencies that need faking implement the target interface directly.
  • Example: sdk/plugin/mock/backend.go — a hand-written logical.Backend implementation that provides a full mock plugin with configurable paths. command/agentproxyshared/sink/mock/mock_sink.go — a hand-written mock for the Sink interface used in agent proxy tests.
  • Assessment: Consistent with Vault’s overall aversion to code generation. Manual fakes are more readable and do not add a go generate dependency, but require more maintenance when interfaces evolve.

Integration tests#

  • Present: Yes — multiple tiers of increasing realism.
  • Tier 1 — In-process cluster: The primary mechanism. helper/testhelpers/ and sdk/helper/testcluster/ support spinning up a full Vault cluster (1-3 nodes, raft storage, TLS, real unseal/seal cycle) inside a single go test process. vault/external_tests/api/api_integration_test.go uses testVaultServer(t) which calls vault.NewCore + http.TestServer directly, giving full API access without network overhead. 335 test files use NewTestCluster or corehelpers.
  • Tier 2 — Docker cluster: sdk/helper/testcluster/docker/environment.go launches real Vault binaries in Docker containers, used for HA/replication scenarios that require separate OS processes.
  • Tier 3 — Blackbox binary tests: vault/external_tests/blackbox/ uses sdk/helper/testcluster/blackbox.New(t) which connects to an already-running Vault instance (address + token from environment variables). TestPostgresDynamicSecrets (blackbox/dynamic_test.go) demonstrates this: it authenticates to a live Vault, configures a PostgreSQL secrets engine, generates credentials, and verifies the full credential lifecycle. This tier requires external services (PostgreSQL, running Vault).
  • Tier 4 — Cloud E2E: builtin/credential/aws/backend_e2e_test.go tests against real AWS IAM — requires AWS credentials.
  • Separation: Directory-based, not build-tag-based. The vault/external_tests/ tree is physically separate from unit tests. No //go:build integration tags found — instead, tests that require services rely on environment variables being set (they skip or fail-fast when vars are absent).

t.Cleanup usage#

  • Usage: 85 occurrences in test files — adopted but not universal. Older code uses defer directly; newer code prefers t.Cleanup which cooperates better with parallel subtests.

RetryUntil pattern#

  • corehelpers.RetryUntil(t, timeout, func() error) is a purpose-built polling helper for async assertions — used when testing background goroutines (lease revocation, replication lag, event bus delivery). Polls every 100ms until the function returns nil or the deadline passes.

Test quality observations#

  • What’s done well:
    • Hierarchical test infrastructure is the standout strength. The three-tier integration approach (in-process → Docker → binary blackbox) lets each test use the cheapest sufficient tier: unit tests use in-process clusters and run fast; replication tests use Docker; true E2E uses the blackbox tier.
    • Aggressive parallelism (1,436 t.Parallel() calls) combined with in-process cluster isolation means the test suite leverages multi-core machines effectively without external service contention.
    • helper/testhelpers/ breadth: a complete per-service ecosystem means tests rarely need to hand-roll service setup. The dbtesting package’s AssertInitialize / VerifyInitialize split (one fatals, one returns error) is a thoughtful API that lets callers choose error-handling policy.
    • Table-driven saturation: 850–964 table-driven test instances means edge cases are systematically enumerated rather than scattered across ad-hoc test functions. The map[string]struct{} style for parallel subtests is a well-known best practice that Vault applies correctly.
    • corehelpers import-cycle discipline: separating test helpers that don’t import vault (corehelpers) from those that do (testhelpers/logical) allows core unit tests to use the helpers without a circular dependency — a non-trivial design challenge in a self-referential system.
    • Security-relevant test patterns: tests for audit-before-action ordering, NonFatalError startup degradation, and barrier view path-traversal prevention show that security properties are first-class test targets, not afterthoughts.
  • What could improve:
    • t.Cleanup adoption is incomplete: 85 uses vs. 1,436 parallel tests suggests many tests still use defer in ways that can misfire with subtests.
    • No build-tag separation for service-dependent tests: tests that need PostgreSQL or AWS silently skip (or fail) when services are absent, rather than being explicitly gated with a //go:build integration tag. This makes CI configuration harder to reason about.
    • testify is limited in scope: only 362 test files use testify; the majority still rely on t.Fatal / t.Errorf + manual assertions. The mixed style means error messages are inconsistently formatted across the codebase.
    • Mock/fake coverage is ad hoc: without a generation tool, fakes are maintained manually and may lag interface changes. No evidence of systematic mock coverage tracking.
  • Patterns worth emulating:
    • The three-tier integration strategy (in-process → Docker → blackbox) scales gracefully: cheap tests stay cheap, realistic tests are available when needed.
    • RetryUntil for async assertions is a clean alternative to time.Sleep-based polling and eliminates flaky timing in concurrent tests.
    • map[string]struct{} table-driven + t.Parallel() inside subtests for truly independent test cases — the gold-standard parallel subtest pattern.
    • dbtesting.VerifyInitialize vs AssertInitialize — exposing both a “returns error” and “fatals on error” variant of the same assertion lets test authors choose the appropriate failure mode.