Nomad — Testing#

Test metrics#

  • Test files: 814 *_test.go files
  • Total Go files: 2,129 (including test files)
  • Ratio (test files / source files): ~62% (814 / ~1315 non-test source files) — very high
  • Test frameworks:
    • github.com/shoenig/test and github.com/shoenig/test/must — primary assertion library in newer code (787 occurrences)
    • github.com/stretchr/testify/require and testify/assert — legacy assertions, still widespread
    • github.com/shoenig/test/wait — async polling assertions
    • stdlib testing — universal base
    • No gomock, ginkgo, or goconvey

Test organization#

  • Placement: Both same-package (package nomad) and _test-suffixed external packages (package nomad_test). The dominant style is same-package testing, giving tests access to unexported types — appropriate for testing internal state machines like the eval broker and scheduler.
  • Helper packages:
    • testutil/ (top-level) — WaitForResult/Wait polling helpers, TestMultiplier() for CI-aware timing, TestServer (fork/exec a real nomad binary), TLS cert generation, Vault test stubs
    • nomad/testing.go — in-process TestServer(t, cb), TestACLServer(t, cb), TestConfigForServer(t) for spinning up a full Nomad server inside the test process; used by all nomad/ package tests
    • nomad/mock/ — rich struct factory package: mock.Job(), mock.Alloc(), mock.Node(), mock.Eval(), mock.ACLManagementToken(), HCL snippets. Essentially a domain-object DSL for tests.
    • api/internal/testutil/ — separate NewTestServer(t, cb) that forks a real binary (avoiding the import cycle between api/ and the server packages)
    • client/testutil/ — Docker availability checks, driver-compatible skip helpers, RPC test utilities
    • drivers/mock/ — full DriverPlugin implementation used as a stand-in driver in integration tests
    • client/serviceregistration/mock/ — ServiceRegistrationHandler implementing the service registration interface for unit testing task runners
    • plugins/csi/fake/ — fake CSI plugin implementation for storage plugin tests
    • client/allocrunner/taskrunner/testing/ — task runner test stubs
    • internal/testing/apitests/ — API integration test helpers
  • Fixtures: testdata/ directories in command/, drivers/docker/, helper/tlsutil/, helper/snapshot/, and client/state/ hold HCL configs, TLS certs, job spec files, and BoltDB snapshots

Test patterns#

Table-driven tests#

  • Prevalence: Heavy — 1,352 matches for testCases, testCases :=, tt.Run, tc.name across test files
  • Style: Anonymous struct slice with a name string field and subtest invocation via t.Run(tc.name, ...). Named-struct style is also common for fingerprinter tests where expected output varies per environment.
  • Example: client/fingerprint/network_test.go:285 — testCases := []struct { name string; ... } with for _, tc := range testCases { t.Run(tc.name, ...) }; scheduler/feasible/feasible_test.go — inline struct slices for constraint evaluation permutations

ci.Parallel pattern#

  • Prevalence: Extremely pervasive — 4,272 occurrences of ci.Parallel(t) at the top of test functions
  • Mechanism: ci/slow.go wraps t.Parallel() with a check for the CI environment variable. When running in CI (GitHub Actions, CircleCI), parallelism is suppressed in favor of serial execution with unrestricted GOMAXPROCS. Locally, tests run in parallel as usual.
  • Slow test gating: ci.SkipSlow(t, reason) skips tests unless NOMAD_SLOW_TEST=1, separating expensive tests (full cluster bootstraps, multi-second waits) from the fast path
  • Assessment: A sophisticated and well-thought-out approach. CI environments get throughput from GOMAXPROCS; local development gets latency from parallelism. This is a pattern worth emulating.

Mocking approach#

  • Strategy: Interface-based hand-rolled fakes — no code-generation tools like mockery or gomock. Each subsystem provides its own mock/ or fake/ package with concrete implementations of the relevant interfaces.
  • Examples:
    • drivers/mock/ — mock.Driver implements the full drivers.DriverPlugin interface, configurable via task config HCL to simulate blocking starts, kill delays, crash behaviors
    • client/serviceregistration/mock/ — records all RegisterWorkload/DeregisterWorkload calls in a slice for assertion; supports injecting error responses via function fields
    • plugins/csi/fake/ — CSI plugin fake with configurable error injection for each RPC
    • nomad/mock/ — struct factories returning fully-populated domain objects, not behavioral mocks
  • Philosophy: The codebase clearly prefers hand-written fakes over generated mocks. Fakes are full implementations that can be configured for test scenarios, which catches more real integration problems than stub-based mocking.

Integration tests#

  • Present: Yes — three distinct levels
  • Level 1 — In-process server: nomad.TestServer(t, cb) creates a fully functional Nomad server (Raft, state store, RPC) inside the test process. Tests in nomad/ call real RPC handlers against a real in-memory state store. These are technically unit tests by build tooling but are functionally integration tests.
  • Level 2 — Fork/exec binary: testutil.NewTestServer(t, cb) in the top-level testutil/ and api/internal/testutil/ packages forks an actual nomad binary as a subprocess, writes a temp config, and waits for it to become healthy. Used by api/ package tests to avoid the import cycle with server internals. Tears down via t.Cleanup.
  • Level 3 — e2e cluster: e2e/ directory contains 68 TestXxx functions organized into ~35 feature subdirectories (acl, cni, connect, csi, deployment, docker, etc.). These require a running Nomad cluster (not spun up by the test itself). e2eutil.NomadClient(t) connects via the API; e2eutil.WaitForLeader(t, ...) and WaitForNodesReady(t, ...) gate execution until the cluster is ready.
  • Separation: e2e tests are in a separate top-level directory (e2e/), not mixed with unit tests. No build tags are used to separate unit from integration — the distinction is by directory and test runner invocation.

WaitForResult pattern#

  • Prevalence: Pervasive in async tests — testutil.WaitForResult(testFn, errorFn) retries testFn every 10ms for 500 * TestMultiplier() iterations (default 5 seconds in CI).
  • Assessment: Cleaner than time.Sleep. TestMultiplier() reads NOMAD_TEST_SLOWNESS env var to scale all timeouts uniformly. The newer code uses shoenig/test/wait which offers a fluent API for the same purpose.

Testify → shoenig/test migration#

  • Both testify/require (classic) and shoenig/test/must (newer) coexist. Newer tests and packages heavily prefer must.NoError(t, err), must.Eq(t, expected, actual) over require.NoError. The migration appears intentional — shoenig/test provides more structured output and composable Option error messages via must.Sprint(...).

Test quality observations#

  • What’s done well:

    • nomad/mock/ as a DSL: The factory package produces realistic, fully-populated domain structs. Tests read naturally because mock.Job() returns something that actually represents a valid job, not a zero-value stub. This pattern — a dedicated fixture factory — is one of the most valuable things about Nomad’s test infrastructure.
    • In-process full-server testing: Running real Raft, real state store, and real RPC handlers inside unit tests catches integration bugs that purely mocked tests would miss. The TestServer(t, cb) pattern with a callback for config customization is ergonomic and flexible.
    • Conditional parallelism (ci.Parallel): Thoughtful CI/local trade-off. 4,272 uses shows this is a genuine discipline, not an afterthought.
    • TestMultiplier / timing multiplier: Centralizing timing sensitivity in one env var (NOMAD_TEST_SLOWNESS) is much better than ad-hoc time.Sleep calls. All wait loops scale proportionally.
    • Fake drivers with configurable behavior: drivers/mock/Driver can be configured to block, crash, or delay via task config HCL — enabling realistic driver lifecycle testing without a real container runtime.

  • What could improve:

    • Two assertion libraries: Simultaneous use of testify/require and shoenig/test/must is a coherence cost. New contributors must know both. A full migration to one would reduce friction.
    • e2e test infrastructure dependency: The e2e/ tests implicitly require a running cluster but this is not enforced by a build tag — tests will fail with cryptic connection errors if run against no cluster. An explicit build tag (e.g., //go:build e2e) would make the dependency explicit.
    • No mock generation: Hand-written fakes are thorough but expensive to maintain when interfaces evolve. Several mock types (e.g., ServiceRegistrationHandler) duplicate the interface signature manually. go generate + mockery for the simplest interfaces could reduce boilerplate.

  • Patterns worth emulating:

    1. mock/ as a domain fixture factory — separating “create a realistic object” from test logic; reused across hundreds of tests
    2. ci.Parallel(t) pattern — environment-aware parallelism that respects CI resource constraints
    3. TestServer(t, cb) with cleanup callback — in-process full-stack server with t.Cleanup(func(){...}) teardown; no global state, each test gets its own server
    4. WaitForResult / TestMultiplier for async assertions — uniform timing scaling without scattered time.Sleep calls
Backward Nomad — API Surface Dapr — Overview Forward