Moby — Testing#

Sampling strategy#

Moby is an XL project. Testing analysis covered: the full internal/testutil/ helper tree, both test entrypoints (integration/container/main_test.go, integration-cli/check_test.go), representative unit tests (client/container_create_test.go, client/client_mock_test.go), a representative integration test (integration/container/create_test.go), the custom suite runner (internal/test/suite/suite.go), and CI workflows (.github/workflows/.test-unit.yml, .github/workflows/test.yml). Patterns result was read first.

Test metrics#

• Test files: 770
• Source files (non-test): 1,375
• Ratio (test/source): ~0.56 — high for an infrastructure project
• Test functions (func Test*): 2,292
• Table-driven test occurrences: 1,214 (t.Run, tc.name, testCases, etc.)
• Test frameworks: gotest.tools/v3 exclusively (not testify); stdlib testing; custom suite runner

Test organization#

Placement#

Both same-package and _test-package tests are used. Client unit tests use package client (white-box, can access unexported helpers). Integration tests use package container, package networking, etc. — always the _test package, keeping them as black-box API tests.

Three test tiers#

Tier 1 — Unit tests (client/, daemon/, individual packages)

• Live alongside source files
• No running daemon required
• Use mock HTTP transport to simulate the Docker API over a fake RoundTripper
• Fast; expected to pass on any platform

Tier 2 — Integration tests (integration/)

• Organized by domain: integration/container/, integration/networking/, integration/image/, integration/daemon/, etc.
• Require a running dockerd (local or remote, controlled by env var DOCKER_HOST)
• Each subdomain has its own main_test.go with OTel setup and environment.Execution initialization
• Platform-conditioned with build tags (//go:build !windows)

Tier 3 — Legacy CLI integration tests (integration-cli/)

• Old-style tests that run the docker CLI binary via icmd.RunCommand
• Use a deprecated internal/test/suite runner (custom reflection-based test suite)
• Comments in the code explicitly label this package as “legacy” and note that new tests should go in integration/
• The checker/ sub-package is similarly marked for removal (// Please remove this package whenever possible)

Helper packages#

internal/testutil/ — the primary test infrastructure library:

internal/test/suite/ — simplified testify/suite replacement (used only by integration-cli/):

• Reflection-based: finds all methods starting with Test on a struct receiver
• Lifecycle: SetUpSuite, SetUpTest, TearDownTest, TearDownSuite interfaces
• Integrates with OTel spans per test via testutil.StartSpan
• Comment: “Please remove this package whenever possible”

Test patterns#

Table-driven tests#

• Prevalence: Very heavy — 1,214 occurrences in test files
• Style: Anonymous struct slice with named doc/name field:

  testCases := []struct {
      doc           string
      image         string
      expectedError string
  }{
      {doc: "image and tag", image: "test456:v1", expectedError: "No such image: test456:v1"},
      {doc: "image no tag", image: "test456",     expectedError: "No such image: test456"},
  }
  for _, tc := range testCases {
      t.Run(tc.doc, func(t *testing.T) {
          t.Parallel()
          // ...
      })
  }

  • File reference: integration/container/create_test.go:40–73
• Assessment: Best-practice style. t.Parallel() inside subtests is used consistently in integration tests to maximize parallelism.

Mock HTTP transport (client unit tests)#

• Strategy: The client/ package defines a testRoundTripper type (exported from the production package) and a WithMockClient(func(*http.Request) (*http.Response, error)) Opt option for test use only.
• Pattern:

  client, err := New(
      WithMockClient(func(req *http.Request) (*http.Response, error) {
          // Assert URL, method, body
          return mockJSONResponse(http.StatusOK, nil, container.CreateResponse{ID: "abc"})(req)
      }),
  )

  Generic helper: mockJSONResponse[T any](statusCode, headers, resp T) marshals resp to JSON and returns a well-formed *http.Response. No network required.
• File reference: client/client_mock_test.go:50–155
• Assessment: Elegant. The testRoundTripper type is defined in production code (client_options.go:144) as an unexported type, so tests in package client can use it freely. This avoids a separate mock-generation step entirely.

Mocking approach (integration tests — no mock framework)#

• Strategy: Real dockerd subprocess + testutil/daemon.Daemon helper. No gomock, no mockery.
• How: Daemon.Start(t) launches a real dockerd binary on a temporary socket; Daemon.Stop(t) shuts it down. Tests interact via the production client.Client.
• Example:

  d := testdaemon.New(t)
  d.Start(t)
  defer d.Stop(t)
  c := d.NewClientT(t)
  // now do real API operations

• Assessment: Higher fidelity than mocks — catches real daemon bugs. The trade-off is test speed and platform requirements. The daemon helper handles cleanup via t.Cleanup.

OTel tracing in tests#

• Unique to Moby: Every integration test wraps its execution in an OpenTelemetry span.
• Pattern:

  func TestMain(m *testing.M) {
      shutdown := testutil.ConfigureTracing()
      ctx, span := otel.Tracer("").Start(context.Background(), "integration/container/TestMain")
      // ...
      span.End()
      shutdown(ctx)
  }
  
  func setupTest(t *testing.T) context.Context {
      ctx := testutil.StartSpan(baseContext, t)
      // ...
      return ctx
  }

  Each t.Run subtest gets its own span; on failure the span status is set to Error. Spans are exported via OTLP if OTEL_EXPORTER_OTLP_ENDPOINT is set.
• Assessment: Forward-looking and unique. Enables distributed tracing of CI test runs, making it possible to debug flaky tests and slow subtests with a trace viewer (Jaeger, Tempo). This goes well beyond typical Go test infrastructure. Worth noting as an emerging best practice.

Polling for async state#

• Usage: 71 occurrences of poll.WaitOn from gotest.tools/v3/poll
• Pattern:

  poll.WaitOn(t, func(log poll.LogT) poll.Result {
      containers, err := apiClient.ContainerList(ctx, ...)
      if err != nil { return poll.Error(err) }
      if len(containers) == expected { return poll.Success() }
      return poll.Continue("waiting for %d containers", expected)
  }, poll.WithTimeout(10*time.Second))

• Assessment: Correct approach for testing eventually-consistent container state. Avoids time.Sleep races. The poll.WithTimeout ensures tests fail clearly rather than hanging.

Golden file tests#

• Usage: 11 occurrences of golden.Assert from gotest.tools/v3/golden
• Used for: Network configuration output (iptables/nftables rules, /etc/hosts content, iptables documentation generation)
• Pattern:

  golden.Assert(t, getEtcHosts(), "TestEtcHostsDisconnect1.golden")

  Golden files live in integration/networking/testdata/ and integration/network/bridge/*/generated/.
• Assessment: Appropriate for complex text output where char-by-char comparison is the right check. Updating goldens requires a flag; not overused.

Platform skipping#

• Usage: 132 occurrences of skip.If from gotest.tools/v3/skip plus build tags
• Pattern:

  skip.If(t, testEnv.GitHubActions())  // skip flaky test in CI
  skip.If(t, runtime.GOOS == "windows")
  //go:build !windows        // build tag for whole file

• Assessment: The dual approach (build tags for whole files, skip.If for individual tests) is slightly inconsistent but pragmatic. The build-tag approach prevents compilation failures on Windows for tests that use Linux-only syscalls.

Integration tests#

Present#

Yes — integration/ is the primary and integration-cli/ is the legacy integration test location. Together they contain the majority of the 770 test files.

How they work#

Tests connect to a real Docker daemon (either local or pointed to by DOCKER_HOST). The environment.Execution struct negotiates the API version with the daemon and determines capabilities:

// environment.go — auto-detected per test run:
type Execution struct {
    client              client.APIClient
    DaemonInfo          system.Info   // OS, kernel, storage driver
    DaemonMinAPIVersion string
    PlatformDefaults    PlatformDefaults
    protectedElements   protectedElements  // images/containers to preserve
}

protectedElements prevents test cleanup from deleting images/containers that were present before the test suite started.

Separation#

• By directory: integration/ vs integration-cli/
• Within integration/: by domain (container/, networking/, daemon/, build/, image/, etc.)
• Build tags for platform: //go:build !windows, //go:build linux
• No //go:build integration tag — integration tests are separated by directory, not tag

CI workflow#

• Unit tests: Containerized, run with docker buildx bake; matrix: ["", "firewalld"]
• Integration tests: Require a VM with dockerd installed; use TEST_INTEGRATION_DIR env var to select domains
• Platforms: Linux (Ubuntu 24.04), Windows (2022, 2025), ARM64 — separate workflow files

Test quality observations#

What’s done well#

1. Rich test helper ecosystem. internal/testutil/ is a first-class library with 15+ sub-packages covering daemon lifecycle, fake registries, fake git repos, fake build contexts, and network namespace sanity checks. Teams clearly invest in test infrastructure.

2. OTel tracing in tests. The integration of OpenTelemetry spans into the test harness is architecturally notable. When OTEL_EXPORTER_OTLP_ENDPOINT is set, CI runs produce distributed traces that show exactly which test and subtest caused slowness or failure. This closes the feedback loop between CI observability and test debugging — a practice rare in the Go ecosystem.

3. No mock framework dependency. Client tests use hand-written RoundTripper mocks. Integration tests use a real daemon. This means test failures point to real bugs, not mock divergence. The avoidance of gomock/mockery is deliberate and pays off at this project’s scale.

4. gotest.tools/v3 is a better fit than testify. Moby is the primary user of gotest.tools (it’s the same maintainer ecosystem — Aaron Lehmann and Sebastiaan van Stijn are both Docker contributors). The assert.Check (non-fatal) vs assert.Assert (fatal) distinction enforces a discipline that testify’s require vs assert naming obscures.

5. t.Parallel() discipline. Integration tests consistently call t.Parallel() inside subtests. Combined with the CheckNotParallel helper in testutil, the project enforces the distinction between tests that must be serialized (daemon lifecycle) and those that can parallelize.

6. Protected elements prevent test pollution. The environment.Execution.protectedElements mechanism prevents cleanup from deleting images/containers pre-existing in the daemon — crucial when running tests against a shared daemon in CI.

What could improve#

1. Legacy integration-cli/ is a maintenance burden. The package runs Docker CLI commands via icmd.RunCommand, coupling tests to CLI output format rather than the API. The code is explicitly marked for migration, but as of this analysis it still contains hundreds of tests that have not been migrated to integration/. This creates two parallel test suites testing overlapping functionality.

2. No inter-tier consistency enforcement. The two test tiers (unit mock transport, real daemon integration) can drift: a client API method test might pass because the mock returns the right JSON, while the actual daemon behavior differs. There’s no intermediate contract test layer.

3. Test context management is complex. The testContextStore (map of TestingT → context.Context) managed by SetContext/GetContext/CleanupContext adds global mutable state to the test harness. The CheckNotParallel guard is a workaround for the fact that this approach is unsafe under t.Parallel() in some patterns.

4. sync.Once in ConfigureTracing with multiple calls. The tracingOnce.Do(...) in helpers.go combined with the fallback tp = otel.GetTracerProvider().(*trace.TracerProvider) is a type assertion that will panic if a non-TracerProvider was registered. This is a fragile pattern in a shared test helper.

Patterns worth emulating (for the book)#