Buildkite Agent — Testing#

Test metrics#

  • Test files: 120
  • Source files (non-test): ~249
  • Ratio (test files / source files): ~0.48 (roughly 1:2)
  • Test frameworks: testing (stdlib), github.com/stretchr/testify (assert/require), gotest.tools/v3/assert, github.com/buildkite/bintest/v3, net/http/httptest

Test organization#

  • Placement: Primarily same-package (white-box) tests; ~27 files use external _test packages (e.g. api_test, agent_test) for black-box testing of exported APIs
  • Helper packages:
    • internal/job/integration/ — ExecutorTester helper that spawns the actual bootstrap subprocess via exec.Cmd and uses bintest to intercept external binary calls; includes gitRepository helper for in-process git repos
    • internal/e2e/ — end-to-end framework that connects to a real Buildkite API; testcase.go wraps agent lifecycle, job triggering, and result fetching
    • internal/replacer/testdata/ — fixture files for the streaming redactor tests
  • Fixtures: Minimal; mostly generated inline (shell scripts written to os.MkdirTemp); testdata/ only present for the replacer package; e2e test cases reference YAML fixture files

Test patterns#

Table-driven tests#

  • Prevalence: Heavy — 150 instances of t.Run, testCases, tt.Run, or tc.name patterns across 120 test files
  • Style: Named anonymous struct slices — for _, test := range []struct{ name string; ... }{ {...}, {...} } followed by t.Run(test.name, func(t *testing.T) { ... })
  • Example: api/secrets_test.go:18 — table of success/failure scenarios for GetSecret, each with distinct HTTP status codes, tokens, and expected errors; agent/integration/job_runner_integration_test.go:25 — table-driven pre-bootstrap hook test with platform-specific cases appended at runtime

Mocking approach#

  • Strategy: Layered, with different tools at different abstraction levels:
    1. Interface fakes (unit level): Each package defines its own narrow APIClient interface and tests supply a hand-written struct implementing only the needed methods (e.g. internal/artifact, internal/secrets). No mock generation framework.
    2. httptest.NewServer (API level): 38 usages. Tests for the api/ package spin up a real HTTP server with a custom handler; the api.Client under test hits the real network stack against 127.0.0.1. This catches serialization bugs and HTTP header issues that interface mocks would miss.
    3. bintest (subprocess level): Integration tests use github.com/buildkite/bintest/v3 — Buildkite’s own binary proxy library. tester.MustMock(t, "git") creates a real executable on $PATH that intercepts OS-level subprocess invocations and feeds responses from within the test process. Supports Expect().Once(), WithAnyArguments(), AndCallFunc(...), and PassthroughToLocalCommand().
  • Example: internal/job/integration/hooks_integration_test.go:50:
    git := tester.MustMock(t, "git").PassthroughToLocalCommand().Before(func(i bintest.Invocation) error {
    return bintest.ExpectEnv(t, i.Env, "MY_CUSTOM_ENV=1", "LLAMAS_ROCK=absolutely")
})
git.Expect().AtLeastOnce().WithAnyArguments()
    This lets the real git command run while also asserting that specific environment variables were propagated to the subprocess.

Integration tests#

  • Present: Yes — two layers
  • Layer 1 — Bootstrap integration (internal/job/integration/, agent/integration/):
    • 14 *_integration_test.go files covering hooks, plugins, redaction, checkout, git mirrors, artifact upload, Docker, job API socket, config allowlisting, job verification, and environment propagation
    • Run with go test ./internal/job/integration/... — no build tag required; these run locally against the OS
    • TestMain in internal/job/integration/main_test.go starts a bintest.StartServer() for all tests to share, then re-runs itself as a buildkite-agent bootstrap binary for the subprocess to call back into
    • ExecutorTester.RunAndCheck(t, ...) shells out to the real bootstrap binary; tests validate actual environment variable propagation, hook execution order, and output content
    • Platform-aware: Windows conditional paths for .bat/.ps1 hooks, runtime.GOOS guards, UDS path-length workaround (/tmp instead of system temp on Linux for socket names ≤ 108 chars)
  • Layer 2 — E2E (internal/e2e/):
    • Gated by //go:build e2e build tag; requires a real Buildkite API token and registered cluster
    • TestMain authenticates via api.Client.GetTokenIdentity, discovers org/cluster slugs
    • Tests start a live agent process (tc.startAgent()), trigger a real build (tc.triggerBuild()), poll for completion with a context timeout, and fetch/assert logs
    • Dockerfile-e2e and pipeline.e2e.yml show these run in CI on actual Buildkite infrastructure
  • Separation: Integration tests live in integration/ subdirectories under both internal/job/ and agent/; e2e tests are isolated in internal/e2e/ with a //go:build e2e tag

Test quality observations#

  • What’s done well:

    • Three-tier test pyramid is explicit and intentional: unit tests with interface fakes → bootstrap integration with bintest → live e2e. Each tier tests a different failure mode; the layers don’t duplicate each other’s concerns.
    • bintest is architecturally clever: Because the agent’s most critical behavior is what environment variables and arguments reach external processes (git, ssh, docker), mocking at the OS binary level — rather than the Go interface level — catches an entire class of bugs invisible to interface mocks. The pattern of PassthroughToLocalCommand().Before(assertEnvFunc) combines real behavior with assertion, avoiding the need to stub return values.
    • TestMain dual-mode trick in integration tests: main_test.go detects whether it’s being invoked as a test runner or as a buildkite-agent subprocess. This allows the test binary to serve as the bootstrap executable itself, eliminating the need to pre-build a separate binary for integration tests.
    • Heavy use of t.Parallel(): Most integration tests call t.Parallel() at the top, making full use of Go’s parallel subtest support. Combined with bintest’s shared server, this means dozens of real subprocess invocations run concurrently without port conflicts.
    • httptest.NewServer for API layer: API tests avoid interface mocks entirely; the real api.Client serialization/deserialization is exercised on every test run.
    • Platform coverage: Windows paths are explicitly handled in integration tests with runtime.GOOS guards and appended test cases, not hidden behind a separate CI-only job.

  • What could improve:

    • No mock generation: All interface fakes are hand-written. For the current set of narrow interfaces this is fine, but as the APIClient interface in core/ grows it will require manual maintenance.
    • ExecutorTester setup is heavyweight: Creating a temp home directory, git repo, and multiple mock processes for each test is expensive. Tests rely on t.Parallel() to amortize this, but there is no shared fixture or TestMain-level setup for the heavier filesystem work.
    • e2e tests require live credentials: There is no recorded-response or contract-test alternative, so e2e tests cannot run in forks or offline. A cassette-based approach (e.g. go-vcr) could enable offline replay for a subset of scenarios.
    • Limited assertion libraries: The mix of testify/assert, gotest.tools/v3/assert, and raw if got != want comparisons is inconsistent. gotest.tools is used in integration tests and testify in unit tests, with no apparent policy.

  • Patterns worth emulating:

    • bintest-style OS-level binary mocking for systems that shell out to external processes — far more realistic than interface mocks for CLI agents, CI runners, and build tools.
    • TestMain dual-mode binary in integration tests — allows the test binary to impersonate the real binary without a separate build step, keeping integration tests self-contained and fast to iterate.
    • Narrow per-package interface fakes — each package defines only the API methods it consumes; tests supply minimal implementations. This keeps test code small and focused on the package under test.
    • httptest.NewServer + real client for HTTP API testing — exercises the full serialization stack without infrastructure, catching JSON field name mismatches, header handling, and status code parsing that interface mocks miss entirely.
Backward Buildkite Agent — API Surface restic — Overview Forward