Buildkite Agent — Patterns#

Concurrency patterns#

Worker Pool#

  • Usage: AgentPool manages N parallel AgentWorker goroutines with sync.WaitGroup
  • Example: clicommand/agent_start.go:1536var wg sync.WaitGroup loops over workers, wg.Add(1) / go func() { defer wg.Done(); worker.Run() }()
  • Assessment: Idiomatic and effective. Pool size is user-configured (number of concurrent agents). The pool waits for all workers before shutdown.

Bounded Fan-out with Semaphore#

  • Usage: internal/secrets/secret.go:71 — parallel secret key fetching limited by a weighted semaphore from golang.org/x/sync/semaphore
  • Example: sem := semaphore.NewWeighted(int64(concurrency)) → goroutines per key → sem.Acquire(ctx, int64(concurrency)) to barrier-wait all goroutines
  • Assessment: Clean use of the weighted semaphore as both a goroutine limiter and a barrier. Context-aware (sem.Acquire respects cancellation).

Pipeline Processing (Streaming I/O)#

  • Usage: Bootstrap subprocess stdout → pipe → process.BufferLogStreamer → Buildkite API chunks, with internal/redact sitting in the pipeline as an io.Writer wrapper
  • Example: clicommand/kubernetes_bootstrap.go:199io.MultiWriter(os.Stdout, socket) for simultaneous local and remote output; internal/job/integration/executor_tester.go:293io.MultiWriter(buf, w)
  • Assessment: Well-designed pipeline. The redactor plugs in transparently as an io.Writer, enabling streaming secret redaction without buffering the full output.

Context Cancellation (Pervasive)#

  • Usage: 344 usages of context.Context across the codebase; 71 select {} blocks
  • Example: process/process.go:256case <-ctx.Done(): // kill subprocess; clicommand/agent_start.go:811ctx, cancel := context.WithCancel(ctx) for per-worker cancellation
  • Assessment: Consistent and correct. Contexts propagate from the CLI entry point down through workers to subprocess execution. Cancellation cleanly terminates the subprocess hierarchy.

Graceful Shutdown (Signal + Context)#

  • Usage: signal.Notify for SIGTERM/SIGINT → calls AgentPool.StopGracefully() or StopUngracefully()
  • Example: clicommand/agent_start.go:1418signal.Notify(signals, syscall.SIGTERM, ...) with separate goroutine handling the signal
  • Assessment: Distinguishes graceful (wait for current job to finish) vs ungraceful (kill job immediately) shutdown modes, each triggered by different signals or repeat signal presses.

Done Channel with sync.Once (Safe Close)#

  • Usage: Used in AgentWorker, process.Process, kubernetes.Runner to signal completion without double-close panics
  • Example: agent/agent_worker.go:98stopOnce sync.Once; kubernetes/runner.go:72doneOnce, interruptOnce sync.Once; close(a.stop) guarded by stopOnce.Do(...)
  • Assessment: Idiomatic pattern for multi-producer/single-consumer done signals. Correctly uses sync.Once to prevent the double-close panic.

Debouncer (Custom Coordination Primitive)#

  • Usage: agent/agent_worker_debouncer.go — event-debouncing loop between the streaming loop and the action handler loop
  • Example: The debouncer collapses consecutive streaming events (pause/resume/pause) into a single correct final action, rather than passing all events to the action handler and risking stale-state ordering bugs
  • Assessment: Sophisticated and well-documented with clear comments explaining why naive forwarding would fail. This is a project-specific pattern that solves a real distributed systems problem (at-least-once delivery with last-write-wins semantics).

Baton (Mutual Exclusion Between Ping Modes)#

  • Usage: agent/baton.go and agent/agent_worker_debouncer.go — a custom “baton” struct coordinates between the streaming ping loop and the traditional HTTP poll loop
  • Example: The streaming loop “holds the baton” while healthy, blocking the ping loop from running. When the stream is unavailable, the baton is released, allowing the poll loop to take over.
  • Assessment: Inventive custom primitive for graceful mode-switching between two acquisition strategies. More explicit than a mutex or atomic flag; the baton holder is named, which aids debugging.

Rate Limiting via time.Ticker#

  • Usage: agent/run_job.go:501 — jitter-based ticker for job cancellation checking; agent/agent_worker_ping.go:100 — ping ticker for HTTP polling interval
  • Example: rejitterTicker := time.Tick((runLength + 1) * processInterval) — re-jitters the polling interval dynamically based on the job’s run length to spread API load
  • Assessment: Simple and effective for the use case. Re-jittering based on run duration is an interesting technique to prevent thundering herd when many jobs finish at similar times.

Error handling#

  • Style: Mixed, with fmt.Errorf %w as the dominant approach; errors.Is/errors.As used for typed error checking; errors.New for simple sentinel errors
  • Error types defined:
    • clicommand/errors.goExitError (carries exit code for process exit), SilentExitError (suppresses output)
    • internal/shell/shell.go:671ExitError (shell-level exit code)
    • internal/secrets/secret.go:27SecretError (secret fetch failures)
    • api/client.go:348ErrorResponse (API HTTP error with status code)
    • internal/socket/client.go:16ErrorResponse (socket protocol error)
    • clicommand/pipeline_upload.go:727-758gitRevParseError, gitLogError, gitMergeBaseError, gitDiffError (git operation errors for structured retry logic)
  • Wrapping approach: fmt.Errorf("context: %w", err) throughout; errors.As used to unwrap ErrorResponse for HTTP status checks (api/client.go:367)
  • Examples:
    • api/pings_streaming.go:21return nil, fmt.Errorf("parsing endpoint: %w", err) — contextual wrapping
    • api/secrets_test.go:134errors.As(err, &aerr) — typed unwrapping for API error handling

Retry Pattern (buildkite/roko)#

  • Library: github.com/buildkite/roko (Buildkite’s own retry library)
  • Usage: Consistent across API calls, annotations, job updates, OIDC token requests
  • Example: clicommand/annotate.go:173roko.NewRetrier(roko.WithMaxAttempts(5), roko.WithStrategy(roko.Constant(1*time.Second)), roko.WithJitter()).DoWithContext(ctx, ...)
  • Strategies used: Constant, ExponentialSubsecond, WithJitter
  • Assessment: The project uses its own retry library (roko) rather than generic alternatives. This gives consistent retry behavior across all API-touching code, with context support and configurable jitter.

Configuration pattern#

  • Approach: Config struct per command with CLI flag struct tags + env var binding; functional options for internal components
  • Example (command config): clicommand/agent_start.goAgentStartConfig struct with fields bound to CLI flags and BUILDKITE_AGENT_* env vars via urfave/cli; setupLoggerAndConfig[T]() generic helper normalizes the config
  • Example (functional options): internal/agenthttp/client.go:62-67WithAuthBearer, WithAuthToken, WithAllowHTTP2, WithTimeout, WithTLSConfig options for the HTTP client; internal/shell/shell.go:93WithCommandLog for the Shell
  • Multi-source priority: CLI flags > env vars > .cfg file (INI format) > defaults
  • Config-to-env bridge: JobRunner.createEnvironment() translates AgentConfiguration struct fields into BUILDKITE_* env vars for the bootstrap subprocess, which then decodes them back into BootstrapConfig — effectively using the OS environment as an IPC channel between the two processes

Dependency injection#

  • Approach: Manual constructor injection; no DI framework
  • Evidence: agent_start.go wires everything explicitly: api.ClientAgentWorkerJobRunner. All components receive dependencies as constructor arguments or *Config structs. The core/ package defines an APIClient interface enabling test substitution.
  • Assessment: Appropriate for the project’s scale. The lack of a DI framework keeps startup code readable and avoids reflection-based magic. The single APIClient interface in core/ is the main seam for testing.

Other notable patterns#

Feature Flag Registry (Experiment System)#

  • internal/experiments/experiments.go — string-keyed experiment registry with three states: known, promoted, unknown
  • Experiments enabled via BUILDKITE_AGENT_EXPERIMENT env var; checked via experiments.IsEnabled(ctx, key) at decision points
  • Promoted experiments become permanent features; unknown experiments log a warning
  • Elegant lifecycle management for incremental feature rollout

Streaming Redactor (Aho-Corasick-style Multi-pattern Filter)#

  • internal/redact/redact.go and internal/replacer/replacer.go — an io.Writer wrapper that performs streaming multi-pattern string replacement
  • Secrets are registered at job start and can be added dynamically during execution
  • Sits in the log pipeline between the bootstrap process and the API uploader
  • Assessment: Architecturally clean — plugs into the io.Writer pipeline without requiring buffering or special integration

Table-Driven Tests#

  • Prevalence: Heavy (139 usages of t.Run, testCases, tt.Run patterns)
  • Style: Named anonymous struct slices; both t.Run(tc.name, ...) and inline subtests
  • Example: api/secrets_test.go — table of expected errors and responses for secret fetching scenarios

Modern Go Iterators (iter.Seq2)#

  • api/pings_streaming.go:16func (c *Client) StreamPings(...) (iter.Seq2[*agentedgev1.StreamPingsResponse, error], error)
  • Uses Go 1.23 range-over-function iterators for the gRPC/Connect streaming ping response
  • Assessment: Forward-looking adoption of a new stdlib pattern; makes streaming iteration look like a for-range loop at the call site

Interface-as-Local-Seam#

  • Small, focused interfaces defined next to their consumers: internal/artifact/api_client.go:APIClient, internal/secrets/secret.go:APIClient, internal/cache/cache.go:CacheClient, core/client.go:APIClient
  • Each package defines its own narrow interface for the API subset it uses, rather than sharing one large interface
  • Assessment: Excellent application of the Interface Segregation Principle; each interface has exactly the methods its consumer needs, making mocking and testing straightforward