Buildkite Agent — Architecture#

Architectural style#

Two-process, layered agent with a pluggable execution backend.

The system is architecturally split into two cooperating processes:

  1. Agent process (buildkite-agent start): long-running daemon that registers with Buildkite’s SaaS API, polls/streams for jobs, and orchestrates one or more worker goroutines.
  2. Bootstrap process (buildkite-agent bootstrap): short-lived subprocess spawned per job; executes the actual job phases (plugin download, git checkout, user commands) in complete isolation from the agent process.

This two-process design is a deliberate security and reliability boundary: the bootstrap process runs with the job’s environment, can be killed independently, and logs its output back to the parent via a pipe. The agent process survives bootstrap crashes.

Within each process, the code is layered: CLI wiring → orchestration → execution engine → support libraries.

Component diagram (textual)#

┌──────────────────────────────────────────────────────────────────┐
│                       Agent Process                              │
│                                                                  │
│  main.go                                                         │
│    └─▶ clicommand/commands.go  (urfave/cli command registry)     │
│           └─▶ AgentStartCommand.Action()                         │
│                  │  reads config (flags + env + .cfg file)       │
│                  │  creates api.Client                           │
│                  │  registers N agents with Buildkite API        │
│                  │                                               │
│                  └─▶ agent.AgentPool                             │
│                          └─▶ [N] agent.AgentWorker (goroutines)  │
│                                  │                               │
│                                  ├─▶ heartbeat loop              │
│                                  ├─▶ ping loop  ──┐              │
│                                  ├─▶ streaming loop ─▶ debouncer │
│                                  └─▶ action loop  ◀──────────────┘
│                                          │                       │
│                                          └─▶ agent.JobRunner     │
│                                                  │               │
│                                                  ├─▶ LogStreamer  │
│                                                  ├─▶ HeaderTimesStreamer
│                                                  └─▶ process.New / kubernetes.NewRunner
│                                                          │       │
└──────────────────────────────────────────────────────────│───────┘
                                                           │ (subprocess via pipe)
┌──────────────────────────────────────────────────────────▼───────┐
│                     Bootstrap Process                            │
│                                                                  │
│  clicommand.BootstrapCommand.Action()                            │
│    └─▶ internal/job.Executor.Run(ctx)                            │
│              │                                                   │
│              ├─▶ PluginPhase()     — download & configure plugins │
│              ├─▶ CheckoutPhase()   — git clone/fetch/checkout     │
│              ├─▶ VendoredPluginPhase() — install vendored plugins │
│              └─▶ CommandPhase()    — run user's build commands    │
│                       (with pre/post hooks around each phase)    │
│                                                                  │
│  internal/shell.Shell — subprocess execution + PTY + env         │
│  internal/job/hook    — hook script lifecycle                    │
│  internal/redact      — streaming log redaction                  │
└──────────────────────────────────────────────────────────────────┘

Kubernetes variant replaces process.New with kubernetes.NewRunner:
  agent process ──▶ kubernetes.Runner (listens on Unix socket)
                        ◀── container A: kubernetes-bootstrap (checkout)
                        ◀── container B: kubernetes-bootstrap (command)

Core components#

AgentPool#

  • Package: github.com/buildkite/agent/v3/agent
  • File: agent/agent_pool.go
  • Responsibility: Manages N parallel AgentWorker goroutines. Spawns each worker in its own goroutine, waits for all to complete, and aggregates errors. Also starts the optional HTTP health/metrics/status server.
  • Key types: AgentPool struct { workers []*AgentWorker; idleTimeout time.Duration }
  • Dependencies: agent.AgentWorker, status, logger, metrics (Prometheus handler)

AgentWorker#

  • Package: github.com/buildkite/agent/v3/agent
  • Files: agent/agent_worker.go, agent_worker_ping.go, agent_worker_streaming.go, agent_worker_heartbeat.go, agent_worker_action.go, agent_worker_debouncer.go
  • Responsibility: A single logical agent registered with Buildkite. Runs four concurrent loops: (1) heartbeat, (2) ping/poll, (3) SSE streaming (gRPC-based), (4) debouncer + action handler. The action handler receives job acquisition signals from either the ping loop or the debouncer and invokes JobRunner.
  • Key types: AgentWorker, AgentWorkerConfig, actionMessage, baton (coordination primitive between ping and streaming)
  • Dependencies: api.Client, core.Client, metrics.Collector, process.Signal

JobRunner#

  • Package: github.com/buildkite/agent/v3/agent
  • Files: agent/job_runner.go, agent/run_job.go
  • Responsibility: Bridges the worker loop and the bootstrap subprocess. Assembles the full job environment (40+ BUILDKITE_* env vars), starts the bootstrap as a subprocess, streams logs to the API, checks for job cancellation, and reports the final exit status back to Buildkite.
  • Key types: JobRunner, JobRunnerConfig
  • Dependencies: process.New (standard) or kubernetes.NewRunner (k8s mode), LogStreamer, headerTimesStreamer, core.Client, api.Client

Executor (bootstrap)#

  • Package: github.com/buildkite/agent/v3/internal/job
  • File: internal/job/executor.go
  • Responsibility: The heart of job execution. Runs sequentially: PluginPhase → CheckoutPhase → VendoredPluginPhase → CommandPhase. Each phase wraps the actual work with agent/environment/plugin hooks (pre-, post-). This is the code that replaced the original shell bootstrap script.
  • Key types: Executor, ExecutorConfig
  • Dependencies: internal/shell.Shell, internal/job/hook, internal/redact, internal/secrets, agent/plugin, api, tracetools

Shell#

  • Package: github.com/buildkite/agent/v3/internal/shell
  • Responsibility: Cross-platform abstraction for spawning subprocesses (with optional PTY), managing the environment, running hook scripts, and reporting output. Used by Executor for every command it runs.
  • Key types: Shell (functional-options constructor), WriterLogger
  • Dependencies: process, logger, env, OS-level PTY primitives

API Client#

  • Package: github.com/buildkite/agent/v3/api
  • Responsibility: Typed REST client for the Buildkite SaaS API. Covers agent registration, job ping/accept/start/finish, log chunk upload, artifact management, pipeline upload, metadata, OIDC, and more. Also contains generated protobuf/Connect RPC code for the streaming ping path (api/proto/).
  • Key types: Client, AgentRegisterResponse, Job, Chunk
  • Dependencies: net/http, connectrpc.com/connect (for gRPC-over-HTTP streaming), generated proto types

core.Client#

  • Package: github.com/buildkite/agent/v3/core
  • Responsibility: Higher-level, retry-aware wrapper around api.Client. Provides StartJob, FinishJob, UploadChunk, etc. with backoff via buildkite/roko. Also defines Controller and JobController for the agent’s job acceptance state machine.
  • Key types: Client, Controller, JobController, ProcessExit, APIClient (interface)
  • Dependencies: api.Client, buildkite/roko (retry), logger

kubernetes.Runner#

  • Package: github.com/buildkite/agent/v3/kubernetes
  • Responsibility: Alternative execution backend for Kubernetes environments. Instead of spawning a local bootstrap subprocess, it opens a Unix domain socket and waits for BUILDKITE_CONTAINER_COUNT sidecar containers to connect. Each container runs buildkite-agent kubernetes-bootstrap which receives env vars over the socket and executes its assigned bootstrap phases.
  • Key types: Runner, RunnerConfig
  • Dependencies: internal/socket, logger

LogStreamer#

  • Package: github.com/buildkite/agent/v3/agent
  • File: agent/log_streamer.go
  • Responsibility: Consumes the job process output buffer in chunks and uploads them to the Buildkite API concurrently (up to 3 parallel uploads). Handles backpressure and respects chunk size limits set by the server.
  • Key types: LogStreamer, LogStreamerConfig
  • Dependencies: core.Client, api.Chunk, process.Buffer

Data flow#

Standard job execution (non-Kubernetes):

1. Buildkite SaaS sends job assignment
   ↓
2. AgentWorker.runPingLoop() or runStreamingPingLoop()
   receives job ID via HTTP poll or SSE/gRPC stream
   ↓
3. actionMessage { jobID, action="job" } → action loop
   ↓
4. AgentWorker.AcquireAndRunJob(ctx, jobID)
   → api.Client.AcceptJob() → Buildkite API (HTTP POST)
   ↓
5. JobRunner.NewJobRunner() assembles:
   - 40+ BUILDKITE_* env vars (createEnvironment)
   - LogStreamer (concurrent chunk upload goroutines)
   - headerTimesStreamer (section timing)
   - process.New(bootstrap-script, env=BUILDKITE_*) 
   ↓
6. JobRunner.Run() starts the bootstrap subprocess:
   - r.client.StartJob() → Buildkite API
   - subprocess stdout/stderr → pipe → process.Buffer → LogStreamer → API chunks
   - Concurrently: jobCancellationChecker polls for cancel signal
   ↓
7. Bootstrap subprocess (Executor.Run()):
   PluginPhase:   download plugins, run plugin env hooks
   CheckoutPhase: git clone/fetch/checkout with hook wrappers
   VendoredPluginPhase: install vendored plugins
   CommandPhase: run pre-command hooks → user command → post-command hooks
   ↓
8. Bootstrap exits → process.Buffer drains → LogStreamer finishes
   ↓
9. JobRunner.cleanup() → core.Client.FinishJob() → Buildkite API
   → AgentWorker returns to ping loop (or disconnects if --disconnect-after-job)

Kubernetes job execution replaces step 5-7:

  • kubernetes.NewRunner (UnixSocket server) replaces process.New
  • Container A (checkout sidecar) connects via socket, runs kubernetes-bootstrap with checkout env
  • Container B (command sidecar) connects via socket, runs kubernetes-bootstrap with command env

Initialization / Bootstrap#

main() 
  → cli.App.Run(os.Args)
  → AgentStartCommand.Action(c *cli.Context)
      1. setupLoggerAndConfig[AgentStartConfig]()
         - Reads CLI flags into AgentStartConfig struct
         - Loads .cfg file (INI format) via cliconfig package
         - Applies env var overrides (via struct tags)
      2. Resolves tags from EC2/GCP metadata (if configured)
      3. Creates api.Client (with token, endpoint, TLS config)
      4. Creates metrics.Collector (Prometheus backend)
      5. Sets up signing/verification keys (JWKS file, AWS KMS, GCP KMS)
      6. For each spawn (1..N):
         - api.Client.Register() → Buildkite API
         - agent.NewAgentWorker(logger, registrationResponse, mc, apiClient, config)
      7. agent.NewAgentPool(workers, agentConf)
      8. Runs startup hook (agentStartupHook)
      9. Sets up OS signal handlers (SIGTERM, SIGINT → StopGracefully/Ungracefully)
     10. pool.Start(ctx) — blocks until all workers finish
     11. Runs shutdown hook (agentShutdownHook via defer)

No dependency injection framework is used. All wiring is explicit/manual in agent_start.go. The AgentWorker, JobRunner, and Executor all receive their dependencies as constructor arguments. The core/ package defines a thin APIClient interface, but the rest of the codebase wires concrete types directly.

Configuration#

Multi-source, priority-ordered:

  1. CLI flags (highest priority) — defined in clicommand/agent_start.go as cli.Flag entries with EnvVar set
  2. Environment variables — matched automatically by urfave/cli via the EnvVar field
  3. Config file (.cfg, INI format) — loaded by cliconfig package; path set by --config flag or BUILDKITE_AGENT_CONFIG
  4. Defaults — baked into flag definitions

Config is decoded into strongly-typed structs (AgentStartConfig, BootstrapConfig) using struct tags (cli:"flag-name", normalize:"filepath|list|commandpath"). A setupLoggerAndConfig helper in clicommand/ handles the decoding and normalization pipeline for all commands.

The two processes share configuration through environment variables: JobRunner.createEnvironment() translates the AgentConfiguration struct fields into BUILDKITE_* environment variables that the bootstrap subprocess reads into its own BootstrapConfig.

Feature flags are managed by internal/experiments: each experiment is identified by a string key and enabled via the BUILDKITE_AGENT_EXPERIMENT env var. The experiments.IsEnabled(ctx, key) API is used at decision points throughout the agent and executor.

Key design decisions#

  1. Two-process architecture as a security and reliability boundary. The agent process and the job executor are separate binaries communicating through environment variables and a pipe. This means a runaway job cannot corrupt the agent’s state, and the agent can cleanly cancel or time-out the bootstrap process via OS signals. It also allows users to invoke buildkite-agent bootstrap directly for debugging.

  2. Dual job acquisition modes: polling vs. streaming. AgentWorker implements both a traditional HTTP ping loop and a streaming (gRPC/Connect) loop. A baton primitive coordinates between them: the streaming loop “holds the baton” while healthy, falling back to polling when the stream is unavailable. This dual-mode approach allows graceful degradation while still benefiting from low-latency streaming when it works.

  3. Bootstrap phases replace the original shell script. The internal/job.Executor runs plugin download, git checkout, and user commands in explicit Go code (not a shell script), with hook wrappers around each phase. This enables cross-platform operation, structured error handling, and testability that a bash script could not provide. The --bootstrap-script flag still allows overriding with a custom script for advanced use cases.

  4. Kubernetes execution via Unix socket sidecar protocol. Rather than embedding a Kubernetes SDK or using a separate controller, the Kubernetes execution model reuses the same JobRunner abstraction with a swapped process backend (kubernetes.Runner). The agent container acts as a coordinator; sidecar containers connect via a Unix domain socket and receive their env/phases. This avoids coupling the main agent binary to Kubernetes while still enabling K8s-native multi-container jobs.

  5. Log redaction as a streaming pipeline. The internal/redact + internal/replacer packages implement an Aho-Corasick-like multi-pattern streaming replacer that sits in the I/O pipeline between the bootstrap process output and the LogStreamer. Secrets are added to the redactor at job start (from env vars) and dynamically during execution (via the Job API). This approach prevents secrets from appearing in uploaded log chunks without buffering the full output.

Backward Buildkite Agent — Dependencies Buildkite Agent — Patterns Forward