Prometheus — Testing#

Test metrics#

• Test files: 259
• Source files (non-test): 438
• Ratio (test files / source files): ~0.59 — solid coverage for a project of this complexity
• Test frameworks: testify/require (primary assertion library), google/go-cmp (deep equality with custom comparers), prom_testutil from prometheus/client_golang (metrics assertion), go.uber.org/goleak (goroutine leak detection), testing/synctest (Go 1.24 deterministic goroutine scheduling)
• Benchmark functions: 133
• Fuzz functions: 8

Test organization#

• Placement: Overwhelmingly same-package (247 files use package <pkg>, only 10 use package <pkg>_test). Tests get full access to unexported symbols, which the team clearly prefers over strict black-box testing.
• Helper packages:
  • util/testutil — a curated set of test utilities:
    • cmp.go: RequireEqual / RequireEqualWithOptions wrapping go-cmp with a custom labels.Equal comparer, so labels.Labels are compared by content rather than Go struct equality.
    • testing.go: TolerantVerifyLeak(m *testing.M) — wraps goleak.VerifyTestMain with pre-configured ignores for known false-positive goroutines from opencensus, k8s klog, and client-go workqueue.
    • context.go: MockContext and MockContextErrAfter — simple stub implementations of context.Context for testing cancellation paths without timers.
    • directory.go, port.go, roundtrip.go: utility helpers for temp dirs, free port allocation, and HTTP round-trip testing.
    • synctest/: thin wrapper around Go 1.24’s testing/synctest package, providing Test(t, f) and Wait() for deterministic concurrency tests.
  • util/teststorage — a real TSDB-backed TestStorage that sets up an in-process Prometheus TSDB with relaxed block/retention windows. Registered via t.Cleanup so callers don’t need to close it. Used by scrape, PromQL, and rules tests to write and query real time-series data.
  • util/fuzzing — a dedicated package containing all FuzzXxx functions (8 total), covering: text format parsing, OpenMetrics parsing, metric selector parsing, PromQL expression parsing, XOR chunk encoding/decoding (two variants), protobuf parsing.
  • promql/promqltest — a full DSL-based test engine (described in detail below).
• Fixtures: Each major package has a testdata/ directory with YAML, text, or binary fixtures: config/testdata/ (config files), scrape/testdata/ (target configs), model/textparse/testdata/ (wire format samples), promql/promqltest/testdata/ (DSL test scripts), tsdb/testdata/, web/api/v1/testdata/ (HTTP response fixtures), and others.

Test patterns#

Table-driven tests#

• Prevalence: Heavy — 382 occurrences of table-driven test patterns across *_test.go files.
• Style: Anonymous struct slice, e.g., tests := []struct{ name string; input ...; expected ... }{} followed by for _, tc := range tests { t.Run(tc.name, ...) }. Named structs are rare; anonymous inlined structs dominate.
• Example: model/relabel/relabel_test.go:30 — TestRelabel uses an anonymous struct with inputLabels, cfg []*config.RelabelConfig, outputLabels, etc.; 20+ cases covering all relabeling rules in a single function.
• Benchmark tables: Also used for benchmarks — model/relabel/relabel_test.go:1069 uses b.Run(tt.name, ...) for parametric benchmarks across label set sizes.

DSL-driven PromQL testing (promqltest)#

• What it is: A custom domain-specific language for declaratively specifying PromQL evaluation tests. Lives in promql/promqltest/testdata/*.test files.
• Format:

  load 5m
    http_requests{job="api-server", instance="0"} 0+10x10
  
  eval instant at 50m sum by (group) (http_requests)
    {group="production"} 300

  Commands: load, eval instant, eval range, clear. Modifiers: eval_fail, eval_warn, eval_ordered. The expect directive can assert on warning/info annotations by regex.
• Why it matters: The test engine (promqltest/test.go) uses //go:embed to bundle all .test files into the binary. Any PromQL engine implementing the promql.QueryEngine interface can be run against this suite — the test corpus is the authoritative specification for PromQL semantics. Third-party PromQL implementations (e.g., Thanos, Cortex) import and run this suite.
• Scale: Testdata files cover: aggregators, at_modifier, collision, duration_expression, extended_vectors, fill-modifier, functions, histograms, info, limit, and more. Hundreds of eval assertions.

Mocking approach#

• Strategy: No gomock or mockery. Prometheus uses two strategies:
  1. Real in-process implementations: Most tests use util/teststorage.New(t) to get a real TSDB, httptest.NewServer for HTTP handlers, and real discovery managers. This makes tests more realistic but heavier.
  2. Manual fakes via interfaces: Storage interfaces (storage.Appender, storage.QueryableFunc, etc.) are satisfied by small hand-written fakes in test files. For example, scrape_test.go contains nopAppender, mutatingAppender, and other local structs that implement storage.Appender with custom behavior.
• Notable: util/testutil.MockContext provides a lightweight context stub for testing cancellation without real timers — lighter than full context with context.WithCancel.
• prom_testutil: Tests of metrics-emitting components use github.com/prometheus/client_golang/prometheus/testutil to assert on Prometheus metric values: testutil.ToFloat64(counter), testutil.CollectAndCompare(reg, expectedMetricsReader). Used 209 times across test files.
• httptest: 201 occurrences of httptest.NewServer / httptest.NewRecorder — the standard Go HTTP test server is heavily used for testing the scrape, remote write, and web API layers.

Goroutine leak detection (goleak)#

• Present: Yes — goleak is used in 30 places across test packages.
• Pattern: TestMain functions call testutil.TolerantVerifyLeak(m) (which wraps goleak.VerifyTestMain) at the package level. Individual tests may also call defer goleak.VerifyNone(t) for targeted checks (e.g., scrape_test.go:1272).
• Known false positives: The TolerantVerifyLeak wrapper ignores goroutines from opencensus, k8s klog, and client-go workqueue that are background daemons started by imported libraries and cannot be stopped per-test.
• Significance: Goroutine leak detection is rare in Go projects of this size. Its presence reflects the project’s awareness that scrape loops, discovery managers, and notification senders are prone to goroutine leaks if shutdown paths are incorrect.

Deterministic concurrency testing (synctest)#

• Present: Yes — the notifier package uses util/testutil/synctest (wrapping Go 1.24’s testing/synctest).
• Example: notifier/manager_test.go:704 — tests for alert send retry logic use synctest.Test(t, func(t *testing.T) { ... }) so that time.Sleep calls inside the notifier’s retry loop use fake time. synctest.Wait() advances fake time until all goroutines are blocked.
• Why notable: This is a very new Go testing facility (Go 1.24). Prometheus is an early adopter, likely because testing retry/backoff logic in the notifier requires precise control over elapsed time without real sleeps.

Integration tests#

• Dedicated integration test files: None found (*_integration_test.go, *_e2e_test.go — zero matches).
• How integration is achieved: Instead of separate files, Prometheus uses:
  1. Test flags: TestVersionUpgrade in cmd/prometheus/main_upgrade_test.go is gated by --test.version-upgrade=true (a custom flag.Bool). It runs in a dedicated CI job that downloads historical Prometheus release binaries and exercises TSDB upgrade/downgrade paths.
  2. TSDB isolation flag: CI runs go test ./tsdb/ -test.tsdb-isolation=false as a separate step — a custom test flag exposed by the TSDB package to toggle transaction isolation behavior, exercising a different code path.
  3. Multi-arch testing: CI runs GOARCH=386 go test ./... to catch 32-bit arithmetic bugs.
  4. Build tag matrix: CI runs go test --tags=dedupelabels ./..., --tags=slicelabels, --tags=forcedirectio — three different label storage implementations and a direct-IO TSDB path are each tested in separate CI jobs.

CI structure#

Five distinct Go test jobs in .github/workflows/ci.yml:

• test_go: make GO_ONLY=1 (standard go test ./...) + TSDB isolation variant.
• test_go_more: Three build-tag variants (dedupelabels, slicelabels+race, forcedirectio+race) + GOARCH=386 + proto regeneration check.
• test_version_upgrade: Version upgrade/downgrade test with real binary downloads.
• test_go_oldest: Runs the full suite against Go N-1 to catch version regressions.
• fuzzing: Matrix job running each FuzzXxx function for 4 minutes, triggered on schedule.

Test quality observations#

What’s done well#

• Goroutine leak detection via goleak in 30 packages — rare discipline for a project this large, directly connected to the concurrency-heavy scrape and discovery layers.
• Real storage in tests — util/teststorage gives tests a real TSDB, making scrape and PromQL tests high-fidelity. Bugs that would slip through a mock appender are caught.
• promqltest DSL — a genuinely innovative testing technique. The declarative format is human-readable, version-controllable, and reusable across third-party PromQL implementations. It is the authoritative PromQL specification.
• Build-tag matrix — testing three separate label storage implementations (dedupelabels, slicelabels, default) and a direct-IO path ensures that experimental optimizations don’t silently break semantics.
• 133 benchmarks — performance is first-class. The AGENTS.md contributor guide requires benchmarks with benchstat output for any performance PR.
• 8 fuzz targets — covering the parsing hot paths (text format, OpenMetrics, PromQL, XOR chunks, protobuf) where malformed input from scrape targets could crash the server.
• go-cmp with custom label comparer — labels.Labels has non-obvious equality semantics (different underlying representations can be logically equal). The RequireEqual wrapper enforces correct comparison without burdening every test author.
• Early adoption of testing/synctest — the notifier retry tests benefit from deterministic time control, eliminating flaky time.Sleep-based assertions.

What could improve#

• Coverage is uneven: The scrape and PromQL packages are extensively tested; some discovery providers have thin test coverage (particularly newer cloud providers added recently).
• Same-package bias: The heavy preference for same-package tests (247 vs 10 external) means the public API surface is less systematically tested. A consumer’s perspective on API ergonomics could be missed.
• No end-to-end HTTP test suite: There is no self-contained integration test that boots a real Prometheus server, scrapes a target, evaluates a query, and asserts on results. The closest is TestVersionUpgrade, which is gated behind a flag and run in a dedicated CI job.
• goleak coverage gaps: Only 30 of 259 test packages have goroutine leak detection. Given the concurrency model, broader coverage would be valuable.

Patterns worth emulating#

• promqltest DSL: Any project with a complex query or evaluation engine should consider a declarative test DSL. The cost of building the parser is paid once; the benefit is hundreds of readable, maintainable test cases.
• util/teststorage: The pattern of a New(t testing.TB, opts ...Option) *TestStorage helper that uses t.Cleanup for teardown is widely applicable. It gives every test a real storage backend with zero boilerplate.
• TolerantVerifyLeak with documented false-positive exceptions: The explicit comment-annotated ignore list in testutil.TolerantVerifyLeak is instructive — it names the upstream issue links, making it clear that these ignores are not laziness but documented limitations of third-party libraries.
• Build-tag matrix for compile-time configuration: When a project has compile-time-selected implementations (via build tags), exercising each in CI is the correct approach. Prometheus makes this visible as separate CI jobs rather than hidden in a single script.