Grafana — Testing#

Test metrics#

  • Test files: 1,692
  • Total Go files: 5,867
  • Ratio (test files / source files): ~29% — approximately 1 test file for every 3.5 source files
  • Test frameworks: testify (require + assert) in 1,739 files — effectively universal; stdlib testing for everything else; gomock not used; mockery v2.53.4 for code-generated mocks in specific packages

Test organization#

  • Placement: Predominantly same-package (white-box). 125 files use the _test package suffix (black-box), concentrated in packages with a stable public API surface (e.g. pkg/apimachinery/, some pkg/services/ packages).
  • Helper packages:
    • pkg/tests/testinfra — StartGrafana() / StartGrafanaEnv() utilities that spin up a complete Grafana server (SQLite-backed, real Wire DI, listening on a random TCP port) for HTTP-level integration tests. This is the backbone of pkg/tests/api/** and pkg/tests/apis/**.
    • pkg/tests/testsuite — testsuite.Run(m) TestMain wrapper invoked in every integration test package. Provides consistent test-suite lifecycle (shared setup, global cleanup).
    • pkg/tests/apis — apis.NewK8sTestHelper(t, GrafanaOpts{...}) convenience wrapper that boots a live server, configures feature toggles, and returns a Kubernetes dynamic client + REST config for testing the new k8s-style API server.
    • pkg/services/ngalert/tests/fakes/ — large hand-written fake collection for alerting storage contracts (RuleStore, KVStore, permissions, provisioning). Each fake carries a t *testing.T, an in-memory data store, a Hook func(cmd any) error for injecting errors, and a RecordedOps []any slice for call history assertions.
    • pkg/services/*/fakes/ — pattern repeated across dozens of domain services (datasources, secrets, cloud migrations, auth API, library elements, etc.).
    • pkg/services/accesscontrol/mock/ — manual mock for access control, separate from fakes; used where you need controllable return values without full store semantics.
    • pkg/storage/unified/sql/db/mocks/ — mockery-generated mocks for the database abstraction interfaces (DB, Tx, Row, Rows), used in storage unit tests.
    • pkg/util/testutil/ — general utilities including SkipIntegrationTestInShortMode(t) guard and mocks/ sub-package.
    • apps/dashboard/pkg/migration/testutil/ — bespoke ChecksumStore for golden-checksum verification of schema migration outputs.
    • Numerous testdata/ directories containing JSON fixtures, golden snapshots, SQL migration files, and binary data.
  • Fixtures: testdata directories across ~90 packages; JSON response snapshots in pkg/tests/api/alerting/test-data/; golden checksum JSON files in apps/dashboard/pkg/migration/testdata/. Schema migration test inputs are collections of versioned JSON dashboard documents.

Test patterns#

Table-driven tests#

  • Prevalence: Heavy — 13,629 occurrences of table-driven patterns (tests := [], testCases, t.Run, tc.name etc.) in *_test.go files
  • Style: Anonymous struct slices are dominant. Pattern:
    tests := []struct {
    name     string
    input    SomeType
    expected SomeType
    wantErr  bool
}{
    {name: "...", input: ..., expected: ..., wantErr: false},
}
for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) { ... })
}
  • Example: pkg/tsdb/loki/parse_query_test.go — dozens of named cases covering Loki query expression transformations; pkg/services/ngalert/eval/ — evaluation path tests over alert rule variants.

Mocking approach#

  • Strategy: Two-tier:

    1. Hand-written fakes — used in high-churn domain packages (alerting, provisioning, datasources, secrets). Fakes implement the service interface and maintain in-memory state. They often accept a t *testing.T and expose RecordedOps so callers can assert what calls were made. Injection point hooks (Hook func(cmd any) error) allow targeted error injection without subclassing.
      // pkg/services/ngalert/tests/fakes/rules.go
type RuleStore struct {
    t   *testing.T
    mtx sync.Mutex
    Rules map[int64][]*models.AlertRule
    Hook  func(cmd any) error
    RecordedOps []any
}
    2. Generated mocks via mockery v2 — used for low-level interface contracts (database abstractions, Git operations, provisioning repository interfaces). //go:generate mockery --name Foo --with-expecter annotations on the interface file; generated files carry a DO NOT EDIT header and use testify/mock with typed expectation chaining (.EXPECT().Method().Return(...)).

  • Dominant choice: Hand-written fakes in the business logic layer; mockery in infrastructure/storage boundaries. This reflects the team’s philosophy: fakes evolve alongside domain logic, while infrastructure interfaces are stable enough for generation.

Integration tests#

  • Present: Yes — 19+ _integration_test.go files and hundreds of tests in pkg/tests/api/ and pkg/tests/apis/ that start a full Grafana server.
  • How: Two mechanisms:
    1. Full server tests (pkg/tests/apis/) — testinfra.StartGrafanaEnv(t, ...) spins up a complete Grafana process (full Wire-initialized DI graph, SQLite database, real HTTP listeners on random ports). Tests then use a Kubernetes dynamic client or plain HTTP to exercise the system end-to-end. Feature toggles are passed as GrafanaOpts.EnableFeatureToggles to gate the k8s API migration paths.
      helper := apis.NewK8sTestHelper(t, testinfra.GrafanaOpts{
    AppModeProduction:    true,
    EnableFeatureToggles: []string{"..."},
})
client := helper.NewDiscoveryClient()
    2. Named integration tests (_integration_test.go) — cover storage-level concerns: Redis/Memcached cache, PostgreSQL/MySQL-specific SQL, email sending, LDAP, CloudWatch calls. These require external services and are run with make test-go-integration (or with specific devenv sources).
  • Separation: Two-pronged:
    • Naming convention: _integration_test.go files for external-service-dependent tests.
    • Guard call at the top of every integration test function: testutil.SkipIntegrationTestInShortMode(t) — 904 usages. This skips when testing.Short() is true, allowing go test -short to run only fast unit tests.
    • Some tests are additionally gated on db.IsTestDbSQLite() or t.Skip("test only on sqlite for now") for DB-engine-specific behavior.
  • Async assertions: require.Eventually used 232 times — primarily in alerting scheduler tests where goroutines must converge to a state, and in integration tests waiting for background services to start.

Snapshot / golden file tests#

  • API snapshots: pkg/tests/api/alerting/test-data/alert-notifiers-v1-snapshot.json and similar — JSON responses captured at a known-good state. Tests re-read the snapshot and compare; a flag path in the test can rewrite it when intentional changes occur.
  • Golden checksum store: apps/dashboard/pkg/migration/testutil.ChecksumStore — SHA-256 checksums of migration output files stored in a JSON manifest. TestMain loads/saves checksums; set REGENERATE_CHECKSUMS=true to update. This is a space-efficient alternative to storing full golden files for large migrations.
  • OpenAPI snapshots: pkg/tests/apis/openapi_snapshots/ — captures the OpenAPI spec output for each API group. Regression-tested to detect inadvertent API schema changes.

Test quality observations#

What’s done well#

  • testinfra is exemplary. Spinning up the real Grafana process in tests — full DI graph, real database — is the gold standard for catching integration issues. It means API tests cover the full stack: Wire wiring, middleware, authentication, database, and business logic simultaneously.
  • Separation of fast and slow tests via SkipIntegrationTestInShortMode is consistent and enables CI to run unit tests cheaply while heavy integration tests run in a dedicated job.
  • Hand-written fakes with Hook injection are more flexible than mockery for domain-heavy code. The RecordedOps pattern (recording every call for later assertion) is a clean substitute for gomock’s expectation DSL in stateful scenarios.
  • Golden checksum pattern in schema migration tests is a compact way to regression-test a large combinatorial space (many dashboard JSON versions × many migration paths) without storing MBs of golden files.
  • require.Eventually is used appropriately for timing-sensitive tests involving goroutines and lifecycle state machines, rather than sleeping.
  • t.Cleanup() is used pervasively for teardown, ensuring cleanup runs even on failure and eliminating the need for defer chains.

What could improve#

  • Two divergent mocking philosophies (hand-written fakes vs. mockery) create maintenance overhead. New contributors must learn which pattern applies where. A project-wide decision recorded in AGENTS.md would help.
  • No //go:build integration build tag — integration test isolation relies solely on the SkipIntegrationTestInShortMode call inside the test function. A build tag would allow go test ./... to exclude integration tests at the package level without loading their (sometimes heavy) imports.
  • Some _test.go packages for pkg/tests/api/ have expensive compilation due to the full Grafana dependency graph. The AGENTS.md warns that pkg/api/ can take ~2 minutes to compile, discouraging iterative local testing.
  • Missing test coverage metrics — no visible coverage threshold enforcement in CI (no go test -coverprofile or codecov gate). For a project of this size, enforcing coverage minimums per package would catch gaps.

Patterns worth emulating#

  • testinfra.StartGrafana() pattern — the idea of a StartTestServer(t, opts) function that boots the full production DI graph with an in-memory DB is highly portable to any Wire-based service. It provides integration confidence that a mock-heavy approach can never match.
  • Hook func(cmd any) error in hand-written fakes — a simple escape hatch that avoids subclassing or mock-framework complexity. Any test can inject arbitrary errors at precise call points without modifying the fake.
  • ChecksumStore for migration golden tests — the pattern of storing SHA-256 checksums of deterministic outputs rather than the outputs themselves scales well to large generated-file workloads and is worth generalizing.
  • require.Eventually over time.Sleep — using testify’s Eventually(condition, timeout, interval) instead of sleeping keeps async tests fast when the condition is met early and provides a meaningful failure message when it isn’t.
Backward Grafana — API Surface Hugo — Overview Forward