Grafana — Patterns#

Concurrency patterns#

Anonymous goroutine spawning#

  • Usage: 315 go func(...) occurrences across the codebase. Used heavily for background work, event listeners, and one-off async tasks.
  • Example: pkg/api/http_server.go:503 — errg, _ := errgroup.WithContext(ctx) followed by goroutines for each listener
  • Assessment: Idiomatic Go, though the raw count reflects the scale of the project. Most spawning happens inside BackgroundService.Run() implementations that are themselves supervised by the lifecycle manager.

errgroup — structured parallel error collection#

  • Usage: golang.org/x/sync/errgroup appears in several key packages. Used when multiple goroutines must run concurrently and their errors collected atomically.
  • Example: pkg/api/http_server.go:503 — HTTP server spawns listeners (HTTP + HTTPS + admin) in an errgroup, so any listener failure propagates to a clean shutdown.
  • Assessment: Excellent idiomatic use. errgroup replaces the ad-hoc WaitGroup+channel error collection that often leads to goroutine leaks.

Worker pools#

  • Usage: Explicit worker pool pattern in the provisioning controllers (pkg/registry/apis/provisioning/controller/). A workerCount integer controls how many goroutines dequeue and process items from a rate-limiting queue.
  • Example: pkg/registry/apis/provisioning/controller/connection.go:130 — for i := 0; i < workerCount; i++ { go cc.runWorker(ctx, ...) } after the queue is started.
  • Assessment: Well-structured. Uses Kubernetes’s workqueue.TypedRateLimitingInterface for the task queue, bringing backpressure and retry logic along with the pool.

WaitGroup — fan-out/fan-in#

  • Usage: 345 occurrences of wg.Add/Done/Wait. Used for coordinating parallel operations before returning a unified result.
  • Example: Spread throughout pkg/services/ngalert/ (alerting) and pkg/services/provisioning/.
  • Assessment: Standard usage. Not unusual in a codebase of this size. Generally not mixed with raw channel fan-in, which keeps the patterns clean.

Context cancellation — universal shutdown signal#

  • Usage: 200 ctx.Done() / <-ctx.Done() calls; 14,694 context.Context parameter occurrences (essentially every function that does I/O or blocks carries context).
  • Example: Every BackgroundService.Run(ctx context.Context) error blocks on <-ctx.Done() as its shutdown trigger. The ManagerAdapter cancels the shared context on SIGTERM, propagating shutdown to all goroutines without explicit stop channels.
  • Assessment: Exemplary. Context is passed everywhere from the top-level signal handler down to SQL queries. No global stop booleans, no manual cancellation channels — all shutdown is ctx-based.

Select statements#

  • Usage: 375 select { blocks. Covers both channel mux (multiple event sources) and non-blocking channel operations.
  • Assessment: Idiomatic. Common in lifecycle loops that must respond to both work items and ctx cancellation.

Rate limiting#

  • Usage: golang.org/x/time/rate for HTTP endpoint rate limiting; workqueue.DefaultTypedControllerRateLimiter for work queue throttling in controllers.
  • Example: pkg/api/frontend_logging.go:115 — rate.NewLimiter(rate.Limit(hs.Cfg.GrafanaJavascriptAgent.EndpointRPS), burst) guards the JavaScript agent logging endpoint.
  • Assessment: Two different rate limiting mechanisms are used contextually: token-bucket for HTTP and exponential backoff via Kubernetes workqueue limiter for controllers. Both are well-placed.

Graceful shutdown#

  • Usage: Signal handling via the lifecycle manager. SIGTERM/SIGINT → context cancel → all BackgroundService.Run() goroutines unblock → HTTP server Shutdown() called → 30-second timeout before forced exit.
  • Example: pkg/api/http_server.go:498 — hs.httpSrv.Shutdown(context.Background()) called inside the service’s Run method when ctx is cancelled.
  • Assessment: Excellent. The BackgroundService interface makes graceful shutdown a first-class contract. Every service is structurally required to respect context cancellation.

Error handling#

  • Style: Mixed — standard library errors + fmt.Errorf %w wrapping dominates. Custom error types used for structured domain errors. pkg/errors (third-party) essentially not used (3 occurrences total).
  • Error types defined:
    • ConversionError, ConversionDataLossError — apps/dashboard/pkg/migration/
    • MigrationError, MinimumVersionError — apps/dashboard/pkg/migration/schemaversion/
    • QuotaExceededError — apps/provisioning/pkg/quotas/
    • InvalidLocalFolderError — apps/provisioning/pkg/repository/local/
    • ValidationError — apps/dashvalidator/pkg/validator/
    • URLValidationError, AnnotationError — pkg/api/
  • Wrapping approach: fmt.Errorf("%w", err) is overwhelmingly dominant at 2,925 occurrences. The codebase has fully migrated to Go 1.13+ error wrapping idioms.
  • errors.Is/errors.As: 1,498 occurrences — used extensively for matching sentinel errors and typed error unwrapping in handlers and test assertions.
  • errors.New: 1,707 occurrences — used for leaf sentinel errors with no wrapping.
  • Examples:
    • apps/dashboard/pkg/migration/schemaversion/errors.go — MigrationError and MinimumVersionError implement Error() string and Unwrap() error, enabling structured error handling in migration pipelines.
    • Domain errors like QuotaExceededError carry structured fields (e.g. Quota, Used) so callers can extract context without string parsing.

Configuration pattern#

  • Approach: Constructor injection via ProvideService(cfg *setting.Cfg, ...) functions, fed by Google Wire. No functional options for the main service graph — configuration flows through the *setting.Cfg megastruct.
  • Functional options: Used locally within specific packages for test setup (APITestServerOption func(hs *HTTPServer) in pkg/api/common_test.go:289) and for package-scoped builder types like FolderManagerOption, RepositoryResourcesOption, ConfigOption. Not used as the primary DI mechanism.
  • Example of Wire constructor:
    // pkg/services/secrets/manager/manager.go
func ProvideSecretsService(
    store secrets.Store,
    kv kvstore.KVNamespacer,
    enc encryption.Internal,
    cfg *setting.Cfg,
    features featuremgmt.FeatureToggles,
    ...
) (*SecretsService, error)
    Wire reads this signature and resolves all dependencies from previously registered providers.
  • Feature flag gating: features.IsEnabled(ctx, featuremgmt.FlagXxx) is used at call sites to gate behavior. Flag names are generated constants (e.g. featuremgmt.FlagKubernetesSnapshots). This is the primary mechanism for incremental rollout of the k8s API migration.

Dependency injection#

  • Approach: Google Wire compile-time code generation. No reflection, no service locator at runtime.
  • Evidence:
    • pkg/server/wire.go — the injector spec (build-tag-guarded, never compiled)
    • pkg/server/wire_gen.go — 1,939-line generated function Initialize() that explicitly constructs every service in dependency order
    • pkg/server/wireexts_oss.go / wireexts_enterprise.go — build-tag-based OSS vs. Enterprise composition
  • Convention: Every Wire provider function is named Provide<ServiceName> and lives in the same package as the service it constructs. wire.Bind(new(Interface), new(*Impl)) entries tie interface types to concrete implementations. Wire verifies the full graph at go generate time, catching missing providers and circular dependencies at compile time.
  • Scale: Hundreds of services are wired. The generated Initialize() is the single most important artifact for understanding what runs in the Grafana process.

Other notable patterns#

BackgroundService — universal lifecycle interface#

Every long-running service in the Grafana process implements:

// pkg/registry/registry.go:25
type BackgroundService interface {
    Run(ctx context.Context) error
}

The ManagerAdapter (wrapping grafana/dskit’s module manager) starts all registered BackgroundService implementations concurrently after the init phase. Shutdown is purely ctx-based: when the root context is cancelled (SIGTERM), all Run() goroutines unblock and return. This is the most important structural pattern in Grafana’s backend.

APIGroupBuilder — extensibility pattern for k8s-style APIs#

New resource types (dashboards, folders, alerting rules, etc.) are registered using the builder.APIGroupBuilder interface:

// pkg/registry/apis/userstorage/register.go:21
var _ builder.APIGroupBuilder = (*UserStorageAPIBuilder)(nil)

func RegisterAPIService(..., apiregistration builder.APIRegistrar, ...) *UserStorageAPIBuilder {
    b := &UserStorageAPIBuilder{...}
    apiregistration.RegisterAPI(b)
    return b
}

This registry + builder pattern lets each API group declare its own schema, storage, and handlers without modifying a central registry. It mirrors the Kubernetes controller/admission webhook registration model.

Feature flag gating — pervasive progressive rollout mechanism#

Feature flags (featuremgmt.FlagXxx) are used at 10+ call sites per major subsystem to gate the k8s API migration. Pattern:

if hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagKubernetesSnapshots) {
    // new code path
} else {
    // legacy code path
}

Flag constants are generated from a canonical definition file (pkg/services/featuremgmt/toggles_gen.go), preventing typos and enabling tooling. This is dual-use: both as a safety net and as an A/B migration strategy.

Registry pattern#

Used at multiple levels:

  • BackgroundServiceRegistry — collects all background services before startup
  • PluginRegistry — tracks loaded plugins
  • APIRegistrar — collects all k8s API group builders
  • UsageStatsProvidersRegistry — aggregates telemetry providers Pattern: each registry holds a slice of interface values; registration is done at Wire time (constructor injection), not at runtime lookup.

In-process event bus (observer)#

pkg/bus/InProcBus provides synchronous publish/subscribe for domain events (SignUpStarted, SignUpCompleted, etc.). Handlers register via bus.AddEventListener(func). The bus is declining in new code — direct interface injection is preferred — but remains in legacy cross-service paths.

  • Example: pkg/api/signup.go:70 — hs.bus.Publish(ctx, &events.SignUpStarted{...})
  • Assessment: The synchronous bus is a historical artifact. It couples publishers and subscribers to a shared type. New services avoid it in favor of explicit interface dependencies wired by Wire.

Generics (Go 1.18+)#

Used sparingly and purposefully:

  • CanViewTargets[T any](...) — pkg/registry/apis/iam/authorizer/resource_permissions.go:75 — generic batch permission filter
  • WithServiceIdentityFn[T any](...) — pkg/apimachinery/identity/context.go:134 — generic identity-aware context executor
  • toInterfaceSlice[T any](s []T) []interface{} — test utility
  • ptrTo[T any](v T) *T — test utilities for pointer boxing No use of generic data structures in core domain code. Generics appear primarily in utility functions and test helpers where type erasure would otherwise require interface{}.

Type switches#

198 switch .(type) occurrences, concentrated in:

  • JSON/protobuf deserialization (converting interface{} to concrete types)
  • k8s runtime object handling (runtime.Object switch)
  • Alerting expression type dispatch (pkg/expr/)
  • Error type checking in API handlers

Sync primitives#

582 total sync primitive usages (sync.Mutex, sync.RWMutex, sync.Once, sync.WaitGroup, sync.Map, atomic.*):

  • sync.Once — lazy initialization of singletons (plugin loader, schema registry)
  • sync.RWMutex — concurrent read-heavy caches (plugin registry, local cache)
  • sync.Map — concurrent-safe maps in alerting models and test utilities
  • atomic.* — counters and flags in performance-sensitive paths

Interface satisfaction assertions#

Pervasive compile-time interface check pattern:

var _ builder.APIGroupBuilder = (*UserStorageAPIBuilder)(nil)

Found throughout pkg/registry/apis/ for every API builder, ensuring concrete types satisfy their contracts. This is idiomatic Go for catching interface drift early.

CUE-driven code generation#

Not a Go pattern per se, but architecturally significant: resource schemas are defined in CUE (kinds/, apps/*/kinds/*.cue) and Go structs are generated from them. The generated Go code is annotated and never hand-edited. This connects to the pattern of trusting code generation over manual synchronization — also seen in Wire (wire_gen.go) and protobuf (*.pb.go).

Backward Grafana — Architecture Grafana — Interfaces Forward