Usage: The primary concurrency backbone. Every long-running subsystem is registered as a run/interrupt function pair in a run.Group. When any one actor exits (normally or with error), all others are interrupted.
Example:cmd/prometheus/main.go:1180 — var g run.Group followed by ~10 g.Add(runFunc, interruptFunc) calls for: signal handler, web handler, two discovery managers, scrape manager, TSDB opener, rule manager, config reload handler, remote storage.
Assessment: Highly idiomatic and effective for a process where partial failure is more dangerous than full restart. Enforces structured concurrency — no goroutine leak is possible if all long-lived goroutines are in the group. Makes the startup/shutdown topology fully legible in one place.
Usage: 165 bare go func goroutines, with sync.WaitGroup coordinating parallel operations across scrape pools at reload time.
Example:scrape/manager.go:254 — reload() acquires a mutex, then launches one goroutine per scrape pool to call sp.Sync(groups) concurrently, waiting for all to finish with wg.Wait(). The comment says: “Run the sync in parallel as these take a while and at high load can’t catch up.”
Assessment: Straightforward and correct. The goroutine captures loop variables explicitly as function arguments go func(sp, groups){}(sp, groups) — the classic Go gotcha is avoided.
Usage: The Discovery Manager exposes a SyncCh() <-chan map[string][]*targetgroup.Group channel. The Scrape Manager blocks on this channel in its Run() loop, processing target group updates as they arrive.
Example:discovery/manager.go:221 — func (m *Manager) SyncCh() <-chan map[string][]*targetgroup.Group { return m.syncCh }. The manager internally publishes updates with a non-blocking send: case m.syncCh <- m.allGroups(). Tests wire them as: go scrapeManager.Run(discoveryManager.SyncCh()).
Assessment: A clean, typed pipeline. The read-only channel return type (<-chan) enforces directionality at the type level. The non-blocking send pattern (with default: drop) in the manager ensures producers don’t stall when the consumer is busy.
errgroup for bounded parallelism (cloud API fan-out)#
Usage: Used in AWS service discovery (discovery/aws/rds.go, discovery/aws/msk.go) to fan out concurrent cloud API calls with a rate limit.
Example:discovery/aws/rds.go:437 — errg, ectx := errgroup.WithContext(ctx); errg.SetLimit(d.cfg.RequestConcurrency) followed by errg.Go(func() error { ... }) for each ARN. Uses errg.SetLimit to cap concurrency at the configured request limit.
Assessment: Idiomatic use of golang.org/x/sync/errgroup. The context threading (ectx from WithContext) ensures that a single API failure cancels all in-flight sibling requests. However, errgroup usage is limited to only these AWS discovery providers — other cloud providers use simpler sequential loops.
Usage: Pervasive. 732 occurrences of context.Context, 452 as first-argument parameters (ctx context.Context). Context is threaded through the entire call stack from HTTP request handlers down to TSDB chunk reads.
Example:scrape/scrape.go — every scrape loop iteration uses a derived context with deadline based on scrape_timeout. promql/engine.go — query evaluation uses context for cancellation by the caller.
Assessment: Correct and thorough. Context is consistently the first argument, never stored in structs. Timeout propagation through the scrape path means a slow target cannot block the scrape loop indefinitely.
Usage: Shutdown is two-phase: oklog/run interrupts all actors via their interrupt functions; individual components have configurable drain behavior (ScrapeOnShutdown, DrainOnShutdown).
Example:scrape/manager.go:147 — ScrapeOnShutdown bool option causes a final scrape pass before closing. notifier/manager.go:70 — DrainOnShutdown bool causes the notifier to attempt to flush all queued alerts before exiting. notifier/sendloop.go:129-133 — drain logic with a warning if the queue is not fully drained.
Assessment: Well-designed. The actor model handles process-level shutdown coordination; drain options give operators control over data completeness at shutdown boundaries.
Usage: 10 occurrences. Used where a channel must be closed exactly once regardless of which execution path (SIGTERM, web quit, timeout) triggers shutdown.
Example:cmd/prometheus/main.go:1152-1155 — var once sync.Once; closeOnce := func() { once.Do(func() { close(ch) }) } — described in a comment as ensuring a channel can be closed at different execution stages.
Assessment: Correct use of the idiom. Avoids panic from double-close.
Usage: 135 occurrences of atomic. across the codebase. Used in TSDB for hot-path statistics (samples appended, series created) and in the scrape engine for loop state flags.
Assessment: Appropriate usage — atomics only appear where measurements are taken concurrently across goroutines without needing a full critical section.
Style: Predominantly wrapping with fmt.Errorf %w (809 occurrences), combined with errors.Is (177) and errors.As (39) at call sites. Sentinel errors are rare; the project prefers wrapped contextual errors.
Error types defined:
storage.AppendPartialError (storage/interface_append.go:87) — carries per-exemplar errors while allowing partial success; used across TSDB head, agent WAL, fanout, and remote write OTLP handler.
storage/remote.RecoverableError (storage/remote/client.go:258) — embeds error and adds retryAfter model.Duration; used by the queue manager to implement exponential backoff with Retry-After header support.
storage/remote.HTTPError (storage/remote/codec.go:50) — carries HTTP status code for non-2xx remote write responses.
web/api/v1.apiError (web/api/v1/api.go:116) — internal type with errorType classification for structured JSON API error responses.
promql/parser.ParseErrors (promql/parser/parse.go:235) — a []ParseErr slice implementing error; allows a single parse call to return all syntax errors at once.
scrape.appendErrors — local struct aggregating out-of-order, out-of-bounds, and duplicate timestamp errors during a scrape.
Wrapping approach:fmt.Errorf("context: %w", err) is the standard throughout. errors.Wrap from pkg/errors is not used; the project uses stdlib wrapping exclusively.
Examples:
discovery/aws/rds.go:437 — fmt.Errorf("failed to describe DB cluster %s: %w", arn, err)
model/rulefmt/rulefmt.go:110 — errs = append(errs, fmt.Errorf("%d:%d: Groupname must not be empty", node.Groups[j].Line, node.Groups[j].Column))
storage/remote/client.go — RecoverableError is checked with errors.As to decide whether the queue manager should retry or drop a batch.
web.Handler, readyStorage (adapter wrapping local TSDB)
Example: At config reload time, main.go calls each of these in sequence via the reloaders slice. Each implementor validates the relevant section and atomically updates its internal state. Failures in any reloader cause the entire reload to abort and the old config to remain active.
Functional options: Used selectively for constructor-time configuration: tsdb/chunks.WriterOption (WithUncachedIO, WithSegmentSize), model/textparse.OpenMetricsOption (WithOMParserSTSeriesSkipped, WithOMParserTypeAndUnitLabels), util/teststorage.Option. The pattern is type Opt func(*options) + func With*(val) Opt { return func(o *options) { o.field = val } }.
Approach: Manual wiring in main(). No DI framework (no wire, dig, or fx).
Evidence:cmd/prometheus/main.go is ~1700 lines. The first 800 lines construct all components in dependency order: readyStorage → remote.Storage → fanoutStorage → two discovery.Managers → scrape.Manager → promql.Engine → rules.Manager → web.Handler. Each constructor receives its dependencies as explicit arguments. The initialization sequence is entirely explicit and readable as a single linear flow.
Assessment: The absence of a DI framework is a deliberate choice. The dependency graph is fixed and well-understood; a framework would add indirection without benefit. The cost is a large, non-modular main function.
Widespread: 20+ occurrences, mostly in the discovery/ package where every cloud provider implements discovery.DiscovererMetrics.
Example: discovery/azure/metrics.go:22 — var _ discovery.DiscovererMetrics = (*azureMetrics)(nil). This gives a compile-time guarantee that the struct satisfies the interface, without requiring a runtime allocation.
Worth noting: the discovery package uses this as a consistent convention across all 20+ provider implementations.
Every service discovery provider calls discovery.RegisterConfig(&SDConfig{}) in its init() function (e.g., discovery/azure/azure.go:103, discovery/consul/consul.go:86). The main package imports the providers via blank imports, triggering the registrations. The discovery manager then looks up the registered factory when parsing config.
This is the init-based registry pattern: providers self-register via side effects, making adding a new SD provider a matter of adding a blank import and registering the config type — zero changes to the core.
model/labels.Builder and model/labels.ScratchBuilder provide a mutable API for constructing immutable labels.Labels values. Three build modes exist depending on the underlying label storage format (slice, string, dedupe).
labels.NewBuilder is called 29 times in production code — primarily in the scrape path and notifier for relabeling. The builder avoids allocating intermediate label sets.
util/zeropool/Pool[T any] wraps sync.Pool to avoid the statica SA6002 linter warning about storing non-pointer types. Uses two sync.Pool instances internally: one for items, one for the pointers to those items. Copied from github.com/colega/zeropool (documented in the comment: “little copying is better than little dependency”).
Used for buffer pooling in the text parsing and scrape hot paths.
Pattern: generics appear in data-structure and algorithm utilities, not in application-level code. The constraint style (T Postings, BC BucketCount) is used where type behavior is needed, any where only type safety is needed.
promql/parser.ParseErrors []ParseErr — a slice type implementing error that collects all parse errors from a PromQL expression, rather than stopping at the first. Returned from parser.ParseExpr.
scrape.appendErrors — a local struct collecting outOfOrder, outOfBounds, and tooOld counters during a single scrape append; surfaced as a single error with counts in the message.
This pattern of “collect all errors, report once” avoids the round-trip cost of fixing and re-parsing.
storage/fanout.go:29 — unexported fanout struct implements storage.Storage. Its Appender(ctx) method returns an appender that writes to both the primary (TSDB) and a list of secondaries (remote storage) with a single Append() call. Errors from secondaries are logged but do not fail the primary append.
The consumer (scrape manager, rule engine, OTLP receiver) never knows it’s writing to two backends. This is the transparent proxy/decorator pattern applied to storage.
1127 defer calls — heavily used for: mu.Unlock() after mu.Lock(), appender.Rollback() if commit is skipped, resp.Body.Close() after HTTP calls, span finishing in tracing. This is idiomatic Go; no unusual patterns observed.
Only 2 type switch occurrences in production code — Prometheus strongly prefers interface-based dispatch over type assertions. The one notable exception is in PromQL’s AST evaluator where node types are switched for evaluation dispatch.