Prometheus — Architecture#

Architectural style#

Modular Monolith with Actor-based Concurrency

Prometheus is a single-process monolith composed of well-delineated subsystems that run as concurrent actors and communicate through interfaces and channels. There is no microservices split; all subsystems — TSDB, scrape engine, query engine, rules evaluator, service discovery, notifier, and HTTP server — run in-process and are coordinated by the main() function.

The key characteristics are:

  • Actor concurrency via oklog/run: Each subsystem registers a run/interrupt function pair in a run.Group. When any actor exits, all are interrupted — giving clean structured concurrency without goroutine leaks.
  • Interface-based decoupling at storage boundaries: storage.Queryable, storage.Appendable, and storage.Storage separate the query engine and scrape engine from the TSDB implementation. This is the primary seam that lets Thanos, Cortex, and Mimir swap in their own backends.
  • Fanout pattern for write path: A storage.NewFanout adapter fans appends to both local TSDB and remote storage simultaneously, without either side knowing about the other.
  • Dual operating mode in one binary: The same binary runs as a full server (TSDB + query engine + rules + web) or as an Agent (WAL-only scrape relay) controlled by the --agent flag.

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────────────┐
│                        Prometheus Process                               │
│                                                                         │
│   ┌──────────────┐        ┌──────────────────┐        ┌─────────────┐  │
│   │  Discovery   │──SD──▶ │  Scrape Manager  │──────▶ │   Fanout    │  │
│   │  Manager     │        │  (pull metrics)  │        │   Storage   │  │
│   │ (scrape SD)  │        └──────────────────┘        │             │  │
│   └──────────────┘                                    │  ┌─────────┐│  │
│                                                       │  │  TSDB   ││  │
│   ┌──────────────┐        ┌──────────────────┐        │  │(local)  ││  │
│   │  Discovery   │──SD──▶ │    Notifier      │──HTTP─▶│  └─────────┘│  │
│   │  Manager     │        │  (Alertmanager   │        │  ┌─────────┐│  │
│   │(alerting SD) │        │   dispatch)      │        │  │ Remote  ││  │
│   └──────────────┘        └──────────────────┘        │  │Storage  ││  │
│                                                       └─┬───────────┘  │
│   ┌──────────────┐        ┌──────────────────┐         │              │
│   │  Rule        │──eval──│  PromQL Engine   │◀────────┘              │
│   │  Manager     │        │  (query engine)  │──query──▶ Queryable    │
│   └──────────────┘        └──────────────────┘                        │
│                                    ▲                                   │
│   ┌──────────────────────────────┐ │                                   │
│   │        Web Handler           │─┘                                   │
│   │  HTTP API + UI + OTLP recv   │                                     │
│   └──────────────────────────────┘                                     │
│                                                                         │
│   ┌────────────┐   ┌──────────────┐   ┌──────────────────────────────┐ │
│   │  Config    │   │  Reload      │   │   Tracing Manager (OTLP)     │ │
│   │  Loader    │──▶│  Dispatcher  │──▶│ (OpenTelemetry integration)  │ │
│   └────────────┘   └──────────────┘   └──────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘

Core components#

Discovery Manager (×2)#

  • Package: github.com/prometheus/prometheus/discovery
  • Responsibility: Discovers scrape targets (one instance) and Alertmanager endpoints (second instance) via 30+ provider integrations. Produces a stream of targetgroup.Group updates via the SyncCh() channel.
  • Key types: Manager, Discoverer interface, targetgroup.Group
  • Dependencies: 30+ cloud-provider SDKs (AWS, Azure, GCP, Consul, Kubernetes, etc.), config

Scrape Manager#

  • Package: github.com/prometheus/prometheus/scrape
  • Responsibility: Receives target groups from Discovery Manager, maintains per-target scrape loops, fetches /metrics over HTTP at configured intervals, parses Prometheus text format or protobuf, and appends samples to Fanout Storage.
  • Key types: Manager, scrapePool, scrapeLoop, Target
  • Dependencies: storage.Appendable, discovery.Manager (via SyncCh()), config, model/textparse, model/relabel

Fanout Storage#

  • Package: github.com/prometheus/prometheus/storage
  • Responsibility: A write-multiplexer (fanoutStorage) that forwards appends to both local TSDB and remote storage. Implements storage.Storage, making the dual-write transparent to producers (scrape manager, rules engine, OTLP receiver).
  • Key types: fanoutStorage (unexported), Storage interface, Appendable interface
  • Dependencies: local tsdb.DB, remote.Storage

TSDB (Local Time-Series Database)#

  • Package: github.com/prometheus/prometheus/tsdb
  • Responsibility: An embedded columnar time-series database. Manages a write-ahead log (WAL), in-memory head block, and persistent immutable blocks. Handles compaction, retention, out-of-order ingestion, and serves as the primary Queryable.
  • Key types: DB, Head, Block, Compactor; sub-packages: wlog (WAL), chunkenc, chunks, index, tombstones
  • Dependencies: OS filesystem; no external storage engine

Remote Storage#

  • Package: github.com/prometheus/prometheus/storage/remote
  • Responsibility: Forwards samples to external systems via remote write (protobuf over HTTP), and reads from remote systems via remote read. Also receives remote write from external senders when enabled. Serializes via prompb.
  • Key types: Storage, WriteStorage, ReadStorage, QueueManager
  • Dependencies: prompb, HTTP client, external Prometheus-compatible receivers

PromQL Engine#

  • Package: github.com/prometheus/prometheus/promql
  • Responsibility: Parses, plans, and evaluates PromQL queries. Contains a hand-written lexer/parser (promql/parser), an AST evaluator, and result types. Evaluates recording rules and responds to API queries.
  • Key types: Engine, Query, EvalNodeHelper; sub-package parser (lexer, AST nodes)
  • Dependencies: storage.Queryable (reads samples), model/labels, util/annotations

Rule Manager#

  • Package: github.com/prometheus/prometheus/rules
  • Responsibility: Loads recording and alerting rule files, evaluates them on a configurable interval using the PromQL engine, appends recording rule results to storage, and fires alerts to the Notifier.
  • Key types: Manager, Group, RecordingRule, AlertingRule
  • Dependencies: promql.Engine, storage.Appendable, storage.Queryable, notifier.Manager

Notifier#

  • Package: github.com/prometheus/prometheus/notifier
  • Responsibility: Queues and dispatches firing alerts to one or more Alertmanager instances discovered by the notify Discovery Manager. Manages batching, deduplication, and retries.
  • Key types: Manager, Alert
  • Dependencies: discovery.Manager (notify SD), HTTP client

Web Handler#

  • Package: github.com/prometheus/prometheus/web
  • Responsibility: Serves the HTTP API (/api/v1/*), Prometheus UI (embedded React/Mantine app), console templates, federation endpoint, remote write receiver, and OTLP write receiver. Also exposes the /reload and /quit lifecycle endpoints.
  • Key types: Handler, Options; sub-package web/api/v1 (HTTP handler functions)
  • Dependencies: promql.Engine, scrape.Manager, rules.Manager, notifier.Manager, storage.*

Config / Reload System#

  • Package: github.com/prometheus/prometheus/config + cmd/prometheus/main.go
  • Responsibility: Loads YAML configuration files, validates them, and propagates changes to all subsystems through a list of reloader functions. Config reload is triggered by SIGHUP, HTTP POST /-/reload, or auto-reload file-hash polling.
  • Key types: Config, GlobalConfig, ScrapeConfig, AlertingConfig
  • Dependencies: All subsystems implement ApplyConfig(*Config) error

Data flow#

Scrape → Storage path (the hot path)#

1. Discovery Manager (scrape) polls cloud APIs / DNS / k8s watches
2.   → publishes targetgroup.Group updates on SyncCh() channel
3. Scrape Manager reads SyncCh(), starts/stops scrape loops per target
4.   → each scrape loop sends HTTP GET /metrics to target at scrape_interval
5.   → response parsed by model/textparse (text/protobuf format)
6.   → relabeling applied (model/relabel)
7.   → fanoutStorage.Appender(ctx).Append(ref, labels, ts, val)
8.      ├─▶ tsdb.Head.Appender → WAL record written → in-memory series updated
9.      └─▶ remote.QueueManager → batched → HTTP POST to remote_write endpoint

Query path#

1. HTTP GET /api/v1/query{_range} → web/api/v1 handler
2.   → promql.Engine.NewInstantQuery / NewRangeQuery
3.   → parser.ParseExpr → AST node tree
4.   → Engine.Eval(ctx, expr, mint, maxt)
5.      → storage.Queryable.Querier(mint, maxt) → tsdb or fanout querier
6.      → tsdb: Block + Head scan via posting lists + chunk decoding
7.   → []promql.Sample / Matrix result → JSON marshalled to HTTP response

Rule evaluation path#

1. rules.Manager ticks on evaluation_interval (e.g., 15s)
2.   → each Group calls EngineQueryFunc (wraps promql.Engine)
3.   → RecordingRule result appended to fanoutStorage
4.   → AlertingRule: if condition fires → Alert pushed to notifier.Manager queue
5.      → notifier batches alerts → HTTP POST to Alertmanager(s)

Initialization / Bootstrap#

The entire startup is orchestrated in cmd/prometheus/main.go through a manual dependency injection pattern — no DI framework is used. The sequence is:

  1. Flag parsing (kingpin) — all configuration is captured in flagConfig
  2. Feature flag processing — cfg.setFeatureListOptions() translates --enable-feature flags into typed config fields on multiple subsystems
  3. Config file pre-validation — config.LoadFile() validates the YAML before any subsystem starts
  4. Component construction (in dependency order):
    • readyStorage wrapper (initially blocks queries until TSDB is ready)
    • remote.Storage (needs readyStorage.StartTime)
    • storage.NewFanout(localStorage, remoteStorage)
    • Two discovery.Manager instances (scrape SD and notify SD)
    • scrape.Manager (needs fanoutStorage)
    • promql.Engine (server mode only; needs Queryable)
    • rules.Manager (server mode only; needs Engine + Appendable)
    • web.Handler (needs everything above)
  5. reloaders slice — each subsystem registers an ApplyConfig(*config.Config) error function; the reload handler calls all of them in order on every config change
  6. oklog/run.Group population — each actor (discovery managers, scrape manager, rule manager, web handler, TSDB opener, reload handler, signal handler) is added as a run/interrupt pair
  7. TSDB open — happens in its own actor; once open it sets localStorage.Set(db, ...) and closes the dbOpen channel, which unblocks the initial config loader
  8. Initial config load — fires reloadConfig() once TSDB is ready; closes reloadReady.C to start scrape/rule/tracing actors
  9. g.Run() — blocks until any actor returns an error or signal; then tears down all actors in parallel via their interrupt functions

Agent mode follows the same path but skips TSDB, query engine, and rule manager; it opens the WAL-based agent storage instead.

Configuration#

Prometheus uses a two-tier configuration system:

  1. CLI flags (via kingpin): Startup parameters that cannot be changed at runtime — storage paths, retention, listen addresses, feature flags, resource limits (--auto-gomaxprocs, --auto-gomemlimit).
  2. YAML config file (prometheus.yml): Runtime-reloadable configuration — global scrape/evaluation intervals, scrape jobs, rule file paths, remote write/read endpoints, Alertmanager addresses, relabeling rules.

Config reload works through the reloaders slice. Each subsystem implements ApplyConfig(*config.Config) error, and the reload handler calls them sequentially. Reloads are triggered by:

  • SIGHUP
  • POST /-/reload (requires --web.enable-lifecycle)
  • File hash polling (requires --enable-feature=auto-reload-config)

There is no Viper or external config framework. YAML unmarshaling uses gopkg.in/yaml.v2 with custom UnmarshalYAML methods on config structs.

Key design decisions#

  1. oklog/run as the concurrency backbone. Every long-running subsystem is registered in a run.Group. This enforces the discipline that any actor failure tears down the whole process — appropriate for a monitoring system where partial failure is more dangerous than full restart. It also makes the startup/shutdown sequence completely explicit in main.go.

  2. storage.Queryable / storage.Appendable as the primary architectural seam. These two interfaces decouple the PromQL engine and scrape engine from the TSDB. Thanos, Cortex, and Mimir all exploit this: they provide their own Queryable implementations backed by object storage, while running the same PromQL engine. This decision made Prometheus’s query layer reusable across the ecosystem.

  3. Fanout write path, not event bus. Rather than publish-subscribe, the write path uses synchronous fanout: a single Appender call writes to both TSDB and remote storage. This keeps the hot path latency bounded and the code simple, at the cost of write amplification.

  4. Manual dependency injection in main.go. All wiring is done explicitly in main(). There is no DI framework, no reflection-based injection. This makes the dependency graph trivially readable (600 lines of main.go is the entire initialization story) and avoids runtime surprises, at the cost of a large, non-modular main function.

  5. Dual-mode binary (server vs agent). The --agent flag activates a write-only mode that replaces the TSDB with a WAL-only buffer, skips the query engine and rule manager, and relies entirely on remote write to forward metrics. This allows the same binary to serve both full server and edge-collector roles without code duplication, controlled by a single flag and compile-time-equivalent conditional initialization.

Backward Prometheus — Dependencies Prometheus — Patterns Forward