Nomad — Architecture#

Architectural style#

Layered Distributed System with Plugin-based Extension and Raft Consensus

Nomad’s architecture is best described as a distributed layered system where:

  1. Consensus layer (Raft via hashicorp/raft) guarantees strong consistency for all cluster state mutations.
  2. Coordination layer (Serf via hashicorp/serf) handles cluster membership gossip and multi-region federation.
  3. Scheduling layer (scheduler/ package) — deliberately isolated from the server — performs bin-packing and constraint evaluation.
  4. Execution layer (client/ package) runs allocations locally, using a plugin-based task driver system.
  5. API layer (command/agent) exposes HTTP REST + RPC, translating external requests into internal Raft-backed state mutations.

The project additionally uses a plugin-based extension model for task drivers (Docker, exec, QEMU, Java) via hashicorp/go-plugin, where each driver runs as a child subprocess communicating over RPC.

Evidence: nomad/server.go calls setupRaft(), setupSerf(), setupWorkers() and scheduler.BuiltinSchedulers as distinct steps; scheduler/structs/interfaces.go defines the Scheduler interface with no import of the nomad/ server package; drivers/ each implement the plugins/drivers interface and are loaded via pluginmanager.

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────┐
│                         Nomad Binary                            │
│                                                                 │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │              command/agent.Agent                         │   │
│  │  (lifecycle owner; holds both Server and Client refs)    │   │
│  │                                                          │   │
│  │  ┌─────────────────────┐  ┌────────────────────────┐   │   │
│  │  │  command/agent HTTP  │  │  nomad.Server          │   │   │
│  │  │  (REST API server)   │  │                        │   │   │
│  │  │  - /v1/* endpoints   │──│  - Raft FSM (nomadFSM) │   │   │
│  │  │  - WebSocket streams │  │  - nomad/state (memdb) │   │   │
│  │  └──────────┬──────────┘  │  - EvalBroker          │   │   │
│  │             │ RPC call    │  - PlanQueue            │   │   │
│  │             ▼             │  - Workers[]            │   │   │
│  │  ┌─────────────────────┐  │  - Serf (membership)   │   │   │
│  │  │  nomad.rpcHandler   │  │  - PeriodicDispatcher  │   │   │
│  │  │  (net/rpc over TCP) │  │  - DeploymentWatcher   │   │   │
│  │  │  forward to leader  │  │  - NodeDrainer         │   │   │
│  │  └─────────────────────┘  └────────────┬───────────┘   │   │
│  │                                         │ dequeues      │   │
│  │                           ┌─────────────▼───────────┐  │   │
│  │                           │  scheduler.Scheduler    │  │   │
│  │                           │  (service/batch/system) │  │   │
│  │                           │  State + Planner ifaces │  │   │
│  │                           └─────────────────────────┘  │   │
│  │                                                          │   │
│  │  ┌──────────────────────────────────────────────────┐  │   │
│  │  │  client.Client                                    │  │   │
│  │  │  - Fingerprinter (CPU/mem/net/GPU)                │  │   │
│  │  │  - AllocRunner per allocation                     │  │   │
│  │  │  - PluginManager (task drivers, CSI, device)      │  │   │
│  │  │  - BoltDB state persistence                       │  │   │
│  │  │  - Service registration (Consul/Nomad native)     │  │   │
│  │  │                                                    │  │   │
│  │  │  AllocRunner → TaskRunner → Driver subprocess     │  │   │
│  │  │  (executor/docker_logger/logmon via go-plugin RPC)│  │   │
│  │  └──────────────────────────────────────────────────┘  │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

External:   Consul (service discovery, catalog)
            Vault (secrets, ACL bootstrap)
            Raft peers (TCP, other Nomad servers in region)
            Serf peers (UDP/TCP gossip, multi-region)

Core components#

Agent#

  • Package: command/agent
  • Responsibility: Top-level lifecycle container for a running Nomad process. Owns both nomad.Server and client.Client (a single binary can be server-only, client-only, or both). Initializes Consul clients, loads plugin catalog, starts the HTTP API, manages TLS metrics and shutdown sequencing.
  • Key types: Agent struct (agent.go)
  • Dependencies: nomad.Server, client.Client, command/agent/consul, helper/pluginutils/loader

HTTP API Server#

  • Package: command/agent (http.go + *_endpoint.go files)
  • Responsibility: REST HTTP server exposing /v1/* endpoints and WebSocket-based streaming. Translates HTTP requests into RPC calls against the local server or client. Does not hold its own state — it’s a thin adapter between HTTP and the RPC layer.
  • Key types: HTTPServer struct
  • Dependencies: nomad.Server (via RPC), net/http stdlib

Nomad Server#

  • Package: nomad
  • Responsibility: The consensus and orchestration engine. Maintains the authoritative cluster state via Raft. Exposes an RPC interface (net/rpc over TCP + msgpack) that both the HTTP API and other servers call. Runs scheduler workers that dequeue evaluations and execute scheduling algorithms. Manages the evaluation broker, plan queue, deployment watcher, node drainer, periodic dispatcher, and keyring.
  • Key types: Server (server.go), nomadFSM (fsm.go), EvalBroker (eval_broker.go), Worker (worker.go), planner (plan_endpoint.go)
  • Dependencies: hashicorp/raft, hashicorp/serf, nomad/state, scheduler, nomad/structs

Raft FSM and State Store#

  • Package: nomad (fsm.go) + nomad/state
  • Responsibility: nomadFSM implements raft.FSM. Every mutating operation (job registration, allocation update, node registration) is serialized as a Raft log entry. Apply() decodes the log and dispatches to the in-memory state store built on go-memdb. The state store is the single source of truth for all server-side data; it is replicated across all Raft peers.
  • Key types: nomadFSM, StateStore (nomad/state/state_store.go)
  • Dependencies: hashicorp/raft, hashicorp/go-memdb, bbolt (raft log backend)

Evaluation Broker#

  • Package: nomad
  • Responsibility: In-memory priority queue for pending evaluations. The broker dequeues evaluations for scheduler workers to process. Supports nacking (re-enqueue on worker failure), blocking when the broker is disabled (non-leader), and priority ordering. It is only active on the leader.
  • Key types: EvalBroker (eval_broker.go)
  • Dependencies: none external; communicates via channels

Scheduler Workers#

  • Package: nomad (worker.go) + scheduler
  • Responsibility: Each Worker goroutine loops, dequeuing evaluations from the EvalBroker and invoking scheduler.NewScheduler().Process(eval). The scheduler produces a Plan (a set of proposed allocations), which is submitted back to the server’s PlanQueue for leader commit. Workers run on every server node but only process evaluations when the broker is enabled (i.e., on the current leader).
  • Key types: Worker, scheduler.Scheduler interface, scheduler.BuiltinSchedulers (service, batch, system, sysbatch)
  • Dependencies: scheduler/structs.State, scheduler/structs.Planner

Scheduler#

  • Package: scheduler
  • Responsibility: Standalone scheduling algorithms with no import dependency on nomad/ (server). Consumes read-only State and write-only Planner interfaces. Implements bin-packing (service jobs), spread, feasibility checking (constraints, affinities, resource availability), and reconciliation (deployment diffs). The four built-in scheduler types each implement the single-method Scheduler interface.
  • Key types: Scheduler interface (scheduler/structs/interfaces.go), GenericScheduler (generic_sched.go), SystemScheduler
  • Dependencies: nomad/structs (domain types), scheduler/structs (interfaces), scheduler/feasible

Nomad Client#

  • Package: client
  • Responsibility: The agent subsystem that runs on worker nodes. Fingerprints host resources, registers with servers, polls for allocation assignments, and runs AllocRunner per allocation. Persists state to BoltDB to survive restarts. Manages plugin lifecycle (task drivers, device plugins, CSI plugins via pluginmanager). Handles service registration with Consul or Nomad’s native service discovery.
  • Key types: Client (client.go), AllocRunner interface (client/interfaces)
  • Dependencies: client/allocrunner, client/pluginmanager, client/fingerprint, client/state (BoltDB), plugins/

AllocRunner and TaskRunner#

  • Package: client/allocrunner + client/allocrunner/taskrunner
  • Responsibility: AllocRunner manages the lifecycle of a single allocation (a co-located group of tasks). It runs pre/post hooks (volume mounting, service registration, identity injection) and starts a TaskRunner per task. TaskRunner invokes the task driver plugin and manages task restarts, log collection, and health checking.
  • Key types: AllocRunner (allocrunner.go), TaskRunner (taskrunner/task_runner.go)
  • Dependencies: plugins/drivers (via go-plugin RPC), client/allocrunner/interfaces

Plugin System#

  • Package: plugins/ (SDK) + drivers/ (implementations) + client/pluginmanager
  • Responsibility: Defines the extension interfaces for task drivers (plugins/drivers), device plugins (plugins/device), and CSI plugins (plugins/csi). Built-in drivers (Docker, exec, Java, QEMU, rawexec) implement these interfaces. At runtime each driver runs as a child subprocess of the Nomad client, communicating via gRPC over stdio using hashicorp/go-plugin. This provides isolation: a crashing driver does not take down the client.
  • Key types: DriverPlugin interface (plugins/drivers/driver.go), go-plugin gRPC transport
  • Dependencies: hashicorp/go-plugin, gRPC, protobuf

Domain Model#

  • Package: nomad/structs
  • Responsibility: Canonical structs for all core domain concepts: Job, TaskGroup, Task, Allocation, Node, Evaluation, Deployment, NodePool, ACLPolicy, Variable, etc. Imported by nearly every package in the codebase. This is a deliberate monolithic domain model (not split by subdomain) — a central coupling point that enables uniform serialization and state management.
  • Key types: Job, Allocation, Node, Evaluation, Plan
  • Dependencies: stdlib only (no internal Nomad deps)

Data flow#

Job submission (primary path)#

1. Operator runs: nomad job run my.nomad

2. CLI command/job_run.go
   → HTTP POST /v1/jobs  (JSON body)

3. command/agent/job_endpoint.go HTTPServer.jobRegister()
   → calls server RPC: Job.Register(JobRegisterRequest)

4. nomad/job_endpoint.go Job.Register()
   → j.srv.forward("Job.Register", ...) 
     [if not leader: forwards to leader via TCP RPC]
   → ACL check
   → validates job (HCL already parsed by API client)
   → raft.Apply(RegisterJobRequestType, msgpack-encoded request)

5. Raft replicates log entry to quorum of servers

6. nomadFSM.Apply() on each server
   → decodes RegisterJobRequestType
   → state.UpsertJob() → go-memdb write

7. Leader creates an Evaluation (trigger: job changed)
   → raft.Apply(EvalUpdateRequestType)
   → FSM: state.UpsertEvals()
   → evalBroker.Enqueue(eval)

8. Worker goroutine (on leader) dequeues eval
   → scheduler.NewScheduler("service", ...).Process(eval)
   → reads state (nodes, existing allocs, constraints)
   → feasibility.CheckNodeConstraints() per candidate node
   → bin-packing: selects nodes for each task group
   → produces Plan{NodeAllocation: map[nodeID][]Allocation}

9. Worker submits Plan → planQueue
   → leader's plan_apply.go applies plan
   → raft.Apply(AllocUpdateRequestType) for accepted allocs
   → FSM: state.UpsertAllocs()

10. Client nodes poll server: Node.GetAllocs() RPC (blocking, index-based)
    → server returns delta when state index advances

11. client.Client receives new Allocation assignment
    → creates AllocRunner(alloc)
    → AllocRunner runs hooks (volume mount, identity, services)
    → starts TaskRunner per task
    → TaskRunner.Run() → driver.StartTask() via go-plugin RPC
    → task subprocess starts (e.g. docker run)

12. TaskRunner monitors health, collects logs via logmon subprocess
    → updates alloc status back to server: Node.UpdateAlloc() RPC
    → server FSM updates allocation state in memdb

Blocking queries (watch semantics)#

Clients and the HTTP API use blocking RPCs: every read RPC accepts a MinQueryIndex and blocks until the state store’s index advances past that value. This is Nomad’s polling-efficient watch mechanism, built on go-memdb watch sets.

Initialization / Bootstrap#

Subprocess init trick (main.go)#

The main.go blank-imports several packages whose init() functions check os.Args[0] or environment variables and, if they match, immediately drop into subprocess logic (executor, logmon, docker_logger, getter, template-render). This happens before the hashicorp/cli framework initializes, keeping memory overhead minimal for child processes that don’t need the full binary’s imports.

Agent startup sequence (NewAgent)#

main() → Run() → cli.Run() → command/agent.Command.Run()
  → NewAgent(config, logger, logOutput, inmem)
      1. setupConsuls()         — Consul API clients (may be multiple)
      2. setupServer()          — if server mode:
           → convertServerConfig()
           → nomad.NewServer()
               a. TLS configuration
               b. ConnPool (yamux multiplexed TCP)
               c. EvalBroker + BlockedEvals
               d. RPC handler + RPC server (net/rpc)
               e. setupRaft() → BoltDB/WAL log store, InmemStore, NetworkTransport, FSM
               f. setupSerf() → gossip cluster
               g. setupWorkers() → N scheduler worker goroutines
               h. setupDeploymentWatcher(), setupVolumeWatcher(), setupNodeDrainer()
               i. monitorLeadership() goroutine (enables/disables EvalBroker on election)
               j. startRPCListener() goroutine
               k. IsReady() — blocks until keyring decrypted
      3. setupClient()          — if client mode:
           → client.NewClient()
               a. BoltDB state restore (surviving allocs)
               b. Plugin catalog load + plugin manager start
               c. Fingerprinter goroutines (CPU, memory, network, GPU)
               d. RPC connection to server(s)
               e. registerNode() RPC
               f. allocSync goroutine (polls for new allocations)
      4. setupEnterpriseAgent() — no-op in CE
      5. Start HTTP server (command/agent.NewHTTPServer)

Dependency injection: Entirely manual. No DI framework (no Wire, Dig, or Fx). Components are constructed by explicit NewXxx(deps...) calls in the setup functions above. Interfaces (e.g., scheduler.State, scheduler.Planner) are passed as constructor arguments. The Server struct embeds *planner and *nodeHeartbeater directly.

Configuration#

Configuration is layered and merged from multiple sources:

  1. HCL/JSON config files — primary mechanism; supports full HCL2 expressions. Parsed in command/agent/config_parse.go using github.com/hashicorp/hcl/v2.
  2. CLI flags — override individual config fields. Defined via standard Go flag package in command/agent/command.go.
  3. Environment variables — a subset of options can be set via env vars (e.g., NOMAD_ADDR, VAULT_TOKEN).
  4. Consul-based server discovery — clients can find servers via Consul service catalog if no explicit server_join is configured.

Config structs live in command/agent/config.go (agent-level) and nomad/structs/config/ (subsystem-level). The convertServerConfig() function in agent.go maps the HCL-parsed agent config into the nomad.Config struct used by the server.

No Viper. No centralized config registry. Configuration is resolved at startup and stored in typed structs.

Key design decisions#

1. Raft without external dependencies#

Nomad bundles Raft consensus (hashicorp/raft) directly, eliminating the need for etcd or ZooKeeper. The state store uses go-memdb (in-memory, index-based) replicated via Raft logs. This gives strong consistency for scheduling decisions without external operational complexity — a key differentiator from Kubernetes. The backend is pluggable between BoltDB and the newer WAL (hashicorp/raft-wal) via the raftBackend interface.

2. Scheduler isolated from server via interfaces#

The scheduler/ package has zero import dependency on nomad/ (server package). It communicates exclusively through scheduler/structs.State (read-only memdb view) and scheduler/structs.Planner (write-only plan submission). This decoupling means the scheduling algorithms can be tested in complete isolation, benchmarked independently, and — in principle — replaced. The four built-in scheduler types (service, batch, system, sysbatch) are registered in a map and instantiated by name.

3. Task driver subprocess isolation via go-plugin#

Every task driver runs as a separate OS process, connected to the Nomad client via gRPC over stdio. A crashing Docker driver does not crash the Nomad client; it is simply restarted. This pattern (from hashicorp/go-plugin) also allows third-party driver authors to ship plugins as separate binaries without forking Nomad. The plugins/ SDK defines the versioned gRPC interfaces; drivers/ contains the first-party implementations.

4. Single binary multi-role subprocess pattern#

The init() trick in main.go allows the same nomad binary to serve as: CLI, server, client agent, executor subprocess, logmon subprocess, and docker_logger subprocess. The blank imports at the top of main.go register init functions that intercept execution before the CLI framework loads. This avoids shipping separate binaries and simplifies deployment, at the cost of some architectural complexity in the startup path.

5. Centralized domain model in nomad/structs#

All core domain types (Job, Allocation, Node, Evaluation, Deployment, etc.) live in one package imported by nearly every other package. This creates intentional coupling: there is one authoritative representation of each concept, which simplifies serialization, state diffing, and cross-subsystem communication. The tradeoff is that nomad/structs cannot be changed without potentially affecting the entire codebase — a deliberate pragmatic choice for a system with pervasive cross-cutting concerns.

Backward Nomad — Dependencies Nomad — Patterns Forward