Orchestrator Architecture Comparison: Kubernetes, k3s, Nomad#

Summary#

These three orchestrators solve the same core problem — placing and running workloads across a fleet of machines — but represent meaningfully different answers to how much complexity is acceptable. Kubernetes maximises extensibility at the cost of operational complexity; Nomad maximises operational simplicity at the cost of ecosystem breadth; k3s occupies an unusual third position by repackaging Kubernetes itself into a single binary, revealing through its architecture just how much Kubernetes’s distributed design was a choice, not a necessity.

Comparison dimensions#

State management and consistency model#

Narrative: The most architecturally significant divergence between Nomad and the Kubernetes family is here. Kubernetes externalises consensus entirely — etcd is a separate cluster, and kube-apiserver is the sole gatekeeper. This gives Kubernetes a clean separation of concerns but requires operators to manage etcd separately, and introduces an API server as a single bottleneck. The SharedInformer pattern (each controller maintains a local cache synchronised via watch streams) is a direct response to this bottleneck: it prevents every controller from hammering the API server at reconcile time.

Nomad embeds Raft directly. There is no separate etcd, no API server, and no local cache layer — scheduler workers read directly from go-memdb via WatchSets. The blocking query pattern (MinQueryIndex) provides watch semantics at the RPC layer without a separate cache infrastructure. This makes Nomad far simpler to operate but means every write is a Raft log entry flowing through the server binary, making strong leader affinity an architectural necessity.

k3s’s Kine layer is an elegant hack: it translates the etcd v3 wire protocol to SQL, allowing SQLite to serve as the state store for single-node deployments without modifying any upstream Kubernetes code. The boundary is exactly at the etcd client API, showing that Kubernetes’s coupling to etcd is entirely at the protocol level, not at any deeper architectural level.

Component topology and deployment model#

Narrative: Kubernetes’s multi-process design is a deliberate architectural choice to maximise fault isolation and independent scaling — a crashing controller manager doesn’t kill the scheduler. The inter-process-communication-through-the-API-server rule is the single most important constraint in Kubernetes’s architecture: it enables stateless components, crash recovery, and horizontal scaling at the cost of significant operational complexity.

k3s inverts this at the extreme: all Kubernetes components run as goroutines inside a single process. This eliminates container startup overhead (k3s boots in under 10 seconds on constrained hardware versus several minutes for a full Kubernetes cluster) and makes single-node operation trivial. The Executor interface (pkg/daemons/executor/) is the seam that makes this work — it is the single place in k3s that isolates “how we launch Kubernetes components” from “upstream Kubernetes code.” The build tag no_embedded_executor can swap in an alternative or test implementation.

Nomad’s single-binary approach differs from k3s’s in a key way: Nomad’s single binary deliberately separates server and client roles that can run together or apart, while k3s’s single binary is Kubernetes with the distribution layer thinned. Nomad’s subprocess isolation for task drivers (go-plugin gRPC) provides real OS-level fault isolation where it matters most — a crashing Docker driver cannot bring down the scheduler or other running tasks.

Scheduling model#

Narrative: The Kubernetes scheduler is a framework: built-in predicates (NodeAffinity, ResourceFit, Taints/Tolerations) are just plugins registered at startup. This enables out-of-tree schedulers and scheduler plugins (batch job packers, GPU-aware schedulers) without forking. The extension point ordering (PreFilter → Filter → PostFilter → PreScore → Score → NormalizeScore → Reserve → Permit → Bind) is well-documented and stable.

Nomad’s scheduler is architecturally isolated in a different way: the scheduler/ package has zero import dependency on the nomad/ server package. It receives only two interfaces — scheduler.State (read-only memdb view) and scheduler.Planner (write-only plan submission). This enables pure-unit testing of scheduling logic without a running server, and the algorithms (bin-packing, spread, constraint evaluation) are substantially more testable than their Kubernetes counterparts. The tradeoff is no plugin system for community-contributed schedulers; built-in types are registered by name in a map.

The Kubernetes plugin framework’s extensibility is essential for the operator ecosystem (cluster autoscalers, custom node selectors, batch schedulers). Nomad’s approach prioritises correctness and testability over extensibility.

Extensibility and plugin model#

Narrative: Kubernetes’s extensibility model evolved in stages: first in-tree plugins, then admission webhooks, then CRDs + operators. Today the dominant extension mechanism is the CRD + operator pattern: users extend Kubernetes’s data model by defining Custom Resources, then deploy a controller that reconciles them. The entire model is built on the same API server, SharedInformer, and reconciliation loop infrastructure that powers built-in resources. This is the most powerful extensibility model of the three — any domain concept can be first-class in Kubernetes — but it requires users to write a Go controller using client-go (or controller-runtime).

Nomad’s go-plugin approach for task drivers is the most principled subprocess isolation of the three: each driver is a separate OS process connected via gRPC over stdio. A crashing Docker driver does not take down the client. Third-party driver authors ship a separate binary implementing the versioned gRPC DriverPlugin interface. The plugins/ SDK defines the versioned contract; the pluginmanager handles subprocess lifecycle. This is a production-grade plugin system that Kubernetes deliberately avoided in favour of interface-based in-process plugins (at the cost of putting the burden on webhook authors to operate separate HTTPS servers).

k3s’s Executor interface is an architectural seam, not a true plugin system. It exists to isolate k3s orchestration code from upstream Kubernetes, and the no_embedded_executor build tag enables test or alternative implementations. It is not designed for third-party extension.

Consensus and leader election#

Narrative: Kubernetes’s use of an API resource (a Lease object in etcd) as the distributed lock for leader election is elegant: no additional infrastructure, the lock is visible via kubectl, and the same mechanism works for both the controller manager and external operators. The downside is etcd must be available for any leader election.

Nomad’s embedded Raft gives it a fundamentally different operational profile: consensus and leader election are intrinsic to the server binary. The monitorLeadership() goroutine pattern — watching a Go channel from Raft and calling broker.SetEnabled() — is a clean example of how to implement leader-only activation in Go. The EvalBroker’s enabled flag (guarded by a mutex, toggled atomically during leadership change) ensures scheduler workers pause cleanly on leadership transitions without a separate coordination service.

Concurrency patterns#

Narrative: All three projects converge on context cancellation for graceful shutdown, but at different levels of adoption. Kubernetes is the most complete: context.Context is threaded through virtually every public function (16,784 usages), and the API server’s handler chain attaches a request-scoped context at entry. Nomad shows its age: older code uses shutdownCh chan struct{} pervasively (307 close() calls), with context.Context appearing primarily in newer subsystems (CSI, event streaming). k3s uses context consistently, layered over a sync.WaitGroup for clean shutdown.

k3s’s readiness channel pattern (<-executor.APIServerReadyChan()) for startup ordering is elegant and self-documenting. The method name and <-chan struct{} return type together form a clear contract: “block here until this component is ready.” It avoids polling or arbitrary sleeps — the channel closes exactly once when the component signals readiness. Kubernetes achieves the same goal via post-start hooks and readiness probes (HTTP checks), which are more sophisticated but also more complex.

Nomad’s blocking query pattern (MinQueryIndex) is the most distinctive concurrency feature: every read RPC accepts an index value and blocks at the go-memdb WatchSet level until the relevant state changes. This is watch semantics without a separate subscription stream, implemented directly in the RPC layer. It eliminates the need for a SharedInformer-style local cache, but places constraints on the server’s ability to handle many simultaneous blocked RPCs.

Configuration approach#

Narrative: The three projects reflect different eras and communities. Kubernetes predates the config-file revolution and remains flag-only in production; the two-stage Options → CompletedConfig pattern is rigorous but verbose. k3s’s configfilearg approach is clever: it converts YAML config files into CLI flags before the CLI library runs, giving a single unified surface without duplicating flag definitions. Nomad’s HCL2 approach is the most powerful (full expression language, hierarchical merge, multi-file composition) and reflects HashiCorp’s investment in a configuration language used across their entire product line.

None of the three use Viper, which is notable given how commonly Viper appears in the broader Go ecosystem. All three rely on typed structs with explicit field assignments rather than automatic unmarshalling into configuration objects, which makes the configuration flow auditable at the cost of verbosity.

Common patterns#

All three projects share these architectural commitments:

1. Manual dependency injection. None use google/wire, uber/dig, or uber/fx. Every component is wired by explicit New(deps...) calls. This is a deliberate choice: at scale, a DI container adds complexity without benefit when package boundaries are well-defined.

2. Interface as the primary extension point. The Executor interface (k3s), the Scheduler interface (Nomad), and the scheduler Framework interfaces (Kubernetes) are all interface-based seams defined by consumers, not providers. This reflects idiomatic Go design.

3. Conservative generics adoption. All three use Go generics only for utility functions (generic stacks, LRU caches, slice helpers) and not in domain logic. The pre-generics architectures (interfaces + type assertions) remain dominant in all three codebases.

4. Single binary multi-role dispatch. All three projects route multiple roles through a single binary: Kubernetes via separate binaries but symlinked kubectl; k3s via the multicall launcher and reexec registry; Nomad via the init() subprocess dispatch trick. Operational simplicity trumps architectural cleanliness at the distribution boundary.

5. Level-triggered reconciliation. All three approach desired-state reconciliation by reading current state at reconcile time rather than tracking events. Kubernetes makes this explicit (“level-triggered, not edge-triggered”); Nomad’s scheduler reads fresh from memdb at evaluation time; k3s inherits the Kubernetes approach.

Divergent choices#

Consensus ownership: This is the deepest architectural divide. Kubernetes offloads consensus entirely to etcd; Nomad owns it. Kubernetes’s choice enables a simpler codebase (no Raft implementation, no gossip) but introduces operational complexity (etcd must be kept healthy, backed up, and upgraded separately). Nomad’s embedded Raft makes the Nomad binary a complete distributed system but tightly couples the scheduling layer to Raft’s single-leader model.

Plugin isolation strategy: Kubernetes accepts that scheduler plugins and controllers run in-process with the scheduler and controller manager — a crashing custom plugin can destabilise the system. Nomad isolates at the OS process boundary for task drivers. k3s accepts no isolation (all Kubernetes components in one process). The spectrum of “speed vs isolation” maps directly to these choices.

Watch semantics implementation: Kubernetes builds an entire local cache tier (SharedInformer) to shield the API server from read load. Nomad builds watch semantics into the RPC protocol (blocking queries with index). k3s inherits Kubernetes’s approach. The Nomad approach is simpler but places a ceiling on the number of concurrent blocking RPCs the server can handle; the Kubernetes approach scales further but adds a cache consistency layer to reason about.

Domain model coupling: Nomad deliberately centralises all domain types in nomad/structs (no internal Nomad dependencies). Kubernetes distributes types across staging/src/k8s.io/api* with a Scheme-based registry for serialisation. Nomad’s centralised approach is simpler but creates a wide coupling surface; Kubernetes’s registry approach enables API versioning and evolution at the cost of complexity.

Recommendations for practitioners#

Choose Kubernetes (or a Kubernetes distribution) when:

• You need the operator pattern — CRDs + controllers that extend the platform’s data model
• You have a team familiar with Kubernetes APIs and the CNCF ecosystem
• You need fine-grained RBAC, admission webhooks, or aggregated API servers
• You are running more than ~100 nodes and need independent component scaling

Choose k3s when:

• You need standard Kubernetes API compatibility (existing Helm charts, operators, tooling)
• But you have constrained hardware, limited operators, or need sub-10-second startup
• Edge computing, IoT, CI environments, or single-developer clusters

Choose Nomad when:

• You need to run non-containerised workloads (VMs via QEMU, raw binaries, batch jobs, Java apps) alongside containers
• You want operational simplicity over ecosystem breadth — Nomad’s single binary with embedded Raft genuinely reduces cluster management overhead
• Your team is already in the HashiCorp ecosystem (Consul, Vault, Terraform integration is first-class)
• You are not invested in Kubernetes-specific tooling (Helm, operators, CRDs)

Book angle#

The story this comparison tells is: the same problem, solved at three different points on the complexity/capability trade-off curve — and Go’s idioms appear at each level, stretched in different directions.

The chapter lesson is architectural seams. Each project defines a single critical interface that isolates the most volatile part of the system:

• Kubernetes: storage.Interface — the seam between the API machinery and etcd; everything above it is portable to any backend.
• k3s: executor.Executor — the seam between k3s’s orchestration logic and upstream Kubernetes; 17 methods that allow a single goroutine-per-component implementation today and an out-of-process implementation behind a build tag.
• Nomad: scheduler.State + scheduler.Planner — the seam between the scheduling algorithm and the server’s state machine; zero import dependency in either direction, enabling pure-unit testing of bin-packing.

In each case the seam is defined as a small, focused interface. In each case it was designed by the consumer (the code that calls it), not the provider. And in each case the interface is the single most important decision in the project’s architecture — the line that makes the rest of the code testable, replaceable, and auditable. That is the lesson worth a chapter.