K3s — Architecture#

Architectural style#

Microkernel / Plugin-via-interface, In-Process Monolith

K3s presents an interesting duality: from the outside it looks like a microkernel (a thin launcher that dispatches to extracted binaries), but the extracted runtime binary is a true in-process monolith — every Kubernetes control-plane and data-plane component (API server, scheduler, controller-manager, etcd, kubelet, kube-proxy, containerd, flannel) runs as goroutines within a single OS process.

The microkernel quality comes from the Executor interface in pkg/daemons/executor/, which acts as a mandatory seam between k3s orchestration logic and k8s component invocations. The single concrete implementation (pkg/executor/embed/) calls upstream Kubernetes app.Run() entry points directly via Go function calls — no subprocess forking, no IPC, no plugin RPC. The interface exists to permit future or alternate executor implementations (e.g., a test executor or an external-process executor) while keeping all current calls in-process.

Evidence:

  • pkg/executor/embed/embed.go imports apiapp "k8s.io/kubernetes/cmd/kube-apiserver/app", sapp "k8s.io/kubernetes/cmd/kube-scheduler/app", kubelet "k8s.io/kubernetes/cmd/kubelet/app" — the same package entry points used by upstream Kubernetes, called as Go functions.
  • executor.Set(&Embedded{}) is called in init() and triggered by a blank import in main.go.
  • The Executor interface has 17 methods, one per Kubernetes component or lifecycle hook.

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────┐
│  Distribution Launcher  (cmd/k3s/main.go)                       │
│  ┌──────────┐  ┌──────────────────────────────────────────┐    │
│  │ runCLIs  │  │ stageAndRun: extract archive → exec(k3s) │    │
│  │(symlinks)│  │ crictl / kubectl / ctr / check-config    │    │
│  └──────────┘  └──────────────────────────────────────────┘    │
└──────────────────────┬──────────────────────────────────────────┘
                       │ exec()
┌──────────────────────▼──────────────────────────────────────────┐
│  In-Process Runtime  (main.go)                                  │
│                                                                  │
│  configfilearg.MustParse → urfave/cli app                       │
│                                                                  │
│  ┌────────────────────┐   ┌────────────────────────────────┐   │
│  │  pkg/cli/server    │   │  pkg/cli/agent                 │   │
│  │  Run()             │   │  Run()                         │   │
│  └────────┬───────────┘   └──────────────┬─────────────────┘   │
│           │ PrepareServer / StartServer   │ RunAgent             │
│  ┌────────▼───────────┐   ┌──────────────▼─────────────────┐   │
│  │  pkg/server        │   │  pkg/agent                     │   │
│  │  Config{} assembly │   │  setup flannel, containerd,    │   │
│  │  HTTP handler reg. │   │  tunnel, netpol, loadbalancer  │   │
│  └────────┬───────────┘   └──────────────┬─────────────────┘   │
│           │                               │                      │
│  ┌────────▼───────────────────────────────▼─────────────────┐   │
│  │  pkg/daemons/control                                      │   │
│  │  Prepare(): certs, bootstrap data, tunnel, authenticator  │   │
│  │  Server(): cluster.Start() → executor calls               │   │
│  └────────────────────────┬──────────────────────────────────┘   │
│                           │ calls via executor package           │
│  ┌────────────────────────▼──────────────────────────────────┐   │
│  │  pkg/daemons/executor (interface)  ←──── pkg/executor/embed│  │
│  │  Executor interface:                    Embedded struct     │  │
│  │  APIServer, Kubelet, KubeProxy,         calls upstream k8s │  │
│  │  Scheduler, ControllerManager,          app.Run() as Go    │  │
│  │  ETCD, Containerd, CNI, ...             function calls     │  │
│  └───────────────────────────────────────────────────────────┘   │
│                                                                  │
│  ┌───────────────┐  ┌──────────────┐  ┌──────────────────────┐  │
│  │  pkg/cluster  │  │  pkg/etcd    │  │  pkg/daemons/config  │  │
│  │  HA bootstrap │  │  etcd lifecycle  │  Node/Agent/Control  │  │
│  │  kine backend │  │  S3 snapshots│  │  shared types        │  │
│  └───────────────┘  └──────────────┘  └──────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Core components#

Distribution Launcher#

  • Package: cmd/k3s/
  • Responsibility: The binary shipped to end users. Handles symlink-based multicall dispatch (crictl, kubectl, ctr), extracts the embedded archive to a versioned data directory, and exec’s into the extracted k3s binary passing the original argv. Also manages PATH setup to include CNI and bundled binaries.
  • Key types: No exported types; all logic is in main.go functions (extract, stageAndRun, runCLIs, externalCLI, internalCLIAction).
  • Dependencies: pkg/data, pkg/datadir, pkg/dataverify, pkg/flock, pkg/untar, pkg/configfilearg

CLI + Config Assembly Layer#

  • Package: pkg/cli/server/, pkg/cli/agent/, pkg/cli/cmds/
  • Responsibility: Parses CLI flags, reads config files (via configfilearg preprocessing), assembles the large server.Config / cmds.Agent structs, and calls into the server/agent startup functions. Also validates build environment and initializes logging.
  • Key types: cmds.Server (100+ fields), cmds.Agent, server.Config
  • Dependencies: urfave/cli/v2, configfilearg, pkg/server, pkg/agent, pkg/signals, pkg/rootless, pkg/vpn

Executor Interface + Embedded Implementation#

  • Package (interface): pkg/daemons/executor/
  • Package (impl): pkg/executor/embed/
  • Responsibility: The central architectural seam. The Executor interface defines one method per Kubernetes component lifecycle call. The Embedded struct implements all 17 methods by calling upstream Kubernetes app.Run() entry points directly as Go function calls. The implementation is registered via init() and activated by a blank import in main.go.
  • Key types: executor.Executor interface, embed.Embedded struct, executor.ETCDConfig
  • Dependencies (embed): k8s.io/kubernetes/cmd/kube-apiserver/app, cmd/kube-scheduler/app, cmd/kube-controller-manager/app, cmd/kubelet/app, cmd/kube-proxy/app

Server Control Daemon#

  • Package: pkg/daemons/control/, pkg/server/
  • Responsibility: control.Prepare() loads bootstrap data, sets up the WebSocket tunnel server, and initializes the authenticator. control.Server() starts the cluster storage backend and calls the executor to launch apiserver, scheduler, controller-manager, and etcd. pkg/server assembles the HTTP handler and runs addon controllers (Helm, deploy, node controllers) after the API server is ready.
  • Key types: config.Control (the primary server config struct), server.Config
  • Dependencies: pkg/cluster, pkg/etcd, pkg/daemons/executor, rancher/wrangler, k3s-io/helm-controller

Cluster / Storage Backend#

  • Package: pkg/cluster/
  • Responsibility: Manages HA cluster bootstrap and storage backend selection. k3s supports embedded etcd (default) or an external datastore via Kine (SQLite, PostgreSQL, MySQL). cluster.Start() bootstraps from the datastore and initiates leader election.
  • Key types: cluster.Cluster
  • Dependencies: k3s-io/kine, pkg/etcd, pkg/clientaccess, pkg/bootstrap

Agent Daemon#

  • Package: pkg/agent/, pkg/daemons/agent/
  • Responsibility: Manages the node-level Kubernetes components. Sets up containerd (or Docker/CRI-dockerd), flannel CNI, the WireGuard VPN (if configured), network policy enforcement, and the client-side load balancer for server endpoints. Also establishes and maintains the WebSocket tunnel connection to the server.
  • Key types: daemonconfig.Node, daemonconfig.Agent
  • Dependencies: pkg/agent/containerd, pkg/agent/flannel, pkg/agent/tunnel, pkg/agent/loadbalancer, pkg/daemons/executor

Shared Config Types#

  • Package: pkg/daemons/config/
  • File: pkg/daemons/config/types.go
  • Responsibility: Defines the central data model used across all components. Node holds node-level config (containerd, flannel, agent config). Control / ControlConfig hold server-level config (datastore, TLS, ports, feature flags, runtime paths). Runtime accumulates derived state (file paths, channels, live clients).
  • Key types: Node, Agent, Control, ControlConfig, Runtime, EtcdS3, Containerd, Flannel

Data flow#

Server startup (k3s server):#

1. cmd/k3s/main.go
   → extract embedded archive → exec k3s-server binary

2. main.go (in-process runtime)
   → init(): executor.Set(&Embedded{})  [blank import triggers registration]
   → configfilearg.MustParse(os.Args)   [YAML config → CLI args expansion]
   → urfave/cli parses flags
   → server.Run(ctx)

3. pkg/cli/server/server.go: run()
   → EvacuateCgroup2(), InitLogging()
   → signals.SetupSignalContext()       [SIGTERM/SIGINT → context cancel]
   → assemble server.Config{} from CLI flags
   → server.PrepareServer(ctx, wg, config, cfg)

4. pkg/server/server.go: PrepareServer()
   → control.Prepare(ctx, wg, &config.ControlConfig)
      → setupDataDirAndChdir()          [create /var/lib/rancher/k3s/server/]
      → cluster.Bootstrap()             [load bootstrap data from datastore]
      → setupTunnel()                   [WebSocket tunnel server handler]
      → authenticator.FromArgs()        [basic-auth + client-CA authenticator]
   → handlers.NewHandler()              [HTTP handler for supervisor API]

5. pkg/server/server.go: StartServer()
   → control.Server(ctx, wg, &config.ControlConfig)
      → cluster.Start()                 [start kine/etcd storage backend]
      → executor.APIServer(ctx, args)   [kube-apiserver.app.Run() in goroutine]
      → <-executor.APIServerReadyChan() [wait for apiserver health]
      → executor.Scheduler(...)
      → executor.ControllerManager(...)
   → start addon controllers (wrangler):
      deploy controller, helm controller, node controller, secrets-encrypt controller
   → write admin kubeconfig
   → optionally: agent.Run() in-process [if --disable-agent not set]

6. pkg/agent/ (if agent enabled):
   → executor.Bootstrap()              [initialize agent config, ready channels]
   → executor.Containerd()             [start containerd in goroutine]
   → <-executor.CRIReadyChan()         [wait for CRI]
   → executor.Kubelet(ctx, args)
   → executor.KubeProxy(ctx, args)
   → executor.CNI(ctx, wg, node)       [start flannel in goroutine]
   → agent/tunnel: WebSocket tunnel to server

Initialization / Bootstrap#

Init sequence for the embedded executor (build-tag registration pattern):

  1. main.go has a blank import: _ "github.com/k3s-io/k3s/pkg/executor/embed"
  2. This triggers embed.init() which calls executor.Set(&Embedded{}), storing the concrete implementation in a package-level variable in pkg/daemons/executor/
  3. All subsequent calls to executor.APIServer(...), executor.Kubelet(...) etc. are forwarded to the Embedded struct’s methods
  4. This is guarded by build tag //go:build !no_embedded_executor — omitting the blank import and setting the tag allows building k3s without embedded k8s components

Dependency injection: Manual wiring via function parameters and large config structs. No DI framework (no wire, dig, or fx). The executor singleton is the one instance of global mutable state used for wiring.

Readiness channels: Components signal readiness by closing channels (apiServerReady, etcdReady, criReady). Dependent components block on <-chan struct{} reads. This is the primary synchronization mechanism for startup ordering.

Configuration#

k3s uses a two-phase configuration approach:

  1. YAML config file preprocessing (pkg/configfilearg/): Before urfave/cli parses arguments, MustParse(os.Args) scans for --config / -c flags, reads the YAML file, and expands key-value pairs into equivalent CLI flags (--key=value). This happens before the CLI library runs, giving a unified flag/file config surface with no duplicated parsing logic.

  2. CLI flags (pkg/cli/cmds/): urfave/cli/v2 with extensive flag definitions. The cmds.ServerConfig and cmds.AgentConfig are package-level structs populated by flag actions.

  3. Config struct assembly: pkg/cli/server/server.go:run() manually copies all CLI flag values into server.Config and config.Control. There is no automatic binding — it is an explicit, verbose mapping (200+ lines of assignments).

  4. Environment variables: Selected settings (K3S_DATA_DIR, K3S_DEBUG, K3S_TOKEN, CRI_CONFIG_FILE) are read directly from the environment before CLI parsing, providing a parallel configuration pathway.

  5. Runtime config: config.Runtime in pkg/daemons/config/types.go accumulates derived paths, live client references, and channels during startup. It is not user-facing but acts as a mutable context object passed through the system.

Key design decisions#

  1. In-process Kubernetes components via the Executor interface seam. Running apiserver, kubelet, scheduler, and etcd as goroutines in the same process eliminates container/process startup overhead and allows k3s to boot in under 10 seconds on constrained hardware. The Executor interface (17 methods) provides a clean separation between k3s orchestration logic and upstream k8s code, and allows the no_embedded_executor build tag to exclude the upstream k8s dependency entirely for testing or alternative implementations.

  2. Two-tier binary design with embedded archive. The distribution binary (cmd/k3s/) is a thin launcher that self-extracts. This allows k3s to ship as a single file while still supporting exec-based isolation for tools like crictl and kubectl that cannot share the same process namespace. The versioned extraction directory (/var/lib/rancher/k3s/data/<hash>/) also supports atomic upgrades: the symlink current → previous swap allows rollback.

  3. Kine as a storage backend abstraction. Rather than requiring etcd, k3s layers kine between the k8s storage layer and the actual database. This translates etcd v3 API calls to SQL queries, enabling SQLite (the default for single-node) or PostgreSQL/MySQL for HA setups — without modifying any upstream Kubernetes code.

  4. Wrangler for Kubernetes controllers. All k3s-specific controllers (addon deploy, Helm chart, node label/annotation, secrets encryption) are implemented using Rancher Wrangler, a generated-controller framework that provides strongly-typed client/informer/cache wrappers. This is a Rancher-ecosystem choice that reflects k3s’s origins as a Rancher Labs project.

  5. WebSocket tunnel for agent-server communication. Rather than requiring agents to have direct API server access, k3s establishes a WebSocket tunnel from each agent to the server’s supervisor port. The server multiplexes API server traffic through this tunnel, enabling air-gapped and NAT-traversal deployments where agents can initiate outbound connections to the server but the server cannot reach agents directly.

Backward K3s — Dependencies K3s — Patterns Forward