K3s — Structure#

Layout pattern#

Custom — Dual-Binary Distribution Layout

K3s uses a variation of Standard Go Layout (cmd/ + pkg/) with a critical twist: it maintains two separate main entry points — a root main.go (the in-process embedded k8s binary) and cmd/k3s/main.go (the distribution launcher that extracts and dispatches). The internal/ directory is absent; all packages live under pkg/. There is no vendor/ directory. The layout reflects the project’s unusual distribution model: it ships a single binary that bootstraps by unpacking other binaries from embedded data assets.

Directory map#

k3s/
├── main.go                 # Root: unified in-process binary (embeds all k8s components)
├── cmd/                    # Individual sub-command entry points
│   ├── k3s/main.go         # Distribution launcher: multicall wrapper + binary extractor
│   ├── server/main.go      # Standalone server subcommand binary
│   ├── agent/main.go       # Standalone agent subcommand binary
│   ├── kubectl/main.go     # Thin kubectl wrapper
│   ├── ctr/main.go         # Thin ctr (containerd CLI) wrapper
│   ├── containerd/main.go  # Containerd binary entry
│   ├── cert/main.go        # Certificate management subcommand
│   ├── encrypt/main.go     # Secret encryption subcommand
│   ├── etcdsnapshot/main.go # etcd snapshot subcommand
│   ├── completion/main.go  # Shell completion subcommand
│   └── token/main.go       # Token management subcommand
├── pkg/                    # All Go packages (no internal/)
│   ├── agent/              # Node agent logic (CRI, CNI, flannel, tunnels)
│   ├── authenticator/      # Basic auth, password file, hash
│   ├── bootstrap/          # Server bootstrap data handling
│   ├── certmonitor/        # TLS certificate rotation watcher
│   ├── cgroups/            # cgroup detection helper
│   ├── cli/                # CLI command definitions (urfave/cli)
│   ├── clientaccess/       # Client credential negotiation
│   ├── cloudprovider/      # K3s cloud provider for Kubernetes
│   ├── cluster/            # HA cluster management, storage bootstrap
│   ├── configfilearg/      # Config file + CLI arg merging
│   ├── containerd/         # Containerd configuration generation
│   ├── ctr/                # ctr passthrough helper
│   ├── daemons/            # Core abstraction layer
│   │   ├── agent/          # Agent daemon setup
│   │   ├── config/         # Shared config types (Node, Agent, Control, etc.)
│   │   ├── control/        # Server control daemon
│   │   └── executor/       # Executor interface definition
│   ├── data/               # Embedded binary asset loading
│   ├── datadir/            # Data directory resolution
│   ├── dataverify/         # Binary integrity verification
│   ├── deploy/             # Addon/manifest deployer
│   ├── etcd/               # Embedded etcd lifecycle + S3 snapshots
│   ├── executor/           # Executor implementation
│   │   └── embed/          # Embedded executor (all k8s in-process)
│   ├── flock/              # File locking utility
│   ├── kubeadm/            # kubeadm token format helpers
│   ├── kubectl/            # kubectl passthrough helper
│   ├── metrics/            # Prometheus metrics helpers
│   ├── node/               # Node label/annotation controllers
│   ├── nodeconfig/         # Node configuration annotation
│   ├── nodepassword/       # Node password authentication
│   ├── passwd/             # Password file management
│   ├── proctitle/          # Process title setting
│   ├── profile/            # pprof server
│   ├── rootless/           # Rootless mode support
│   ├── rootlessports/      # Rootless port forwarding
│   ├── secretsencrypt/     # Kubernetes secrets encryption at rest
│   ├── server/             # HTTP server, API handlers, TLS setup
│   ├── signals/            # OS signal handling
│   ├── spegel/             # libp2p-based distributed registry mirror
│   ├── static/             # Embedded static files (helm charts, etc.)
│   ├── untar/              # Tar extraction utility
│   ├── util/               # Misc utilities
│   │   ├── bindata/        # Embedded binary data helpers
│   │   ├── errors/         # Error utilities
│   │   ├── home/           # Home directory resolution
│   │   ├── logger/         # Logging helpers
│   │   ├── metrics/        # Metrics utilities
│   │   ├── mux/            # HTTP multiplexer helpers
│   │   ├── permissions/    # File permission utilities
│   │   └── services/       # Kubernetes service helpers
│   ├── version/            # Version constants
│   └── vpn/                # WireGuard VPN integration
├── manifests/              # Kubernetes manifests deployed at startup
├── scripts/                # CI/build shell scripts
├── tests/                  # All test types
│   ├── docker/             # Docker-based integration tests
│   ├── e2e/                # End-to-end tests (Vagrant/VM)
│   ├── fixtures/           # Test fixtures
│   ├── install/            # Install script tests
│   ├── integration/        # Go integration tests
│   ├── mock/               # Mock implementations
│   └── perf/               # Performance tests
├── docs/                   # Architecture decision records, release notes
├── contrib/                # Contrib tools (ansible, test templates)
├── package/                # RPM packaging
├── updatecli/              # Dependency update automation
└── Makefile                # Build orchestration (delegates to scripts/)

Entry points#

K3s has a two-tier binary architecture — the distribution launcher and the in-process runtime:

Tier 1: Distribution Launcher (cmd/k3s/main.go)#

The binary shipped to end users. On invocation it:

  1. Detects if called via symlink (crictl, kubectl, ctr) and dispatches directly
  2. For internal sub-commands (server, agent), extracts a compressed data archive from embedded assets into a data directory
  3. Exec’s into the extracted k3s binary or delegates to bundled binaries

Tier 2: In-Process Runtime (main.go at root)#

The binary embedded inside the data archive. It runs all k8s components in-process:

  • k3s server → starts API server, controller-manager, scheduler, etcd, and agent in-process goroutines
  • k3s agent → starts kubelet, kube-proxy, containerd, flannel in-process goroutines
  • All other subcommands (kubectl, crictl, etcdsnapshot, cert, token, secretsencrypt, completion) run as distinct extracted binaries

Individual sub-command binaries (cmd/*/main.go)#

Thin wrappers compiled as separate binaries that get embedded into the data archive:

  • cmd/server/main.go — server binary (embedded in archive)
  • cmd/agent/main.go — agent binary (embedded in archive)
  • cmd/kubectl/main.go — kubectl wrapper
  • cmd/ctr/main.go — ctr (containerd) wrapper
  • cmd/cert/main.go, cmd/encrypt/main.go, cmd/etcdsnapshot/main.go, etc.

Package organization#

Internal packages (all under pkg/ — no internal/ directory)#

Core abstraction layer:

  • pkg/daemons/config — Shared configuration types (Node, Agent, Control, ControlConfig) used across all components; the central data model
  • pkg/daemons/executor — Defines the Executor interface, the key architectural boundary between the CLI/control layer and the actual k8s component implementations

Server-side packages:

  • pkg/server — HTTP/HTTPS server setup, kubeconfig generation, API handler registration
  • pkg/cluster — HA cluster formation, storage backend bootstrapping, leader election
  • pkg/etcd — Embedded etcd lifecycle management, S3 snapshot upload/restore, snapshot scheduling
  • pkg/deploy — Addon deployer: watches for YAML manifests in a directory and applies them via the k8s API
  • pkg/secretsencrypt — Secrets encryption configuration management

Agent-side packages:

  • pkg/agent — Node agent startup (containerd, flannel, tunnel, netpol, loadbalancer)
    • pkg/agent/config — Agent configuration loading from server
    • pkg/agent/containerd — Containerd startup and configuration
    • pkg/agent/flannel — Flannel CNI configuration
    • pkg/agent/tunnel — WebSocket tunnel between agent and server
    • pkg/agent/loadbalancer — Client-side load balancer for server endpoints

Infrastructure packages:

  • pkg/executor/embed — Concrete Executor implementation: runs kube-apiserver, kubelet, scheduler, etc. as in-process goroutines using their app.Run() entrypoints
  • pkg/configfilearg — Merges YAML config file arguments with CLI args before urfave/cli parses them
  • pkg/clientaccess — Negotiates and stores server credentials on the agent
  • pkg/bootstrap — Serializes/deserializes server bootstrap data for agent distribution

Utility packages:

  • pkg/version — Version constants (Program, Version, GitCommit)
  • pkg/signals — OS signal handling (SIGTERM, SIGINT → context cancellation)
  • pkg/flock — File-based locking to prevent concurrent data directory writes
  • pkg/spegel — Embedded OCI registry mirror via libp2p (Spegel integration)
  • pkg/vpn — WireGuard VPN integration for pod networking

Public packages (pkg/)#

All packages are technically public (no internal/ restriction). In practice, the package structure is clearly layered for internal use; nothing here is designed as an importable library by external consumers.

Layering#

The package dependency graph follows a rough layered architecture:

CLI layer       (pkg/cli/*, cmd/*)
     ↓
Server/Agent    (pkg/server/*, pkg/agent/*)
     ↓
Daemon control  (pkg/daemons/control/*, pkg/cluster/*)
     ↓
Executor        (pkg/daemons/executor/ interface ← pkg/executor/embed/ impl)
     ↓
Config types    (pkg/daemons/config/)
     ↓
Utilities       (pkg/signals/, pkg/flock/, pkg/version/, pkg/util/*)

The Executor interface in pkg/daemons/executor/ is the main architectural seam: everything above it is k3s-specific logic; everything below it (the embed implementation) is k8s upstream code called via Go function invocations.

Build system#

  • Build tool: Docker-based build via Makefile → scripts/build. Production builds happen entirely inside multi-stage Docker containers (Alpine-based) with static CGO linking.
  • Key targets:
    • make local-binary — Builds k3s binary and assets inside Docker
    • make ci — Runs full CI: validate, build, test
    • scripts/build — Core build script: compiles all sub-binaries, creates the data archive (tar + zstd), embeds it into the main binary using go-bindata-style embedding
    • scripts/package — Packages the final binary into tarballs and RPMs
  • Docker: Yes, multi-stage. The build image (Dockerfile.local) installs all native deps (SQLite, btrfs, libseccomp, WireGuard) for static linking. The test image (Dockerfile.test) runs integration tests.
  • Output: A single statically-linked binary containing all sub-binaries and CNI plugins as a compressed embedded archive.

Notable structural decisions#

  1. No internal/ directory: Every package is technically importable by external code. This is an unusual choice for a project of this size; it likely reflects k3s’s origins as a rapid extraction from a larger internal codebase and/or the need to allow the distribution launcher (cmd/k3s/) to import freely from pkg/.

  2. The Executor interface as a build-tag seam: pkg/daemons/executor/executor.go defines the interface; pkg/executor/embed/embed.go provides the implementation, gated by //go:build !no_embedded_executor. This allows building k3s without the embedded k8s components — useful for testing or alternative executor implementations. The import of _ "github.com/k3s-io/k3s/pkg/executor/embed" in main.go is a blank import that triggers init() to register the concrete implementation.

  3. Two-tier binary design: The distribution binary (cmd/k3s/) is a thin launcher that extracts embedded binaries; the extracted binary (main.go) runs everything in-process. This design allows k3s to ship a single file while still supporting exec-based isolation for certain commands (crictl, kubectl) that cannot share the same process.

  4. Data embedding via archive: The bundled binaries (kubectl, crictl, ctr, CNI plugins, etc.) are not embedded individually via //go:embed — they are packed into a compressed tar archive and embedded as a single binary blob, then extracted to a versioned data directory at runtime. The pkg/data/ package handles this archive loading.

  5. pkg/configfilearg preprocessing: Rather than using a standard config-file library, k3s preprocesses os.Args before passing them to urfave/cli. This MustParse() call (visible in main.go) expands YAML config file keys into CLI flag equivalents, enabling a unified CLI/file config system without reimplementing flag parsing.

  6. tests/ as a top-level peer: All test types (unit, integration, e2e, Docker, perf) live under a single top-level tests/ directory rather than colocated with source packages. This is unusual for Go projects and reflects k3s’s heavy reliance on VM-based E2E tests that cannot run alongside source packages.

Backward K3s — Overview K3s — Dependencies Forward