K3s — Patterns#

Concurrency patterns#

Readiness channels (primary startup synchronization)#

  • Usage: The dominant synchronization mechanism for component startup ordering. Each Kubernetes component (API server, etcd, CRI) signals readiness by closing a chan struct{}. Dependent components block on <-chan struct{} reads before proceeding.
  • Example: pkg/daemons/executor/executor.go:47-50 — APIServerReadyChan(), ETCDReadyChan(), CRIReadyChan() are methods on the Executor interface; callers block with <-executor.APIServerReadyChan(). The concrete Embedded impl closes these channels when the respective upstream app.Run() signals ready.
  • In config struct: pkg/daemons/config/types.go stores ready-channels in Runtime (the mutable context object): multiple ready channels threaded through the entire call stack.
  • Assessment: Idiomatic and effective for strictly ordered startup without busy-polling. The pattern is self-documenting — a <-chan struct{} return type on a method named XReadyChan() is immediately readable. One concern: nil channel returns when executor is not set would block forever; guarded by nil-executor checks in forwarding functions.

Goroutine-per-component launch#

  • Usage: Each Kubernetes component runs as a detached goroutine inside the in-process monolith. 52 anonymous goroutine launches (go func()) across non-test code.
  • Example: pkg/daemons/control/server.go:184 — nodeReady channel closed inside a goroutine that waits for apiserver readiness then runs RBAC setup. pkg/daemons/agent/agent.go:40-47 — separate goroutines for agent startup side effects.
  • Assessment: Natural for a system that must start many long-running services without blocking. Goroutines are tied to the root context.Context; when it is cancelled (SIGTERM), all components begin shutdown.

WaitGroup for lifecycle tracking#

  • Usage: A *sync.WaitGroup is created at the top of run() in pkg/cli/server/server.go:82 and threaded through the entire call stack to control.Prepare, control.Server, and executor methods. Downstream goroutines call wg.Add(1) / defer wg.Done() so the server can wait for clean shutdown.
  • Example: pkg/cli/server/server.go:94 — defer wg.Wait() ensures all background goroutines complete before the process exits. pkg/daemons/config/types.go:319 — StartupHooksWg *sync.WaitGroup stored on Runtime for startup hooks.
  • Assessment: Correct pattern for clean shutdown. Passing the WaitGroup by pointer through 4–5 call levels is slightly verbose but explicit. No use of errgroup for the main lifecycle (reserved for isolated parallel tasks in pkg/spegel).

errgroup for bounded parallel tasks#

  • Usage: Used sparingly; only in pkg/spegel/bootstrap.go:254 for parallel peer bootstrapping and in test helpers.
  • Example: pkg/spegel/bootstrap.go:254 — eg, ctx := errgroup.WithContext(ctx) to fan out to multiple bootstrap peers with a shared cancellable context.
  • Assessment: Appropriate scoping — errgroup is used where a bounded set of goroutines must all succeed or the operation fails, rather than globally.

select-based event loops#

  • Usage: 32 select {} statements; used for multiplexing channel receives (signal handling, timeout + cancellation, reading from multiple ready channels concurrently).
  • Example: pkg/cli/cmds/log_linux.go:89 — signal channel with select for SIGHUP log rotation. pkg/daemons/control/server.go:446 — make(chan error, 1) + select pattern for racing a goroutine result against context cancellation.
  • Assessment: Standard Go idiom, used well. The buffered chan error + select pattern (allocate a 1-capacity error channel, launch goroutine that sends to it, select against ctx.Done()) appears 3 times and is a reliable concurrency primitive for timeout-bounded async calls.

Graceful shutdown via context propagation#

  • Usage: pkg/signals.SetupSignalContext() (called once at startup) returns a context that is cancelled on SIGTERM/SIGINT. This root context is propagated through every component. No other shutdown mechanism.
  • Example: pkg/cli/server/server.go:81 — ctx := logger.NewContext(signals.SetupSignalContext(), version.Program); all downstream calls receive this context.
  • Assessment: Clean and idiomatic. The single shutdown signal propagates through the entire system via context cancellation, with wg.Wait() ensuring clean drain.

Error handling#

  • Style: Mixed — errors.New for static sentinel errors, fmt.Errorf %w for contextual wrapping, custom struct types for domain-specific errors. The pkg/errors library is NOT used; k3s has its own thin shim (pkg/util/errors/errors.go) that adds WithStack, WithMessage, WithMessagef helpers layering on top of stdlib.
  • Error types defined:
    • executor.ErrNotInitialized — sentinel returned when executor package-level var is nil (guards all forwarding functions).
    • etcd.membershipError / etcd.memberListError — typed errors for etcd cluster membership operations (pkg/etcd/etcd.go:133,148).
    • nodepassword.passwordError — wraps node password validation failures (pkg/nodepassword/nodepassword.go:26).
    • etcd/s3.secretError — wraps S3 credential secret errors (pkg/etcd/s3/config_secret.go:18).
  • Wrapping approach: fmt.Errorf("context: %w", err) is used consistently throughout. The custom pkg/util/errors wrappers add stack traces and messages for improved diagnostics in deeply nested calls.
  • Canonical context.Canceled guard: Every cmd/*/main.go entry point guards the final error with !errors.Is(err, context.Canceled) before printing and exiting non-zero. This prevents false alarms when the server is cleanly shut down by SIGTERM.
    // cmd/server/main.go:88
if err := app.Run(...); err != nil && !errors.Is(err, context.Canceled) {
    logrus.Fatal(err)
}
  • Assessment: Disciplined and consistent. The context.Canceled guard is particularly clean — a single idiom applied uniformly across all binaries. Custom error types are used only where callers need to distinguish error categories (errors.As), not gratuitously.

Configuration pattern#

  • Approach: Two-phase YAML-file-to-CLI-args preprocessing + explicit struct assignment. The primary config innovation is pkg/configfilearg/: before urfave/cli parses flags, MustParse(os.Args) reads YAML config files and injects them as CLI flags. This gives a single unified flag/file surface with no duplicated logic.
  • Large config struct: cmds.ServerConfig (100+ fields, pkg/cli/cmds/server.go) populated by urfave/cli flag actions, then manually copied into config.Control and server.Config in 200+ explicit assignment lines. No auto-binding (no Viper, no mapstructure).
  • Functional options for HTTP clients: pkg/clientaccess/token.go:49 defines type ClientOption func(*http.Client), type RequestOption func(*http.Request), type ValidationOption func(*Info) with constructors like WithCACertificate, WithClientCertificate, WithTimeout. This is the only area using functional options — applied to the narrow scope of configuring HTTP client behavior.
  • Example of functional options usage:
    // pkg/clientaccess/token.go
info, err := clientaccess.ParseAndValidateToken(url, token,
    clientaccess.WithCACertificate(caFile),
    clientaccess.WithTimeout(30*time.Second),
)
  • Assessment: The YAML-to-CLI approach elegantly solves the file/flag unification problem without framework overhead. The large-struct approach for server config trades elegance for explicitness — every field mapping is visible, making the code verbose but auditable.

Dependency injection#

  • Approach: Manual wiring via large config structs passed through the call stack. No DI framework.
  • The Executor Singleton Pattern (key DI mechanism):
    • pkg/daemons/executor/executor.go declares a package-level var executor Executor and exposes package-level forwarding functions (Bootstrap(), Kubelet(), APIServer(), etc.) that check for nil and delegate to the concrete implementation.
    • Registration is done via executor.Set(&Embedded{}) called from init() in pkg/executor/embed/embed.go, triggered by a blank import _ "github.com/k3s-io/k3s/pkg/executor/embed" in main.go.
    • Guarded by build tag //go:build !no_embedded_executor — setting this tag and omitting the blank import builds k3s without the upstream k8s dependencies.
    • All forwarding functions guard: if executor == nil { return ErrNotInitialized }.
  • Driver registry for storage backends: pkg/cluster/managed/drivers.go maintains a []Driver slice. RegisterDriver(d Driver) appends, Default() returns drivers[0], Registered() returns all. Drivers register themselves (etcd is the only bundled driver). This is a simple slot-based registry — not injection, but enables selection at runtime.
  • Evidence summary: The executor init()/blank-import pattern is the most architecturally significant DI mechanism in the codebase. The blank import is a build-time activation switch; the init() function is the registration hook; executor.Set() is the injection point.

Other notable patterns#

Build-tag feature flags#

  • Usage: //go:build tags control which code is compiled for different feature sets. Observed tags: no_stage, no_embedded_executor, linux && cgo, linux && cover, !linux || !cgo.
  • Example: pkg/cli/cmds/stage.go vs pkg/cli/cmds/nostage.go — mutually exclusive files controlled by no_stage tag. pkg/cli/cmds/log_linux.go vs log_default.go — cgo vs non-cgo log setup.
  • Assessment: Effective use of Go’s build system for configuration. The no_embedded_executor tag is architecturally significant — it enables test builds or alternative executor implementations without upstream k8s code.
  • Usage: cmd/k3s/main.go checks filepath.Base(os.Args[0]) to dispatch to crictl, kubectl, ctr, or check-config. The distribution binary is one file that behaves as multiple programs depending on the invocation name.
  • Example: cmd/k3s/main.go:168 — if progName matches a known tool, calls externalCLI(progName, dataDir, os.Args[1:]).
  • Assessment: Classic Unix multicall pattern (busybox, BusyBox). Reduces deployment footprint to a single binary. Combined with the embedded archive self-extraction, this allows the entire Kubernetes distribution to be shipped as a 50–100 MB single file.

reexec registry for process re-entry#

  • Usage: moby/sys/reexec registers named process entry points (containerd, kubectl, crictl, ctr) in cmd/server/main.go:33-36. On startup, reexec.Init() checks os.Args[0] and re-enters the process as the named function.
  • Example: cmd/server/main.go:42 — if reexec.Init() { return } — if the binary is re-entered as “containerd”, it runs containerd.Main and exits, bypassing all k3s initialization.
  • Assessment: Complements the multicall binary pattern for the in-process runtime binary (which is different from the distribution launcher). Allows the single extracted binary to serve as containerd, kubectl, etc. without symlinks.

sync.Once for one-time initialization#

  • Usage: 9 instances. Used for: log setup (pkg/cli/cmds/log.go:44), HTTPS server start (pkg/agent/https/https.go:19), embedded executor (pkg/executor/embed/embed.go:52), S3 client initialization (pkg/etcd/s3/s3.go:55), cert monitor startup (pkg/certmonitor/certmonitor.go:71).
  • Assessment: Correctly scoped — sync.Once is used where a resource may be initialized from multiple goroutines but must only run once (e.g., starting the HTTPS listener, creating the S3 client). The pkg/etcd/s3/s3.go usage is particularly correct — S3 client creation is expensive and must be idempotent.

Generics (limited use)#

  • Usage: Used in pkg/util/lru.go (Cache[T any]) and pkg/util/patch.go (Patcher[T runtime.Object], controllerPatcher[T], clientPatcher[T]).
  • Example: pkg/util/lru.go:7 — type Cache[T any] struct { lru *lru.Cache[string, T] } — a generic LRU wrapper.
  • Assessment: Conservative adoption. Generics appear only in utility wrappers where the type parameter is genuinely needed to avoid interface{} casts. Not used in the main application logic. The Patcher[T runtime.Object] usage shows awareness of type-parameterized interfaces over a type constraint.

Interface-based extensibility (seam pattern)#

  • Usage: Interfaces are used as deliberate architectural seams rather than speculative abstractions. Key examples: Executor (17 methods), managed.Driver (15 methods), etcdproxy.Proxy (3 methods), agent/proxy.Proxy, mux.Handler, spegel.DeferredStore.
  • Assessment: The Executor interface is the best example of an intentional seam — it exists to isolate k3s orchestration logic from upstream k8s code, enabling build-tag-controlled substitution. The managed.Driver interface enables the etcd backend to be swapped for alternative cluster storage. Both interfaces are larger than ISP ideals (17 and 15 methods), but the breadth reflects the scope of the component lifecycle being abstracted, not interface design carelessness.

Startup hook function type#

  • Usage: type StartupHook func(context.Context, *sync.WaitGroup, StartupHookArgs) error defined in pkg/cli/cmds/server.go:24. Allows registering callbacks that run during server startup, with access to the root context and WaitGroup.
  • Assessment: A lightweight extensibility point. The function type as first-class value is idiomatic Go — avoids a full interface for a single-method extension point.

Error sentinel at package boundary#

  • Usage: executor.ErrNotInitialized = errors.New("executor not initialized") in pkg/daemons/executor/executor.go:23. Every forwarding function in the package checks for nil executor and returns this sentinel.
  • Assessment: Defensive programming at a true system boundary — the executor singleton can only be set by a blank import, so ErrNotInitialized would only fire in tests or broken builds. The sentinel enables errors.Is checking by callers without string matching.
Backward K3s — Architecture K3s — Interfaces Forward