K3s — Testing#

Test metrics#

  • Test files: 78 (out of 323 total Go source files)
  • Ratio (test files / source files): ~0.24 (1:4) — moderate for a systems project with heavy integration-level testing
  • Test frameworks: stdlib testing, go.uber.org/mock/gomock (generated mocks), github.com/onsi/gomega (assertions), github.com/onsi/ginkgo/v2 (BDD for integration + Docker tests)

Test organization#

  • Placement: Unit tests use same-package style (black-box: package foo_test not used; all in package foo). Integration/Docker/E2E tests are in external tests/ tree.
  • Helper packages:
    • tests/unit.go (package tests) — GenerateRuntime(), GenerateDataDir(), CleanupDataDir(): sets up a temp /tmp/k3s/* directory tree with generated TLS certs/credentials, replicating the production data directory layout for in-process unit tests.
    • tests/client.go (package tests) — Kubernetes API helpers shared across all tiers: CheckDefaultDeployments, ParseNodes, AllPodsUp, NodesReady, PodReady, etc. These are used by integration, Docker, and E2E tests alike, making cross-tier assertions consistent.
    • tests/mock/ — gomock-generated Executor mock plus hand-written composition helpers (see Mocking approach below).
    • tests/integration/integration.go (package integration) — Helpers for starting/stopping k3s as a subprocess: K3sServer type, K3sStartServer, K3sKillServer, K3sCleanup, K3sCmd, log scanning (SearchK3sLog), and file lock coordination (K3sTestLock backed by /tmp/k3s-test.lock).
  • Fixtures: testdata/ directories under individual integration test suites (etcdrestore/testdata, localstorage/testdata, etc.) hold YAML manifests for workload deployment. tests/fixtures/etcd/ holds etcd snapshot data for restore tests.

Test patterns#

Table-driven tests#

  • Prevalence: Heavy use — 66 table-driven indicators across test files
  • Style: Anonymous struct ([]struct{ name string; ... }) is the dominant style, consistent with the gotests-generated format. Named test structs are used in more complex cases.
  • Tooling: K3s provides contrib/gotests_templates — custom templates for the gotests tool. Tests are expected to be auto-generated via this tool, which is integrated with VS Code’s Go extension. The TESTING.md explicitly documents this workflow.
  • Naming convention: All unit test functions use Test_Unit<FunctionName> or Test_Unit<Receiver>_<Method> naming (e.g. Test_UnitServer, Test_UnitParser_findStart). This convention, unique among Go projects, makes it easy to run only unit tests: go test ./pkg/... -run Unit.
  • Example: pkg/configfilearg/parser_test.go:9 — Test_UnitParser_findStart with 8 table cases covering nil args, found/not found, and subcommand variations.
  • Example: pkg/daemons/control/server_test.go:30 — Test_UnitServer with 4 cases covering ControlPlane+ETCD, ETCD-only, ControlPlane+Kine, and auth config combinations.

Mocking approach#

  • Strategy: go.uber.org/mock/gomock with code-generated mocks. The primary mock target is the Executor interface (17 methods), the central architectural seam in k3s.
  • Generated mock: tests/mock/executor.go — fully generated via mockgen --source pkg/daemons/executor/executor.go. All 17 methods are mocked with EXPECT() recorder pattern.
  • Composite mock helper: tests/mock/executor_helpers.go — NewExecutorWithEmbeddedETCD(t *testing.T) creates a mock executor but delegates ETCD-specific methods (ETCD, ETCDReadyChan, Bootstrap, CurrentETCDOptions, IsSelfHosted) to a hand-written fakeExecutor struct that wraps the real embedded etcd. Ready channels (APIServerReadyChan, CRIReadyChan) are mocked to return immediately-closed channels. This pattern lets unit tests start a real embedded etcd cluster while mocking away all upstream Kubernetes components.
  • Gomega-Gomock bridge: tests/mock/matchers.go exports GM(gm types.GomegaMatcher) *gomockGomegaMatcher — an adapter that wraps any Gomega matcher (e.g. ContainElement, Not, And) for use as a gomock argument matcher. This enables expressive argument assertions like:
    matchLeaderElectArgs := mock.GM(ContainElement(ContainSubstring("--leader-elect=false")))
executor.EXPECT().Scheduler(gomock.Any(), gomock.Any(), matchLeaderElectArgs).MinTimes(1)
    The GM() bridge is a small but architecturally significant piece of test infrastructure — it avoids duplicating assertion logic across gomock and gomega worlds.
  • Gomega dot-import: Integration and unit tests that use gomega import it with . "github.com/onsi/gomega" (dot import). The code disables the revive linter for this: //revive:disable:dot-imports. This is a common Ginkgo/Gomega convention.

Integration tests#

  • Present: Yes — 14 named integration test suites under tests/integration/
  • How: Process-level black-box testing. Integration tests compile and run the actual k3s binary (dist/artifacts/k3s) as a subprocess via exec.Cmd. Tests then interact via the Kubernetes API or log scanning. This approach tests real inter-package behavior including initialization, networking, and certificate handling — things impossible to test in-process without enormous setup.
  • Framework: Ginkgo v2 + Gomega (BDD style). BeforeSuite/AfterSuite manage server lifecycle. Describe/When/It blocks describe scenarios. Eventually with timeout/polling is used pervasively for asynchronous assertions (e.g. Eventually(func() error { return CheckDefaultDeployments(...) }, "120s", "5s").Should(Succeed())).
  • Test suites: startup, etcdsnapshot, etcdrestore, cacertrotation, certrotation, custometcdargs, dualstack, flannelipv6masq, flannelnone, kubeflags, localstorage, longhorn, secretsencryption — covering etcd operations, TLS lifecycle, network config, and storage providers.
  • Separation: Build-tag-free; tests are run directly with go test ./tests/integration/.... A compile-time variable existingServer = "False" is linkable via -ldflags to allow running against an already-running k3s cluster. File lock /tmp/k3s-test.lock (via pkg/flock) prevents concurrent integration test runs on the same host.

Docker tests#

  • Present: Yes — 16 suites under tests/docker/ (autoimport, basics, bootstraptoken, cacerts, conformance, dualstack, etcd, hardened, lazypull, nixsnapshotter, secretsencryption, skew, snapshotrestore, svcpoliciesandfirewall, token, upgrade)
  • How: Uses a tests/docker package that provisions Docker containers as k3s nodes (config.ProvisionServers(1), config.ProvisionAgents(1)). Tests then check cluster health via the Kubernetes API. Ginkgo v2 + Gomega, same pattern as integration tests. Run in Drone CI test stage.
  • Scope: Tests configuration that requires actual container images — upgrade paths, image auto-import, secrets encryption, skew testing between server/agent versions.

E2E tests#

  • Present: Yes — 17 suites under tests/e2e/ using Vagrant VM provisioning
  • How: Each E2E suite stands up multi-node clusters (typically 1-3 nodes) via Vagrant. Tests cover features requiring real VMs: btrfs, rootless, dualstack, Tailscale, WASM, private registry, external IP. Nightly CI via .github/workflows/e2e.yaml.

Install tests#

  • Present: Yes — 8 OS-specific Vagrant configurations under tests/install/
  • How: Validate the install script on each distro (CentOS 9, Rocky 8/9, Fedora, OpenSUSE Leap/MicroOS, Ubuntu 24.04, Alma 10). Run via vagrant up with named provisioners for each health check step. Nightly CI via .github/workflows/nightly-install.yaml.

Performance tests#

  • Present: Yes — tests/perf/ with Terraform-based density tests
  • How: Tests large-scale deployments (600, 2000, 5000 nodes). Uses Terraform for cluster provisioning. Separate from all CI pipelines — intended for manual/scheduled performance benchmarking.

CI integration#

Five GitHub Actions workflows cover the test pyramid:

  • integration.yaml — unit + integration tests on PR
  • e2e.yaml — nightly E2E (Vagrant)
  • install.yaml / nightly-install.yaml — install script tests
  • build-k3s.yaml — build + Docker tests (Drone)
  • govulncheck.yml + codeql.yml — security scanning

Test quality observations#

What’s done well#

  • Tiered test taxonomy explicitly documented: tests/TESTING.md is a comprehensive guide covering all 6 test types, their purpose (unit = white box, integration = black box), naming conventions, tooling, and how to run each tier. This is unusually thorough for an OSS project of this size.
  • Naming convention enforces type separation: The Test_Unit* prefix for all unit test functions allows go test -run Unit to execute only unit tests, independent of the Ginkgo integration tests. This is pragmatic and clear.
  • GM() bridge is a genuine contribution: The Gomega-Gomock adapter in tests/mock/matchers.go is elegant and reusable. It solves the real problem of wanting rich Gomega assertion expressiveness inside gomock EXPECT() calls without writing custom matchers for each case.
  • NewExecutorWithEmbeddedETCD hybrid mock: Composing the generated mock with a real embedded etcd (fakeExecutor.ETCD calls etcd.StartETCD) lets Test_UnitServer test the full control plane startup path — including leader election arg generation — with real cluster storage. This is more valuable than a fully-mocked test would be.
  • Kubernetes API helpers in shared package: tests/client.go functions like CheckDefaultDeployments and NodesReady are consumed across integration, Docker, and E2E tiers. This prevents drift between tiers in how cluster health is assessed.
  • File lock for integration tests: Using pkg/flock to serialize integration test runs on shared CI hosts is operationally mature — avoids port conflicts and etcd data dir collisions.
  • gotests templates: Providing contrib/gotests_templates to auto-generate table-driven tests removes friction and enforces the convention. This is a deliberate investment in test culture.

What could improve#

  • Low unit test coverage of core packages: The pkg/daemons/, pkg/agent/, pkg/cluster/, and pkg/etcd/ packages have very few unit tests relative to their importance. Much of the k3s-specific logic (startup sequencing, certificate generation, cluster membership) is validated only at the integration tier, requiring a built binary and root privileges to test. This makes fast iteration more expensive.
  • No test for the multicall dispatch or reexec path: cmd/k3s/main.go and cmd/server/main.go contain the multicall binary and reexec registry logic, which are tested only incidentally by integration tests. A unit test for the dispatch table would catch regressions without requiring a full binary build.
  • Integration tests require root + built binary: The integration test infrastructure explicitly checks IsRoot() and looks for dist/artifacts/k3s. This means integration tests cannot run in typical developer environments without a full build step and elevated privileges. The tiered design compensates somewhat, but onboarding friction is higher than projects that run integration tests in-process.
  • E2E suite has no shared Go test runner: Unlike the integration and Docker tiers (which are standard Go test packages), the E2E tests are heterogeneous — some use Vagrant+Ginkgo, others use Ansible playbooks (e2e_test_playbook.yaml), and the test scripts are shell-based. This fragmentation makes it harder to get consistent pass/fail signals in CI.

Patterns worth emulating#

  • The GM() adapter pattern for bridging two assertion libraries (gomock and gomega) is directly transferable to any project using both. It’s 37 lines of code that pays compound dividends in test expressiveness.
  • Tiered test taxonomy + naming convention (Test_Unit* prefix + 6-tier documentation) is a model for how to grow a test suite across multiple abstraction levels without the tiers conflating. The explicit documentation of when each test type should be written (not just how) is the key differentiator.
  • Hybrid mock: generated + real component composition. NewExecutorWithEmbeddedETCD shows that gomock and real implementations can be selectively composed per test case. Generating the full mock first, then wrapping specific methods with real behavior, gives maximum control over test fidelity without abandoning generated code.
Backward K3s — API Surface Helm — Overview Forward