Kubernetes — Testing#

Sampling note: Kubernetes is an XL project (~9,573 non-test Go files, 3,014 test files). Testing analysis used targeted grep across the full repository (excluding vendor/) for counts and representative examples. Deep-reads were focused on: pkg/controller/deployment/ (unit tests), test/integration/deployment/ and test/integration/secrets/ (integration tests), test/e2e/ framework, staging/src/k8s.io/client-go/kubernetes/fake/ (generated fakes), and pkg/controller/testutil/ (test helpers).

Test metrics#

  • Test files: 3,014 (*_test.go files, excl. vendor)
  • Non-test source files: 9,573
  • Ratio (test / source): ~0.315
  • Test frameworks:
    • stdlib testing — the primary framework for all unit and integration tests
    • Ginkgo v2 + Gomega — used exclusively in test/e2e/ and test/e2e_node/ (6,650+ ginkgo. usages in e2e)
    • github.com/google/go-cmp — 2,211 uses of cmp.Diff/cmp.Equal as the comparison engine for assertion failures (preferred over reflect.DeepEqual)
    • klog/v2/ktesting — 3,051 uses; provides a testing.T-backed klog logger so structured log output is captured per-test
    • github.com/stretchr/testify — peripheral use; appears in some staging packages (apimachinery, cluster-bootstrap) but is explicitly avoided in core packages (see apitesting/close.go comment: “assertNoError simulates assert.NoError without adding testify as a non-test dependency”)

Test organization#

Placement#

Tests are split across three tiers, each with its own package conventions:

  1. Unit tests — *_test.go files co-located with source, typically in a separate package foo_test (external black-box) or occasionally in package foo (white-box). Both conventions coexist without a uniform rule.

  2. Integration tests — test/integration/ (388 Go files, 40+ subdirectories organized by feature area: deployment, apiserver, auth, dra, scheduler, etc.). These use stdlib testing directly, not Ginkgo.

  3. End-to-end tests — test/e2e/ (553 Go files) and test/e2e_node/ (137 Go files, linux-only build tag). Both use Ginkgo/Gomega exclusively.

Helper packages#

  • pkg/controller/testutil/ — controller-specific helpers: UpdatePodStatus, CreateFakePod, action-inspecting utilities for the fake client tracker. Used across most controller unit tests.
  • staging/src/k8s.io/client-go/testing/ — the fake object tracker (fixture.go). Implements an in-memory REST store that intercepts CRUD calls and dispatches them to registered ReactFn chains. This is the foundational layer for all fake client testing.
  • staging/src/k8s.io/client-go/kubernetes/fake/ — code-generated (client-gen) fake clientset implementing clientset.Interface. Every resource type gets a typed fake (e.g., fakeappsv1.FakeDeployments) that delegates to the shared object tracker.
  • cmd/kube-apiserver/app/testing/ and staging/src/k8s.io/apiserver/pkg/testing/ — StartTestServerOrDie helpers that spin up a real API server in-process for integration tests.
  • test/e2e/framework/ — a large framework (~553 Go files collectively) with test lifecycle helpers, cluster clients, event watchers, polling utilities, and the gomega.Expect-based assertion patterns.

Fixtures#

  • test/fixtures/ — YAML fixtures for kubectl and CLI tests (doc-yaml examples).
  • testdata/ directories scattered throughout staging (e.g., code-generator/cmd/validation-gen/output_tests/*/testdata) for golden-file testing of code generators.
  • No global fixtures database; each package owns its fixture data.

Test patterns#

Table-driven tests#

  • Prevalence: Extremely heavy — 7,246 occurrences of testCases, tests :=, or t.Run(tc.name in *_test.go files. This is the single most uniformly applied coding convention in the entire project.

  • Style: Anonymous struct slices with a mandatory name string field. Subtests always use t.Run(tc.name, ...).

    // pkg/controller/deployment/deployment_controller_test.go (representative)
tests := []struct {
    name      string
    d         *apps.Deployment
    rsList    []*apps.ReplicaSet
    podList   *v1.PodList
    wantErr   bool
    expected  int32
}{
    {
        name:     "no replica sets",
        d:        newDeployment("d", 3, nil, nil, nil, nil),
        expected: 0,
    },
    // ...
}
for _, tc := range tests {
    t.Run(tc.name, func(t *testing.T) {
        // exercise
    })
}

  • Assessment: The pattern is not enforced by linting tools but is so culturally ingrained that deviations are effectively absent in reviewed code. Every regression bug fix adds a new table row, making test files serve as living specifications. This is the gold standard of table-driven testing in the Go ecosystem.

Mocking approach#

  • Strategy: Interface-backed generated fakes, not framework-generated mocks. The project does not use gomock or mockery.

  • The fake client pattern:

    clientset.Interface  ←── real generated client (HTTPS to API server)
      ↑
fake.NewClientset()  ←── generated fake (in-memory object tracker)

    fake.NewClientset(objects...) initializes an object tracker pre-populated with seed objects. Tests inject the fake clientset into the component under test via its constructor. API calls from the component are intercepted, recorded as testing.Action values, and can be inspected:

    // pkg/controller/deployment/deployment_controller_test.go
client := fake.NewSimpleClientset()
// ... run controller with client ...
actions := client.Actions()
// assert specific verbs/resources were called

  • ReactFn chains: Tests can register custom reactor functions (client.AddReactor("get", "pods", ...)) to inject faults or override specific behaviors, enabling precise failure scenario testing.

  • Manual fakes for domain types: Components like the SharedInformerFactory, EventRecorder, and scheduler framework plugins provide hand-written fakes (e.g., record.NewFakeRecorder(100)) rather than generated ones.

  • No gomock: The avoidance of interface mocking frameworks is intentional — the project relies on real fake implementations of its own interfaces rather than expecting test authors to set up call expectations.

Integration tests#

  • Present: Yes — a dedicated test/integration/ tree with 388 Go files spanning 40+ functional areas.

  • How: Integration tests spin up a real API server in-process plus a real etcd instance using StartTestServerOrDie:

    // test/integration/secrets/secrets_test.go
server := kubeapiservertesting.StartTestServerOrDie(
    t, nil,
    framework.DefaultTestServerFlags(),
    framework.SharedEtcd(),  // reuses a shared in-process etcd
)
defer server.TearDownFn()
client := clientset.NewForConfigOrDie(server.ClientConfig)

    The API server runs as a goroutine in the same process, not in Docker. SharedEtcd() manages a single etcd instance shared across tests in the same binary, improving speed. t.Cleanup / defer TearDownFn() ensures teardown.

  • Separation: No build tags separate integration from unit tests — they live in a completely different directory tree (test/integration/ vs co-located *_test.go). CI scripts select them by path. This is simpler than build tags but means you cannot accidentally co-locate integration tests with unit tests.

  • Scope: Integration tests exercise full API server request handling (admission, validation, storage round-trips via etcd, watch propagation) against actual components, not fakes.

E2e tests#

  • Framework: Ginkgo v2 + Gomega, with a large custom framework (test/e2e/framework/).
  • Pattern: BDD-style Describe/It blocks organized by feature area (apps, auth, network, dra, etc.).
  • Target: A running Kubernetes cluster. E2e tests do not spin up the control plane; they receive --kubeconfig as input and exercise live resources.
  • Node e2e: test/e2e_node/ targets a single node running kubelet directly; marked //go:build linux.
  • Separation: E2e tests are isolated by directory convention and by the Ginkgo runner (hack/ginkgo-e2e.sh), not by build tags.

Test quality observations#

What’s done well#

  • Table-driven tests as cultural norm: The consistency is exceptional. 7,246+ instances means almost every TestFoo function uses the pattern. New contributors learn it immediately, and test additions are always predictable in structure.

  • Generated fakes are comprehensive: client-gen produces fake clients for every API group and version. Tests never need to mock networking. The object tracker (testing/fixture.go) faithfully implements strategic merge patch, JSON patch, and optimistic locking, making fakes a high-fidelity stand-in for the real server.

  • In-process integration testing: Spinning up a real API server + etcd in-process (rather than Docker containers or external services) is fast, hermetic, and gives near-production fidelity. It catches admission webhook bugs, etcd encoding issues, and watch event fan-out problems that fake clients would miss.

  • go-cmp for diffs: Using cmp.Diff over reflect.DeepEqual gives human-readable failure messages showing exactly which field diverged. The 2,211 usages represent a deliberate, project-wide preference.

  • ktesting integration: Capturing structured klog output per-test (klog/v2/ktesting) means controller logs appear in test output only for failing tests — no global log pollution.

  • Three-tier test pyramid respected: Unit (fast, isolated, fakes) → Integration (in-process API server) → E2e (live cluster). Each tier has a clear scope and toolchain.

What could improve#

  • testify avoidance creates verbose assertions: The explicit avoidance of testify means test assertion code is more verbose (if !cmp.Equal(...) { t.Errorf(...) } vs. require.Equal(t, ...)). go-cmp partially compensates, but control-flow assertions (require.NoError) still require hand-rolled if err != nil { t.Fatalf(...) }.

  • Inconsistent package placement: Some tests use package foo, others use package foo_test, without a clear rule. This creates occasional import cycles and makes it harder to know which internal state is accessible in tests.

  • E2e framework size: The test/e2e/framework/ package has grown very large. Without reading it exhaustively, it shows signs of accumulation (bugs.go, flake_reporting_util.go) that suggest it could benefit from decomposition.

  • No fuzz testing infrastructure visible: With Go’s native testing.F available since 1.18, Kubernetes-scale input parsing (YAML/JSON API objects) would benefit from fuzzing, but no FuzzXxx functions are present in sampled code.

Patterns worth emulating#

  1. Table-driven tests with t.Run: The most portable, readable, and extensible Go testing pattern. Kubernetes’s consistent application is the canonical example.

  2. In-process real-server integration testing: The StartTestServerOrDie + SharedEtcd pattern proves that integration tests do not need Docker. An API server and storage backend can live in-process, keeping CI times manageable while exercising real codepaths.

  3. Generated fakes with a shared object tracker: The fake.NewClientset() + ReactFn pattern decouples tests from networking entirely while preserving API semantics. Replicable for any project that exposes a client interface.

  4. go-cmp with cmpopts: Using cmp.Diff with options (e.g., cmpopts.IgnoreFields, cmpopts.EquateEmpty) gives precise, self-documenting assertion failures. Better than reflect.DeepEqual for complex nested structs.

  5. ktesting for structured logging in tests: Integrating the project’s own logger with testing.T so per-test log output is captured and only shown on failure. Any project using klog (or slog) should adopt this pattern.

Backward Kubernetes — API Surface Moby — Overview Forward