Helm — Testing#

Test metrics#

  • Test files: 228
  • Total Go files: 534
  • Ratio (test files / source files): ~43% — healthy coverage density
  • Test frameworks: stdlib testing as the base; github.com/stretchr/testify (assert, require, suite) for assertions; no gomock, ginkgo, or gocheck

Test organization#

Placement#

All 228 test files declare the same package as the code they test (e.g., package action, package cmd, package driver). No _test external package suffix is used anywhere. This gives tests full access to unexported symbols — a deliberate choice given that much of Helm’s interesting behavior lives in unexported helpers.

Helper packages#

Helm ships two purpose-built internal test helper packages and one public fake:

internal/test/ — Golden file assertion framework

  • AssertGoldenString(t, actual, filename) and AssertGoldenFile(t, actualFile, expectedFile) compare strings to reference files in testdata/
  • --update flag (var updateGolden = flag.Bool("update", ...)) regenerates golden files in-place, eliminating manual golden file maintenance
  • Normalizes CRLF → LF so tests are portable across Windows/Linux CI

internal/test/ensure/ — Environment isolation helpers

  • HelmHome(t *testing.T) sets XDG and Helm-specific env vars to t.TempDir(), giving each test a hermetic Helm home directory that cleans up automatically
  • TempFile(t, name, data) creates a scoped temp file via t.TempDir() — no manual cleanup needed

pkg/kube/fake/ — Fake Kubernetes client (public, usable by external code)

  • PrintingKubeClient — implements kube.Interface entirely in-process; all operations succeed and serialize their input to an io.Writer (typically io.Discard or a bytes.Buffer)
  • FailingKubeClient — embeds PrintingKubeClient and adds per-method error fields (CreateError, DeleteError, WaitError, …); tests inject specific failures without mocking a whole interface
  • Both types assert interface satisfaction at compile time: var _ kube.Interface = &FailingKubeClient{}

pkg/repo/v1/repotest/ — In-process chart repository server

  • Wraps net/http/httptest.NewServer / httptest.NewTLSServer with chart-serving logic
  • NewTempServer(t, opts...) using functional options (WithTLSConfig, WithMiddleware, WithChartSourceGlob) — the same With* idiom used across Helm’s production APIs (see patterns analysis)
  • Also embeds a real in-process OCI registry (via github.com/distribution/distribution) for registry integration tests

Fixtures (testdata)#

22 testdata/ directories, one per package. Contents vary:

  • Chart tarballs and YAML files for chart-loading tests
  • Golden output/*.txt files for CLI output comparison (heavily used in pkg/cmd/testdata/)
  • RBAC manifests and Kubernetes resource YAML for action-layer tests
  • Renderer inputs/outputs for template engine tests

Test patterns#

Table-driven tests#

  • Prevalence: Heavy — 568 occurrences of t.Run, tests :=, tt., or testCases in *_test.go files
  • Style: Two variants in use:
    1. Slice of anonymous struct (dominant): tests := []struct{ name string; ... }{ {...}, {...} } iterated with for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ... }) }
    2. Map-keyed cases (less common, used when order doesn’t matter): testCases := map[string]struct{ ... }{ "case_name": {...} } — e.g., pkg/cmd/helpers_test.go:165 (dry-run flag strategy tests)
  • Example: pkg/action/install_test.go contains 30+ top-level test functions; most iterate a local tests slice through t.Run

Golden file testing (CLI layer)#

The pkg/cmd/ package uses a specialized cmdTestCase struct:

type cmdTestCase struct {
    name      string
    cmd       string    // full Helm CLI invocation as a string
    golden    string    // path to expected output file (relative to testdata/)
    wantError bool
    rels      []*release.Release
    repeat    int       // 0 = run once; >0 = repeat N times for flakiness checks
}

runTestCmd(t, tests) parses cmd into args with shellwords.Parse, constructs a root Cobra command wired to in-memory storage + kubefake.PrintingKubeClient, captures output into a bytes.Buffer, and asserts it against the golden file via test.AssertGoldenString. This makes every CLI command test a single-line declaration — adding a new case is as cheap as adding a struct literal. The repeat field is notable: it re-runs a test case N+1 times to confirm stability after historically flaky behavior.

Example (pkg/cmd/install_test.go:52):

{
    name:   "basic install",
    cmd:    "install aeneas testdata/testcharts/empty --namespace default",
    golden: "output/install.txt",
},

Mocking approach#

  • Strategy: Manual fakes over interface contracts — no code generation, no gomock or mockery
  • pkg/kube/fake.FailingKubeClient is the primary dependency substitute for the action layer. Tests inject specific error scenarios by field assignment:
    cfg.KubeClient = &kubefake.FailingKubeClient{
    PrintingKubeClient: kubefake.PrintingKubeClient{Out: io.Discard},
    WaitError: errors.New("wait failed"),
}
  • In-memory storage driver (driver.NewMemory()) replaces Kubernetes secrets/configmaps storage — a real implementation variant, not a mock
  • k8s.io/client-go/kubernetes/fake (from upstream) is used for a few tests needing Kubernetes API server behavior
  • No HTTP-level mocking (no httptest.NewRecorder for most tests) — the repotest.Server is used instead as a real in-process HTTP server

Integration tests#

  • Present: Yes — in the pkg/registry/ package
  • How: Four test suites (HTTPRegistryClientTestSuite, TLSRegistryClientTestSuite, InsecureTLSRegistryClientTestSuite, RegistrySuite) each spin up a real in-process OCI registry (github.com/distribution/distribution) using httptest.Server, then run login/push/pull/tag operations against it
  • Framework: testify suite.Suite — used specifically here because the registry tests need SetupSuite/TearDownSuite lifecycle for the shared server
  • Separation: No build tags or separate directories — registry suite tests are co-located with unit tests in pkg/registry/. They are slower but self-contained
  • No Docker or testcontainers — all dependencies are embedded in-process

Test quality observations#

What’s done well#

  • Golden file + --update flag: The pattern eliminates the maintenance burden of keeping expected outputs current. Updating all golden files after a formatting change is a single go test ./... -update invocation.
  • cmdTestCase as declarative DSL: Adding a new CLI test is one struct literal. The test harness handles wiring, execution, and comparison. This scales to 50 test cases per command file without noise.
  • actionConfigFixture centralizes wiring: The single actionConfigFixture(t) function in pkg/action/action_test.go is shared across all 23 action test files. Changing how tests are wired (e.g., adding a new capability or registry client) requires changing one place.
  • FailingKubeClient error injection model: Per-method error fields mean tests can specify exactly which operation fails without implementing a full mock. Unexpectedly elegant for a hand-rolled fake.
  • t.TempDir() and t.Setenv() throughout: Helm has fully adopted the Go 1.14+ cleanup API. No manual defer os.Remove(tmpDir) scattered through tests.
  • repeat field in cmdTestCase: Explicit mechanism for regression-testing flaky behavior. The intent is documented in the struct comment rather than buried in individual test bodies.
  • Testify suite scoped to where it adds value: The suite.Suite is used only in the registry package, where SetupSuite/TearDownSuite is genuinely needed for a shared server. The rest of the codebase doesn’t pay the ceremony cost.

What could improve#

  • No _test package boundary: Every test file is in the production package, which gives access to unexported symbols but makes it easy for tests to rely on internal state that shouldn’t be part of the contract. A few packages (especially pkg/action/) would benefit from an external _test package for integration-style tests that should only observe public behavior.
  • pkg/kube/fake.FailingKubeClient error model is flat: All methods share the same WaitError field for waiter operations, even when tests need different errors from Wait vs. WaitWithJobs. The RecordedWaitOptions field (added later) suggests the type is accumulating special cases.
  • Registry tests not separated from unit tests: The OCI registry suite tests are significantly slower than unit tests (they start real servers) but run unconditionally with go test. Build tags like //go:build integration would let CI separate fast/slow passes.
  • No benchmarks found: No Benchmark* functions in test files — surprising for a tool that processes potentially large chart files and Kubernetes manifests.

Patterns worth emulating#

  1. Golden file testing with --update — applicable to any project with human-readable output (CLI tools, code generators, report tools). The cost of adoption is one flag.Bool and one os.WriteFile call.
  2. cmdTestCase declarative DSL pattern — wrapping CLI execution in a minimal struct + runner function scales to hundreds of integration-style tests without per-test boilerplate. Works for any Cobra-based CLI.
  3. FailingKubeClient error injection via struct fields — cleaner than generating mocks when the interface has a small, stable method set. Embed the “happy path” fake and override individual methods with error conditions.
  4. actionConfigFixture(t) shared constructor — centralizing test wiring prevents test drift and makes the dependency graph of what’s under test explicit.
Backward Helm — API Surface Istio — Overview Forward