MinIO — Testing#

Test metrics#

  • Test files: 247 (*_test.go)
  • Total Go files: 902 (cmd: 453, internal: 423, other: 26)
  • Ratio (test files / source files): ~27% (247/902)
  • Test frameworks: stdlib testing only — no testify, gomock, ginkgo, or goconvey
  • Benchmark files: 59 files contain func Benchmark*

Test organization#

  • Placement: Same package (package cmd) overwhelmingly. Only 2 files use the external _test suffix — nearly all tests have white-box access to internals.
  • Helper packages:
    • cmd/test-utils_test.go — central infrastructure: TestServer, prepareErasure*(), prepareFS(), sign/auth helpers, random data generators (~700 lines)
    • cmd/benchmark-utils_test.go — benchmark scaffolding over multiple backends (ErasureSD, Erasure16, ErasureSet32)
    • internal/logger/target/testlogger/ — production package that routes log output to testing.TB during individual tests; uses atomic.Pointer[testing.TB] to swap the log target per test
    • internal/grid/debug.goSetupTestGrid(n int) and TestGrid type that bootstraps N in-process grid nodes for cluster communication tests (non-test file deliberately, so benchmarks can import it)
  • Fixtures: cmd/testdata/ contains binary fixtures: xl.meta, xl.meta-v1.2.zst, xl-meta-merge.zip, xl-many-parts.meta, metacache.s2, undeleteable-object.tgz, decryptObjectInfo.json.zst, TLS keys. Similar testdata/ directories under internal/event/target/, internal/s3select/csv/, internal/s3select/json/. These are checked-in binary artifacts used to test backward compatibility of on-disk format parsing.
  • Generated tests: *_gen_test.go files (10+ in cmd/) are generated by tinylib/msgp (//go:generate) and test msgpack round-trip serialization and msgp.Skip behavior for every generated type. They are mechanical but provide complete serialization coverage for all on-disk metadata types.

Test patterns#

Table-driven tests#

  • Prevalence: Heavy use — 699 occurrences of tests :=, testCases, tt.Run, or tc.name in *_test.go files; the patterns analysis noted 2340 total occurrences including production code usages.
  • Style: Anonymous structs with named fields (name, input, and expected fields). Subtests via t.Run(tc.name, ...).
  • Example: cmd/erasure-object_test.go:78TestErasureDeleteObjectBasic uses a testCases := []struct{bucket string; object string; expectedError error} table to cover missing-bucket, missing-object, and valid-object cases in a single loop.

Manual test suite pattern (custom, not gocheck)#

  • MinIO’s cmd/server_test.go defines TestSuiteCommon — a struct holding server state — and a custom check type that embeds *testing.T and adds an Assert(got, expected any) helper using reflect.DeepEqual.
  • runAllTests(suite *TestSuiteCommon, c *check) manually calls each test method in sequence. TestServerSuite instantiates two suite configurations (ErasureSD and ErasureSet) and calls runAllTests for each. This pattern predates t.Run subtests and is kept for backward compatibility; it is not gocheck despite the surface resemblance.
  • The check type pattern is also used in sts-handlers_test.go and admin-handlers-users-race_test.go.

Mocking approach#

  • Strategy: Real implementations only — no mocks, fakes, or interface stubs anywhere in the test suite. Dependencies are satisfied by spinning up real in-process object layers (erasureServerPools) on temporary OS directories.
  • prepareErasure(ctx, nDisks) creates N temporary directories, calls initObjectLayer(), and returns a fully functional ObjectLayer. Callers defer removeRoots(disks) for cleanup.
  • prepareFS(ctx) does the same for single-drive mode.
  • The one concession to testability is newObjectLayerFn() — a package-level indirection that reads globalObjectAPI under a lock, allowing tests to inject a custom ObjectLayer via globalObjLayerMutex + globalObjectAPI = objLayer.
  • The TestServer type wraps httptest.NewUnstartedServer around the full MinIO HTTP handler, producing a complete in-process server including IAM, event notification, and config subsystems.

Global state management#

  • TestMain in cmd/test-utils_test.go:73 sets globalIsTesting = true, disables color output and logging, unsets environment variables that interfere with tests, and calls resetTestGlobals().
  • resetTestGlobals() is called before each top-level test in the suite (admin-handlers_test.go:54,101) to clear global IAM state, bucket metadata, and object layer references between tests. This is the cost of the global-variable DI pattern: tests must manually reset shared state.
  • Only 4 uses of t.Parallel() across the entire codebase — the heavy reliance on global state makes parallel unit tests impractical without significant refactoring.

Integration tests#

  • Present: Yes, but separated by naming convention (bash scripts + CI workflows) rather than Go build tags.
  • How: Shell scripts in buildscripts/ launch a real MinIO binary and verify behavior end-to-end:
    • make verify-healing / buildscripts/verify-healing.sh — starts a 4-drive erasure set, corrupts a drive, replaces it, verifies data heals correctly.
    • make test-resiliencydocs/resiliency/resiliency-tests.sh uses Docker Compose to simulate node failures and network partitions.
    • make test-iam — Go tests targeting TestIAM* functions, run with -race against real LDAP/OIDC backends.
    • make test-replication — multi-site replication scenarios via bash scripts.
  • CI separation: Six specialized GitHub Actions workflows (go-healing.yml, go-resiliency.yml, iam-integrations.yaml, replication.yaml, mint.yml, root-disable.yml) run the integration suites on every PR, separate from the unit test workflow (go.yml).
  • The Mint test suite (run-mint.sh, mint.yml) is an external S3-compatibility test suite run against a live MinIO instance.

Benchmarks#

  • cmd/benchmark-utils_test.go provides BenchmarkPutObject, BenchmarkGetObject, BenchmarkListObjects helpers that run across all backend types.
  • internal/grid/benchmark_test.go benchmarks cluster RPC throughput at scale: BenchmarkRequests tests 2, 4, 8, 16, 32 servers in a sub-benchmark matrix.
  • Race detector is explicitly enabled for the main test suite via buildscripts/race.sh and make test-race.

Test quality observations#

  • What’s done well:
    • No external test frameworks — pure stdlib reduces friction and dependency surface. Tests are immediately readable by any Go developer.
    • Real implementations — testing against actual erasure-coded I/O on temp directories means tests catch real bugs, not interface mismatches. The TestServer integration approach is particularly strong: handler tests execute real S3 request signing, real XML parsing, and real storage, eliminating an entire category of mock-vs-reality divergence.
    • Generated serialization tests — every msgpack type has mechanical round-trip coverage; format regressions are caught automatically.
    • Backward-compatibility fixtures — binary xl.meta artifacts in testdata/ ensure that new code can still read old on-disk formats; this is critical for an on-disk storage system.
    • Dedicated test logger (internal/logger/target/testlogger) — routes production log output to t.Log() so test failures include server-side error context without polluting passing test output.
    • Table-driven tests are systematic and comprehensive, covering error boundary cases at the storage layer (quorum failures, corrupt drives, missing metadata).
    • Specialized CI workflows — healing, resiliency, IAM, replication, and S3-compatibility each have their own CI pipeline, so regressions in one subsystem don’t get lost in a monolithic test run.
  • What could improve:
    • Global state coupling — the global* variable model forces resetTestGlobals() calls between tests and makes t.Parallel() nearly impossible. This is the biggest structural testing debt. Tests that spin up full object layers are slow and cannot be parallelized.
    • No build tags separating slow tests — tests like TestErasureObject* that initialize 16 real disk directories are in the same package and run with go test ./.... There’s no //go:build integration to skip them in fast-feedback loops.
    • Manual test suite patternTestSuiteCommon / runAllTests predates t.Run and loses per-subtest failure isolation and reporting granularity. Migrating to t.Run subtests would give better output.
  • Patterns worth emulating:
    • TestServer + prepareErasure* factory pattern — a clean way to bootstrap a complete subsystem under test without mocks, using real temp directories and defer removeRoots(). Translatable to any system with a pluggable storage layer.
    • Dedicated testlogger package — registering a test-aware logger as a production-code target at init() time means logs appear in go test -v output automatically, zero instrumentation in test code. The atomic.Pointer[testing.TB] swap trick is elegant.
    • Generated round-trip tests — if a project uses code generation for serialization, generating corresponding _gen_test.go round-trip tests is low-effort insurance.
    • Per-concern CI workflows — separating unit, healing, resiliency, and compatibility tests into independent GitHub Actions workflows gives faster feedback and clearer failure attribution.