Temporal — Testing#

Test metrics#

• Test files: 741
• Total Go files: 2,555
• Ratio (test files / source files): ~29% (741 / 2555)
• Test frameworks: github.com/stretchr/testify (require + suite), go.uber.org/mock/gomock

Test organization#

Placement#

Tests are placed in two distinct tiers:

1. Unit tests — *_test.go files living alongside source files in the same package (white-box), or as package foo_test (black-box). This is standard Go layout.
2. Functional tests — collected in the top-level tests/ directory as a separate package (package tests), each file containing one or more testify suites that inherit from testcore.FunctionalTestBase.

Helper packages#

Temporal has an unusually rich common/testing/ directory with purpose-built testing utilities — arguably one of the largest in-project testing libraries in the 50-project set:

Fixtures#

• common/auth/testdata/ — TLS certificates for auth tests
• service/worker/scheduler/testdata/ — scheduler test fixtures
• service/worker/workerdeployment/testdata/ — deployment test data
• tools/tdbg/testdata/, tools/testrunner/testdata/ — CLI/tool fixtures

Test patterns#

Table-driven tests#

• Prevalence: Heavy — 1,775 occurrences of t.Run, tt.Run, testCases, and similar patterns across *_test.go files
• Style: Both named struct slices (type testCase struct { name string; ... }) and anonymous []struct{ ... } inline definitions are used. Named structs dominate in longer tests; anonymous structs appear in compact unit tests.
• Example: common/persistence/versionhistory/version_history_test.go — large table-driven test with anonymous structs covering many version-history edge cases.

Mocking approach#

• Strategy: go.uber.org/mock/gomock (the community fork of the original golang/mock) with code generation via mockery/go generate. 126 *_mock.go files exist — the highest count in the 50-project set.
• Placement: Mock files live next to the interfaces they mock:
  • api/adminservicemock/, api/historyservicemock/, api/matchingservicemock/, api/testservicemock/ — entire gRPC service client mocks
  • common/persistence/mock/ — persistence layer mocks
  • common/testing/mocksdk/ — Temporal SDK interface mocks
  • Many individual *_mock.go files alongside source
• Example: chasm/lib/scheduler/handler_test.go:117 — env.MockEngine.EXPECT().StartExecution(gomock.Any(), gomock.Any(), gomock.Any()) — standard gomock expectation setup using a pre-built MockEngine from the test environment.
• Generation command: go generate directives are present across source files; dedicated common/testing/mockapi/generate.go and common/testing/mocksdk/generate.go contain the generation directives for the large service mocks.

Integration tests#

• Present: Yes — extensive
• How: Two mechanisms:
  1. In-process “OneBox” server — tests/testcore/onebox.go defines TemporalImpl, which runs all four Temporal services (Frontend, History, Matching, Worker) in-process within a single fx.App per test suite. This gives functional tests a real end-to-end server without Docker, using SQLite or a real Cassandra/PostgreSQL instance.
  2. Containerized databases in CI — GitHub Actions runs functional test jobs with Docker Compose spinning up Cassandra, PostgreSQL, MySQL, Elasticsearch, and OpenSearch for full persistence coverage.
• Separation: Functional tests are in tests/ directory (separate package package tests). Unit tests are co-located with source. The Makefile provides separate targets: unit-test-coverage, functional-test-coverage, functional-test-ndc-coverage, functional-test-xdc-coverage, integration-test.
• Fault injection: config.FaultInjection + testcore.WithFaultInjectionConfig(...) allows tests to inject persistence faults into specific operations (e.g., tests/acquire_shard_test.go injects failures during shard acquisition to test recovery paths).

The parallelsuite pattern#

Temporal has built a custom generic suite on top of testify:

// parallelsuite.Suite[T testingSuite] — each Test* method runs in a fresh
// suite instance in its own t.Parallel() goroutine.
func Run[T testingSuite](t *testing.T, s T, args ...any) { ... }

Rules enforced at runtime via reflection:

• Suite struct must be named with suffix "Suite"
• Suite must have exactly one field (the embedded Suite[T]) — extra fields must become args
• All exported non-Test* methods on the suite cause a panic (no accidental exported helpers)
• Each Test* method gets a fresh suite copy initialized to its own *testing.T

This is a significant investment in test isolation and parallelism: unlike standard testify suites where test methods share state, parallelsuite gives every test method a private suite instance, making parallel execution safe.

The testhooks pattern#

A notable production-testing integration:

// In testhooks/hooks.go — typed hook keys parameterized by value type T and scope S
var MatchingDisableSyncMatch = newKey[bool, namespace.ID]()
var UpdateWithStartInBetweenLockAndStart = newKey[func(), namespace.ID]()

Production code checks these hooks at key execution points. Tests inject hook values via testcluster.InjectHook(key, value, scope). The type parameters encode both the value type and the scope (namespace vs. global), so mismatches are caught at compile time. This approach lets functional tests reach into production code paths without modifying source files — analogous to dependency injection but via a typed side-channel.

The testvars pattern#

common/testing/testvars.TestVars generates deterministic test data from the test name:

tv := testvars.New(t) // seeds from t.Name()
tv.WorkflowID()       // always same string for same test
tv.TaskQueue()        // deterministic task queue name
tv.NamespaceID()      // deterministic namespace ID

This avoids random UUIDs that make test failures hard to reproduce, while still isolating parallel tests from each other.

The historyrequire pattern#

Domain-specific assertions for workflow history event sequences:

s.EqualHistoryEvents(`
  1 WorkflowExecutionStarted
  2 WorkflowTaskScheduled
  3 WorkflowTaskStarted
  4 WorkflowTaskCompleted
`, actualEvents)

The history string format uses event-type names (not integers), and the assertion produces a readable diff on failure. This is a significant improvement over asserting raw protobuf equality — it shows exactly which events diverge.

CI configuration#

• Three test tiers: unit, integration (persistence), functional (end-to-end)
• Persistence matrix: Cassandra+Elasticsearch, Cassandra+OpenSearch, Cassandra+OpenSearch3, PostgreSQL, SQLite — tests run against all backends
• Retry on flakiness: MAX_TEST_ATTEMPTS=3 — if a test fails, the CI monitor script reruns the binary up to 3 times before declaring failure. This is an explicit admission that some tests are flaky at scale.
• Shard splitting: Functional test jobs are split into 3 shards (SHARD_COUNT: 3) to parallelize across GitHub Actions runners
• Separate NDS/XDC jobs: functional-test-ndc-coverage and functional-test-xdc-coverage specifically test multi-cluster replication scenarios
• Flaky test tracking: A dedicated flaky-tests-report.yml workflow aggregates flaky test data from JUnit XML artifacts

Test quality observations#

What’s done well#

• Testing library depth: The common/testing/ package is among the most comprehensive in-project testing libraries seen. Domain-specific assertions (historyrequire, protorequire), injectable interceptors (grpcinject, testhooks), deterministic test data (testvars), and in-memory backends (testtelemetry) eliminate a huge class of fragile test setup.
• OneBox server: Running all four services in-process removes Docker as a test dependency for most tests, dramatically lowering the barrier to running full functional tests locally.
• Typed test hooks: The testhooks pattern is an elegant solution to the “how do I test timing-sensitive production paths” problem. The generic type parameters prevent stale hooks from silently succeeding.
• parallelsuite enforcement: The panic-on-incorrect-usage enforcement (naming convention, field count, method naming) prevents the subtle suite-shared-state bugs that plague larger testify suite-heavy codebases.
• Fault injection: First-class persistence fault injection through config.FaultInjection allows testing recovery paths that are otherwise impossible to trigger in integration tests.
• Mock coverage: 126 gomock-generated files cover every service boundary, enabling pure unit tests of complex components without live gRPC connections.

What could improve#

• Flaky test acknowledgment: MAX_TEST_ATTEMPTS=3 in CI is a pragmatic fix but signals underlying non-determinism. The flaky-tests-report.yml workflow suggests this is a known, ongoing issue rather than an occasional occurrence.
• Test suite inheritance depth: FunctionalTestBase embeds suite.Suite, *require.Assertions, ProtoAssertions, HistoryRequire, UpdateUtils, and carries 10+ fields. This is deeply stateful and makes individual test methods hard to reason about in isolation. The newer parallelsuite approach solves this structurally for pure unit tests, but the functional test tier hasn’t migrated.
• Test file size: Some functional test files (e.g., activity_test.go) are very large, mixing multiple suites and many hundreds of lines. Splitting by feature area could improve discoverability.

Patterns worth emulating#

1. testvars — deterministic test data from test name: Any project that generates UUIDs or random strings in tests should consider seeding from t.Name() instead. Reproducing failures becomes trivial.
2. historyrequire — domain assertions with readable diffs: Instead of require.Equal(t, expectedEvents, actualEvents) (which produces a protobuf noise dump), writing a small DSL for your domain’s primary data structure pays off in long-term test maintainability.
3. testhooks — typed injectable hooks: For testing timing-sensitive distributed systems code paths, a typed hook registry is far safer than ad-hoc testing.Short() checks or build-tagged stub functions.
4. parallelsuite — enforced isolation via generics: The pattern of enforcing no-shared-state via reflection panics at test startup, rather than hoping developers are careful, is worth adopting in any codebase with heavy testify suite use.
5. OneBox integration testing: Spinning up the full server in-process (rather than requiring Docker) is the right trade-off for a complex distributed system — it eliminates infrastructure flakiness while still testing real code paths.