Temporal — Patterns#

Concurrency patterns#

Custom goroutine lifecycle management (goro package)#

  • Usage: Project-wide. The common/goro package is Temporal’s answer to the question “how do we manage long-lived background goroutines?” It is used across all four services wherever a component has a Start()/Stop() lifecycle.
  • Types:
    • goro.Handle — wraps a single goroutine with a context.CancelFunc and done chan struct{}. Callers call h.Cancel() to request stop and <-h.Done() to wait. Error is stored atomically and retrieved via h.Err().
    • goro.Group — manages a set of goroutines under a shared cancellable context. Group.Go(f) spawns them; Group.Cancel() + Group.Wait() drains them.
  • Example: common/goro/goro.go:35 — Handle.Go(f) launches f(ctx) and stores any returned error atomically.
  • Assessment: Highly idiomatic. The separation of NewHandle from Go (so the handle can be stored into a struct field before the goroutine starts) neatly avoids a common race condition. The package doc explicitly tells users to prefer errgroup for request-scoped work — a nice signal about intended scope.

Adaptive worker pool#

  • Usage: common/goro/adaptive_pool.go. Used by task queue processing loops to auto-scale goroutine count in response to queue depth.
  • Example: goro.AdaptivePool starts with minWorkers goroutines and adds more (up to maxWorkers) when queued work is piling up, then shrinks back by a configurable shrinkFactor. Sizing is driven by measuring actual task-dispatch delay vs. a targetDelay.
  • Assessment: Unusually sophisticated for in-process worker pool management. The pool avoids the overhead of a fixed large pool while maintaining low latency for bursting workloads. Notably backed by clock.TimeSource (injectable mock clock), making it fully testable.

KeyedSet (generics-based goroutine registry)#

  • Usage: common/goro/keyed_set.go. Used by task queue managers to maintain exactly one goroutine per key (e.g., one goroutine per task queue partition).
  • Example: goro.KeyedSet[K comparable] — Sync(target map[K]struct{}, f func(context.Context, K)) cancels goroutines for keys no longer in target and starts new ones for newly added keys.
  • Assessment: A clean generics design — the key type is constrained to comparable only (not any), which is the correct minimal constraint. The Sync API avoids the common bug of starting duplicate goroutines while also not requiring callers to track individual handles.

Channel-based shutdown broadcast#

  • Usage: common/channel/shutdown_once.go, used throughout services.
  • Example: channel.ShutdownOnce — wraps a chan struct{} with atomic CAS on a status int32. Shutdown() closes the channel exactly once; Channel() exposes it for select statements; IsShutdown() allows polling.
  • Assessment: The idiomatic Go shutdown pattern, properly wrapped to avoid double-close panics. The use of atomic.CompareAndSwapInt32 (rather than a mutex) for the close guard is efficient for the common path.

Fan-out with buffered channels#

  • Usage: client/history/client.go:138-186. Replication status polling fans out gRPC calls across all history nodes.
  • Example:
    respChan := make(chan *historyservice.GetReplicationMessagesResponse, len(requestsByClient))
errChan := make(chan error, 1)
// launches one goroutine per client, collects first error
  • Assessment: Classic fan-out pattern with buffered channels sized to the number of workers (avoiding goroutine leak) and a capacity-1 error channel that captures the first failure without blocking.

Select-based multiplexing (367 sites)#

  • Usage: Pervasive — every long-running loop that needs cancellation, timeout, or task dispatch uses select.
  • Assessment: The project correctly prefers select over polling. The high count (367) is expected for a system with this many concurrent queues, timers, and processors.

Categories checked:#

  • Worker pools: Yes — goro.AdaptivePool, goro.DynamicWorkerPoolScheduler
  • Fan-out/fan-in: Yes — replication client, multi-rate-limiter combiner
  • Pipeline processing: Yes — the common/tasks package is an entire task scheduling mini-framework (see below)
  • Context cancellation: Pervasive — 8,416 context.Context references
  • Graceful shutdown: Yes — channel.ShutdownOnce, fx lifecycle hooks, goro.Group.Cancel()+Wait()
  • Rate limiting: Extreme prevalence — 1,553 references; a dedicated common/quotas package with 6+ rate limiter implementations

Task scheduling mini-framework#

The common/tasks package deserves special mention as a concurrency pattern at a higher level of abstraction. It defines a Scheduler[T Task] interface and provides multiple implementations:

  • FIFOScheduler — simple sequential dispatch
  • SequentialScheduler — per-key sequential execution (ordered within a key, parallel across keys)
  • GroupByScheduler[K comparable, T Task] — generic fan-out to per-key sub-schedulers
  • ExecutionQueueScheduler — priority-aware queue-based scheduler
  • DynamicWorkerPoolScheduler — dynamically sized goroutine pool for task dispatch
  • InterleavedWeightedRoundRobin — weighted interleaving for multi-queue dispatch (used for namespace isolation)
  • RateLimitedScheduler — wraps any scheduler with a rate-limiter

This is a notable pattern: rather than using raw goroutines and channels, History’s queue processors compose from this typed scheduler hierarchy. The use of generics (GroupByScheduler[K, T]) keeps the framework type-safe while remaining composable.

Error handling#

  • Style: Mixed — sentinel errors + gRPC-typed errors + fmt.Errorf wrapping
  • Error types defined:
    • Sentinel errors.New(...) variables for internal state machine conditions (ErrStateMachineNotFound, ErrUseCurrentExecution, ErrTaskDiscarded)
    • gRPC-aligned typed errors from go.temporal.io/api/serviceerror: serviceerror.NewInvalidArgument(...), serviceerror.NewNotFound(...), serviceerror.NewUnavailable(...), serviceerror.WorkflowExecutionAlreadyStarted (pointer type for errors.As)
    • Package-local sentinel errors prefixed with err (unexported, e.g., errRetryLimitExceeded, errRequestTimedOut)
    • Queue-specific queueerrors.NewUnprocessableTaskError(...) for dead-letter-queue routing
  • Wrapping approach: fmt.Errorf("context: %w", err) — consistent use of %w throughout the codebase. pkg/errors is not used.
  • Error inspection: errors.Is and errors.As are heavily used for behavior-based error handling (e.g., checking if an error is WorkflowExecutionAlreadyStarted to skip duplicate starts in the scheduler).
  • Examples:
    • chasm/lib/scheduler/scheduler.go:88-94 — package-level sentinel vars using gRPC-typed errors for API-facing errors
    • chasm/lib/scheduler/invoker_tasks.go:173 — fmt.Errorf("failed to read component: %w", err) for wrapping
    • chasm/lib/scheduler/invoker_tasks.go:669 — errors.As(err, &expectedErr) for typed error checking

Configuration pattern#

  • Approach: Builder pattern (method chaining on value receivers) for retry/backoff policies; functional options (type XxxOption func(*XxxOptions)) for CHASM framework configuration; go.uber.org/fx providers for service-level config injection.
  • Retry/backoff example (common/backoff):
    policy := backoff.NewExponentialRetryPolicy(100*time.Millisecond).
    WithBackoffCoefficient(2.0).
    WithMaximumInterval(10*time.Second).
    WithExpirationInterval(time.Minute)
    All WithXxx methods return *ExponentialRetryPolicy, enabling chaining. Three policy types exist: ExponentialRetryPolicy, ConstantDelayRetryPolicy, ErrorDependentRetryPolicy.
  • CHASM functional options (chasm/registrable_component.go):
    type RegistrableComponentOption func(*RegistrableComponent)
func WithEphemeral() RegistrableComponentOption { ... }
func WithSingleCluster() RegistrableComponentOption { ... }
func WithDetached() RegistrableComponentOption { ... }
  • Dynamic config (common/dynamicconfig): Settings are defined as typed keys (generated by cmd/tools/gendynamicconfig) and accessed via a Collection that resolves values from a polled YAML file. The setting type uses two generic parameters — setting[T any, P any] — where T is the value type and P encodes the precedence (namespace vs. global vs. task-queue scoped) at the type level, preventing misuse at compile time. This is a sophisticated use of generics as type-level documentation.

Dependency injection#

  • Approach: go.uber.org/fx throughout — the entire server and all four services are wired via fx.
  • Evidence:
    • temporal/fx.go — TopLevelModule is a fx.Options(...) composing all server-level providers.
    • service/history/fx.go — History’s Module is a large fx.Options(...) with 30+ fx.Provide and fx.Supply calls.
    • Each service runs in a nested fx.App instance. Common resources (logger, metrics, persistence, membership) are provided in the parent graph and passed to service graphs via fx.Supply(...) — working around fx’s graph isolation.
    • Component workers in service/worker/common/fx.go use a generic helper: AnnotateWorkerComponentProvider[T any](f func(t T) WorkerComponent) fx.Option — a generics helper that avoids repetitive fx annotation boilerplate.
  • Assessment: One of the most complete fx deployments in open-source Go. The nested graph pattern is unusual but necessary for the architecture (independent service scaling). The trade-off is opacity in error messages and an extra indirection layer.

Other notable patterns#

Interface guard (compile-time satisfaction check)#

Used heavily — var _ SomeInterface = (*SomeImpl)(nil) appears 126+ times across the codebase, concentrated in CHASM components and mock files.

  • chasm/lib/nexusoperation/operation.go:8-9 — double guard: var _ chasm.Component = (*Operation)(nil) and var _ chasm.StateMachine[...] = (*Operation)(nil)

Table-driven tests#

1,031 references to test case arrays/structs in *_test.go files — the dominant testing style. Both named struct slices (type testCase struct { name string; ... }) and anonymous structs are used.

Code generation#

Two major code generation passes:

  1. cmd/tools/gendynamicconfig — generates common/dynamicconfig/setting_gen.go (700+ lines) containing every typed dynamic config key. The //go:generate directive lives at the top of collection.go.
  2. gomock/mockery — generates 126 *_mock.go files, one per interface requiring a mock (e.g., MockTaskProcessor, MockContext, MockMutableContext). This is the heaviest mock generation setup seen in the 50-project set.

Generics usage (Go 1.18+)#

Targeted and meaningful — not pervasive but used where abstraction pays off:

  • goro.KeyedSet[K comparable] — goroutine registry keyed by any comparable type
  • common/tasks.GroupByScheduler[K comparable, T Task] — generic task grouping scheduler
  • common/dynamicconfig.setting[T any, P any] — type-safe dynamic config settings with phantom type parameter for precedence
  • common/dynamicconfig.GradualChange[T any] — generic gradual rollout container
  • chasm.Field[T any], chasm.ParentPtr[T any] — CHASM framework typed node references
  • chasm.SideEffectTaskHandlerBase[T any] — base type for task handlers parameterized on component type
  • Pattern: generics are used for framework primitives (schedulers, goroutine sets, config keys) where the type parameter provides safety across many callers, not for business logic.

Functional options on type aliases#

chasm/engine.go:91 — type TransitionOption func(*TransitionOptions) with func WithSpeculative() TransitionOption, func WithBusinessIDPolicy(...) TransitionOption, func WithRequestID(...) TransitionOption. This is the functional-options-as-first-class-types variant: options are named types rather than a bare func(*T), which allows package-level documentation per option.

Registry pattern#

service/history/hsm.Registry — a central registry for state machine type definitions. Components register via Registry.RegisterMachine(StateMachineDefinition) at server startup (via fx fx.Invoke). The registry maps state machine type names to definitions, enabling the new HSM execution model to dynamically dispatch transitions.

Interleaved Weighted Round Robin#

common/tasks/interleaved_weighted_round_robin.go — a FIFO-with-weights scheduler used to give each namespace a fair share of processing capacity. This is the core mechanism for multi-tenant isolation within a single History shard.

Jitter-on-retry (global anti-thundering-herd)#

common/backoff/retrypolicy.go:178-186 — jitter is automatically added to exponential backoff intervals using a package-level atomic.Pointer[rand.Rand] that is lazily initialized with CAS. This is a robust approach: jitter is opt-in via WithJitter on non-exponential policies, but automatically included in ExponentialRetryPolicy because distributed systems always need it.

Invariant enforcement via var _ Interface = (*Impl)(nil)#

Beyond compile-time checking, Temporal uses this as documentation: each CHASM component file starts with two guard lines declaring which interfaces it satisfies. This makes the interface set of a type immediately visible at the top of a file without reading the method list.

Backward Temporal — Architecture Temporal — Interfaces Forward