Temporal — Architecture#

Architectural style#

Layered distributed system with event-sourced core and fx-based dependency injection

Temporal is a multi-service distributed system built from four cooperating gRPC services (Frontend, History, Matching, Worker) that together constitute the “Temporal cluster.” Each service is an independently deployable process, yet all are compiled from a single binary (temporal-server) and started selectively via the --service flag. Internally, the server uses Uber’s fx dependency injection framework to wire every service’s component graph — making this one of the most sophisticated real-world fx deployments in open-source Go.

The architecture is anchored on event sourcing: workflow execution state is never stored directly; instead, an append-only history of events is persisted, and state is always reconstructed by deterministic replay. This core commitment propagates into every architectural decision — from how History shards are managed, to why the Matching service exists at all.

The system also contains an in-progress architectural migration from its original Cadence-derived execution model (in service/history/workflow/) to a new Hierarchical State Machine model (in service/history/hsm/ and chasm/), giving it a “dual execution model” structure during this transition.

Evidence:

  • temporal/fx.go: TopLevelModule wires the four services via fx.New and nested fx.App instances.
  • temporal/fx.go: HistoryServiceProvider, MatchingServiceProvider, FrontendServiceProvider, WorkerServiceProvider each create independent fx.App graphs for their service.
  • service/history/fx.go: History Module is a large fx.Options(...) block composing 30+ providers.
  • docs/architecture/README.md: The official architecture documentation confirms the four-service model and the event-sourcing premise.

Component diagram (textual)#

                        ┌──────────────────────────────────────────┐
                        │              Temporal Cluster            │
                        │                                          │
  User App (SDK)  ──gRPC──>  ┌──────────────────────┐            │
                        │    │   Frontend Service    │            │
  Worker (SDK)    ──gRPC──>  │  (public gRPC + REST  │            │
                        │    │   gateway, auth, RL)  │            │
                        │    └──────┬───────────────┘            │
                        │           │ internal gRPC               │
                        │    ┌──────▼───────────────┐            │
                        │    │   History Service     │            │
                        │    │  (shard-based WF state │           │
                        │    │   machine + event log) │           │
                        │    └──────┬───────────────┘            │
                        │           │ internal gRPC               │
                        │    ┌──────▼───────────────┐            │
                        │    │  Matching Service     │            │
                        │    │  (task queue routing, │            │
                        │    │   long-poll dispatch) │            │
                        │    └───────────────────────┘            │
                        │                                          │
                        │    ┌──────────────────────┐             │
                        │    │   Worker Service      │             │
                        │    │  (internal workflows: │             │
                        │    │   archival, timers,   │             │
                        │    │   replication, sched) │             │
                        │    └───────────────────────┘            │
                        │                                          │
                        │    ┌──────────────────────┐             │
                        │    │  Persistence Layer    │             │
                        │    │  (Cassandra / SQL /   │             │
                        │    │   Elasticsearch)      │             │
                        │    └───────────────────────┘            │
                        └──────────────────────────────────────────┘

  Gossip membership: all services participate in Ringpop (or static host config)
  Internal gRPC: all inter-service communication uses generated protobuf services

Core components#

Frontend Service#

  • Package: service/frontend
  • Responsibility: Public-facing gRPC gateway for the Temporal API. Handles all client-originated requests (StartWorkflow, SignalWorkflow, QueryWorkflow, etc.) and worker poll requests (PollWorkflowTaskQueue, PollActivityTaskQueue). Applies authentication (ClaimMapper, Authorizer), rate limiting, and namespace routing before forwarding to History or Matching.
  • Key types: Handler (implements workflowservice.WorkflowServiceServer), AdminHandler (implements adminservice.AdminServiceServer), Config (from service/frontend/configs)
  • Dependencies: client/history, client/matching, common/authorization, common/quotas, common/namespace, common/rpc

History Service#

  • Package: service/history
  • Responsibility: The heart of the system. Manages individual workflow executions as event-sourced state machines. Handles workflow lifecycle RPCs, appends history events, updates mutable state, and enqueues transfer/timer tasks for further processing. Partitioned into shards — fixed at cluster creation — each shard owns a subset of workflows.
  • Key types: Handler (implements historyservice.HistoryServiceServer), HistoryEngine (per-shard execution engine), workflow.MutableState (in-memory workflow execution state), shard.Context (per-shard resource manager), hsm.Registry (new-generation state machine registry)
  • Dependencies: common/persistence, service/matching (via client/matching), common/namespace, common/tasks, components/callbacks, components/nexusoperations

Matching Service#

  • Package: service/matching
  • Responsibility: Manages task queues being polled by Temporal Worker processes. Routes workflow and activity tasks from History to the appropriate worker. Splits task queues into partitions for throughput; implements a tree-based forwarding protocol to match pollers with tasks across partitions.
  • Key types: Handler (implements matchingservice.MatchingServiceServer), Engine (task queue management), taskQueueManager (per-task-queue state)
  • Dependencies: common/persistence, common/membership, common/tasks

Worker Service#

  • Package: service/worker
  • Responsibility: Runs Temporal’s own internal Workflow and Activity workers. Houses background jobs that are implemented using the Temporal SDK itself: archival (moving old histories to S3/GCS), replication (cross-cluster event sync), scheduler (cron/calendar schedule workflows), batcher (bulk workflow operations), and worker deployment management.
  • Key types: Service, individual worker components (scanner, scheduler, replicator, batcher, workerdeployment)
  • Dependencies: All four services (it acts as a Temporal worker pointing at the Temporal cluster itself)

Persistence Layer#

  • Package: common/persistence, common/persistence/sql, common/persistence/cassandra
  • Responsibility: Multi-backend storage abstraction. Defines interfaces for all storage operations (ExecutionManager, ShardManager, NamespaceManager, QueueV2, etc.) and provides implementations for Cassandra (via gocql), PostgreSQL/MySQL/SQLite (via SQL drivers), and Elasticsearch/OpenSearch (for workflow visibility/search). A DataStoreFactory pattern allows runtime selection of backend.
  • Key types: ExecutionManager, ShardManager, VisibilityManager, DataStoreFactory, AbstractDataStoreFactory
  • Dependencies: Database driver libraries loaded via blank imports in cmd/server/main.go

Dynamic Config System#

  • Package: common/dynamicconfig
  • Responsibility: A runtime-tunable configuration system. Settings (rate limits, feature flags, timeouts) can be changed without restart via YAML files or a custom Client implementation. Generated typed key accessors (via cmd/tools/gendynamicconfig) provide compile-time safety. All services read from this system for tunable parameters.
  • Key types: Collection (typed key accessor), Client (interface), FileBasedClient, NoopClient
  • Dependencies: Observed by all services; changes propagate on a polling interval

Cluster Membership#

  • Package: common/membership, common/membership/ringpop, common/membership/static
  • Responsibility: Gossip-based or static host membership for service discovery within the cluster. Ringpop (temporaryhio/ringpop-go) is the production backend, allowing services to discover which node owns which History shard without a central coordinator. Static membership is used in single-node dev deployments.
  • Key types: Monitor (interface), ServiceResolver (maps shard/task-queue to host), ringpop.Monitor
  • Dependencies: temporaryhio/ringpop-go, gRPC transport

HSM / CHASM Subsystem (next-generation execution)#

  • Package: service/history/hsm, chasm/
  • Responsibility: A new hierarchical state machine framework replacing the original monolithic workflow execution model. hsm.Registry maps state machine types to definitions; hsm.Node trees represent hierarchical execution state; chasm/ extends this with the CHASM protocol for coordinated async execution. Components (callbacks, nexusoperations) register as child state machines in this framework.
  • Key types: hsm.StateMachineDefinition, hsm.Node, hsm.Registry, hsm.Environment, chasm.Engine
  • Dependencies: common/persistence, service/history/workflow, components/

Data flow#

Typical workflow execution lifecycle#

1. Client calls StartWorkflowExecution
   → Frontend: validates auth + namespace, applies rate limits
   → Frontend: routes to History service via internal gRPC (shard = hash(workflowId) % numShards)

2. History (shard handler):
   → Loads or creates MutableState for the workflow
   → Appends WorkflowExecutionStarted event to history log
   → Creates a TransferTask (representing "schedule WorkflowTask in Matching")
   → Persists atomically: history events + mutable state + transfer task

3. History Queue Processor:
   → Reads TransferTask from the queue
   → Calls Matching service: AddWorkflowTask(taskQueue, workflowId, ...)

4. Matching Service:
   → Receives AddWorkflowTask
   → Stores task in task queue partition
   → When a Worker is polling: dispatches task to worker

5. Worker:
   → Receives WorkflowTask
   → Replays workflow history to reconstruct current state
   → Executes workflow code until blocked (e.g., schedules an Activity)
   → Calls RespondWorkflowTaskCompleted with commands (e.g., ScheduleActivityTask)

6. History (RespondWorkflowTaskCompleted handler):
   → Appends WorkflowTaskCompleted + ActivityScheduled events
   → Creates TransferTask for the activity
   → Persists atomically

7. (Cycle continues for each workflow step)

Event sourcing is the consistency mechanism#

The atomicity of step 2 and 6 (events + mutable state + tasks persisted together) is the key consistency guarantee. If the process crashes after persisting but before tasks are enqueued to Matching, the queue processor re-reads the task from the database and retries. This is the “at-least-once” delivery model that makes durable execution possible.

Initialization / Bootstrap#

Process startup sequence#

1. cmd/server/main.go: main()
   → urfave/cli parses flags (--service, --config-file, ...)
   → config.Load(...) reads YAML config
   → log.NewZapLogger builds the root logger
   → authorization.GetAuthorizerFromConfig creates the authorizer plugin
   → temporal.NewServer(...) called with functional options

2. temporal.NewServer(opts...):
   → Creates ServerFx via NewServerFx(TopLevelModule, opts...)
   → TopLevelModule is an fx.Options combining all top-level providers

3. fx.New(TopLevelModule):
   → ServerOptionsProvider: resolves all options, creates metrics handler,
     dynamic config client, TLS provider, ES client
   → ApplyClusterMetadataConfigProvider: reads cluster metadata from DB,
     validates and reconciles config
   → TaskCategoryRegistryProvider: registers history task categories
   → PersistenceFactoryProvider: creates DB connection factories
   → HistoryServiceProvider, MatchingServiceProvider, FrontendServiceProvider,
     WorkerServiceProvider: each creates a NESTED fx.App for the service
     (only for services listed in --service)
   → dynamicconfig.Module, pprof.Module, TraceExportModule, chasm.Module
   → fx.Invoke(ServerLifetimeHooks): registers start/stop lifecycle hooks

4. s.Start() → s.app.Start(ctx):
   → fx runs lifecycle Start hooks in dependency order
   → Each service's nested fx.App starts:
       - gRPC server binds to configured port
       - Membership monitor starts (Ringpop joins the ring)
       - Shard controllers start (History only)
       - Queue processors start (History only)
       - Worker pollers start (Worker service only)
   → Server blocks on interrupt channel (SIGINT/SIGTERM)

5. s.Stop() → s.app.Stop(ctx):
   → fx runs lifecycle Stop hooks in reverse dependency order
   → Services drain in-flight RPCs, stop processors, close DB connections

Dependency injection pattern#

Each service runs in its own fx.App instance (a nested graph). Common dependencies (config, logger, metrics, persistence factory, membership) are instantiated in the top-level server graph and injected into service graphs via ServiceProviderParamsCommon — a large fx.In struct that gets realized in the server graph and then re-provided as fx.Supply(...) to each service graph. This is an unusual pattern (commented on in the source: “we want an fx.In object in the server graph, and an fx.Out object in the service graphs”) that works around fx’s graph isolation boundaries.

Within each service graph, providers follow the standard fx idiom: constructors are registered with fx.Provide, startup/shutdown logic registered with fx.Invoke(ServiceLifetimeHooks).

Configuration#

Layered configuration: static YAML + runtime dynamic config

  1. Static YAML config (primary): Loaded at startup via config.Load(...). Sources:

    • --config-file <path>: explicit path (recommended for containers)
    • --config <dir> + --env <name> + --zone <name>: legacy convention (config/<env>.yaml)
    • Embedded config (fallback for local dev)
    • Environment variables: TEMPORAL_CONFIG_FILE, TEMPORAL_ENVIRONMENT, etc.
    • The config struct (common/config.Config) covers: persistence backends, cluster metadata, TLS, metrics, archival, authorization, dynamic config client settings, and per-service gRPC listener addresses.

  2. Dynamic config (common/dynamicconfig): Runtime-tunable parameters (rate limits, timeouts, feature flags, queue processing parameters). Backed by a YAML file polled at a configured interval (FileBasedClient) or a no-op client if not configured. Settings are accessed via typed keys generated by cmd/tools/gendynamicconfig. Changes take effect within the polling interval without restart.

  3. Authorization config: Extracted from the static config via authorization.GetAuthorizerFromConfig. Supports JWT-based auth via go.temporal.io/server/common/authorization.

Key design decisions#

  1. Event sourcing as the foundation of durability
    Every workflow state transition is expressed as an event appended to an append-only history log. State is reconstructed by replay, never stored directly. This eliminates the need for distributed transactions across the server and the user’s worker processes — the server only needs to durably store events; the worker reconstructs state by replaying them. All other architectural decisions flow from this commitment.

  2. Fixed shard count, hash-based ownership
    History shards are the fundamental unit of parallelism and consistency. Shard count is fixed at cluster creation (typically 512 or 2048). Each workflow execution maps to a shard via hash(workflowId) % numShards. Shard ownership is tracked in the database; Ringpop membership events trigger shard transfer between History nodes. This gives linear scalability without global coordination — each shard is independent.

  3. Four services as separate scaling units
    Frontend, History, Matching, and Worker scale independently. A cluster with high workflow start volume can scale Frontend + History without scaling Matching. A cluster with many concurrent activities can scale Matching without scaling Frontend. In development, all four run in a single process. In production, they run as separate deployments. The temporal-server binary supports this via the --service flag.

  4. go.uber.org/fx for the entire dependency graph
    Using fx at the scale of an entire distributed system server is unusual. The benefit is that every component’s dependencies are declared explicitly and verified at startup. The trade-off is complexity: the nested fx.App pattern (server graph → service graph) adds indirection, and fx error messages can be opaque. The project commits to fx throughout rather than mixing manual wiring.

  5. Dual execution model during architectural migration
    The original Cadence-derived execution model (service/history/workflow/MutableState) and the new Hierarchical State Machine (service/history/hsm/) coexist. The components/ package (callbacks, nexusoperations) represents the first wave of features built entirely on the new HSM model. The CHASM layer (chasm/) sits above HSM and is experimental. This mid-migration architecture is architecturally significant: it shows how a mature production system incrementally replaces a core abstraction without a flag day.

  6. One package per RPC handler in History
    The History service has 60+ packages under service/history/api/, one per RPC method (e.g., startworkflow/, respondworkflowtaskcompleted/). This radical decomposition keeps handlers small, independently testable, and prevents the “god handler” anti-pattern common in large gRPC services. Each handler package imports only the dependencies it needs, making the dependency graph of each handler visible to the compiler.

Backward Temporal — Dependencies Temporal — Patterns Forward