Messaging Architecture: NATS Server vs Temporal vs Dapr#

Summary#

Three CNCF projects, three completely different answers to the question “what is messaging?” NATS treats messaging as the fundamental substrate of connectivity — a high-performance primitive that everything else is built on top of. Temporal treats messaging as state — workflow events are the application state, durably ordered, and replayed to reconstruct truth. Dapr treats messaging as an API — a portable interface that abstracts away any underlying broker, making the messaging system a configuration choice rather than a code dependency. The architectural implications of these three starting positions propagate into every design decision: delivery guarantees, persistence strategy, routing model, concurrency architecture, and even how the Go code is organized.

Comparison dimensions#

Fundamental messaging paradigm#

NATS is a messaging system — its entire raison d’être is moving bytes between subjects and subscribers as fast as possible. Pub/sub is not a feature; it is the architecture. Every other capability (persistence via JetStream, clustering, MQTT bridging) is layered on top of the core pub/sub engine using the pub/sub engine itself (JetStream streams are internal NATS subscriptions; Raft log replication happens on $NRG.* subjects).

Temporal is not a messaging system in any conventional sense. It is a durable execution platform. The “messages” are workflow events (WorkflowExecutionStarted, ActivityScheduled, WorkflowTaskCompleted) that form an append-only, ordered history log. State is derived from this log by deterministic replay — a Temporal worker does not receive a message and do something; it replays the entire history of a workflow to reconstruct current state, then executes forward from the latest event. This model eliminates distributed transactions and makes recovery automatic. Workers poll task queues for work, pulling the next workflow task; they never receive a raw event stream.

Dapr is a messaging proxy. The daprd sidecar sits beside the application and accepts POST /v1.0/publish/{topic} from the app, then delegates to whichever broker component is configured (Redis Streams, Kafka, Azure Service Bus, etc.). Inbound messages from the broker are delivered to the app via an HTTP callback (POST /dapr/subscribe). Dapr owns neither the message store nor the routing logic — it normalizes the interface so the broker becomes a deployment detail. The pkg/api/universal/ package enforces this: pub/sub API semantics are implemented once and shared by the HTTP and gRPC transport layers.

Narrative: These paradigms exist at different levels of abstraction. NATS is level 0 — the raw network substrate. Dapr is level 1 — a portable API over any level-0 system. Temporal is a different dimension entirely — it uses messaging as a mechanism for building durable state machines, not for generic message routing.

Delivery guarantees#

NATS makes a deliberate split: the core server is at-most-once, optimized for speed (no ACK round-trips, no persistence). JetStream adds at-least-once on top via consumer state: each consumer tracks its last-acknowledged sequence number, and unacknowledged messages are redelivered after the AckWait timeout. The consumer state lives in the same filestore/memstore as the stream data, backed by the same Raft consensus for clustered streams.

Temporal achieves something closer to exactly-once by making each workflow execution identified by a unique workflowId. The Frontend rejects duplicate StartWorkflowExecution calls for an already-running workflow. Within an execution, each event is numbered sequentially in the history log and persisted atomically alongside any resulting tasks. If the History service crashes after persisting event N but before processing the transfer task, the queue processor re-reads from the database and retries — idempotently, because the event is already in the log. Workers that replay history are inherently idempotent: replaying the same events produces the same state.

Dapr’s delivery guarantee is bounded by the underlying broker. The pkg/resiliency package wraps every component call with configurable retry and circuit breaker policies, so Dapr will retry failed deliveries. But Dapr cannot guarantee exactly-once across the broker and the application — the app must implement idempotency for duplicate deliveries. This is an explicit design choice: Dapr is a proxy, not a coordinator.

Message routing and dispatch#

NATS’s routing is its crown jewel. The Sublist type in server/sublist.go is a trie over dot-separated subject tokens that supports two wildcard types: * matches any single token, > matches any suffix. A bounded LRU match cache (slCacheMax = 1024) short-circuits trie traversal for high-frequency subjects. The result is sub-microsecond subject matching that scales with subscriber count, not subject cardinality. Each Account has its own Sublist, providing tenant isolation at zero routing overhead.

Temporal’s routing is two-level. First, a workflow is pinned to a History shard by hash(workflowId) % numShards. The shard count is fixed at cluster creation (typically 512–2048), and Ringpop membership gossip tracks which History node owns each shard. Second, workflow and activity tasks are routed to workers via named task queues, which the Matching service manages. Task queues are further partitioned for throughput; a tree-based forwarding protocol matches pollers with tasks across partitions without requiring a central coordinator.

Dapr delegates routing entirely to the broker. The pkg/api/universal layer maps app subscriptions (declared via POST /dapr/subscribe) to broker subscriptions. The pkg/runtime/processor/subscriber package manages the subscription lifecycle, applying resiliency policies and routing messages to the correct app callback endpoint. Dapr adds a CloudEvents envelope with routing metadata, but the underlying fan-out logic is the broker’s responsibility.

Narrative: The routing designs reflect each project’s philosophy. NATS owns routing as a core capability and optimizes it ruthlessly. Temporal routing is really shard ownership — it is a consistency mechanism, not a fan-out mechanism. Dapr outsources routing to the broker and focuses on normalizing the result.

Message persistence and durability#

NATS Core is explicitly ephemeral — messages not delivered immediately are dropped. JetStream adds durable storage as a first-class optional layer: streams are backed by a file-based WAL (filestore.go) or an in-memory circular buffer (memstore.go), both implementing the StreamStore interface. Replicated streams use NATS’s custom Raft implementation (NRG) to achieve quorum writes before acknowledging producers. The architectural recursion is notable: Raft log replication happens over NATS pub/sub itself.

Temporal’s persistence is non-optional and foundational. Every state transition is an event appended to the history log. The common/persistence layer provides a multi-backend abstraction (ExecutionManager, ShardManager) with implementations for Cassandra, PostgreSQL, MySQL, and SQLite. Durability is guaranteed by atomic persistence: history events, mutable state, and transfer tasks are written in a single transaction. Recovery is automatic — replay the history to reconstruct state.

Dapr’s persistence is entirely delegated. The daprd sidecar holds no message state. Durability, replication, and retention are controlled by the underlying broker’s configuration. Dapr’s pub/sub outbox pattern (pkg/runtime/pubsub/outbox.go) adds optional transactional outbox support, writing messages to a state store before publishing — bridging the gap between state mutations and message delivery — but even this is opt-in.

Concurrency architecture for message processing#

NATS uses the simplest and most direct concurrency model: one readLoop goroutine and one writeLoop goroutine per connection. All 10K connections means 20K goroutines, but each is lightweight and blocked on I/O. The write path uses a sync.Cond-based producer-consumer queue rather than channels — the c.out.pb pending-bytes counter serves as the backpressure signal. High-watermark breaches trigger slow consumer detection and optional client disconnect. The startGoRoutine wrapper registry ensures all goroutines are tracked and drained cleanly on shutdown.

Temporal’s concurrency is more sophisticated than either of its peers. The common/tasks package is a standalone task scheduling mini-framework with seven composable implementations: FIFOScheduler, SequentialScheduler (ordered per key), GroupByScheduler[K, T] (generic fan-out), ExecutionQueueScheduler (priority-aware), DynamicWorkerPoolScheduler (auto-scaling goroutine pool), InterleavedWeightedRoundRobin (multi-tenant fairness), and RateLimitedScheduler. These are composed — a rate-limited, weighted round-robin scheduler over per-namespace sequential schedulers is a one-liner. The goro.AdaptivePool dynamically scales goroutine count based on measured task dispatch delay rather than queue depth alone. The common/quotas package provides 6+ rate limiter implementations for every layer of the stack.

Dapr’s concurrency model centers on the concurrency.RunnerCloserManager from the dapr/kit library. Every long-running subsystem (runtime, HTTP server, gRPC servers, component processor) is registered as a func(ctx context.Context) error runner. The manager propagates context cancellation, collects errors, and executes closers in reverse registration order. This is structured concurrency at the service level. For individual message delivery, Dapr uses go func — acceptable because delivery goroutines are bounded by errgroup or buffered channels. Backpressure is handled by the pkg/resiliency package’s circuit breaker: if the app is overloaded, the breaker opens and stops new deliveries until the app recovers.

Narrative: NATS optimizes for raw throughput — the simplest concurrency that avoids lock contention on the hot path. Temporal optimizes for fairness and tunability — multi-tenant workload isolation at every layer. Dapr optimizes for operational safety — structured concurrency prevents leaks, resiliency policies prevent cascading failures.

Multi-tenancy and isolation#

NATS’s Account model is the most sophisticated isolation mechanism in this group. Each Account has a fully independent Sublist — there is no subject overlap between accounts. Cross-account communication requires explicit export/import declarations in the server configuration. Accounts can be managed without server restart via JWT-signed AccountClaim tokens. The $SYS system account carries server-level advisory events but is isolated from user accounts by design.

Temporal’s Namespace isolation is primarily a routing and rate-limiting boundary. Workflow executions in different namespaces cannot interact directly. The InterleavedWeightedRoundRobin scheduler in common/tasks ensures each namespace gets a fair share of History shard processing capacity — preventing a busy namespace from starving others. Dynamic config keys are scoped at three levels: global, namespace, and task-queue, providing fine-grained tuning per tenant.

Dapr has no native multi-tenancy construct. App ID is a routing label, not an isolation boundary — different app IDs can share the same broker topic if they subscribe to it. Multi-tenancy in Dapr is an infrastructure concern (separate Kubernetes namespaces, separate Dapr deployments) rather than a built-in capability.

Protocol architecture#

NATS runs a custom binary protocol with a handwritten parser (server/parser.go). The parser is specifically designed to avoid allocation on the hot path: it operates on a fixed read buffer and dispatches via direct function calls into the client state machine. NATS also bridges MQTT 3.1.1 (server/mqtt.go, ~6000 lines), mapping MQTT topics to NATS subjects and QoS 1 persistence to JetStream. WebSocket support upgrades connections to NATS protocol over the WebSocket framing. All connection types converge on the same *client struct with a kind discriminant.

Temporal uses gRPC/protobuf exclusively for all interfaces: the public API (workflowservice), internal service communication (historyservice, matchingservice), and admin operations. This is standard and consistent. The proto files live in proto/ and the generated Go code in api/. The REST gateway is Temporal’s own HTTP API that translates to gRPC.

Dapr embraces dual-protocol as a design principle. Both HTTP (FastHTTP via go-fasthttp) and gRPC serve the same API semantics via pkg/api/universal/Universal. The UniversalHTTPHandler[T, U proto.Message] generic adapter (in pkg/api/http/universal.go) bridges between HTTP request bodies and protobuf types, eliminating per-endpoint boilerplate. gRPC servers use grpc.ChainUnaryInterceptor for composable middleware (OTel tracing, metrics, auth).

Message observability#

NATS’s observability is particularly distinctive: advisory events are published as NATS messages on well-known subjects ($JS.EVENT.ADVISORY.*, $SYS.SERVER.CONNECT, etc.). Any tool subscribed to these subjects gets real-time server events without polling. The server also exposes rich HTTP monitoring endpoints at a separate port (/varz, /connz, /jsz, /leafz, /routez) that return JSON snapshots of server internals. The publishAdvisory function is a textbook observer pattern implemented over the pub/sub system itself.

Temporal’s observability is metric-heavy. The 1,553 rate-limiter references and the common/quotas package reflect a system designed to be tuned in production. Dynamic config keys for metric export, sampling rates, and shard processing parameters can be changed without restart. The Admin gRPC API provides deep introspection into workflow history, shard state, and replication status.

Dapr ships complete Grafana dashboards (grafana/) as part of the repository — a rare operational maturity signal. OTel context propagation ensures that traces span from the app through the sidecar to the backend component, giving end-to-end visibility across the language boundary.

Common patterns#

All three projects share:

1. Context propagation as the shutdown mechanism. NATS uses quit chan struct{}, Temporal uses channel.ShutdownOnce and fx lifecycle hooks, Dapr uses RunnerCloserManager — but all three propagate cancellation signals through context/channel hierarchies rather than global flags.

2. Interface abstraction for storage backends. NATS has StreamStore/ConsumerStore, Temporal has ExecutionManager/ShardManager, Dapr delegates to components-contrib interfaces. All three separate the storage contract from the implementation.

3. Rate limiting at multiple layers. None of the three trusts consumers to be well-behaved. NATS disconnects slow consumers; Temporal has 1,553 rate-limiter references; Dapr wraps every component call with configurable retry/circuit-breaker policies.

4. Generics for type-safe framework primitives. NATS uses SubjectTree[T any] and GenericSublist[T comparable] for data structures. Temporal uses GroupByScheduler[K, T] and goro.KeyedSet[K comparable] for scheduling. Dapr uses UniversalHTTPHandler[T, U proto.Message] and Runner[T any] for policy application. In all three, generics appear in framework-level code where type safety across many callers pays off — not in domain logic.

5. No global state for component wiring. All three assemble their subsystems via explicit constructor wiring — whether manual structs (NATS, Dapr) or fx (Temporal). None relies on init() global registries for core subsystem wiring (though Dapr uses init() for component self-registration, which is different).

Divergent choices#

Message durability philosophy#

The sharpest divergence is in how each project thinks about message persistence. NATS treats durability as optional — the core is explicitly ephemeral, and JetStream is additive. Temporal treats durability as non-negotiable — there is no mode in which workflow events are not persisted. Dapr treats durability as someone else’s problem — it delegates completely to the broker.

This is not a quality difference; it reflects genuine use case differences. For IoT telemetry where losing a sensor reading is acceptable, NATS Core’s ephemeral delivery is correct and performant. For a payment workflow that must complete exactly once, Temporal’s non-optional persistence is the only viable approach. For a service mesh that needs to support both Redis and Kafka without code changes, Dapr’s broker agnosticism is the right call.

Concurrency model complexity#

NATS uses the simplest concurrency model: one goroutine pair per connection, direct method calls, no framework. Temporal uses the most sophisticated: a composable task scheduler hierarchy with seven implementations, an adaptive goroutine pool, and an entire goro package for goroutine lifecycle management. Dapr sits in the middle: RunnerCloserManager for structured long-lived goroutines, go func for delivery.

The complexity correlates with the problem. NATS’s throughput goal means minimal indirection — no scheduler overhead on the message dispatch path. Temporal’s multi-tenant fairness goal requires a real scheduling framework. Dapr’s operational safety goal is served by structured concurrency at the service level.

Protocol ownership#

NATS owns its protocol and has the engineering cost to show for it: a handwritten 6000-line client.go parser that handles NATS, MQTT, and WebSocket. The benefit is maximum control over allocation and dispatch on the critical path. Temporal standardizes on gRPC/protobuf throughout — trading control for ecosystem integration (all gRPC tooling works). Dapr runs both HTTP and gRPC simultaneously — trading simplicity for the widest possible language compatibility.

Dependency injection#

NATS uses manual wiring — a deliberate choice that eliminates framework overhead and keeps the package structure flat. Temporal uses go.uber.org/fx throughout — the largest real-world fx deployment in the 50-project set. Dapr uses manual Options-struct wiring despite its scale (~2300 Go files). The Temporal choice enables compile-time dependency graph verification; the NATS and Dapr choices keep the wiring code greppable and debuggable.

Recommendations for practitioners#

Choose NATS when: your primary requirement is throughput and latency in message fan-out. Sub-millisecond delivery to millions of subscribers, IoT at scale, internal microservice communication where at-most-once is acceptable or at-least-once is sufficient via JetStream. Its 10 direct dependencies and single-binary deployment make it the operationally simplest choice.

Choose Temporal when: your application has long-running, stateful workflows that must survive crashes, network partitions, and server restarts. If you are building order processing, approval flows, saga orchestrations, or any multi-step process where partial failure must be handled, Temporal’s event-sourced model eliminates the need for application-level retry logic. The operational cost is higher (four services, a database, Ringpop) but the correctness guarantees are unmatched.

Choose Dapr when: your team works across multiple languages, your broker choice may change over time, or you want to standardize distributed system patterns (pub/sub, state, actors) across microservices without coupling to a specific messaging library. The sidecar model adds latency (~1ms local IPC) but buys language independence and component portability. It is especially well-suited to platform engineering teams building internal developer platforms.

Combine them: NATS and Temporal can coexist — NATS for high-throughput event streaming, Temporal for durable orchestration of business workflows triggered by those events. Dapr can sit in front of either as the app-facing abstraction layer.

Book angle#

This comparison tells the story of three different levels at which messaging can be solved. NATS solves it at the infrastructure level: it is the network, the router, and the broker all at once. Temporal solves it at the application correctness level: messaging is the mechanism by which state transitions are made durable and recoverable. Dapr solves it at the portability level: messaging is an API contract that should survive infrastructure evolution.

The lesson for practitioners is that “messaging” is not a single problem. Before choosing a messaging system, answer three questions:

1. What is your delivery guarantee requirement? At-most-once (NATS Core), at-least-once (JetStream, Dapr + broker), effectively-once (Temporal)?
2. Is your state in the messages or separate from them? If your business logic state is derived from an ordered message history, you are building event sourcing and should look at Temporal’s architecture. If messages are signals and state lives elsewhere, NATS or Dapr are more appropriate.
3. Who owns the broker? If you want to own the full messaging stack (routing, persistence, consensus), NATS is the model. If you want to delegate broker ownership to your infrastructure team, Dapr is the model. If the broker is your database (the event log), Temporal is the model.

The Go code in all three projects reflects these answers with architectural clarity. NATS’s single-package, zero-abstraction-on-hot-paths design maximizes throughput by eliminating indirection. Temporal’s composable scheduler hierarchy and fx-wired component graph maximize correctness and tunability at the cost of complexity. Dapr’s universal.Universal shared handler layer and RunnerCloserManager structured concurrency maximize operational safety and language portability. Each set of trade-offs is internally consistent — none is wrong; they solve different problems.