NATS Server — Architecture#

Architectural style#

Event-driven Network Server with embedded distributed systems subsystems

NATS Server is a purpose-built, high-performance network server organized around a single dominant package (server/). It does not follow clean architecture, hexagonal, or layered patterns. Instead, it embeds all subsystems — message routing, persistence, consensus, multi-protocol support, multi-tenancy, and TLS/auth — directly inside the server package to eliminate cross-package call overhead on hot paths.

The architecture is best described as a single-actor server (one Server struct), where:

  • Core pub-sub is synchronous and lock-free on the read path (trie + cache)
  • Persistence (JetStream) layers on top of pub-sub via internal subscriptions
  • Clustering (routes, gateways, leaf nodes) is handled by client connections of a different kind
  • Consensus (Raft) uses the NATS pub-sub substrate itself for log replication

This design is justified by NATS’s performance requirements: sub-millisecond latency and millions of messages per second demand that message dispatch avoid interface dispatch, heap allocations, and lock contention wherever possible.

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────────────────┐
│                              nats-server binary                             │
│                                                                             │
│   main.go                                                                   │
│   ├── ConfigureOptions()  ← conf/ (lexer/parser) + stdlib flag              │
│   ├── NewServer()                                                           │
│   └── Run() → Start() → AcceptLoop()                                       │
│                                                                             │
│   server.Server  (central orchestrator)                                     │
│   ├── Listener(s): clients, routes, gateways, leafnodes, WS, MQTT          │
│   ├── accounts: sync.Map[name → *Account]                                  │
│   │   └── Account                                                           │
│   │       ├── sl: *Sublist   (per-account subject routing trie)            │
│   │       └── js: *jsAccount (JetStream state per account)                 │
│   ├── clients:  map[cid → *client]  (connected clients)                    │
│   ├── routes:   map[host → []*client]  (cluster peer connections)          │
│   ├── leafs:    map[cid → *client]  (leaf node connections)                │
│   ├── gateway:  *srvGateway  (super-cluster gateway state)                 │
│   ├── js:       atomic.Pointer[*jetStream]                                 │
│   │   ├── accounts: map[name → *jsAccount]                                 │
│   │   │   └── streams: map[name → *stream]                                 │
│   │   │       └── consumers: []*consumer                                   │
│   │   └── cluster: *jetStreamCluster  (Raft-coordinated metadata)          │
│   │       └── meta: RaftNode  (meta-leader election)                       │
│   └── sys: *internal  (system account event bus)                           │
│                                                                             │
│   client (unified connection state machine)                                 │
│   ├── kind: CLIENT | ROUTER | GATEWAY | LEAF | MQTT | WS                  │
│   ├── acc:  *Account  (tenant association)                                 │
│   ├── nc:   net.Conn  (underlying TCP/TLS connection)                      │
│   ├── subs: map[sid → *subscription]                                       │
│   └── type-specific fields: route, gw, leaf, ws, mqtt                      │
│                                                                             │
│   Sublist  (subscription routing — per account)                             │
│   ├── nodes: trie of subject tokens                                         │
│   ├── cache: [1024]entry  (LRU match cache for hot subjects)               │
│   └── stree: Adaptive Radix Tree (server/stree) for subject-space queries  │
│                                                                             │
│   JetStream storage layer                                                   │
│   ├── StreamStore interface                                                 │
│   │   ├── filestore.go  (file-backed, WAL + per-message index)             │
│   │   └── memstore.go   (in-memory circular buffer)                        │
│   └── ConsumerStore interface                                               │
│       ├── filestore (consumer state on disk)                                │
│       └── memstore  (consumer state in memory)                              │
│                                                                             │
│   NATS Raft Group (NRG)  — custom Raft using NATS pub-sub as transport     │
│   ├── RaftNode interface  (raft.go)                                         │
│   ├── raft struct  (implements RaftNode)                                    │
│   └── WAL interface  → filestore / memstore for log persistence            │
└─────────────────────────────────────────────────────────────────────────────┘

Support packages (thin, no domain logic):
  conf/    → .conf file lexer/parser
  logger/  → file, syslog, Windows event log backends
  internal/ocsp, ldap, fastrand, antithesis
  server/stree, avl, gsl, ats, thw, pse, sysmem, tpm

Core components#

Server#

  • Package: server (server/server.go, ~5000 lines)
  • Responsibility: Central lifecycle manager. Owns all listeners, account registry, client map, JetStream reference, and the system account event bus. Coordinates startup and shutdown ordering.
  • Key types: Server struct (350+ fields), Info (JSON sent to connecting clients/peers), Run()
  • Dependencies: Directly holds references to every other major subsystem; there is no DI framework — wiring is manual via pointer fields.

client — Unified Connection State Machine#

  • Package: server (server/client.go, ~6800 lines)
  • Responsibility: Represents every connection type — NATS client, cluster route, super-cluster gateway, leaf node, MQTT session, WebSocket session — through a single struct. The kind field (an int constant) distinguishes connection types. Type-specific state is stored in embedded structs (route *route, gw *gateway, leaf *leaf, ws *websocket, mqtt *mqtt).
  • Key types: client struct, readCache (parse buffer), outbound (write queue), subscription
  • Dependencies: *Server, *Account, net.Conn, *permissions

Sublist — Subject Routing Trie#

  • Package: server (server/sublist.go)
  • Responsibility: Matches published subjects against all registered subscriptions, handling * (single-token) and > (multi-token) wildcards. The core routing engine. Uses a two-level trie of maps per token, with a bounded LRU match cache (slCacheMax = 1024) for high-frequency subjects.
  • Key types: Sublist, SublistResult (psubs + qsubs), subscription
  • Dependencies: Pure stdlib; no external deps. The server/stree Adaptive Radix Tree is used separately for subject-space enumeration, not for hot-path matching.

Account — Multi-Tenant Namespace#

  • Package: server (server/accounts.go, ~4800 lines)
  • Responsibility: Provides tenant isolation. Each account has its own Sublist, JetStream limits, import/export mappings (cross-account subject bridging), and auth policy. The $G (global) account is the default; the $SYS (system) account carries internal advisory events.
  • Key types: Account, AccountResolver interface, MemAccResolver, DirAccResolver, URLAccResolver
  • Dependencies: Sublist, *Server, JWT/NKey libs

JetStream — Persistence and Streaming Layer#

  • Package: server (server/jetstream.go, stream.go, consumer.go, filestore.go, memstore.go)
  • Responsibility: Adds durable message storage and replay to the ephemeral pub-sub core. Streams capture published subjects into an ordered log. Consumers track per-subscriber progress and delivery state. Implemented as internal NATS subscriptions — JetStream is NATS talking to itself.
  • Key types: jetStream, jsAccount, stream, consumer, StreamStore interface, ConsumerStore interface
  • Dependencies: RaftNode (for clustered replication), filestore / memstore (storage backends)

NATS Raft Group (NRG) — Custom Consensus#

  • Package: server (server/raft.go, ~5100 lines; server/jetstream_cluster.go, ~10900 lines)
  • Responsibility: Provides Raft-based consensus for JetStream metadata (stream/consumer definitions) and stream replica state. Uniquely, the Raft log transport uses NATS pub-sub itself ($NRG.* subjects) rather than raw TCP — simplifying the networking layer and reusing auth/TLS.
  • Key types: RaftNode interface, raft struct, WAL interface (log storage), jetStreamCluster
  • Dependencies: WAL → filestore or memstore; sends via NATS internal pub-sub

Multi-Protocol Bridges (MQTT, WebSocket)#

  • Package: server (server/mqtt.go, ~6000 lines; server/websocket.go)
  • Responsibility: Expose NATS messaging through alternative wire protocols. MQTT clients are mapped to NATS accounts/subjects. WebSocket clients use the standard NATS protocol over a WebSocket upgrade. Both are represented as client instances with kind=MQTT or kind=WS.
  • Key types: srvMQTT, srvWebsocket, per-connection mqtt / websocket structs embedded in client
  • Dependencies: *Server, *Account; MQTT uses JetStream for QoS 1 persistence

Options / Configuration#

  • Package: server (server/opts.go, ~6500 lines)
  • Responsibility: Defines the entire server configuration surface. ConfigureOptions() merges CLI flags and config file. Options is a flat struct with ~200 fields covering TLS, auth, clustering, JetStream, MQTT, monitoring, etc.
  • Key types: Options, ClusterOpts, LeafNodeOpts, JetStreamConfig
  • Dependencies: conf/ package for .conf file parsing; stdlib flag

Data flow#

Core pub-sub (CLIENT → CLIENT)#

TCP accept → goroutine per client → client.readLoop()
  → bufio.Reader.Read() → client.parse()  [server/parser.go]
  → processMsgResults(sub, hdr, msg)
  → acc.sl.Match(subject)  [Sublist.Match — trie lookup + cache]
  → for each matching subscription:
      if local:  client.addMsgToSend() → out.pb queue → sendLoop goroutine → net.Conn.Write()
      if remote route: route client.queueOutbound()
      if gateway:      gateway forwarding logic

The critical path is: parse → Sublist.Match → enqueue to subscriber outbound queue. No heap allocation occurs in this path for cache-hit subjects.

JetStream publish (stream capture)#

client publishes to subject S
  → Sublist.Match finds internal subscription owned by *stream
  → stream.processInboundMsg()
  → filestore/memstore.StoreMsg()  [write to WAL / index]
  → if replicated (R>1): RaftNode.Propose(entry)
      → NRG: serialize as NATS pub to $NRG.<group>.<seq>
      → peer servers' raft goroutines receive, apply to their WALs
      → on quorum ack: committed entry applied to stream state
  → deliver to active consumers (push) or wait for pull request

JetStream consumer pull#

client sends $JS.API.CONSUMER.MSG.NEXT.stream.consumer request
  → JetStream API handler (jetstream_api.go)
  → consumer.getNextMsg()
  → filestore.LoadMsg(seq)  or memstore.LoadMsg(seq)
  → reply to client's reply subject

Initialization / Bootstrap#

main()
  1. flag.FlagSet setup
  2. server.ConfigureOptions(fs, args)     — parse flags + load .conf file
  3. server.NewServer(opts)                — allocate Server struct
      a. generate server ID (nuid)
      b. init accounts (global + system)
      c. load TLS config
      d. set up NKey/JWT trust chain
      e. init sublist, gateway, leaf node state
  4. s.ConfigureLogger()                   — attach file/syslog/stderr logger
  5. server.Run(s) → s.Start()
      a. startRateLimitLogExpiration
      b. SetSystemAccount / SetDefaultSystemAccount  — start event bus
      c. StartMonitoring()                 — HTTP /varz /connz etc.
      d. AccountResolver.Start()           — optional: URL/Dir resolver
      e. EnableJetStream()                 — init JS engine, recover streams
          i.  scan store dir for existing stream data
          ii. if clustered: startJetStreamClustering()
              → elect meta-leader via NRG Raft
      f. startOCSPMonitoring
      g. startGateways (if configured)
      h. startWebsocketServer (if configured)
      i. startLeafNodeAcceptLoop (if configured)
      j. solicitLeafNodeRemotes (if configured)
      k. startMQTT (if configured)
      l. StartRouting (if cluster configured)  — goroutine
      m. close(startupComplete)
      n. AcceptLoop(clientListenReady)     — blocks, goroutine per accepted conn
  6. s.WaitForShutdown()                  — blocks on shutdownComplete channel

No dependency injection framework is used. All wiring is manual: NewServer builds the Server struct with direct pointer assignments. Subsystem initialization happens sequentially inside Start().

Configuration#

  • Primary mechanism: Custom .conf file format, parsed by conf/ (standalone lexer + parser producing a generic map). Loaded by server.ConfigureOptions() which merges into the flat Options struct.
  • CLI flags: stdlib flag package; flags are a subset of what the config file supports.
  • Operator/account config: Decentralized via JWT tokens signed by NKeys (NaCl keypairs). Accounts, users, and permissions are encoded in JWTs; the server validates them against trusted operator public keys. This allows account management without server restarts.
  • Hot reload: SIGHUP triggers server.Reload() (server/reload.go). The reload compares old and new options, applies changes that are safe to apply live (e.g., log settings, TLS certs, account permissions), and restarts listeners when the bind address changes. Config reload is guarded by reloadMu (a separate sync.RWMutex from the main mu).
  • Viper: Not used. Configuration is entirely bespoke.

Key design decisions#

  1. Single-package server with no internal interfaces on hot paths. The server package (~180 files) owns all domain logic. There are no interface abstractions between the connection state machine (client), the routing engine (Sublist), and the account model — these call each other via direct struct method calls. This eliminates virtual dispatch overhead on the ~100ns message dispatch path. The cost is a package that is very large by Go conventions.

  2. client as a unified, polymorphic connection struct. Rather than having separate types for NatsClient, RouteConn, GatewayConn, etc., every connection is a *client with a kind field. Type-specific behavior is handled by if c.kind == ... checks and type-specific embedded struct fields (route, gw, leaf, ws, mqtt). This means one connection management loop, one metrics system, one auth system, and one send-queue implementation for all connection kinds — at the cost of a very large struct with many conditional code paths.

  3. JetStream built on top of NATS itself. JetStream streams are implemented as internal NATS subscriptions. The JetStream API is served via $JS.API.* subject handlers. Raft log replication uses $NRG.* subjects. This architectural recursion means JetStream gets auth, TLS, and account isolation “for free” — and that adding JetStream does not require a separate network stack. The trade-off is that JetStream performance is inherently bounded by the core pub-sub engine.

  4. Custom Raft consensus, not etcd/raft. NATS implements its own Raft (NRG — NATS Raft Group) rather than using etcd/raft or a similar library. The custom implementation is tightly integrated with NATS subjects and the account system. This required building from scratch (~5100 lines) but avoids an external dependency and allows Raft traffic to flow over existing NATS connections with the same TLS/auth guarantees.

  5. StreamStore / ConsumerStore interface abstraction for JetStream storage. The two storage backends (filestore, memstore) implement well-defined interfaces (store.go). This is one of the few places where the codebase uses interface-based polymorphism for behavioral substitution — it enables memory-only streams (for ephemeral use cases) and disk-backed streams (for durability) to be selected at stream creation time without server restart.

Backward NATS Server — Dependencies NATS Server — Patterns Forward