Usage: Every accepted connection — NATS client, cluster route, leaf node, MQTT, WebSocket — spawns exactly two goroutines: one for reading (readLoop) and one for writing (writeLoop / sendLoop). This is the foundational concurrency unit in the server.
Example:server/client.go:1369 (readLoop), server/client.go:1278 (writeLoop). Same pair invoked for MQTT at server/mqtt.go:647-648 and leaf nodes at server/leafnode.go:1384-1389.
Assessment: Classic and idiomatic for network servers. The separation of read and write into independent goroutines avoids head-of-line blocking: a slow writer does not stall parsing, and a slow parser does not stall outbound delivery. The write goroutine blocks on c.out.pb (outbound buffer), draining it via net.Conn.Write. Effective and well-understood.
Usage: All server-managed goroutines are started via s.startGoRoutine(f func(), tags ...pprofLabels) rather than raw go. The wrapper gates on s.grRunning (prevents new goroutines after shutdown begins) and calls s.grWG.Add(1) before launching. The function f is responsible for calling s.grWG.Done() when it exits.
Example:server/server.go:4071-4084. Called 30+ times across server.go, mqtt.go, leafnode.go, raft.go, accounts.go.
Assessment: Elegant lifecycle management. The grWG WaitGroup allows WaitForShutdown() to block until all tracked goroutines exit. The grRunning flag prevents goroutines from being started during or after shutdown — avoiding a common race. The optional pprofLabels (used by Raft nodes) adds goroutine-level labels visible in go tool pprof.
Usage: Long-running goroutines receive a quit chan struct{} or closeCh chan struct{} that is closed on shutdown. select {} statements (675 occurrences) multiplex over quit channels, timers, and work channels. The Raft node has its own n.quit chan struct{} (server/raft.go:241). MQTT session managers use a server-level s.quitCh that is threaded down through session creation (server/mqtt.go:1172-1196).
Assessment: Idiomatic Go. Closing a channel broadcasts to all receivers simultaneously, making this pattern superior to sending a signal value when multiple goroutines need to observe the same shutdown event.
Usage: 31 sync.Pool instances are defined, concentrated in server/raft.go. Separate pools exist for CommittedEntry, Entry, appendEntry, appendEntryResponse, and other Raft message types. Get/Put calls wrap every allocation on the Raft hot path.
Assessment: Correct and effective. Raft replication is latency-sensitive; avoiding GC pressure on message structs is important. Pools are scoped to specific types (not a generic byte-buffer pool), which is the right granularity. The pattern requires careful Put discipline — callers must zero fields before returning to the pool.
Assessment: Appropriate for hot-path state flags that do not require lock-level ordering guarantees. The mix of old and new atomic styles is a sign of organic growth; the newer typed fields are cleaner and less error-prone.
Usage: A custom rate-limiter for warning logs prevents log flooding. It uses s.rateLimitLogging sync.Map (key = log message format string, value = last-logged timestamp) plus a s.rateLimitLoggingCh chan time.Duration that feeds a dedicated goroutine (s.logRateLimitLogger). The goroutine periodically sweeps rateLimitLogging and emits suppressed messages.
Example:server/server.go:356-357, server/server.go:4717-4737. Used via s.rateLimitFormatWarnf(...) in jetstream_cluster.go.
Assessment: An unusual but well-justified pattern. JetStream clustering can produce bursts of identical warning messages during leader elections or snapshot generation. Rate-limiting at the log call site (using sync.Map rather than a per-message mutex) avoids thundering herd on the logger. The design is slightly opaque but effective.
Usage:Server.WaitForShutdown() blocks on s.shutdownComplete chan struct{}. Shutdown sequence: (1) close s.quitCh (signals all long-running goroutines), (2) call s.grWG.Wait() (blocks until all registered goroutines exit), (3) close s.shutdownComplete (unblocks WaitForShutdown). Raft nodes also maintain their own wg sync.WaitGroup for sub-goroutines.
Assessment: Clean and correct. Two-phase (signal → wait → confirm) shutdown avoids resource leaks and ordering bugs. The use of s.grRunning to gate new goroutine starts during shutdown prevents races between in-flight requests and cleanup.
Usage: JetStream API handlers, Raft log replication, system account advisories, and MQTT bridging all communicate via NATS subjects on internal (in-process) subscriptions rather than direct function calls. $JS.API.* for JetStream API, $NRG.* for Raft consensus, $SYS.* for server events. Internal subscriptions are created via s.subscribeInternal / s.sys.client.
Example:server/server.go:1882 (s.internalSendLoop), server/jetstream_events.go:23 (publishAdvisory), server/raft.go:737 (Raft client registered with system account).
Assessment: This is NATS’s most distinctive architectural pattern. Using the message bus as an inter-subsystem RPC mechanism gives JetStream and Raft free access to account isolation, TLS, and auth without any additional wiring. The trade-off is indirection and slightly harder debugging — a function call becomes a message on a subject. It is architecturally elegant but requires understanding the subject namespace to trace data flow.
Style: Mixed — sentinel errors for connection/protocol-level conditions, structured ApiError for JetStream wire-protocol errors, and custom error types for config parsing. The mix is deliberate: each layer has its own error contract.
Error types defined:
errors.go: ~40 exported sentinel errors (ErrConnectionClosed, ErrAuthentication, ErrMaxPayload, ErrLeafNodeLoop, etc.) covering the core server protocol.
jetstream_errors.go: ApiError struct (Code + ErrCode + Description) for JetStream API responses; ErrorIdentifier uint16 enum; ApiErrors map of ~100 pre-defined API error instances; IsNatsErr() utility for matching by ErrCode.
errors.go (private): configErr, unknownConfigFieldErr, configWarningErr, processConfigErr with Error() methods for config file parsing errors.
stream.go: BatchFlowErr for JetStream batching flow control.
accounts.go: ClaimUpdateError for JWT claim update failures.
Wrapping approach:fmt.Errorf + errors.New (1108 combined occurrences). errors.Is / errors.As used for checking. No github.com/pkg/errors dependency. IsNatsErr() is the project-specific idiom for checking JetStream error codes.
Approach: Flat config struct (Options, ~200 fields in server/opts.go). No functional options for the main server. This is a deliberate choice: every field has a zero value meaning “disabled” or “use default”, and ConfigureOptions() does a multi-pass merge (defaults → flags → config file → env overrides).
Limited functional options:DirResOption func(s *DirAccResolver) error (server/accounts.go:4562) for account resolver optional parameters, and ErrorOption func(*errOpts) (server/jetstream_errors.go:12) for error construction. These appear in subsystems where the Options struct would be overkill.
Config reload:server.Reload() in server/reload.go. Hot-reloadable fields are implemented as typed reloaders — each config field that can change at runtime has a diffOpts() check and an apply() method. TLS certs, log levels, and account permissions support hot reload; bind addresses and cluster topology require restart.
Approach: Manual wiring. NewServer(opts *Options) builds the Server struct by direct field assignment. Subsystems receive *Server or individual fields as constructor arguments.
Evidence:server/server.go:717 (NewServer), which allocates Server and directly sets all fields. No framework (Wire, dig, fx) is used. The server package’s decision to colocate all subsystems eliminates most cross-package wiring — there are no interfaces between client, Sublist, and Account because they all live in the same package and call each other directly.
Assessment: Appropriate for a single-package server. The absence of DI indirection contributes to the performance profile — no interface dispatch, no registry lookups. The cost is that NewServer is a ~300-line function and Server is a 350+-field struct.
server/stree/stree.go:28: SubjectTree[T any] — generic Adaptive Radix Tree for subject-space storage; used for mapping and stream subscriptions.
server/gsl/gsl.go:57: GenericSublist[T comparable] — a generic version of the core subscription trie, parameterized for reuse across different value types.
server/jetstream_cluster.go:7766: sysRequest[T any] — generic helper that sends a system request and decodes the response into *T, reducing boilerplate for the ~20 JetStream API callers.
Assessment: Restrained, purposeful generics adoption. The data-structure packages (stree, gsl) benefit clearly from type parameterization — the same ART/trie logic can be reused for different value types without code duplication. The sysRequest[T] helper shows Go generics used as a simple type-safe wrapper, not abstract machinery. No generics in the core hot path.
ApiError implements error; configErr, processConfigErr, etc. implement error. Error wrapping via errors.As pattern for config error chains. Standard Go idiom, well applied.
All goroutines started via startGoRoutine can receive pprofLabels tags (a map[string]string). Raft nodes pass stream and consumer labels, enabling per-goroutine filtering in go tool pprof CPU profiles. This is a production observability pattern rarely seen in OSS Go servers.
Outbound message queue with credit-based flow control#
The client.out outbound buffer uses a pending-bytes count (out.pb) that callers increment when enqueueing messages. The writeLoop drains the buffer and decrements. A high-watermark (maxPending) triggers blocking behavior. This is a producer-consumer queue with backpressure, implemented without channels — using a sync.Mutex + condition variable (sync.Cond) for the writeLoop to sleep on when empty.
publishAdvisory(acc *Account, subject string, adv any) (server/jetstream_events.go:23) JSON-marshals advisory structs and publishes them to $JS.EVENT.* or $SYS.SERVER.* subjects on the system account. Observers (external tools, account-level subscriptions) can receive these without server-side callbacks. This is the observer pattern implemented over NATS pub-sub rather than Go channels or callback slices.
Assessment: Clean decoupling. The server does not maintain subscriber lists for advisory events — NATS routing handles fan-out. New event consumers can be added without touching server code.
JetStream consumers with bytes-per-second delivery limits use golang.org/x/time/rate.Limiter (server/consumer.go:437, 2298-2314). This is the one external rate-limiter use; the internal logging rate-limiter (see above) is bespoke.