NATS Server — Testing#

Test metrics#

  • Test files: 127 (_test.go files, excluding vendor)
  • Total Go files: 251
  • Ratio (test files / source files): ~50% — exceptionally high for a project of this complexity
  • Test functions: 3,141 Test* functions; 313 Bench* functions; 3 Fuzz* functions
  • Test frameworks: Pure stdlib testing package only — no testify, gomock, ginkgo, or any third-party assertion library

Test organization#

Placement#

Tests are colocated with source in the same package (white-box / internal testing). The server package tests declare package server, giving direct access to unexported symbols. The external test/ directory provides a separate package (package test) for black-box protocol-level tests that interact over raw TCP.

Helper packages#

  • test/test.go (660 lines, package test): Classic protocol-level test scaffold for the original NATS wire protocol. Provides RunServer, RunServerWithConfig, sendFun/expectFun function types for sending raw protocol commands and matching regex responses over net.Conn. Functions like checkMsg, expectNothing, doConnect, setupRoute let tests interact at the byte level without a NATS client library. Used by the test files in test/ (cluster, gateway, leafnode black-box tests).
  • server/test_test.go (package server): Defines the custom require_* assertion family — require_True, require_False, require_NoError, require_NotNil[T], require_Contains, require_NoPanic, and more (all generic or variadic where appropriate). Also defines DefaultTestOptions, RunServer, RunRandClientPortServer. Every assertion integrates with the Antithesis SDK (see below).
  • server/jetstream_helpers_test.go (2,236 lines, package server): The most significant test infrastructure file in the codebase. Defines cluster and supercluster types that spin up multi-server in-process clusters with full JetStream, routing, and gateway topology. Provides dozens of helper methods: waitOnLeader, waitOnStreamLeader, waitOnConsumerLeader, waitOnPeerCount, createJetStreamCluster, createJetStreamSuperCluster, createMixedModeCluster, and so on. Its init() tunes Raft timing constants (heartbeat interval, election timeouts) to speed up leader elections in test scenarios. This file is the foundation for nearly all JetStream cluster tests.
  • internal/antithesis/: A thin wrapper around the Antithesis SDK. When built without the enable_antithesis_sdk tag, all assertion functions are NOOPs. When built with the tag (on the Antithesis platform), they fire fault-injection assertions. Used in test helpers to annotate timeout conditions and invariant violations so the platform can trigger targeted inputs.
  • internal/testhelper/: DummyLogger (a sync.Mutex-protected logger capture type) used in tests that verify log output.

Fixtures#

  • test/configs/: NATS .conf files, TLS certificates, NKey files, and JWT-related fixtures used by the black-box test suite. No testdata directories elsewhere — fixtures are config files and in-memory constructed strings.
  • No embedded fixtures (//go:embed) or generated test data. Config strings for JetStream cluster templates are defined as Go string literals in jetstream_helpers_test.go.

Test patterns#

Table-driven tests#

  • Prevalence: Occasional — 49 occurrences of table-driven patterns (tests := [], testCases, tt.Run, tc.name)
  • Style: Anonymous struct slice with t.Run subtests for parametric cases; most tests are single-scenario integration tests rather than table-driven unit tests
  • Example: server/subject_transform_test.go uses a struct slice with input/expected fields to test subject transforms across many combinations

Mocking approach#

  • Strategy: No mocking whatsoever. Tests use real in-process servers exclusively. Dependencies are not mocked — instead, a real NATS server is started with RunServer(opts) (which calls NewServer and s.Start() in a goroutine) and torn down in defer s.Shutdown() or defer c.shutdown().
  • Example: A JetStream cluster test calls c := createJetStreamClusterExplicit(t, "R3S", 3) which starts three real servers bound to random ports, forms a cluster, waits for readiness, and returns the cluster handle. Tear-down is defer c.shutdown().
  • Consequence: All tests are integration tests by nature. There is no unit test isolation below the server level — no stub transports, no mock accounts. This is a deliberate philosophy: NATS’s correctness lies in its distributed behavior, which cannot be exercised with mocks.

Integration tests#

  • Present: Yes — effectively all server/ tests are integration tests (spin up real servers)
  • How: In-process servers bound to 127.0.0.1 with Port: -1 (OS-assigned random port). Cluster tests use Cluster.Port: -1 as well to avoid conflicts. No Docker, no testcontainers, no external processes — the full server runs in-process in the test binary.
  • Separation by build tag: Tests are partitioned into named subsets using Go build tags and function naming conventions, managed by scripts/runTestsOnTravis.sh:
    • TestJetStream* — non-clustered JetStream tests (tagged skip_js_cluster_tests, etc.)
    • TestJetStreamCluster* — split across 4 CI jobs using skip_js_cluster_tests_1/2/3/4 tags
    • TestJetStreamSuperCluster* — gateway-bridged multi-cluster
    • TestNRG* — Raft group tests (run in raft_tests job)
    • TestMQTT*, TestMsgTrace*, TestJWT* — dedicated CI jobs
    • TestNoRace* in norace_1_test.go and norace_2_test.go — tagged !race && !skip_no_race_[12]_tests — run without -race flag because they exercise timing-sensitive or throughput paths where the race detector overhead would cause false failures or extreme slowdowns
  • Race detection: CI conditionally applies -race based on branch: PRs and feature branches run with -race; main/release branches do not (presumably for throughput). The RACE env variable is set in the workflow and passed to go test.

Fuzz tests#

Three Fuzz* functions using Go 1.18+ native fuzzing:

  • FuzzParser (server/parser_fuzz_test.go): Exercises the NATS protocol parser with arbitrary byte sequences; the corpus includes valid NATS commands to guide mutation
  • FuzzServerTLS (server/server_fuzz_test.go): Fuzzes TLS handshake handling
  • FuzzSubjectsCollide (server/subject_fuzz_test.go): Fuzzes the SubjectsCollide function with pairs of subject strings, seeded from known collision/non-collision pairs

Benchmarks#

313 benchmark functions, concentrated in:

  • test/bench_test.go and test/fanout_test.go: Classic pub/sub throughput benchmarks (fanout to N subscribers, queue groups, request-reply latency)
  • server/mqtt_ex_bench_test.go: MQTT-specific throughput
  • server/store_test.go and related: FileStore and MemStore write/read throughput

CI architecture#

GitHub Actions (tests.yaml) runs ~15 parallel jobs, each with a 30-minute timeout:

  • store, js-no-cluster, raft, js-consumers, js-cluster-1/2/3/4, js-supercluster, no-race-1/2, mqtt, msgtrace, jwt, server-pkg-non-js, non-server-pkg
  • All test jobs depend on build-latest, build-supported, and lint
  • Separate nightly.yaml and long-tests.yaml workflows for extended runs
  • cov.yaml for coverage reporting

The naming convention (TestJetStreamCluster_, build tags for splitting) is a hand-crafted test-sharding scheme that predates native Go test sharding — a pragmatic solution for a 3,000-test suite where JetStream cluster tests dominate runtime.

Test quality observations#

What’s done well#

  • In-process cluster infrastructure (jetstream_helpers_test.go) is the standout achievement. The cluster and supercluster helpers allow writing distributed correctness tests (leader election, stream replication, snapshot/restore) in ~10 lines of test code. This infrastructure is arguably as complex as many standalone projects.
  • checkFor polling helper (server/server_test.go:57) — a universal retry loop checkFor(t, totalWait, sleepDur, func() error) — eliminates flaky time.Sleep calls. Distributed tests poll state with this pattern rather than asserting synchronously.
  • No third-party test libraries — pure stdlib means no dependency churn and predictable behavior. The hand-rolled require_* family (with generics for require_NotNil[T]) covers all needed assertions without importing testify.
  • Antithesis integration is unusually sophisticated. Test assertions are wired to a fault-injection platform that can replay specific executions with targeted perturbations. The internal/antithesis package cleanly decouples this from normal test runs (NOOP by default).
  • Fuzz test corpus quality: FuzzSubjectsCollide seeds with semantically meaningful cases (known collision pairs) rather than random bytes, dramatically improving fuzzer efficiency.
  • !race partitioning is explicit and documented: norace_1_test.go/norace_2_test.go carry build tag comments explaining why -race is excluded. Tests are not silently skipping race detection.
  • Raft timing tuning in init() (jetstream_helpers_test.go:53-63): Reducing hbInterval from production values to 50ms and capping election timeouts dramatically speeds up Raft leader election in tests without changing production behavior.

What could improve#

  • t.Parallel() adoption is minimal (22 uses). Given that each test starts servers on random ports, many tests could safely parallelize. The current approach relies on CI job sharding instead of in-process parallelism, which is coarser.
  • Table-driven tests are underused in the server/ package. Many tests that exercise parametric behavior (e.g., config option combinations) repeat boilerplate start/stop sequences rather than using subtests. The test/ package is better here.
  • No t.Cleanup for server shutdown — tests use defer s.Shutdown() or defer c.shutdown(). While functionally equivalent, t.Cleanup would run cleanup before subtests exit and would be more consistent with modern Go test style.
  • jetstream_helpers_test.go is a single 2,236-line file with no internal organization. As a shared dependency for ~70 test files, it has become a catch-all. Splitting by topology type (single-cluster, super-cluster, leaf-node, gateway proxy) would improve navigability.

Patterns worth emulating#

  • The checkFor polling idiom is a generalizable pattern for any asynchronous or eventually-consistent system test. It keeps tests from sleeping fixed durations and makes timeout failures self-describing.
  • In-process cluster construction as a test strategy — starting real servers rather than mocking — eliminates an entire class of mock-drift bugs where the mock’s behavior diverges from the real implementation over time. For infrastructure software with complex distributed protocols, this is often the right trade-off despite higher setup cost.
  • Build-tag test sharding by naming convention (prefixes + tags) is a workable pattern for large test suites before native sharding is available. The explicit script (runTestsOnTravis.sh) documents the partition logic clearly.
  • Antithesis-aware assertions — wrapping test assertions with platform-specific hooks (that are NOOPs in normal runs) allows plugging a distributed fault-injection platform in later without modifying test logic. This is a forward-compatible pattern for mission-critical systems.
Backward NATS Server — API Surface Temporal — Overview Forward