Architecture and Testability: What Co-Occurs Across 51 Go Projects#

Summary#

Across 51 production Go projects, the architectural trait that most reliably co-occurs with high testability is T4: injecting time, I/O, and randomness as explicit parameters rather than making ambient calls. Projects scoring T4≥2 account for 74% of the high-hermetic-suite cases (U1≥2) compared to 60% in the T4=1 cohort, and six of the eight projects with U1=3 (fully hermetic suites) score T4≥2. Narrow consumer-side interfaces (T1≥2) appear in every one of the top-nine testability projects, suggesting a necessary-but-not-sufficient role. In contrast, two of the most-discussed architectural virtues — eliminating global state (T2) and using concrete internal collaborators (T6) — show almost no predictive power for testability in this corpus: the top testability projects (CockroachDB U=15, Go stdlib U=14, Prometheus U=13) all score T2=1, and CockroachDB (the clear testability leader) scores T6=1, meaning it uses extensive internal interface abstraction. The most striking finding is the outlier pair: Grafana and Temporal each score among the four highest architectural composites (T_arch ≥ 20) yet fall in the bottom third of testability scores, while Hugo achieves top-quintile testability (U=10) with a below-average architectural composite (T_arch=14) by investing in a single runtime decision — routing all filesystem access through the afero abstraction.

Methodology#

This synthesis cross-tabulates two independently produced blind scorecards. The arch-traits scorecard (X22a) scored each of the 51 projects on eight architectural properties (T1–T8) by reading only non-testing Phase 1 reports (overview, structure, dependencies, architecture, patterns, interfaces, api-surface). The testability scorecard (X22b) scored each project on six testability properties (U1–U6) by reading only the project’s --testing.md report. Neither scorer was permitted to read the other’s sources. The synthesis now cross-tabulates the two independent sets of scores.

Four risks bound every claim in this document.

Selection bias. All 51 projects are successful, maintained, widely-used open-source software. There are no abandoned projects, no notoriously untestable codebases, and no failed experiments in this set. The range of testability scores (4–15) and architectural scores (6–22) is real but truncated at the low end. Relationships that appear weak here might be stronger or weaker in a sample that includes genuinely poor codebases.

Confounding by scale and domain. Infrastructure-heavy distributed systems (Kubernetes, Istio, Argo-CD, K3s) cannot achieve hermetic test suites in the same sense that a library like viper or cobra can. A distributed system’s most valuable tests — the ones that find bugs — require real network protocols, real consensus behavior, real filesystem interactions. Comparing their U1 scores to a configuration library’s U1 score conflates architectural quality with domain difficulty. The synthesis distinguishes domain-constrained scores from architectural choices wherever evidence permits.

Reverse causation. The data cannot establish whether good architecture produces testable code or whether teams that care about testability also produce cleaner architecture. Hugo’s use of afero might reflect a team that valued testability first and built an architecture to serve that value, rather than an architecture that happened to enable testability. CockroachDB’s injectable effects (T4=3) may have been introduced specifically to enable the hermetic test suite (U1=2), not independently of it. Throughout, “co-occurs with” is the correct phrase; “causes” is not warranted.

Single-reader rubric. Both scorecards were produced by a single model reading phase-one analysis reports (which were themselves produced by the same model on different runs). Rubric calibration variance is estimated at ±1 point per trait for borderline cases. The confidence note at the end of this document identifies the cells most likely to shift under a different reader.

The Data at a Glance#

The table below summarizes co-occurrence patterns. For each architectural trait, it shows the average testability composite of the top-3 scorers on that trait vs. the bottom-3 scorers, and identifies which testability quartile those groups land in (Q1 = bottom 25%, Q4 = top 25%; the testability composite range is 4–15 across the corpus, Q4 threshold ≈ 10).

Several observations jump out immediately. T1 (narrow interfaces) produces the sharpest contrast: the nine projects scoring T1=3 average a testability composite of 10.1, while the two scoring T1=0 (beego and fzf) average 7.5 — but the difference is partly confounded by domain and size. T6 produces the flattest contrast: the seven projects scoring T6=3 average 7.9, virtually identical to the two scoring T6=0 (drone at 4, rclone at 10, averaging 7.0). T4’s signal is positive but noisier than expected: the five T4=3 projects average 8.4, the same as T2=3 and slightly below T4=2’s average of 8.9 — the outlier P09-cockroach (T4=3, U=15) anchors the T4=3 group, while P50-temporal (T4=3, U=6) drags it back toward the mean.

The most striking observation is that the overall correlation between the architectural composite (sum of T1–T8, range 6–22) and testability composite (sum of U1–U6, range 4–15) is weak. The five projects with the highest architectural composites (etcd 22, wireguard-go 21, nats-server 21, grafana 20, temporal 20) have testability composites of 11, 12, 7, 5, and 6 respectively — a range of 7 points, nearly the full corpus spread. You cannot predict a project’s testability from its architectural score.

Architectural Traits that Co-Occur with High Testability#

T4 — Injected Effects and Hermetic Suites#

The clearest co-occurrence in the data links T4 (time/IO/randomness parameterization) with U1 (hermetic-by-construction test suites). The mechanism is direct and legible: when a system routes all external effects through injected interfaces or function parameters, a test can substitute an in-process fake and eliminate external dependencies without rewriting business logic.

Hugo (P06) is the canonical demonstration. Its architectural composite (T_arch=14) is below the corpus median, and its T4 score (2) is not exceptional. But it uses the afero filesystem abstraction throughout: the same Hugo pipeline that writes to disk in production runs against afero.MemMapFs in tests. The result is U1=3 — the entire test suite, including full rendering integration tests, runs in-process with no Docker, no network, no real filesystem. A single architectural seam eliminated an entire category of test infrastructure.

Cobra (P31) provides a different angle. Its T4 score is 3: SetIn, SetOut, and SetErr inject all three standard I/O streams explicitly, and the parent-command fallback chain propagates them to every subcommand automatically. The result is U1=3: tests capture command output via Go buffers, exercise real command execution, and never spawn external processes. This is not a complex architecture — cobra is a small, flat library — but the T4 decision enables complete hermeticity.

WireGuard-go (P44) extends the pattern to the network layer. Its ChannelBind and ChannelTUN replace OS network interfaces with channel-backed in-memory implementations. TestTwoDevicePing runs a full encrypted packet flow between two virtual devices without opening a single OS socket (U1=3). The architecture’s T4 score (2) reflects that the //go:linkname fastrandn ambient random source wasn’t parameterized, but the I/O seams that matter for hermeticity — network and TUN device — were.

PocketBase (P48) applies the pattern to databases. Rather than mocking the ORM or abstracting the database, it runs every test against a real in-process SQLite instance initialized from a fixture snapshot in a temp directory. The “injection” here is via the test setup calling NewTestApp() rather than an interface injection, but the effect is the same: external infrastructure is replaced by an in-process equivalent.

The projects with the highest T4 scores that do not achieve high testability (Kubernetes T4=3, U=7; Temporal T4=3, U=6) reveal an important ceiling: T4 is necessary but not sufficient for top-tier testability. Kubernetes routes external effects through parameters but still requires a real cluster for its most important tests (E2E), and its internal mock infrastructure (U3=2) prevents it from reaching the highest tier. Temporal has exemplary injectable effects (common/clock package, injectable TimeSource) but deploys 126 generated mock files at internal seams (U3=0), which limits its testability score despite the architectural investment.

T1 — Narrow Consumer-Side Interfaces#

Every project in the top-nine testability cohort (U≥10) scores T1≥2. The co-occurrence is consistent but the causal mechanism is subtler than T4’s. Narrow interfaces do not themselves cause hermetic tests; rather, they enable the real-implementations-as-test-implementations pattern. When an interface has 1–3 methods, it is easy to provide a second real implementation — an in-memory fake, a channel-backed stub, a SQLite-backed alternative — without building a mock. When an interface has 30 methods, the cost of providing a real test implementation is prohibitive, so teams reach for generated mocks instead.

Restic (P38) makes this concrete. Its backend.Backend interface is narrow (7 methods covering the full I/O contract for a backup storage backend), and the internal/backend/mem implementation is a genuine in-memory backend used in tests. Tests run the real archiver and real cryptographic pipeline against the in-memory backend. U3=2: mocks are confined to external boundaries (cloud storage APIs), and internal logic uses real implementations. The narrow interface made a real fake affordable.

rclone (P40) scales this to 70+ backends. Its fs.Fs has 5 methods; optional capabilities are expressed as ~25 separate 1-method interfaces (checked via type assertion). Because the core interface is narrow, rclone can provide the local filesystem as the canonical test backend for every operation, and the generic fstest/fstestcase conformance suite (U4=3) runs every backend through the same test corpus. A single narrow interface enabled both hermetic testing and conformance enforcement at scale.

The two T1=0 projects — beego and fzf — are instructive contrasts. Beego’s wide interfaces reflect its framework-as-platform philosophy; testing requires real databases for meaningful coverage (U1=1). fzf’s T1=0 reflects the opposite: it uses no narrow interfaces at all (just two wide TUI backend types), but it achieves reasonable testability (U=7) through pure-function algorithm testing. fzf demonstrates that T1=0 is compatible with moderate testability when the most testable part of the system (fuzzy matching) is already pure.

T5 — Functional Core and Low Mock Burden#

The co-occurrence between T5 (pure-logic isolation) and U3 (low mock burden) is real but modest. The reasoning is similar to T1’s: when pure-logic packages exist that import no I/O, their tests are necessarily mock-free — there is nothing to mock. The tests just call the functions with inputs and assert on outputs.

The clearest examples are sqlc (P29, T5=3, U3=3) and viper (P30, T5=2, U3=3). sqlc’s parse→catalog→IR pipeline stages are pure transformations that produce outputs from SQL inputs with no side effects; the tests drive these stages with SQL files and golden-file assertions. Viper’s internal/encoding/* codec implementations are pure format-conversion packages tested without any I/O. In both cases, T5≥2 directly enabled U3=3.

The counter-examples are important. Grafana (P05, T5=3, U3=1) has excellent functional core separation — pkg/infra/ has no domain imports, pkg/services/ implements business logic — but testing still reaches into generated mocks for datasource and store interfaces because the service layer itself interfaces with external infrastructure. T5 being high doesn’t prevent the other layers from being mock-heavy. Temporal (P50, T5=3, U3=0) is the extreme case: common/tasks, common/goro, and common/backoff are pure packages, but the service layer surrounding them uses 126 generated mock files. Functional core isolation is a local property that can coexist with pervasive mocks elsewhere.

Architectural Traits that Do NOT Predict Testability#

T2 — Eliminating Globals Does Not Unlock Testability#

The most surprising finding in the cross-tabulation is T2’s near-zero predictive power. Projects with T2=1 (some global state or init() side effects) average a testability composite of 8.8 — higher than T2=2’s average of 7.7 and almost as high as T2=3’s 8.4. The T2=1 cohort includes the corpus’s best testability performers: Prometheus (T2=1, U=13), CockroachDB (T2=1, U=15), Go stdlib (T2=1, U=14), and PocketBase (T2=1, U=12).

The reason is structural: the init() patterns that lower T2 scores in these projects are plugin/service-discovery registration hooks that populate immutable registries before main() runs. Prometheus’s service discovery providers call discovery.RegisterConfig() in init(); this doesn’t affect test isolation because tests can still use real in-process storage and query engines. The global registry is write-once and doesn’t contaminate test state. Similarly, CockroachDB’s CCL feature hooks (T2=1) are enterprise build integration points that don’t affect the core testability properties the rubric measures.

The opposite finding — T2=3 projects that aren’t especially testable — includes Grafana (T2=3, U=5), air (T2=3, U=5), nats-server (T2=3, U=7), and crush (T2=3, U=9). Eliminating globals does not substitute for the test infrastructure investment that actually determines testability scores (exported test helpers, conformance suites, goroutine leak checks).

The practical implication is that architects who eliminate global state primarily for testability reasons may be misidentifying the causal path. Global-state elimination has real benefits for reasoning about code and preventing test contamination between parallel test runs — but in this corpus, those benefits did not translate into materially higher testability scores. The projects that score high on T2 because they genuinely have no process-wide mutable state (wireguard-go, air, nats-server) are architecturally clean, but their testability is determined more by U1/U3 investment than by the absence of globals.

T6 — Concrete Internal Collaborators: The Contrarian Trait#

T6 was designed as the contrarian hypothesis: the idea that using concrete internal collaborators (rather than interface-abstracting every dependency) correlates with testability because it forces teams to use real implementations rather than mocks. The data partially supports this logic but in an unexpected direction.

The five projects scoring T6=3 (wireguard-go, nats-server, air, fzf, cobra) do tend toward real-implementation testing. But the T6=1 projects — Kubernetes (U=7), CockroachDB (U=15), Tailscale (U=7), Grafana (U=5), Syncthing (U=5) — split widely. CockroachDB uses extensive internal interface abstraction (T6=1, 1,481 interface definitions at internal seams) yet achieves the highest testability composite in the corpus. Grafana also has T6=1 with heavy internal interfaces but terrible testability. The interface density inside these projects is the same; the outcome is wildly different.

The confound is that T6=1 projects in this corpus include both “mock-everything distributed systems” and “real-implementations distributed systems.” CockroachDB’s internal interfaces enable in-process substitution with real embedded backends, not with generated mocks. Kubernetes’s interfaces, by contrast, have accumulated a substantial mock infrastructure (kube/client-go FakeClient, mockgen-generated types). Both have the same T6 score but opposite relationships to testability. T6 cannot discriminate between these two very different patterns because it only measures the presence of internal interfaces, not what those interfaces enable.

For practitioners: T6 captures a signal about interface design philosophy but does not reliably predict testability in either direction. A low T6 score (many internal interfaces) is compatible with both excellent and poor testability; the differentiating factor is whether the interfaces were designed to enable real in-process implementations or to enable mock substitution.

T3 — Composition Root: Necessary but Insufficient#

T3 (explicit composition root) has the highest mode of any trait in the corpus (24 of 51 projects score 3), and its average testability in the T3=3 cohort (8.4) exceeds T3=2 (7.2) and T3=1 (7.1). The positive correlation is real but mild, and the most important observation is the number of T3=3 projects that have mediocre testability: Grafana (T3=3, U=5), drone (T3=3, U=4), air (T3=3, U=5), nats-server (T3=3, U=7), and temporal (T3=3, U=6).

An explicit composition root tells you that dependencies are injected. It does not tell you whether the injected dependencies are real implementations or mocks, whether the suite is hermetic, or whether there are exported test helpers for downstream users. Drone’s wire_gen.go is a textbook compile-time composition root, yet drone’s testability is the second-lowest in the corpus (U=4). Its U3=1 reveals hand-written mock stores used broadly across handler and service layers; the Wire-generated DI made those mocks easy to inject but didn’t create pressure to avoid them.

T3 is best understood as an enabling condition for testability, not a predictor of it. A project without an explicit composition root has poor testability for structural reasons — it’s hard to substitute dependencies when there’s no single wiring point. But a project with a perfect composition root (T3=3) may still choose mock-heavy testing or require external infrastructure for reasons unrelated to the DI pattern.

T7 and T8 — Structural Traits Without Testability Signal#

T7 (public helper/builder API) and T8 (package boundary discipline) show the weakest correlations with testability. Both traits primarily measure how a project presents itself to external consumers and how it enforces internal visibility — properties that matter for ecosystem integration but are largely orthogonal to how the project tests itself internally.

T8’s near-zero predictive power is partly explained by the Go compiler: even without explicit internal/ directories, Go projects tend to avoid import cycles, making T8 a floor measure rather than a discriminating one. No project scores T8=0. The 13 projects scoring T8=3 include both excellent-testability (etcd U=11, gorm U=7, delve U=9) and mediocre-testability (restic U=9, gh U=7) projects.

T7’s weak signal is similarly structural: whether a project exports a builder API affects downstream developers’ ability to test code that depends on the project, but it doesn’t determine whether the project’s own test suite is well-engineered. Fyne (T7=3, U=10) and syncthing (T7=3, U=5) illustrate the non-relationship: both score 3 on public test helper availability, but one has a thoughtfully hermetic test suite and the other doesn’t.

Outliers#

Grafana: Excellent Architecture, Poor Testability#

Grafana (P05) is the most striking outlier in the dataset. Its architectural composite is 20 — the fourth highest in the corpus, tied with Temporal — yet its testability composite is 5, placing it in the bottom 15% of the corpus. The scores are not in error. The architecture is genuinely good: Wire-generated DI (T3=3) eliminates globals and produces a compile-time-verified 1939-line composition root; the layering is clean (T5=3) with pkg/infra/ separated from pkg/services/; service implementations are concrete structs throughout (T6=3).

The testability failure is visible in the evidence: many integration tests require a running Grafana instance with PostgreSQL, or external datasources. The score on U1=1 reflects that the hermetic ceiling is at basic unit tests, not integration logic. U3=1 reveals that generated mocks are used extensively for datasource and store interfaces across internal packages — not confined to external boundaries. The Wire-validated DI was not leveraged to build real in-process implementations for testing; instead, it was leveraged to make mock injection easy.

This is the most important single data point for practitioners: architectural cleanliness and test infrastructure culture are separable. A team can have excellent DI practices, clean layering, and no globals yet still accumulate a mock-heavy, infrastructure-dependent test suite. The architectural properties in this rubric measure what the code looks like, not what the test philosophy demands.

Temporal: Textbook Architecture, Zero Mock Discipline#

Temporal (P50) shares Grafana’s top-tier architectural composite (20) but takes the mock story to its logical extreme. Its U3=0 is the only zero in the entire corpus on that trait: 126 *_mock.go files (the highest count in the dataset), generated via go.uber.org/mock/gomock, covering gRPC service boundaries, persistence layer, and internal component interfaces. The fx dependency injection framework mandates interface-typed dependencies throughout (T6=1); since fx provides them as interfaces to inject, the team reached for generated mocks as the natural counterpart.

The paradox is that Temporal has genuinely excellent architecture for testability: a common/clock package with injectable TimeSource (T4=3), pure domain logic packages (common/tasks, common/goro), explicit fx composition (T3=3). These architectural investments could have supported a real-implementations test suite. Instead, the team’s testing culture chose the opposite path. The architecture created seams; the team chose what to put at those seams.

A secondary consequence is visible in the U1=2 score: the OneBox in-process server enables most functional tests without Docker, but the full persistence matrix (Cassandra, PostgreSQL) requires Docker Compose. The architectural investment in in-process execution (OneBox) partially delivers hermetic testing, but the mock-heavy culture prevents it from achieving the highest tier.

CockroachDB: Below-Average Architecture, Best Testability#

CockroachDB (P09) is the inverse outlier. Its architectural composite is 14 — below the corpus median — yet it achieves the highest testability composite (15) in the entire dataset. The architectural scores that drag it down are T2=1 (CCL feature hooks via init()), T5=2 (not full functional core isolation), T6=1 (1,481 interface definitions at internal seams), and T7=0 (no public test helpers). None of these are architectural failures; they’re accurate scores for a complex distributed database that uses a lot of internal interfaces.

What CockroachDB has that produces U=15: T4=3 (context propagation in 26,543 usages; every I/O-performing function takes context as first parameter; the HLC provides injectable temporal reasoning); U4=3 (KV conformance tests and SQL executor conformance tests that every storage engine must pass); U5=3 (heavy table-driven tests plus the TestLogic file-based SQL format that loads thousands of .sql test files); and U6=3 (goroutine leak checks pervasively integrated as a correctness property).

The CockroachDB data point argues that testability is primarily a function of investment in specific testing practices rather than of broad architectural cleanliness. Teams that invest in conformance suites, hermetic injection points, declarative test formats, and goroutine hygiene produce highly testable codebases regardless of whether their internal architecture is deeply interface-heavy.

Hugo: Targeted Investment, High Return#

Hugo (P06) achieves top-quintile testability (U=10) with an architectural composite (14) that matches CockroachDB’s. Its distinguishing characteristic is the afero filesystem abstraction: every file operation in the Hugo pipeline goes through the afero.Fs interface, and afero.MemMapFs provides a complete in-process substitute. The entire test suite — including rendering integration tests for full Hugo sites — runs without any real filesystem access.

Hugo’s T4 score is 2 (not 3): it uses context and bep/clocks for time mockability, but does not parameterize randomness or all I/O surfaces. Its T5=2 reflects partial separation (the identity package is minimal, converters are pure) without system-wide enforcement. Yet U=10 exceeds most projects that score higher on both T4 and T5.

The lesson is that a single well-chosen injection point, applied consistently throughout a codebase, can have a larger impact on testability than a generalized architectural discipline applied inconsistently. Hugo’s teams chose afero early and used it everywhere; that single decision more than offset the absence of several other architectural virtues.

Archetypes that Emerge#

Three distinct archetypes are visible in the scatter plot of T_arch vs U_testability.

Archetype A: Real Implementations as the Test Implementation. These projects score U3=3 (no mocks for internal collaborators) and U1≥2 (mostly hermetic suites). The test implementation is the production implementation, exercised directly via injectable seams. Examples: Go stdlib, Hugo, Fyne, gin, echo, fiber, viper, cobra, sqlc, fzf, wireguard-go, delve, pocketbase, nats-server, crush (15 of 51 projects). What they share architecturally is not a single trait pattern but a combination of T1≥2 (narrow seams make real fakes affordable) and at least one well-placed T4 seam (filesystem, network, or OS process injection). Their arch composites range from 13 to 21, confirming that this archetype is achievable across the full architecture-quality spectrum.

Archetype B: Infrastructure-Constrained Systems. These projects have genuine domain constraints that require external infrastructure for meaningful tests: operating clusters, real databases, network protocols with external peers. Examples: Istio, Argo-CD, K3s, Consul, Beego, Syncthing, Drone. They score U1=1 and often U3=1 (internal mocks penetrate subsystem boundaries). These scores reflect domain difficulty rather than architectural failure. The distinction matters for how they should be read in the synthesis: their testability scores do not indicate bad architecture, they indicate a harder problem domain. The notable exception within this group is Consul, which has invested in sdk/testutil (U2=2) as a step toward better testability without having solved the hermetic infrastructure problem.

Archetype C: Mock-Heavy Discipline. A small cluster of sophisticated distributed systems adopted a testing culture that treats generated mocks as the standard abstraction for every service boundary. Examples: Temporal (the extreme case, U3=0), Grafana (U3=1), Argo-CD (U3=1). These projects have good-to-excellent architecture but chose mock-based testing at internal seams rather than real implementations. They may reach high unit test coverage counts while still exposing their integration behavior only through expensive E2E suites. The architecture in Archetype C projects actually facilitates this choice — the DI wiring makes mock injection easy — which is why T3=3 is not a predictor of testability: it enables both real and mock implementations equally.

The data does not support a fourth archetype of “testable despite weak architecture” in any systematic sense. The low-testability projects in this corpus tend to have genuinely weak architecture scores (gitea T_arch=6, gogs T_arch=9, k3s T_arch=9), not just weak testability investment. Hugo is the closest counter-example, and as argued above, it achieved testability through targeted investment rather than despite weak architecture — its architecture is adequate, just not exceptional.

The Contentious Middle#

The projects in the Kubernetes / Temporal / Vault / Dapr cluster deserve specific treatment because they are simultaneously the most architecturally sophisticated and the most debated on testability.

Kubernetes (T_arch=16, U=7), Temporal (T_arch=20, U=6), Vault (T_arch=15, U=7), and Dapr (T_arch=12, U=7) form a coherent cluster: good architecture, below-average testability, heavy mock infrastructure (Kubernetes uses kube/client-go FakeClient and mockgen extensively; Temporal has 126 mock files; Vault has 72 sync.Once lazy singletons and mock auth backends; Dapr uses generated mocks at gRPC and component interface boundaries). All four run meaningful unit tests in-process but require real infrastructure for complete testing.

Reading A (mock-heavy as a cost paid for architectural choices). The interface-abstracted architectures in these projects — Kubernetes’s 2,498 interface definitions, Temporal’s fx-driven interface-typed dependencies — made it easy to generate mocks and hard to maintain real in-process implementations. The architectural decision to interface-abstract internal collaborators created structural pressure toward a testing culture that fills those interfaces with generated fakes. Under this reading, teams building distributed systems should prefer concrete internal collaborators (T6=2-3) and invest heavily in real in-process implementations (like CockroachDB’s embedded test nodes) rather than the easier path of mock generation. The cost of not doing so is visible in the testability scores: even excellent architectural investment didn’t translate to high-tier testability.

Reading B (mock-heavy as an appropriate response to domain complexity). Distributed systems at the scale of Kubernetes, Temporal, and Vault have operational properties that cannot be meaningfully tested in a single in-process scenario: network partition behavior, Byzantine failure modes, multi-datacenter replication, clock skew. For these properties, integration tests with real infrastructure or carefully constructed mocks that simulate failure modes are the right tool. The U3 scores in this cluster may be penalizing a rational engineering choice. Under this reading, the U3 rubric (“mocks confined to external boundaries”) was calibrated for single-process applications and libraries, and cannot be applied to multi-process distributed systems without adjustment.

The data cannot adjudicate. CockroachDB (T4=3, U=15) is a direct challenge to Reading B: it is a distributed database at comparable or greater complexity than Temporal, and it achieves the highest testability score in the corpus through a combination of injectable effects, conformance suites, and goroutine leak hygiene — without pervasive generated mocks. But CockroachDB’s team made sustained deliberate investments that took years to develop. Reading B might be saying not that mock-heavy testing is correct for distributed systems, but that it is the rational default given the effort required for the alternative.

The practitioner conclusion is that both readings are partially true, and the right response depends on team investment capacity. Teams with the resources to build real in-process test implementations (embedded database nodes, channel-backed network stacks, in-memory state stores) will achieve significantly better testability outcomes. Teams without those resources will likely default to mock-heavy testing as a pragmatic compromise. The architecture should not foreclose the first path by making interface-abstraction so ubiquitous that real implementations become prohibitively expensive.

What This Means for a Practitioner#

• Prioritize T4 over T2 when designing for testability. Injecting time, I/O, and randomness as explicit parameters has a stronger and more direct connection to hermetic suites (U1) than eliminating global state (T2). Many projects with T2=1 (some globals) achieve excellent hermeticity by routing specific effect types — filesystem, time, network — through injected seams at the boundaries that matter most. The afero pattern in Hugo is a better return on architectural investment than eliminating every package-level variable.

• Design narrow interfaces (T1) not because it’s a best practice but because it makes real fakes affordable. A 3-method interface can have a second real implementation (an in-memory fake) without becoming a maintenance burden. A 30-method interface almost always ends up with only one real implementation (production) and one generated mock (tests). The narrow-interface discipline enforces the pattern that enables real-implementations testing.

• Don’t conflate DI with testability. An explicit composition root (T3=3) and clean DI wiring make mock injection easy — but they don’t create pressure to avoid mocks. Drone (T3=3, U=4) and Grafana (T3=3, U=5) demonstrate that Wire-generated or explicitly-wired DI is compatible with poor testability. The real question is what gets injected at the seams the DI provides.

• Invest in conformance suites when you have replaceable parts. The projects that achieve U4≥2 (CockroachDB, rclone, restic, Prometheus, etcd) do so because they both defined a clear interface for their replaceable parts and wrote a parameterized test suite that runs against every implementation. The corpus suggests this is rare (40 of 51 score U4=0) but high-value: every new implementation gets systematic correctness enforcement for free.

• Goroutine leak hygiene is a systematic blind spot. 47 of 51 projects score U6=0. Projects with significant goroutine concurrency (NATS, Temporal, Headscale, Tailscale, Syncthing) all score 0, meaning goroutine lifecycle is not treated as a first-class correctness property. Adding goleak or equivalent to CI is a one-hour investment with disproportionate returns for any concurrent system.

What This Analysis Cannot Tell You#

The single most important limit of this analysis is selection bias. Every project in this corpus is successful, maintained, and widely adopted. The minimum testability composite in this dataset is 4 (gogs, drone). Real-world software includes projects with testability composites of 0 or 1 — projects with no meaningful test infrastructure at all — and the architectural traits of those projects are not represented here. Findings from this corpus establish co-occurrence patterns among good OSS software; they cannot be extrapolated to predict what happens when architectural discipline is absent entirely.

A related limit is that both scorecards were produced from Phase 1 analysis reports rather than direct source code inspection. The Phase 1 reports are detailed, but they summarize; edge cases in the actual source that would shift a score by 1 point are likely present. The confidence note below identifies the most uncertain cells.

The reverse causation limit deserves restatement here. The Hugo/afero finding — that a single injection point enabled full hermeticity — could be read as “if you add afero, you get hermetic tests.” The causal reading is probably closer to: “the team that cared enough about hermetic testing to use afero throughout also invested in all the other practices that make U1=3 achievable.” Architecture and testing culture co-evolve; isolating the contribution of one is not possible with this data.

Finally, the composite scores used in this analysis are unweighted sums. The architecture rubric treats T1 (narrow interfaces) and T8 (package discipline) as equally important. The testability rubric treats U6 (goroutine leak hygiene) as important as U1 (hermetic suites). These weightings are not validated; a practitioner who believes T4 matters twice as much as T8 should re-weight accordingly. The synthesis pass deliberately avoids weighting because the synthesis pass is not the place to impose value judgments about which architectural properties matter most.

Confidence Note#

Cells most likely to shift under re-reading:

The T4 scores for fyne (T4=1) and delve (T4=1) are accurate per rubric but architecturally misleading: fyne deliberately omits context threading because GUI main-thread constraints demand a different model, and delve uses minimal context because debugger attachment is synchronous. Both are domain-appropriate decisions that score poorly on a rubric calibrated for networked services. The T4-U1 correlation should be interpreted with this domain confound acknowledged: T4=1 in a GUI framework or debugger does not carry the same signal as T4=1 in a distributed service.

The Grafana U3=1 score may be generous. The evidence (“generated mocks used extensively for datasource and store interfaces across internal packages — not confined to external boundaries”) suggests U3=0 or U3=1 is the right range; U3=1 was assigned because some packages do use real implementations. A stricter reader might assign U3=0, which would push Grafana’s testability composite from 5 to 4 and strengthen the Grafana outlier finding.

The CockroachDB T6=1 score captures an architectural fact (1,481 interface definitions at internal seams) but may undercount the extent to which those interfaces enable real embedded implementations rather than mocks. A reader who distinguishes “interfaces designed for real-implementation substitution” from “interfaces designed for mock substitution” might score CockroachDB T6=2, which would slightly reduce its role as a T6 counter-example.

The temporal U4=1 score reflects a multi-database test matrix (Cassandra, PostgreSQL, MySQL, SQLite) that runs the same functional tests against multiple backends. The rubric requires a “generic parameterized acceptance suite” for a score of 2. Whether Temporal’s matrix qualifies depends on how rigorously “formal parameterized suite” is interpreted. A score of 2 would not change the qualitative conclusions.

The synthesis rests most heavily on the T4-U1 co-occurrence, the T1-universality among top testability projects, and the Grafana/Temporal outliers. These findings are robust to ±1 point variation in individual cells. The T2-testability non-correlation and T6-testability non-correlation are also robust: the counter-examples (prometheus, cockroach, go stdlib for T2; cockroach, nats-server for T6) are not borderline cases.

What would strengthen confidence: direct source inspection of test infrastructure in the 10 most uncertain projects (grafana, temporal, consul, tailscale, fyne) to verify the Phase 1 report accuracy; independent scoring by a second reader calibrated against the rubric to measure inter-rater reliability; and expansion of the corpus to include projects with lower quality floors to validate the selection-bias adjustment.