Error Handling Across 51 Go Projects#

Summary#

Error handling in Go is not a solved problem — the 51 projects in this corpus show five distinct strategies ranging from zero-overhead sentinel errors to fully annotated structured error types with cross-process serialization. The dominant pattern post-Go 1.13 is fmt.Errorf("%w", err) with sentinel errors for caller-checkable conditions, but every major project that handles errors at scale has developed custom machinery for behavioral classification (retry/fatal), protocol-boundary translation (HTTP/gRPC status codes), or multi-error aggregation. The key finding: wrapping strategy converges, but type richness diverges sharply with project scale and domain complexity.

Taxonomy#

Approach 1: Minimal — stdlib only, no custom error types#

These projects use errors.New, fmt.Errorf, and fmt.Errorf("%w", err) exclusively. No custom error structs or behavioral interfaces.

Projects using it: air, pop, cobra, fzf (partial), wireguard-go (partial), headscale (partially)
How it works: Errors are plain strings or wrapped stdlib errors. Context is added via fmt.Errorf("failed to X: %w", err). Callers use errors.Is against sentinel values for control flow.
When it’s appropriate: Tools and libraries where error context is consumed by humans (logs, stderr) rather than programmatically. CLI hot paths where error chains are shallow. Projects with a single primary author where consistency is natural.
Exemplar: Air — zero custom types, consistent "failed to <verb> <object>: %w" convention throughout. Headscale wraps its (sparse) sentinels with fmt.Errorf("%w: %d", ErrInvalidNodeID, nodeID) to attach context while preserving errors.Is semantics.

Approach 2: Sentinel-dominant#

Projects export named error values (var ErrXxx = errors.New("...")) covering the full space of distinguishable error conditions. Custom structs are rare; sentinels carry classification.

Projects using it: kubernetes, prometheus, gorm, viper, tekton-pipeline, frp, nats-server (baseline sentinels), gin (via bitmask), cobra
How it works: Exported var Err* error values allow callers to use errors.Is. Complex sentinels are wrapped with fmt.Errorf("%w: additional context", ErrSpecific), which adds detail without losing the identity check. GORM uses 18 sentinel errors accumulated into DB.Error across a query chain via %v; %w joining.
When it’s appropriate: Libraries with stable error taxonomies that don’t need structured data. APIs that need backward compatibility (sentinels have stable identity; struct fields can break callers).
Exemplar: NATS Server — 40+ sentinel errors for client-visible conditions (e.g. ErrConnectionClosed, ErrAuthentication), then a separate ApiError struct for JetStream wire protocol errors that need numeric codes and human descriptions.

Approach 3: Rich custom type hierarchy#

Projects define multiple custom error structs, often organized into domain subsystems. Each type carries structured fields enabling programmatic inspection via errors.As.

Projects using it: moby, cockroachdb, gh, gitea, gogs, delve, drone, syncthing, caddy, dapr, nomad, helm, istio
How it works: Each domain defines its own error types (e.g. Moby’s errdefs package has 12 marker interfaces). Callers use errors.As to extract structured data. Protocol boundaries translate internal types to transport codes. CockroachDB goes furthest: a fork of pkg/errors (github.com/cockroachdb/errors) carries Hint/Detail/IssueLink annotations and transmits error chains over protobuf.
When it’s appropriate: Large-scale services where errors must be distinguished by machine, routed to different handlers, mapped to API response codes, or diagnosed from logs. Debuggers and developer tools where error precision is the product.
Exemplar: Moby’s errdefs package — 12 marker interfaces (IsNotFound, IsUnauthorized, IsConflict, etc.) each with a corresponding New* constructor and As* inspector. Any error satisfying an interface is classified by behavior, not type identity. This idempotent classifier pattern decouples error creation from error handling.

Approach 4: Error-as-behavior (behavioral interface classification)#

A subset of projects treat errors as carriers of behavioral metadata — should this be retried? Does this terminate the process? — detached from the error’s origin type.

Projects using it: rclone, syncthing, restic, buildkite-agent, syncthing, nats-server (partially)
How it works:
- Rclone: Wraps any error in wrappedRetryError, wrappedFatalError, or wrappedNoRetryError types that implement Retrier, Fataler, or NoRetrier interfaces. fserrors.ShouldRetry(err) walks the chain checking for these interfaces, plus HTTP status codes. Centralized retry policy works on any error from any backend.
- Syncthing: svcutil.FatalErr signals the suture supervisor to terminate rather than restart the service. Other errors cause a restart. Error type controls process lifecycle.
- Restic: internal/errors facade exports IsFatal(err) to distinguish errors that should suppress stack traces (clean exits) from unexpected failures. Main uses this to decide whether to print a chain or just the message.
When it’s appropriate: Systems with retry/restart loops or supervisor trees where the handling decision (retry, abort, log-and-continue) should be centralized independently of where the error originated.

Approach 5: Protocol-boundary translation#

Projects that serve multiple clients (HTTP, gRPC, CLI) implement a translation layer that maps internal errors to external protocol codes at handler boundaries.

Projects using it: traefik, caddy, minio, consul, vault, dapr, drone, gin, echo, fiber, pocketbase, nats-server, temporal, etcd
How it works: Internal code raises typed or sentinel errors. An adapter (HTTP middleware, gRPC interceptor, response encoder) calls errors.As or custom helpers (errors.AsStatus, IsNatsErr) to map to HTTP status codes or gRPC status codes. The internal error chain is preserved for logging; the client sees only a status code and message.
- Drone: errors.AsStatus(err) traverses the chain to find the first errors.Error with a status field. Central handler maps it to HTTP status.
- Dapr: Constructs google.golang.org/grpc/status errors with additional metadata, enabling typed gRPC errors for cross-language SDKs.
- MinIO: Three-layer system: S3 API errors → object-layer errors → storage errors. Each layer has a translation function that maps from the layer below to its own type, then the HTTP handler translates the object-layer error to an S3-XML response.
- etcd: Bidirectional translation: internal errors ↔ gRPC status ↔ rpctypes.EtcdError for client library consumption.
When it’s appropriate: Any project serving a defined external API where internal error detail should not leak to clients. Particularly important when the same error must be represented in multiple protocols (HTTP + gRPC + CLI).

Comparison Dimensions#

Wrapping approach#

Project	Primary wrapping	Legacy/alt	%w adoption
kubernetes	`fmt.Errorf("%w", ...)`	errors.New	High (2402 uses)
moby	`fmt.Errorf("%w", ...)`	`pkg/errors` (240 legacy sites)	Mixed, migrating
etcd	`fmt.Errorf("%w", ...)`	None	High
prometheus	`fmt.Errorf("%w", ...)`	None	High (809)
grafana	`fmt.Errorf("%w", ...)`	None	High (2925)
hugo	`fmt.Errorf("%w", ...)`	None	Exclusive
traefik	`fmt.Errorf("%w", ...)`	None	High (1273)
caddy	`fmt.Errorf("%w", ...)`	None	High
cockroachdb	`github.com/cockroachdb/errors`	None	Custom library
fyne	`fmt.Errorf` (no %w in core)	None	Low
minio	`fmt.Errorf("%w", ...)`	None	High
consul	`fmt.Errorf("%w", ...)`	`%v` (older paths)	Mixed
vault	`fmt.Errorf("%w", ...)`	`hashicorp/errwrap` (212)	Mixed
terraform	`fmt.Errorf("%s", ...)`	None	Low (%s over %w)
nomad	`fmt.Errorf("%w", ...)`	None	High (627)
dapr	`fmt.Errorf("%w", ...)`	None	High
k3s	`fmt.Errorf("%w", ...)`	None	High
helm	`fmt.Errorf("%w", ...)`	None	High
istio	`fmt.Errorf("%w", ...)`	`%v` (older paths)	Mixed
argo-cd	`fmt.Errorf("%w", ...)`	None	High
tekton	`fmt.Errorf("%w", ...)`	None	High
go (stdlib)	`fmt.Errorf("%w", ...)`	None	Selective (272)
gin	Custom accumulator	None	Minimal
echo	`fmt.Errorf("%w", ...)`	None	High
fiber	`fmt.Errorf("%w", ...)`	None	High
buffalo	`fmt.Errorf("%v", ...)`	None	Low (loses chains)
beego	`berror.Wrap` + `fmt.Errorf`	None	Custom
gorm	`fmt.Errorf("%w", ...)`	None	High
sqlc	`fmt.Errorf("%w", ...)`	None	High
viper	`fmt.Errorf("%w", ...)`	None	High
cobra	`fmt.Errorf` (rare `%w`)	None	Low
fzf	`errors.New` (concat)	None	Minimal
gh	`fmt.Errorf("%w", ...)`	None	High
gitea	`fmt.Errorf("%w", ...)`	None	High (1326)
gogs	`errors.Wrap/Wrapf` (CDB fork)	None	Custom library
drone	`fmt.Errorf("%w", ...)`	None	High
buildkite-agent	`fmt.Errorf("%w", ...)`	None	High
restic	`errors.Wrap/Wrapf` (facade)	`fmt.Errorf %w`	Hybrid (facade)
syncthing	`fmt.Errorf("%w", ...)`	None	Exclusive (205)
rclone	`fmt.Errorf("%w", ...)`	None	High (1601)
frp	`fmt.Errorf("%w: details", s)`	None	High
headscale	`fmt.Errorf("%w: data", s)`	None	High
tailscale	`fmt.Errorf("%w", ...)`	`%v` (older)	High (297 is/as)
wireguard-go	`fmt.Errorf("%w", ...)`	None	Selective
delve	`fmt.Errorf("%w", ...)`	Type assertion	Mixed
air	`fmt.Errorf("%w", ...)`	None	Exclusive
pop	`fmt.Errorf("%w", ...)`	None	High
pocketbase	`fmt.Errorf("%w", ...)`	None	High
nats-server	`fmt.Errorf + errors.New`	None	High
temporal	`fmt.Errorf("%w", ...)`	None	High
crush	`fmt.Errorf("%w", ...)`	None	High

Key observation: 45/51 projects use fmt.Errorf("%w", ...) as their primary or exclusive wrapping strategy. Only 4 projects use alternative libraries: CockroachDB (own library), Gogs (CDB fork), Restic (internal facade over pkg/errors), Vault (hashicorp/errwrap for partial use). This represents near-total convergence on the Go 1.13 stdlib approach.

Buffalo and Terraform are the notable outliers within the stdlib camp: Buffalo frequently uses %v instead of %w (losing error chains), and Terraform predates Go 1.13, so much of its wrapping uses %s rather than %w. Both represent technical debt in their error handling.

Error type richness#

Tier	Projects	Custom error type count
Zero	air, pop, fzf, cobra	0–1
Minimal (1–5)	crush, wireguard-go, headscale, headscale, k3s, pocketbase	2–8
Moderate (6–15)	gh, traefik, caddy, syncthing, nomad, helm, tekton, rclone, frp, fiber, echo, gorm, viper, tailscale	8–15
Rich (15+)	kubernetes, moby, cockroachdb, gitea, delve, nats-server, dapr, consul, vault	15–50+

Pattern: Error type richness correlates strongly with project scale and the diversity of callers. Developer tools (delve), servers with many client types (NATS, Dapr, Consul), and infrastructure-level libraries (kubernetes, moby) define the most error types. CLI tools and focused libraries define the fewest.

Multi-error aggregation#

Projects that need to accumulate multiple errors before reporting:

Project	Pattern
kubernetes	`field.ErrorList` (validation errors with path attribution)
prometheus	`ParseErrors` slice, `AppendPartialError`
grafana	`errors.Join` (Go 1.20)
terraform	`tfdiags.Diagnostics` (up to 50+ errors, HCL source location)
vault	`hashicorp/go-multierror`
istio	`errors.Join` (Go 1.20+)
gorm	`%v; %w` joining on `DB.Error`
sqlc	`multierr.FileError` with location
nats-server	config parse errors slice
tekton	`errorList` accumulation

Pattern: Multi-error is most common in parsers/validators (terraform HCL, sqlc SQL, NATS config, kubernetes admission) where reporting all errors at once is essential for developer experience. Runtime/server code tends to return the first error encountered.

Trends#

Scale → type richness: The correlation between project scale (XL vs. S tier) and number of custom error types is strong. XL projects (kubernetes, moby, cockroachdb, grafana) universally develop custom error infrastructures. S-tier projects (air, crush, wireguard-go) almost universally avoid them.

Age → wrapping style: Projects predating Go 1.13 (or heavily developed before it) show legacy wrapping: pkg/errors in moby and restic, %v in consul/istio’s older paths, %s in terraform. Post-1.13 projects use %w exclusively. This is the most reliable age signal in the corpus.

API diversity → boundary translation: Projects serving HTTP + gRPC + CLI simultaneously (dapr, etcd, consul) develop the most sophisticated error translation layers. Single-protocol projects (rclone over remote filesystems) rely on behavioral wrappers instead.

Framework vs. application: Web frameworks (gin, echo, fiber, beego) define minimal error types because they expose error handling as a user extension point. Applications built on top (gitea, grafana, pocketbase) define their own rich error hierarchies using the framework’s extension mechanisms.

gRPC adoption → typed wire errors: Projects using gRPC heavily (dapr, temporal, etcd, istio) use google.golang.org/grpc/status errors. This creates typed wire errors that survive serialization across process boundaries — the gRPC equivalent of HTTP status codes with structured details.

Best Practices#

Synthesized from the most sophisticated error handling in the corpus:

Use fmt.Errorf("%w", err) for all wrapping. This is non-negotiable post-Go 1.13. Buffalo’s %v and Terraform’s %s patterns are unambiguously worse — they silently discard error chains.
Sentinel errors for classification, wrapping for context. The pattern fmt.Errorf("%w: additional data", ErrSpecific) — seen in headscale, frp, tekton — preserves errors.Is identity while adding context. This is more powerful than bare sentinels but simpler than custom structs.
Protocol-boundary translation is mandatory for multi-protocol projects. Internal errors should never leak to external clients as-is. Implement a translation layer (HTTP middleware, gRPC interceptor) that maps internal errors to protocol codes once, centrally.
Define custom error types only when callers need to inspect structured fields. The trigger is errors.As usage: if you never call errors.As(err, &target) against a type, that type probably shouldn’t be a struct. Sentinel errors and behavioral interfaces cover most cases with less code.
Behavioral interfaces over type assertions for retry/fatal classification. Rclone’s Retrier/Fataler pattern is more powerful than type-switching on specific error structs: any error, from any source, can be given retry behavior by wrapping it. The policy lives in one place.
For large parsers/validators, accumulate errors. Returning the first parse error in a configuration or schema parser is hostile to users. Terraform, sqlc, NATS, and kubernetes all demonstrate multi-error accumulation patterns. errors.Join (Go 1.20) makes this cheap.
Use an internal/errors facade if switching libraries is likely. Restic’s facade approach (internal/errors wrapping pkg/errors but exposing a stdlib-compatible surface) enabled a migration path. Only worth it if you have >100 call sites and the library choice is contested.
Document error semantics at package boundaries. The errdefs package in Moby is the gold standard: it defines what each error category means (IsNotFound, IsConflict, etc.) in one place, independent of how any error was created. Callers only need to know the behavior, not the origin type.

Anti-Patterns#

1. %v instead of %w in error wrapping (Buffalo, older Consul/Istio paths) Silently discards error chains. errors.Is and errors.As stop working at wrap boundaries. This is the single most common error handling bug in the corpus.

2. String-parsed error codes (Beego) Beego encodes errors as "ERROR-{code}, {msg}" strings, then parses the string to recover the code. This bypasses the entire Go error infrastructure. It cannot participate in errors.Is/errors.As chains and is fragile to message changes.

3. Panic for non-exceptional errors (Beego, etcd STM) stmError in etcd wraps a panic-based retry mechanism for software transactional memory — defensible in that context. Beego uses panics for parse errors more broadly, mixing control flow semantics. Panics should be reserved for truly exceptional, unrecoverable states.

4. Losing error chains at protocol boundaries (frp) frp serializes errors as plain strings in JSON, destroying type information. Callers cannot errors.Is against a wire error. This is acceptable for very simple tools; for anything where callers need to distinguish error types, use a structured error envelope (see NATS ApiError, Drone errors.Error, Dapr gRPC status).

5. Per-error if err != nil { return err } duplication without wrapping context Several projects (older paths in gogs, frp) return bare errors without context. Each hop strips out the call site information. Even minimal wrapping (fmt.Errorf("doing X: %w", err)) is better than nothing.

6. Not using errors.As for structured inspection (Delve, pre-1.13 style) Delve uses if _, ok := err.(*typeConvErr); ok — direct type assertion — for unexported error types. This works but breaks if the error is ever wrapped. The errors.As equivalent (errors.As(err, &target)) unwraps transparently. The pattern survives in Delve’s codebase from a pre-1.13 era.

Exemplars#

1. Moby — errdefs (behavioral classification) The errdefs package is the most architecturally clean error system in the corpus. Marker interfaces (type notFound interface{ NotFound() }) combined with Is* helpers (IsNotFound(err error) bool) create a behavioral vocabulary that any error can satisfy — whether from the OS, a driver, or the core. New error categories can be added without changing callers. Study this when designing error taxonomies for large systems.

2. CockroachDB — structured annotation github.com/cockroachdb/errors is the most feature-rich error library in the corpus, adding Hint/Detail/IssueLink annotations, protobuf serialization for cross-process chains, SQLSTATE codes, and assertion failure guards. AssertionFailedf (4,949 uses) enforces invariants throughout the codebase. Study this for database-tier or distributed systems where error chains need to survive process boundaries.

3. Rclone — behavioral retry classification The fserrors package demonstrates how to centralize retry policy without coupling it to specific error types. Any error can be classified as Retrier/Fataler/NoRetrier by wrapping. ShouldRetry(err) walks the chain and also inspects HTTP status codes, providing a single policy point for 50+ storage backends. Study this for any system with heterogeneous error sources and a shared retry loop.

4. restic — internal/errors facade The internal/errors package re-exports pkg/errors functions with an added IsFatal classifier, while presenting a surface that could be replaced with stdlib. This gave restic a migration path from pkg/errors to stdlib without touching 121 call sites. Study this when inheriting a codebase with legacy error libraries.

5. GitHub CLI (gh) — exit code protocol gh maps sentinel error types to exit codes (1=error, 2=misuse, 4=auth, 8=pending) creating a machine-readable protocol for CI/shell scripts. This is distinct from — and complementary to — human-readable error messages. Study this for any CLI tool consumed programmatically.

Note on fyne and crush#

fyne is a GUI framework. Its error handling is intentionally minimal at the core — the framework surface is interface-based, and errors from GUI operations are typically surfaced through callbacks or dialog boxes rather than returned error chains. The absence of custom error types in fyne core is a deliberate architectural choice consistent with GUI framework conventions, not a gap.

crush is a TUI app with AI-assisted development history. Its error handling profile — fmt.Errorf("%w", ...) dominant, 6 exported sentinels, one structured HTTP error type (proto.Error), consistent "context message: %w" convention — matches the “minimal-stdlib” approach appropriate for a focused S-tier tool. Consult analysis/results/P51-crush--ai-development-profile.md for context on which signals reflect AI assistance vs. tool domain.

The striking consistency in crush’s error handling style (no mixed legacy approaches, no %v vs. %w inconsistencies) aligns with what the ai-development-profile identifies as a signal of AI-assisted development: uniform style throughout with no evidence of incremental accumulation over time. However, this is equally consistent with a small, focused, recently-started project by a single developer following current Go conventions.