restic — Patterns#

Concurrency patterns#

Worker Pool via errgroup + Channel#

  • Usage: The primary parallelism pattern throughout the codebase; used in archiver, restorer, data, and the domain restic package.
  • Example: internal/archiver/file_saver.go:34-55 — newFileSaver creates a buffered channel ch, then launches fileWorkers goroutines via errgroup.Group.Go. Workers range over the channel until it is closed.
  • Assessment: Idiomatic and effective. The errgroup ensures all workers are joined and the first error propagates context cancellation to the rest. Shutdown is triggered by closing the input channel (TriggerShutdown), which terminates the range loop.

Fan-out/Fan-in via errgroup.WithContext + Channel#

  • Usage: internal/restic/parallel.go — ParallelList implements a generic fan-out: one producer goroutine lists file IDs and sends them to a channel; N worker goroutines consume and process them.
  • Example: internal/restic/parallel.go:11-50 — wg, ctx := errgroup.WithContext(ctx) creates a linked context; the producer closes the channel after listing; workers return errors that cancel the shared context.
  • Assessment: Clean textbook fan-out. The use of select { case <-ctx.Done(): ... case ch <- ...: } in the producer avoids a deadlock if a worker returns early. Used as a utility function across the codebase.

Semaphore via Buffered Channel#

  • Usage: internal/backend/sema/semaphore.go — a struct wrapping chan struct{} of capacity N. Used by sema.Backend to limit concurrent backend operations.
  • Example: internal/backend/sema/semaphore.go:12-31 — GetToken() sends to the channel; ReleaseToken() receives from it. Capacity equals max connections (default 5).
  • Assessment: The canonical Go semaphore idiom. Exposed as a decorator (sema.Backend) so all storage drivers get connection limiting without any driver code change. Also used in SFTP startup (sftp.go:175).

errgroup as the Standard Async Coordination Primitive#

  • Usage: 18+ call sites across non-test code; golang.org/x/sync/errgroup is the project’s exclusive higher-level concurrency abstraction.
  • Assessment: restic deliberately avoids raw sync.WaitGroup for error-bearing goroutines. errgroup.WithContext pairs cancellation with error collection cleanly. The wg.SetLimit() (used in ParallelRemove) is also exploited, reducing boilerplate for bounded concurrency.

Context Cancellation as the Primary Stop Signal#

  • Usage: 769 occurrences of context.Context in the codebase. Every I/O-bound function in the backend, repository, archiver, and restorer layers accepts a context.
  • Example: internal/walker/walker.go:59-61 — after each tree node: if ctx.Err() != nil { return ctx.Err() }. This is a consistent pattern: check context at the start of every iteration step.
  • Assessment: Thorough and idiomatic. Context cancellation is the uniform way to communicate shutdown/timeout to all layers; there is no ad-hoc “stop” flag.

Graceful Shutdown via Signal → Context Cancel#

  • Usage: cmd/restic/cleanup.go — createGlobalContext wires SIGINT/SIGTERM to a context.CancelFunc.
  • Example: cleanup.go:14-38 — a dedicated goroutine blocks on a signal channel; on receipt it calls cancel() and logs the signal. The root cobra context propagates this cancellation to all subcommands.
  • Assessment: Single, clean shutdown path. Restic also supports RESTIC_DEBUG_STACKTRACE_SIGINT to dump a full goroutine stack on SIGINT — a production debugging affordance.

Rate Limiting via io.Reader/io.Writer Decoration#

  • Usage: internal/backend/limiter — a Limiter interface wrapping io.Reader/io.Writer for upload/download throttling; injected into the transport layer.
  • Assessment: Rather than a token-bucket goroutine, the rate limiter wraps I/O streams, keeping it transparent to backend logic.

Error handling#

  • Style: Mixed — github.com/pkg/errors (stack-trace wrapping) via the internal/errors facade, plus stdlib errors.Is/errors.As for inspection.
  • Error types defined:
    • fatalError (internal/errors/fatal.go) — wraps a message to signal CLI termination; detected via errors.IsFatal() in main(). Errors returned by command handlers are inspected here; fatalError triggers a non-zero exit without printing the error chain.
    • MultipleIDMatchesError, NoIDByPrefixError (internal/restic/backend_find.go) — typed errors for snapshot ID prefix resolution.
    • alreadyLockedError, invalidLockError (internal/restic/lock.go) — sentinel types for lock management.
    • checker.Error, checker.TreeError — structured errors from repository integrity checks.
    • repository/pack.InvalidFileError, repository/checker.PackError — typed storage errors.
  • Wrapping approach:
    • errors.Wrap / errors.Wrapf from github.com/pkg/errors (121 call sites in non-test code) — used for errors from external calls where a stack trace is valuable.
    • fmt.Errorf with %w (77 call sites) — used for simple contextual additions, usually in the cmd/ layer.
    • errors.Is / errors.As (136 call sites) — used for all error type inspection.
  • The internal/errors facade: restic does not import github.com/pkg/errors directly in most packages. Instead it imports github.com/restic/restic/internal/errors which re-exports Wrap, Wrapf, New, Errorf from pkg/errors and As, Is, Join, Unwrap from stdlib. This allows switching the stack-trace implementation without touching call sites.
  • Examples:
    • cmd/restic/cmd_dump.go:186 — fmt.Errorf("cannot dump to file: %w", err) — simple context addition.
    • cmd/restic/cmd_self_update.go:68 — errors.Wrap(err, "unable to find executable") — external error with stack trace.
    • cmd/restic/cmd_mount.go:138 — errors.Is(err, os.ErrNotExist) — sentinel error check.

Configuration pattern#

  • Approach: Per-command Options structs bound to pflag flags via cobra. Every command defines its own XxxOptions struct (e.g. BackupOptions, RestoreOptions, CheckOptions) and registers flags on the cobra command’s FlagSet. Global configuration lives in global.Options and is passed as a value everywhere.
  • Example: internal/archiver/archiver.go:141-150 — Options struct with fields ReadConcurrency, MaxTreePackSize, NoExtraVerify, PackSize. Passed as a value argument to archiver.New(repo, fs, opts).
  • No functional options pattern: restic uses plain constructor functions New(...) with an Options value arg rather than With* variadic options. This is consistent across archiver, repository, restorer.
  • Feature flags: internal/feature implements an Alpha/Beta/Stable/Deprecated flag lifecycle. RESTIC_FEATURES env var can enable/disable named flags at startup. Example: feature.Flag.Enabled(feature.BackendErrorRedesign).

Dependency injection#

  • Approach: Manual constructor-based wiring. No DI framework (wire, dig, fx).
  • Evidence: internal/global/global.go is the composition root. OpenRepository explicitly constructs the decorator stack:
    driver → sema.NewBackend → logger.New → retry.New → optional cache
    Returns a *repository.Repository value. Commands receive this fully wired value. No service locator, no ambient global state beyond global.Options.
  • Backend registry: The location.Registry (populated via init() side effects from backend/all) acts as a factory; URL-scheme dispatch replaces DI for backend selection at runtime.

Other notable patterns#

Decorator / Middleware Pattern on an Interface (Backend Stack)#

The most architecturally distinctive pattern in restic. backend.Backend is a 12-method interface. Decorators (sema, logger, retry, cache, dryrun, limiter) each wrap a Backend and implement the same interface. global.wrapBackend composes them explicitly. Each decorator also implements backend.Unwrapper (single Unwrap() Backend method), enabling the generic AsBackend[T] function to walk the chain and extract any layer by type.

// internal/backend/backend.go:109
func AsBackend[B Backend](b Backend) B {
    for b != nil {
        if be, ok := b.(B); ok {
            return be
        }
        if be, ok := b.(Unwrapper); ok {
            b = be.Unwrap()
        } else {
            break
        }
    }
    var be B
    return be
}

This is an HTTP middleware stack applied to a storage interface, with generic type-safe introspection.

Generics for Type-Safe File Type Constraints#

  • internal/restic/repository.go:99-154 — FileTypes is a union constraint FileType | WriteableFileType. Generic interfaces SaverUnpacked[FT FileTypes], RemoverUnpacked[FT FileTypes], Unpacked[FT FileTypes] enforce at compile time which parts of the repository allow writes. ParallelRemove[FT FileTypes] is a generic function parameterized on this constraint.
  • internal/backend/location/registry.go:40 — genericBackendFactory[C any, T backend.Backend] captures config type and backend type generically.
  • internal/restorer/hardlinks_index.go — HardlinkIndex[T any] is a generic map keyed by inode/device.
  • Assessment: Targeted, purposeful generics use — not generics for abstraction’s sake. The FileTypes constraint replaces what would have been a runtime check or a WriteableFileType bool field.

Go 1.23 Range-over-Function Iterators (iter.Seq)#

  • internal/data/tree.go:36 — type TreeNodeIterator = iter.Seq[NodeOrError] — tree node iteration via a push-style iterator.
  • internal/data/tree.go:368 — DualTreeIterator takes two TreeNodeIterators and zips them.
  • internal/repository/index/associated_data.go:153 — All() and Keys() on AssociatedSet[T] return iter.Seq2 / iter.Seq.
  • cmd/restic/cmd_copy.go:79-80 — a snapshot filter returns iter.Seq[*data.Snapshot].
  • Assessment: Adoption of Go 1.23 iterator protocol is incremental but deliberate, replacing channel-based iteration in tree traversal to eliminate goroutine overhead for sequential consumers.

Visitor with Optional Callbacks (Struct-based Visitor)#

  • internal/walker/walker.go:26-32 — WalkVisitor is a struct with callback fields:
    type WalkVisitor struct {
    ProcessNode WalkFunc  // mandatory
    LeaveDir    func(path string) error  // optional
}
    Callers fill only the callbacks they need. This avoids both the interface-implement-all-methods burden and the function explosion of separate Walk variants.
  • Assessment: Practical middle ground between the interface visitor pattern and bare function callbacks. Widely used across cmd_find, cmd_ls, cmd_stats, dump.

Sentinel Error for Control Flow in Recursive Traversal#

  • internal/walker/walker.go:13 — var ErrSkipNode = errors.New("skip this node") — if ProcessNode returns ErrSkipNode on a directory, it is not descended into.
  • Assessment: Standard Go pattern (cf. filepath.SkipDir). The named sentinel avoids a separate bool return and keeps the error path singular.

Build-Tag Dual-File Feature Gating#

  • Used for FUSE mount, debug commands, and self-update: every optional command has two files:
    • cmd_mount.go (real implementation, //go:build !nofuse)
    • cmd_mount_disabled.go (stub that registers the command but returns a helpful error, //go:build nofuse)
  • Assessment: Avoids #ifdef-style conditional code inside files. Makes binary footprint controllable at build time without dead code.

sync.Once for Lazy Initialization#

  • internal/repository/repository.go:52-53 — allocEnc, allocDec sync.Once initialize zstd compressor/decompressor pools on first use.
  • internal/restic/config.go:55 — checkPolynomialOnce verifies the CDC polynomial once.
  • internal/ui/termstatus/status.go:35 — outputWriterOnce initializes the terminal writer.
  • Assessment: Standard usage. Avoids init-time side effects for expensive allocations.

Atomic Types for Concurrent Progress Counters#

  • internal/ui/progress/counter.go:20 — value, max atomic.Uint64 — used by progress bars updated from multiple goroutines without a mutex.
  • internal/restorer/restorer.go:675 — atomic.AddUint64(&nchecked, 1) — inline lock-free counter increment.
  • Assessment: Appropriate use for hot path counters. No overuse; mutexes are used where the critical section spans more than a single integer update.

interface{} → any Migration in Progress#

  • interface{} appears 145 times, any 116 times — the codebase is mid-migration to the Go 1.18 any alias. No mechanical consistency yet, but newer code uses any.
Backward restic — Architecture restic — Interfaces Forward