restic — Architecture#

Architectural style#

Layered Monolith with a Dependency-Inversion Anchor

restic is a single-binary CLI application whose internal structure follows a strict, enforced layer hierarchy. The architecture is most accurately described as a “layered monolith with a pluggable backend subsystem.” There is exactly one binary (cmd/restic) and no public library surface. The layering is not just a convention — it is enforced structurally: internal/restic (the domain package) imports no other internal packages, so circular dependencies are mechanically impossible.

Two orthogonal design forces dominate the architecture:

  1. A vertical dependency chain from CLI glue → operations → repository → backend → domain types
  2. A horizontal decorator stack in the backend subsystem, composing cross-cutting concerns (caching, retries, rate limiting, logging, semaphore) over any storage driver without modifying the drivers themselves

Evidence: internal/global/global.go:588-622 explicitly constructs the decorator stack: logger.New(sema.NewBackend(be)) → retry.New(…) → optional test hooks. The domain package internal/restic contains only types and interfaces, no I/O, no implementation.

Component diagram (textual)#

┌──────────────────────────────────────────────────────────────────────────┐
│  cmd/restic  (cobra CLI — one file per command, ~25 commands, 83 files)  │
│  newBackupCommand, newRestoreCommand, newCheckCommand, …                 │
└────────────────────────┬─────────────────────────────────────────────────┘
                         │ uses
                         ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  internal/global  (OpenRepository, CreateRepository, wrapBackend)        │
│  + internal/ui/*  (termstatus, progress, backup/restore UI)              │
└──────┬────────────────────────────────────────────────┬──────────────────┘
       │ uses                                           │ uses
       ▼                                                ▼
┌──────────────────┐   ┌───────────────┐   ┌──────────────────────────────┐
│ internal/archiver│   │internal/restorer│  │ internal/checker             │
│ (backup creation)│   │(snapshot restore)│ │ internal/fuse (FUSE mount)   │
│ internal/walker  │   └───────┬────────┘  │ internal/dump                │
└──────┬───────────┘           │            └───────────┬──────────────────┘
       │                       │                        │
       └───────────────────────┼────────────────────────┘
                               │ uses restic.Repository interface
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  internal/repository  (Repository struct — packing, indexing, encryption)│
│  ├── repository/index  (MasterIndex)                                     │
│  ├── repository/pack   (pack file reading)                               │
│  └── repository/hashing (hashing writer)                                 │
└──────────────────────┬───────────────────────────────────────────────────┘
                       │ uses backend.Backend interface
                       ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  Backend Decorator Stack (assembled at startup in global.wrapBackend)    │
│                                                                          │
│  [cache.Backend]  ←  optional local disk cache                          │
│       ↓                                                                  │
│  [retry.Backend]  ←  15-minute retry with exponential backoff            │
│       ↓                                                                  │
│  [logger.Backend] ←  debug-level operation logging                       │
│       ↓                                                                  │
│  [sema.Backend]   ←  semaphore: concurrent-connection limiter            │
│       ↓                                                                  │
│  Storage Driver (implements backend.Backend)                             │
│  local | sftp | rest | s3 | azure | gcs | b2 | swift | rclone           │
└──────────────────────┬───────────────────────────────────────────────────┘
                       │ types defined in / implements interfaces from
                       ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  internal/restic  (domain core — imports NO other internal packages)     │
│  ID, Blob, BlobHandle, Snapshot, Node, Pack, Index, Config               │
│  Repository interface, Backend interface (via internal/backend)          │
│  FileType (PackFile, KeyFile, LockFile, SnapshotFile, IndexFile, Config) │
└──────────────────────┬───────────────────────────────────────────────────┘
                       │ primitive utilities
                       ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  internal/crypto   (AES-256-CTR + Poly1305-AES, key derivation)         │
│  internal/fs       (filesystem abstraction: OS, virtual, node types)     │
│  internal/filter   (glob-based include/exclude pattern matching)         │
└──────────────────────────────────────────────────────────────────────────┘

Core components#

Domain Core (internal/restic)#

  • Package: github.com/restic/restic/internal/restic
  • Responsibility: Defines all canonical domain types and the key interfaces. Acts as the dependency inversion anchor — everything imports it; it imports nothing internal.
  • Key types: ID (SHA-256 content hash), Blob (content-addressed unit), BlobHandle, Snapshot, Node (file/dir/symlink), Pack, Config
  • Key interfaces: Repository (the full repository contract), BlobSaver, BlobLoader, Loader, Lister, BlobSaverAsync, FindBlobSet, AssociatedBlobSet
  • Dependencies: Only stdlib + internal/backend (for FileType constants), internal/crypto (for *crypto.Key), internal/errors, internal/ui/progress

Backend Interface + Storage Drivers (internal/backend)#

  • Package: github.com/restic/restic/internal/backend
  • Responsibility: Defines the Backend interface for raw object storage; implements 14+ storage drivers plus decorator wrappers.
  • Key types: Backend interface (Save, Load, List, Stat, Remove, Warmup/WarmupWait, IsNotExist, IsPermanentError, Delete, Hasher, Properties), Handle, FileInfo, FileType, Properties
  • Subdirectories: local, sftp, rest, s3, azure, gs, b2, swift, rclone (storage drivers); cache, retry, limiter, sema, logger, dryrun (decorator wrappers); location (URL-based registry/factory), layout (filesystem layout strategies), all (registers all backends), mock/mem/test (testing)
  • Dependencies: stdlib + individual cloud SDKs (aws, azure, google); decorators depend only on backend.Backend

Repository (internal/repository)#

  • Package: github.com/restic/restic/internal/repository
  • Responsibility: Concrete implementation of restic.Repository. Manages encryption at rest, content-defined chunking (via github.com/restic/chunker), blob packing into pack files, index management, locking, and zstd compression.
  • Key types: Repository struct (holds backend.Backend, *crypto.Key, *index.MasterIndex, *cache.Cache), Options (Compression, PackSize, NoExtraVerify), packerManager, packerUploader
  • Dependencies: internal/restic, internal/backend, internal/crypto, internal/repository/index, internal/repository/pack, github.com/klauspost/compress/zstd, github.com/restic/chunker, golang.org/x/sync/errgroup

Archiver (internal/archiver)#

  • Package: github.com/restic/restic/internal/archiver
  • Responsibility: Creates backups (snapshots). Traverses the source filesystem, chunks files using CDC, deduplicates against the existing index, saves new blobs via the Repository, and records the resulting tree structure.
  • Key types: Archiver, Scanner (pre-backup statistics scan), FileSaver, TreeSaver
  • Dependencies: internal/restic, internal/repository, internal/fs, internal/filter, internal/debug, golang.org/x/sync/errgroup

Restorer (internal/restorer)#

  • Package: github.com/restic/restic/internal/restorer
  • Responsibility: Restores a snapshot to a target directory. Parallel file writing with configurable concurrency; handles permission restoration, symlinks, and special files.
  • Key types: Restorer
  • Dependencies: internal/restic, internal/repository, internal/fs

Global / Bootstrap (internal/global)#

  • Package: github.com/restic/restic/internal/global
  • Responsibility: Global flag definition, repository open/create logic, backend wiring (decorator stack assembly), password resolution, cache setup.
  • Key functions: OpenRepository, CreateRepository, innerOpenBackend, wrapBackend, parseConfig, setupTransport
  • Dependencies: Nearly all internal/* packages; this is the composition root for the backend stack

UI (internal/ui/*)#

  • Package: github.com/restic/restic/internal/ui
  • Responsibility: Layered terminal output system. termstatus (raw terminal line control), terminal (output routing and buffering), backup/restore (domain-specific progress rendering), progress (generic counter), table (tabular output), signals (OS signal handling).
  • Key types: ui.Terminal interface (consumed by global.Options), termstatus.Terminal, backup.Progress

Crypto (internal/crypto)#

  • Package: github.com/restic/restic/internal/crypto
  • Responsibility: AES-256-CTR encryption with Poly1305-AES MAC, scrypt key derivation, key management (multiple keys per repository).
  • Key types: Key, EncryptedData
  • Dependencies: stdlib crypto only

Data flow#

Backup operation: restic backup /some/dir#

1. cobra parses flags → runBackup() called
2. global.OpenRepository(ctx, gopts, printer)
   a. readRepo → parse repo URL
   b. innerOpenBackend:
      - location.Parse → identify scheme (e.g. "s3"), create driver config
      - factory.Open → instantiate storage driver (e.g. s3.Backend)
      - wrapBackend → compose decorator stack:
          sema.NewBackend(driver) → logger.New → retry.New(…, 15min)
      - optional cache: s.UseCache(cache.Cache)
   c. repository.New(be, opts) → create Repository struct
   d. s.SearchKey → iterate KeyFile entries, decrypt with password → store *crypto.Key
3. repository.LoadIndex → download IndexFile entries → build in-memory MasterIndex
4. archiver.New(repo, fs, opts) → create Archiver
5. archiver.Archive(ctx, snapshotOptions, targets, progressPrinter)
   a. Scanner.Scan → stat all source files, estimate sizes
   b. For each file: FileSaver.Save
      - read file in chunks (restic/chunker CDC, content-defined chunking)
      - for each chunk: compute SHA-256 ID
      - MasterIndex.Has(blobID) → if known, skip (deduplication)
      - if new: encrypt blob (crypto.Key.Seal), add to packer
      - packerManager: when pack reaches target size (default 16 MiB):
          pack the blobs with a header, encrypt the header
          backend.Save(ctx, Handle{Type: PackFile, Name: packID}, data)
   c. For each directory: TreeSaver.Save → build tree node list → save as tree blob
   d. Finalize: flush remaining packs, save updated index (IndexFile)
   e. Save snapshot metadata (SnapshotFile): host, tags, paths, tree ID, parent ID
6. Print summary (bytes added, files new/changed/unchanged, duration)

Restore operation: restic restore <snapshot-id> --target /dest#

1. OpenRepository (same as above)
2. LoadIndex → MasterIndex populated
3. repo.LoadUnpacked(ctx, SnapshotFile, snapshotID) → decrypt → unmarshal Snapshot
4. restorer.New(repo, snapshot) → Restorer
5. restorer.RestoreTo(ctx, "/dest")
   a. Walk snapshot tree recursively (walker.Walk)
   b. For each node:
      - Dir: mkdir, set permissions
      - File: repo.LoadBlob(ctx, DataBlob, blobID, buf) for each blob
              → MasterIndex.Lookup → identify which pack contains the blob
              → backend.Load(ctx, Handle{PackFile, packID}, length, offset, fn)
              → decrypt blob → write to target file
      - Symlink: os.Symlink
   c. Restore timestamps and permissions in a second pass

Initialization / Bootstrap#

main()
  ├── tweakGoGC()          // lower GOGC 100→50 for memory efficiency
  ├── feature.Flag.Apply() // parse RESTIC_FEATURES env var → enable/disable flags
  ├── global.Options{Backends: all.Backends()}  // register all storage drivers
  ├── termstatus.Setup()   // initialize terminal status line
  ├── createGlobalContext() // sets up signal handling (SIGINT/SIGTERM)
  └── newRootCommand(&globalOptions).ExecuteContext(ctx)
        └── cobra dispatches to subcommand
              └── PersistentPreRunE: globalOptions.PreRun(needsPassword)
                    ├── parse RESTIC_PACK_SIZE env
                    ├── set verbosity level
                    ├── options.Parse(opts.Options) // --option k=v flags
                    └── resolvePassword() // env var → file → command → terminal prompt

Dependency injection: Manual constructor-based wiring. There is no DI framework (no wire, dig, or fx). The composition root is global.OpenRepository / global.CreateRepository, which explicitly constructs and wires the backend decorator stack, repository instance, cache, and index. The global.Options struct acts as the configuration carrier passed by pointer from main() through cobra to every command.

Backend registry: backend/all package calls init() via blank-import side effects to register all drivers into location.Registry. Each driver provides a location.Factory implementation with Open and Create methods. The registry is URL-scheme-keyed (e.g. "s3", "sftp", "rest", "local").

Configuration#

Primary mechanism: CLI flags via github.com/spf13/pflag. Every global flag (repo, password, cache, verbosity, compression, limits, TLS options) is defined in global.Options.AddFlags() (internal/global/global.go:89).

Environment variable fallback: Most flags read their default from environment variables at flag-definition time (e.g. opts.Repo = os.Getenv("RESTIC_REPOSITORY")). The PreRun hook handles cases where the env var must override a flag (e.g. RESTIC_PACK_SIZE).

Extended backend options: --option key=value (may be repeated) is parsed by internal/options into an options.Options map. Backend-specific options (e.g. s3.bucket-lookup, sftp.command) are extracted per-scheme and applied to the driver config struct via reflection or ApplyEnvironment.

Feature flags: internal/feature implements a simple flag registry with defined flags. RESTIC_FEATURES env var can enable/disable specific flags at startup. Useful for gradual rollout of new behavior.

No config file: restic deliberately has no config file format. All configuration is via CLI flags or environment variables. This is a principled choice for a backup tool — the invocation must be explicit and auditable.

Key design decisions#

1. internal/restic as the dependency inversion anchor#

The innermost package defines all domain types and the Repository/Backend interfaces. Every other package imports it; it imports nothing internal. This is the canonical “Dependency Rule” (Clean Architecture) applied at the Go package level. It makes the architecture verifiable: no cycle is possible from internal/restic outward.

2. Backend decorator stack as composable middleware#

Rather than adding cross-cutting concerns (retries, caching, rate limiting, logging) to each storage driver, restic implements them as backend.Backend-wrapping decorators. global.wrapBackend composes them explicitly: sema → logger → retry → [cache]. Each decorator implements backend.Unwrapper to allow introspection through the stack (backend.AsBackend[T] uses generics to walk the chain). This is the same pattern as HTTP middleware, applied to an object-level interface.

3. End-to-end encryption as a non-negotiable invariant#

Every blob written to a backend is encrypted before leaving internal/repository. The internal/crypto package provides AES-256-CTR + Poly1305-AES with scrypt key derivation; the Repository struct holds the key and applies it on every SaveBlob / LoadBlob. There is no code path that writes plaintext to a backend — the type system enforces this by making plaintext data never reach the Backend interface.

4. Content-addressable storage with CDC chunking#

Files are split into variable-size chunks using the Rabin fingerprinting CDC algorithm (github.com/restic/chunker). Each chunk is identified by its SHA-256 hash. Before storing a chunk, MasterIndex.Has(id) is consulted — if the chunk is already in the repository (from any previous snapshot), it is not uploaded again. This gives near-free incremental backups without tracking what changed at the filesystem level.

5. Build-tag-gated optional features without interface pollution#

Platform-conditional features (FUSE mount on Linux/macOS, debug commands, self-update) are handled via dual-file pairs: cmd_mount.go (real implementation, //go:build !nofuse) + cmd_mount_disabled.go (stub that registers a helpful error message). This avoids #ifdef-style conditional compilation inside shared files and keeps the binary footprint minimal on platforms that don’t support FUSE. The same pattern applies to debug and self-update commands.

Backward restic — Dependencies restic — Patterns Forward