Storage System Design: MinIO vs Rclone vs Syncthing vs restic#

Summary#

Four Go projects solve storage problems at different levels of the stack, yet each independently arrived at the same three structural decisions: a central interface as the storage contract, an init()-based or constructor-based composition root, and context-threaded cancellation throughout. Where they diverge sharply is in how they handle extensibility (plugin backends vs. fixed backends vs. decorator stacks) and cross-cutting concerns (retry, caching, rate limiting) — and those divergences reveal coherent philosophies about what the system is for.

Comparison dimensions#

Storage interface design#

The range from 5 methods (rclone fs.Fs) to 40 methods (MinIO ObjectLayer) is not arbitrary — it reflects how much semantic richness the interface must encode. Rclone’s Fs expresses only the universally supportable primitive set (List, NewObject, Put, Mkdir, Rmdir); richer optional operations live in Features as nullable function fields. MinIO’s ObjectLayer must encode full S3 semantics including multipart uploads, object versioning, tagging, healing, bucket lifecycle, and health checks — collapsing these into one large interface is the price of expressing the complete S3 contract as a single seam.

restic’s backend.Backend sits in the middle: 12 methods covering Save, Load, List, Stat, Remove, and warmup/locking operations. It is abstract enough to be satisfied by local disk, SFTP, S3, Azure Blob, GCS, and Backblaze B2 without modification, while being specific enough that the decorator pattern works cleanly (every decorator passes through all 12 methods).

Narrative: The interface size law — interface should be as large as the smallest common denominator of all backends — is obeyed by rclone most strictly and by MinIO least strictly. MinIO’s violation is justified: there is only one backend, so the interface is a testability seam rather than an extensibility point.

Optional capabilities: interface fragmentation vs. struct with function fields#

Rclone’s Features struct deserves extended attention. The classical Go alternative — many small interfaces (Purger, Copier, Mover) with type assertions — leads to N type assertion sites scattered across operations code. Rclone instead concentrates all optional capability declaration in one struct and all capability checks at operation call sites with if do := f.Features().Copy; do != nil. The cost is a Features struct that currently has ~50+ fields and grows with each new optional operation.

restic takes the opposite position: backend.Backend has no optional methods. If a driver cannot implement something (e.g., warmup for cold storage), it provides a no-op. Cross-cutting optional behavior (caching, retries, rate limiting) is layered on top via the decorator stack rather than being optional interface methods.

Backend registration and composition root#

Both rclone and restic use the init()-blank-import-aggregator pattern for backend registration. The key difference is what happens at instantiation time: rclone constructs backends lazily (when the remote string is first referenced) and caches them in a fs/cache.Cache LRU; restic constructs the backend stack once at startup and passes it as a dependency through the entire command.

Syncthing’s lib/syncthing.App.startup() is the clearest example of manual DI wiring in this group: every component is constructed in strict dependency order, all constructors take explicit interface arguments, and the single bootstrapping function is the readable record of the entire dependency graph. The only trick is the lateAddressLister wrapper to break the connections ↔ discover circular dependency.

MinIO’s approach — package-level global variables for every subsystem — is the outlier. It trades testability (tests must reset global state between runs) for zero boilerplate. The concession is newObjectLayerFn(): a function-typed accessor that tests can swap to inject a different ObjectLayer.

Concurrency model for storage I/O#

MinIO’s erasure I/O fan-out is the most sophisticated: errgroup.WithNErrs(n) pre-allocates an error slice of length N so each goroutine writes its error by index without a lock. This is followed by counting non-nil entries to determine quorum. The adaptive hedged reads (erasure-decode.go: readTriggerCh) add another layer: start reads on quorum-many drives and trigger additional reads only when those are too slow, reducing tail latency at the cost of minimal extra I/O.

Rclone and Syncthing both use multi-stage pipeline architectures, but Syncthing’s is strictly ordered with four sequential stages and four WaitGroups — shutdown must proceed in exact stage order (close copyChan → wait copiers → close pullChan → wait puller …). Rclone’s two-stage model (checkers → copiers) is simpler, with the checker stage being purely metadata-bound and the copier stage bandwidth-bound, allowing them to run at different concurrency levels.

restic’s approach is the most uniform: errgroup.WithContext is the exclusive higher-level primitive for all error-bearing goroutines. SetLimit() provides bounded concurrency without a separate channel-as-semaphore. internal/restic/parallel.go’s ParallelList is a reusable fan-out utility used across the codebase.

Cross-cutting concerns: retry, caching, rate limiting#

restic’s decorator stack is the most principled approach to cross-cutting concerns in this group. Instead of weaving retry, caching, and rate limiting into storage driver code or into calling code, restic implements each as a standalone backend.Backend wrapper assembled at startup:

driver → sema.NewBackend → logger.New → retry.New → [cache.Backend]

Each decorator implements backend.Unwrapper (single Unwrap() Backend method), and the generic AsBackend[T] function can walk the chain to extract any layer by type. Adding a new concern (e.g., dryrun.Backend, limiter.Backend) requires no changes to any existing code.

Rclone’s lib/pacer achieves a similar result for retry but distributes it: each backend instantiates its own Pacer and wraps every API call in f.pacer.Call(func() (bool, error) {...}). The shouldRetry closure is per-backend, delegating HTTP status code interpretation to fserrors.ShouldRetryHTTP. This is more flexible (backends can tune retry behavior) but harder to reason about globally.

Syncthing’s multi-semaphore (lib/semaphore.MultiSemaphore) is the most sophisticated rate-limiting design: a single RAII-style requestResponse.Close() releases up to three semaphores atomically, preventing double-counting between global, per-device, and per-folder limits.

Fault isolation model#

Syncthing’s suture-based supervisor tree is the outlier: it is the only project here that treats process-level fault tolerance as a first-class design concern. This is appropriate for a long-running daemon whose users expect it to survive transient failures (network outages, scanner crashes, filesystem errors) without manual restart. Every service implements a single Serve(ctx context.Context) error method and is registered under a root supervisor — adding fault isolation to a new subsystem is one interface method plus one supervisor.Add(service) call.

MinIO’s HTTP recovery middleware (http.Recover equivalent) catches panics in request handlers, preventing a single malformed request from crashing the server, but it does not provide subsystem-level restart. Drive failures are tolerated at the storage level via erasure coding quorum, not at the application level via restarts.

Error handling strategy#

MinIO’s StorageErr string type is architecturally distinctive: sentinel disk-error values are defined as named string constants (var errMaxVersionsExceeded = StorageErr("...")). Callers type-switch on StorageErr to distinguish storage errors from higher-level errors without struct allocation. This works because the error message is the identity — there are no additional fields needed.

Rclone’s behavioral error approach is unique in this group: rather than classifying errors by their origin (storage layer, protocol layer), it classifies them by what should happen next (retry, abort, do nothing). fserrors.ShouldRetry(err) walks the errors.As chain looking for Retrier, Fataler, and NoRetrier interfaces. This keeps retry policy decision-making close to the caller (the pacer) rather than distributed across backends.

Configuration philosophy#

MinIO’s self-referential configuration — cluster-wide settings stored in the same object store the cluster provides — is architecturally bold. It eliminates the need for an external config store (no etcd by default) but creates a bootstrapping dependency: the cluster must reach quorum before it can read its own configuration. The retry loop in initServerConfig() handles this chicken-and-egg problem.

restic’s deliberate absence of a config file is the principled opposite: for a backup tool, every invocation must be explicit and auditable. There is no ambient configuration state to reason about.

Common patterns#

All four projects share:

1. context.Context as the universal cancellation primitive. All I/O-bound functions accept a context. MinIO has 1,714 uses, rclone 3,557, syncthing 359, restic 769. None of them invented a custom cancellation mechanism.

2. init()-based self-registration for extensibility points. Rclone and restic use it for storage backends; rclone also uses it for CLI commands. MinIO does not need it (one backend), but the pattern is otherwise universal in this group.

3. Manual constructor-based DI. None of the four projects uses Wire, Dig, or Fx. MinIO uses package-level globals; the other three use explicit constructor injection at a single composition root.

4. Table-driven tests as the dominant test style. MinIO: 2,340 occurrences; rclone: 541; syncthing: prevalent; restic: prevalent. All four arrived independently at this as the standard form.

5. errgroup for error-bearing concurrent work. MinIO uses a custom errgroup.WithNErrs; rclone, syncthing, and restic use golang.org/x/sync/errgroup. The pattern — start N goroutines, collect errors, cancel on first failure — is universal.

Divergent choices#

Single backend vs. plugin registry. MinIO commits fully to one storage model (Reed-Solomon erasure coding) and optimizes it relentlessly. Rclone and restic both use plugin registries with 70+ and 9+ backends respectively. Syncthing has fixed transports but protocol abstraction. The lesson: plugin registries impose interface design constraints that can limit backend-specific optimization; fixed backends enable deep algorithmic specialization.

Daemon vs. CLI lifecycle. Syncthing (daemon) and MinIO (server) must handle fault tolerance, live configuration reload, and graceful degradation. Rclone and restic (CLI tools) treat each invocation as atomic — errors abort, flags configure, nothing persists between invocations. This single architectural decision cascades into supervisor trees (or their absence), live config reload (or its absence), and long-running background services (or their absence).

Interface size philosophy. MinIO’s 40-method ObjectLayer vs. rclone’s 5-method fs.Fs + 50-field Features struct vs. restic’s 12-method backend.Backend represent three different answers to the interface design question. MinIO chose semantic completeness; rclone chose minimalism + optional extension; restic chose practical uniformity.

Cross-cutting concerns placement. restic places retry, caching, and rate limiting as composable decorators assembled at startup. Rclone places them per-backend (each backend instantiates its own pacer). Syncthing places them as semaphore-bearing subsystems accessed via multi-acquisition. MinIO scatters them across subsystems as globals. The decorator approach (restic) is most testable and replaceable; the global approach (MinIO) is most convenient.

Recommendations for practitioners#

Choose the decorator stack pattern (restic) when: you have a storage interface with multiple implementations and multiple cross-cutting concerns (retry, caching, throttling). Each concern becomes independently testable. Adding a new concern requires zero changes to existing code.

Choose fs.Features optional function fields (rclone) when: backends have highly heterogeneous optional capabilities and you want to avoid N type-assertion sites. Prefer this over many small optional interfaces when the optional capability set is large and growing.

Choose suture supervision (syncthing) when: your application is a long-running daemon that must survive subsystem failures. The pattern eliminates per-subsystem restart logic at the cost of requiring every component to implement a single Serve(ctx) error interface.

Choose a monolithic interface (MinIO) when: you have one implementation and need a testability seam. A large interface is a reasonable trade-off if it defines a complete domain contract (S3) rather than an accidental aggregation.

Avoid config stored in the store itself (MinIO) unless you are willing to handle the bootstrapping chicken-and-egg problem explicitly. For simpler systems, explicit CLI flags + env vars (restic) or a dedicated config file (rclone, syncthing) are more predictable.

Book angle#

The four projects together tell the story of storage abstraction at different granularities. restic is the exemplar of Clean Architecture applied to storage: a domain core that imports nothing, a backend interface at the seam, and decorators for cross-cutting concerns. Rclone is the exemplar of the optional-capabilities design problem: how do you express 70 backends with wildly different feature sets through one interface without making callers do N type assertions? MinIO is the exemplar of the focused monolith: one backend, deep algorithmic specialization, and a large interface that expresses one complete domain contract. Syncthing is the exemplar of daemon architecture: supervisor trees, live config reload, and component-level fault isolation applied to a storage-adjacent problem (file sync).

The lesson for practitioners: the shape of your storage interface reveals what you believe about your users. MinIO’s 40-method interface says “we are the storage layer; callers need all of S3.” Rclone’s 5-method interface says “we provide the minimum common denominator; backends decide what else they support.” restic’s 12-method interface says “we need enough to implement meaningful cross-cutting concerns as decorators.” Each is internally consistent and appropriate for its use case — the mistake is copying one project’s interface size philosophy into another project’s use case.