Rclone — Architecture#

Architectural style#

Microkernel + Init-based Plugin Registry

Rclone is organized around a small, stable kernel (fs/) that defines abstract contracts and a runtime registry, surrounded by 70+ independently compiled plugins (backends) that self-register via Go’s init() mechanism. This resembles a microkernel architecture: the kernel defines the abstractions and orchestration logic; the plugins are loaded by blank-import aggregators at program start. No DI framework, no plugin loader binary — just init() functions and a global []*RegInfo slice.

Evidence:

• rclone.go is 15 lines: main() calls cmd.Main() after blank-importing backend/all and cmd/all
• backend/all/all.go blank-imports every backend package; each package’s init() calls fs.RegisterFs()
• fs/registry.go defines var Registry []*RegInfo and Register() — the central dispatch table
• The kernel (fs/) has zero imports from backend/ or cmd/; dependency direction is strictly one-way

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────┐
│                         rclone.go                            │
│  main() → cmd.Main()                                         │
│  blank imports: backend/all, cmd/all, lib/plugin             │
└───────────────┬─────────────────────────────────────────────┘
                │
      ┌─────────▼──────────┐
      │   cmd/ (Cobra CLI) │  ← cmd/all blank-imports all commands
      │  ~30+ subcommands  │    each registers itself in init()
      └─────────┬──────────┘
                │ uses
      ┌─────────▼──────────────────────────────────────────────┐
      │                   fs/ (kernel)                          │
      │                                                         │
      │  types.go       — Fs, Object, DirEntry interfaces       │
      │  registry.go    — []*RegInfo, Register(), NewFs()       │
      │  features.go    — optional Features struct (fn fields)  │
      │  operations/    — CopyFile, MoveFile, Check, Dedupe     │
      │  sync/          — syncCopyMove pipeline engine          │
      │  march/         — parallel two-tree directory walker    │
      │  walk/          — recursive listing helpers             │
      │  accounting/    — transfer stats + bandwidth throttle   │
      │  cache/         — Fs instance LRU cache                 │
      │  rc/            — JSON-RPC call registry + HTTP server  │
      │  config/        — INI config file + flag binding        │
      │  filter/        — include/exclude rule engine           │
      │  fshttp/        — shared HTTP transport                 │
      │  hash/          — hash type registry                    │
      └──────┬──────────────────────┬──────────────────────────┘
             │                      │
   ┌─────────▼────────┐    ┌────────▼──────────┐
   │  backend/ (70+)  │    │  vfs/ (POSIX layer)│
   │  s3, drive, sftp │    │  VFS wraps fs.Fs   │
   │  b2, dropbox ...  │    │  dir.go, file.go   │
   │  each impl. fs.Fs │    │  vfscache/         │
   └──────────────────┘    └────────┬───────────┘
                                    │ used by
                         ┌──────────▼───────────────────┐
                         │  cmd/mount, cmd/cmount        │
                         │  cmd/serve/{http,ftp,sftp,    │
                         │    webdav,dlna,docker,nfs}    │
                         └──────────────────────────────┘

   lib/           — shared utilities (no cmd/backend imports)
   ├── rest/      — generic REST client
   ├── oauthutil/ — OAuth2 token flows
   ├── pacer/     — retry + rate-limit pacer
   ├── dircache/  — directory ID cache for hierarchical APIs
   ├── http/      — HTTP server used by serve/* commands
   └── ...

   librclone/     — CGo C shared library
   └── exposes fs/rc JSON-RPC API; same wire protocol as rclone rcd

Core components#

fs.Fs Interface#

• Package: github.com/rclone/rclone/fs
• File: fs/types.go
• Responsibility: The universal contract every storage backend must implement. Provides list, get, put, mkdir, rmdir. Five methods on Fs; additional optional capabilities are declared in Features.
• Key types: Fs, Object, ObjectInfo, DirEntry, Info
• Dependencies: fs/hash (hash type registry), stdlib io, context

Backend Registry#

• Package: github.com/rclone/rclone/fs
• File: fs/registry.go
• Responsibility: Global []*RegInfo slice. Each RegInfo holds the backend name, a NewFs factory function, declared Options, and optional Config function. Backends call fs.Register(&RegInfo{...}) from their init().
• Key types: RegInfo, Options, Option
• Dependencies: fs/config/configmap, fs/config/configstruct

Features (optional capabilities)#

• Package: github.com/rclone/rclone/fs
• File: fs/features.go
• Responsibility: The Features struct combines boolean flags (e.g., CaseInsensitive, BucketBased) with optional function fields (Purge, Copy, Move, DirMove, PutStream, OpenChunkWriter, etc.). Backends fill in only the capabilities they support; nil function fields mean “not supported”. Callers check f.Features().Copy != nil before calling.
• Key types: Features
• Dependencies: stdlib only

Fs Instance Cache#

• Package: github.com/rclone/rclone/fs/cache
• File: fs/cache/cache.go
• Responsibility: LRU cache mapping "remote:path" strings to live fs.Fs instances. cache.Get(ctx, "s3:mybucket") returns an existing instance or calls RegInfo.NewFs() to create one. Prevents redundant connection setup for repeated references to the same remote in one invocation.
• Key types: cache.Cache (from lib/cache)
• Dependencies: fs, lib/cache, fs/filter

Sync Engine#

• Package: github.com/rclone/rclone/fs/sync
• File: fs/sync/sync.go
• Responsibility: Implements Sync, CopyDir, MoveDir. Internally uses syncCopyMove struct with a two-stage pipeline: checker goroutines compare src/dst objects; copier goroutines transfer files that differ. Channels (toBeChecked, toBeUploaded) connect the stages.
• Key types: syncCopyMove, pipe
• Dependencies: fs, fs/march, fs/operations, fs/accounting, fs/filter, fs/hash

March (Two-tree Walker)#

• Package: github.com/rclone/rclone/fs/march
• Responsibility: Walks source and destination directory trees simultaneously, pairing up matching entries. Emits callbacks for same-named entries, src-only entries, and dst-only entries. The sync engine’s logic sits entirely in these callbacks.
• Key types: March struct, Marcher interface

VFS Layer#

• Package: github.com/rclone/rclone/vfs
• File: vfs/vfs.go
• Responsibility: Wraps an fs.Fs to provide POSIX filesystem semantics (Stat, Open, Read, Write, Seek, Truncate). All FUSE mounts, WebDAV, SFTP-serve, FTP-serve, DLNA, NFS-serve, and Docker volume plugin use this single VFS struct. vfscache/ provides a local disk cache for pending writes.
• Key types: VFS, Node interface, Dir, File, Handle
• Dependencies: fs, fs/cache, fs/walk, fs/rc, vfs/vfscache, go-billy

Remote Control (rc)#

• Package: github.com/rclone/rclone/fs/rc
• File: fs/rc/rc.go
• Responsibility: A JSON-RPC call registry and HTTP server. Packages throughout the codebase call rc.Add("operations/copyfile", fn) to register callable endpoints. rclone rcd starts this server; rclone rc sends calls to it. librclone also uses this same protocol, making the embedded API identical to the daemon API.
• Key types: Calls registry, Params map, Call struct
• Dependencies: fs, lib/http

Accounting#

• Package: github.com/rclone/rclone/fs/accounting
• Responsibility: Global transfer statistics (bytes transferred, errors, ETA), per-transfer progress tracking, bandwidth throttling (--bwlimit), and stats logging. The sync engine and operations package route all transfers through accounting.Stats.
• Key types: StatsInfo, Account, Pacer (via lib/pacer)

Data flow#

Typical operation: rclone sync s3:mybucket gdrive:myfolder

1. main() → cmd.Main() → cobra dispatches to cmd/sync/sync.go RunE
   
2. cmd/sync/sync.go calls:
   fsrc = cache.Get(ctx, "s3:mybucket")    // instantiates S3 backend
   fdst = cache.Get(ctx, "gdrive:myfolder") // instantiates Drive backend
   fssync.Sync(ctx, fdst, fsrc, createEmptySrcDirs)

3. fs/sync/sync.go creates syncCopyMove{fdst, fsrc, ...}
   Starts goroutines:
   - march.Run() walks fsrc and fdst trees in parallel via List()
   - March emits paired entries to dstFiles map and srcFiles map

4. For each matched/unmatched pair:
   - Src-only → sent to toBeChecked pipe (checkerFn evaluates if transfer needed)
   - Dst-only → collected for potential deletion
   - Both exist → checker compares hash/modtime/size

5. Checker pipeline (ci.Checkers goroutines):
   - Calls operations.CheckIdentical(src, dst) 
   - If differs → sends to toBeUploaded pipe

6. Copier pipeline (ci.Transfers goroutines):
   - Calls operations.CopyFile(ctx, fdst, fsrc, dstFileName, srcFileName)
   - Which calls: fsrc.NewObject → obj.Open → fdst.Put (streaming transfer)
   - Wrapped in accounting.Account for stats/throttling

7. Deletions: after transfers complete, delete dst-only objects if mode=sync

8. cmd.Run() loop: on error, may retry entire operation (--retries flag)

9. cleanup: cache.Clear() shuts down all cached Fs instances

Concrete transfer path (step 6 expanded):

obj.Open(ctx, options) → io.ReadCloser from S3 (HTTP GET stream)
  ↓ 
accounting.Account wraps reader (throttles bandwidth, tracks bytes)
  ↓
asyncreader may buffer ahead (--buffer-size)
  ↓
fdst.Put(ctx, in, srcObj, options) → Google Drive backend
  drives multipart upload or resumable upload via lib/rest + lib/pacer

Initialization / Bootstrap#

1. Go runtime fires all init() functions:
   - backend/*/init() → each calls fs.Register(&RegInfo{...})
     Registers backend name, NewFs factory, Options declarations
   - cmd/*/init() → each calls cmd.Root.AddCommand(...)
     Registers cobra subcommand

2. main() calls cmd.Main():
   a. cobra root command is constructed in cmd/cmd.go with global flags
   b. Config loading is lazy: first call to fs.GetConfig(ctx) or cache.Get()
      triggers reading ~/.config/rclone/rclone.conf via fs/config/configfile

3. cobra parses argv, dispatches to matching command's RunE

4. RunE typically calls cmd.NewFsSrcDst(args):
   - fspath.SplitFs("s3:mybucket") → backend name "s3", path "mybucket"
   - cache.Get(ctx, remote) → looks up "s3" in Registry → calls s3.NewFs()
   - s3.NewFs() reads config via configmap.Mapper, creates aws SDK session

5. cmd.Run(retry, showStats, cmd, func() error {...}):
   - Runs the operation with retry loop
   - Starts stats reporting goroutine if --stats set
   - Calls atexit hooks on completion

Dependency injection: None — rclone uses manual global state everywhere. The fs.Registry global, accounting.GlobalStats() singleton, and fs.GetConfig(ctx) are all package-level globals. context.Context carries per-operation config overrides (fs.AddConfig(ctx, ci)) allowing config mutation without globals — this is the primary DI mechanism for config.

Configuration#

Key config subsystem: fs/config/configstruct uses reflection to map config file keys and env vars to struct fields of backend option types. Backend authors declare Options []fs.Option in their RegInfo and get a configmap.Mapper in NewFs() — they never parse flags directly.

No Viper. Rclone maintains its own config subsystem built on configmap, configstruct, and configfile.

Key design decisions#

1. init()-based plugin registration with blank-import aggregators. The 15-line rclone.go achieves full capability through blank imports alone. A custom build can include any subset of backends by writing a replacement backend/all/all.go. No plugin framework binary, no runtime loading, no code generation required. Trade-off: all backends are linked into the binary regardless of use — the binary is large (~80MB+).

2. fs.Features as a struct of optional function fields. Rather than fragmenting optional capabilities into dozens of small interfaces (Purger, Copier, Mover, DirMover…) and forcing callers to do type assertions, rclone wraps all optional operations into a single Features struct with function-typed fields. nil means “not supported”; non-nil means “call this”. This simplifies capability discovery to a single f.Features() call and keeps the optional surface in one place. Trade-off: Features is a large struct (~50+ fields) that grows with each new optional capability.

3. Two-stage pipeline for sync (checkers + copiers). The sync engine uses two explicitly sized goroutine pools connected by buffered channel pipelines (toBeChecked, toBeUploaded). This lets checkers and copiers run concurrently and at different rates (more checkers than copiers is typical). The pipeline depth is bounded by --checkers and --transfers flags, giving users direct control over concurrency.

4. context.Context carries config overrides. fs.AddConfig(ctx, &ConfigInfo{...}) attaches a modified config to a context; fs.GetConfig(ctx) reads it. This avoids global mutation for per-operation settings (e.g., bandwidth limits per command). The same pattern is used for filters (filter.AddConfig) and accounting. Backends that need config pass the context through all calls.

5. VFS as a shared POSIX shim. Rather than each serve command (WebDAV, SFTP, FTP, HTTP, FUSE) implementing its own filesystem model over object storage, all share vfs.VFS. This amortizes the complexity of translating POSIX semantics (seek, truncate, atomic rename) to object-storage semantics across one package instead of N. vfscache/ handles write buffering, enabling write-back caching to local disk before upload.

6. fs/rc as universal embedding API. The librclone C shared library does not expose rclone functions directly — it exposes the same JSON-RPC call registry used by rclone rcd. This means embedding rclone (in Python, PHP, mobile apps) uses the identical protocol as the network daemon, with zero additional API maintenance burden. The trade-off is that embedders must speak JSON-RPC rather than calling Go functions.