Rclone — Interfaces#

Interface catalog#

fs.Fs#

  • Package: github.com/rclone/rclone/fs
  • File: fs/types.go:17
  • Methods:
    List(ctx context.Context, dir string) (DirEntries, error)
NewObject(ctx context.Context, remote string) (Object, error)
Put(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) (Object, error)
Mkdir(ctx context.Context, dir string) error
Rmdir(ctx context.Context, dir string) error
// + embedded Info (Name, Root, String, Precision, Hashes, Features)
  • Purpose: The universal storage backend contract. Every one of rclone’s 70+ backends implements this interface. It is deliberately minimal — only the 5 operations every object store must support, plus Info metadata.
  • Implementations: All backend/*/ packages: s3.Fs, drive.Fs, sftp.Fs, b2.Fs, dropbox.Fs, etc. Also wrapping backends: crypt.Fs, compress.Fs, chunker.Fs, union.Fs.
  • Design quality: Exceptionally well-segregated. The required surface is kept to an absolute minimum (5 methods) to avoid burdening simple backends. All optional operations are moved to the Features struct (see below). Follows ISP strictly.

fs.Info#

  • Package: github.com/rclone/rclone/fs
  • File: fs/types.go:62
  • Methods:
    Name() string
Root() string
String() string
Precision() time.Duration
Hashes() hash.Set
Features() *Features
  • Purpose: Read-only metadata about a filesystem instance — its name, root path, modtime precision, supported hash types, and optional capabilities. Embedded in Fs.
  • Implementations: Every fs.Fs implementation; also unknownFs (zero-value sentinel).
  • Design quality: Clean separation between “what does the FS support” (Info) and “what can I do with it” (Fs operations). The Features() return is the central capability discovery point.

fs.Object#

  • Package: github.com/rclone/rclone/fs
  • File: fs/types.go:83
  • Methods:
    // Embedded ObjectInfo → DirEntry (Fs, String, Remote, ModTime, Size) + Hash, Storable
SetModTime(ctx context.Context, t time.Time) error
Open(ctx context.Context, options ...OpenOption) (io.ReadCloser, error)
Update(ctx context.Context, in io.Reader, src ObjectInfo, options ...OpenOption) error
Remove(ctx context.Context) error
  • Purpose: Represents a single file in object storage. Extends ObjectInfo (read-only metadata) with the four mutation/read operations needed by the sync engine.
  • Implementations: One per backend (e.g., s3.Object, drive.Object). Wrapping backends wrap the underlying Object.
  • Design quality: Well-balanced. The four required methods are exactly what the sync engine needs. Optional per-object capabilities (MimeTyper, IDer, GetTierer, SetTierer, Metadataer, ObjectUnWrapper) are separate single-method interfaces discovered via type assertion.

fs.DirEntry / fs.ObjectInfo / fs.Directory (embedding chain)#

  • Package: github.com/rclone/rclone/fs
  • File: fs/types.go:115–147
  • Hierarchy:
    DirEntry (Fs, String, Remote, ModTime, Size)
  └── ObjectInfo (+ Hash, Storable)
        └── Object  (+ SetModTime, Open, Update, Remove)
  └── Directory (+ Items, ID)
  • Purpose: DirEntry is the common supertype of both files and directories as returned by List(). Callers type-assert to Object or Directory as needed. This avoids a union-type and lets backends return heterogeneous listings.
  • Design quality: The three-level hierarchy is clean and predictable. FullObject and FullObjectInfo composite interfaces serve as compile-time completeness checks for wrapping backends.

Optional capability interfaces in fs/features.go (~25 interfaces)#

  • Package: github.com/rclone/rclone/fs
  • File: fs/features.go:506–811
  • Selected interfaces:
    Purger        — Purge(ctx, dir) error
Copier        — Copy(ctx, src Object, remote string) (Object, error)
Mover         — Move(ctx, src Object, remote string) (Object, error)
DirMover      — DirMove(ctx, src Fs, srcRemote, dstRemote string) error
ListRer       — ListR(ctx, dir string, callback ListRCallback) error
ListPer       — ListP(ctx, dir string, callback ListRCallback) error
PutStreamer   — PutStream(ctx, in io.Reader, src ObjectInfo, ...) (Object, error)
OpenChunkWriter — OpenChunkWriter(ctx, remote, src, ...) (ChunkWriterInfo, ChunkWriter, err)
ChunkWriter   — WriteChunk, Close, Abort
ChangeNotifier — ChangeNotify(ctx, fn, <-chan Duration)
Abouter       — About(ctx) (*Usage, error)
PublicLinker  — PublicLink(ctx, remote, expire, unlink) (string, error)
UnWrapper     — UnWrap() Fs
Wrapper       — WrapFs() Fs; SetWrapper(f Fs)
Commander     — Command(ctx, name, args, opts) (any, error)
Shutdowner    — Shutdown(ctx) error
RangeSeeker   — RangeSeek(ctx, offset, whence, length) (int64, error)
  • Purpose: Each encodes one optional backend capability. Backends implement the interfaces they support; callers check f.Features().Copy != nil (where the Features struct holds function-typed fields pointing to each method) rather than doing type assertions.
  • Implementations: Varies per backend. S3 implements Copier, Mover, ListRer, OpenChunkWriter. Google Drive implements Copier, Mover, DirMover, ChangeNotifier, etc. Simple backends may implement none.
  • Design quality: Each interface is a single-method contract — perfect ISP. The indirection through the Features struct (which aggregates function fields) means capability discovery is O(1) with no type assertion required by callers. The trade-off: Features itself becomes a large struct that grows with each new capability (~50 fields).

march.Marcher#

  • Package: github.com/rclone/rclone/fs/march
  • File: fs/march/march.go:52
  • Methods:
    SrcOnly(src fs.DirEntry) (recurse bool)
DstOnly(dst fs.DirEntry) (recurse bool)
Match(ctx context.Context, dst, src fs.DirEntry) (recurse bool)
  • Purpose: The callback protocol between the two-tree directory walker (March) and the sync engine. March.Run() walks source and destination trees in parallel and calls these three methods for each entry classification. The sync engine (syncCopyMove) implements Marcher.
  • Implementations: fs/sync.syncCopyMove (sync engine), fs/operations (check operations), other callers that need paired directory walks.
  • Design quality: Minimal and well-designed. Three methods cover all cases exhaustively (src-only, dst-only, both). The recurse bool return allows the caller to short-circuit subtree traversal, enabling pruning for performance.

vfs.Node#

  • Package: github.com/rclone/rclone/vfs
  • File: vfs/vfs.go:57
  • Methods:
    os.FileInfo  // embedded: Name, Size, Mode, ModTime, IsDir, Sys
IsFile() bool
Inode() uint64
SetModTime(time.Time) error
Sync() error
Remove() error
RemoveAll() error
DirEntry() fs.DirEntry
VFS() *VFS
Open(flags int) (Handle, error)
Truncate(size int64) error
Path() string
SetSys(any)
  • Purpose: Common interface for both Dir and File within the VFS POSIX layer. Extends os.FileInfo with VFS-specific operations. All FUSE, WebDAV, FTP, SFTP, HTTP-serve, DLNA, NFS-serve commands operate on Node values.
  • Implementations: *vfs.File and *vfs.Dir — enforced by compile-time checks (var _ Node = (*File)(nil)).
  • Design quality: Appropriately broad for a POSIX shim — it mirrors exactly what the OS-level os.FileInfo + os.File paradigm requires. The DirEntry() bridge method allows transparent transition back to the fs layer.

vfs.Handle#

  • Package: github.com/rclone/rclone/vfs
  • File: vfs/vfs.go:127
  • Methods:
    OsFiler  // embedded: all *os.File methods (Read, Write, Seek, Stat, Close, ...)
Flush() error
Release() error
Node() Node
Lock() error
Unlock() error
  • Purpose: Represents an open file or directory handle within VFS, mirroring *os.File’s API plus FUSE-specific lifecycle methods (Flush, Release). FUSE mounts and POSIX servers use this interface to abstract over ReadFileHandle, WriteFileHandle, and RWFileHandle.
  • Implementations: ReadFileHandle, WriteFileHandle, RWFileHandle, DirHandle — each embedding baseHandle for ENOSYS defaults.
  • Design quality: The baseHandle default-ENOSYS pattern allows concrete handles to implement only the subset of methods they support. Appropriate for FUSE semantics.

configmap.Getter / Setter / Mapper#

  • Package: github.com/rclone/rclone/fs/config/configmap
  • File: fs/config/configmap/configmap.go:25–42
  • Methods:
    Getter: Get(key string) (value string, ok bool)
Setter: Set(key, value string)
Mapper: Getter + Setter  // composed interface
  • Purpose: The config delivery contract between the config subsystem and backends. A Mapper is passed to every backend’s NewFs() factory. Backends call m.Get("access_key") to retrieve their config values — they never read the config file or flags directly.
  • Implementations: configmap.Map (priority-ordered list of Getters with one or more Setters), configmap.Simple (plain map[string]string, used in tests), and various adapter types for env vars, CLI flags, and the config file sections.
  • Design quality: Elegant layering. The priority queue in configmap.Map allows env vars to override config file values, which override defaults — all transparent to backends. Composing Getter+Setter into Mapper follows ISP.

fs/fserrors error classification interfaces#

  • Package: github.com/rclone/rclone/fs/fserrors
  • File: fs/fserrors/error.go
  • Interfaces:
    Retrier       — error + Retry() bool
Fataler       — error + Fatal() bool
NoRetrier     — error + NoRetry() bool
NoLowLevelRetrier — error + NoLowLevelRetry() bool
RetryAfter    — error + RetryAfter() time.Time
CountableError — error + Count(stats *StatsInfo)
  • Purpose: A type-based error classification system for the retry engine. Backends return errors implementing these interfaces to signal to the outer cmd.Run() retry loop whether the operation should be retried, aborted, or retried after a delay. Classified via liberrors.Walk (recursive unwrap traversal).
  • Implementations: retryError, wrappedRetryError, wrappedFatalError, etc. — concrete private types returned by various backends.
  • Design quality: Fine-grained and extensible. Each retry classification is an independent single-method interface. New classification categories can be added without changing existing backends. The recursive unwrap traversal handles wrapped errors correctly.

Interface patterns#

  • Size distribution: Remarkably small per interface. The core fs.Fs has 5 methods (plus embedded Info at 6); fs.Object has 4 (+embedded); all optional capability interfaces have exactly 1 method; march.Marcher has 3; vfs.Node is the outlier at ~12 (because it mirrors os.FileInfo). Average outside VFS is 1–3 methods per interface.

  • Embedding: Used heavily and consistently throughout the type hierarchy:

    • Fs embeds Info
    • Object embeds ObjectInfo which embeds DirEntry
    • Directory embeds DirEntry
    • vfs.Handle embeds OsFiler
    • vfs.Node embeds os.FileInfo
    • configmap.Mapper embeds Getter + Setter
    • FullObject / FullObjectInfo / FullDirectory embed all optional interfaces as compile-time completeness checks.

  • Implicit satisfaction: All interfaces are consumer-defined: fs.Fs is defined in the kernel; backends satisfy it implicitly. Rclone uses var _ fs.Fs = (*MyFs)(nil) compile-time assertions widely to catch missed implementations early.

  • stdlib interfaces used:

    • io.Reader, io.ReadCloser, io.ReadSeeker — heavily used in Object.Open(), Fs.Put(), ChunkWriter
    • io.WriterAt, io.Closer — composed into WriterAtCloser
    • os.FileInfo — embedded into vfs.Node
    • fmt.Stringer — DirEntry.String(), Info.String(), Noder
    • json.Unmarshaler — Flagger interface requires it for flag parsing
    • error — embedded into all fserrors classification interfaces

Key abstractions#

  1. fs.Fs (fs/types.go:17) — The project’s foundational abstraction. Every backend’s existence is defined by this interface. Its deliberate minimalism (5 methods) is what allows 70+ backends to implement it without burden.

  2. Optional capability interfaces in fs/features.go — The answer to the “optional method” problem in Go. Instead of fragmenting optionality across many small interfaces requiring type assertions everywhere, rclone routes them through Features function fields. The ~25 single-method interfaces are still defined (enabling type-safe implementation), but discovery is centralized.

  3. march.Marcher (fs/march/march.go:52) — The cleanest interface in the codebase. Three methods, complete coverage of all directory-entry pairings, recurse bool for pruning. It decouples the tree-traversal mechanism from the sync logic entirely.

  4. vfs.Node / vfs.Handle (vfs/vfs.go:57, 127) — The POSIX shim abstractions that allow 7+ serve commands (FUSE, WebDAV, SFTP, FTP, HTTP, DLNA, NFS) to share a single filesystem model without each reimplementing object-storage-to-POSIX translation.

  5. configmap.Mapper (fs/config/configmap/configmap.go:39) — The config delivery interface that gives backends clean, layered access to their configuration (file → env → flag → default) without any knowledge of the config system’s internals.

Interface-driven extensibility#

Rclone’s extensibility model is almost entirely interface-driven:

Backend extensibility: Any third-party package can become a valid rclone backend by implementing fs.Fs (5 required methods) and calling fs.Register(&RegInfo{...}) from init(). No code generation, no framework registration, no build tags required. Optional capabilities are opt-in by implementing additional single-method interfaces from fs/features.go.

Wrapping backends: The UnWrapper/Wrapper interface pair (features.go:595–606) enables a chain of decorating backends. crypt.Fs wraps any fs.Fs to add encryption; compress.Fs wraps to add compression; chunker.Fs wraps to split large files. The UnWrapFs() utility function traverses the wrapper chain to reach the base backend. This is rclone’s composability mechanism — backends as decorators.

Config extensibility: The configmap.Getter/Setter/Mapper interfaces allow new config sources (env vars, in-memory overrides, test fixtures) to be plugged in without touching backend code.

VFS extensibility: vfs.Node and vfs.Handle allow new serve commands to be added. Any command that implements its protocol (FUSE, WebDAV, NFS, etc.) in terms of Node and Handle automatically inherits all backend support and VFS caching without writing backend-specific code.

Error extensibility: The fserrors classification interfaces allow new retry semantics to be added without changing the retry engine. A backend can return a RetryAfter error to request exponential back-off to a specific time, or a Fataler error to abort immediately, purely through type-based dispatch.

Backward Rclone — Patterns Rclone — API Surface Forward