MinIO — Architecture#

Architectural style#

Layered Monolith with Distributed Storage Backend

MinIO is a single-binary, horizontally scalable object storage server. Its architecture is a layered monolith in the sense that a single minio binary handles all concerns (API, IAM, healing, replication, erasure coding), but it is designed from first principles for distributed operation across many nodes and drives. The system is organized around one central abstraction: ObjectLayer, a large interface (~40 methods) that decouples the S3 API surface from the underlying storage implementation. All application logic routes through this interface, making it the architectural keystone of the system.

The storage backend itself is distributed: data is striped using Reed-Solomon erasure coding across sets of drives, which may span multiple nodes. This gives the system Cassandra-like horizontal write/read paths while retaining a clean, interface-driven control plane.

Evidence from the code:

  • newObjectLayer() in server-main.go:1199 exclusively returns newErasureServerPools() — there is no pluggable alternative at runtime.
  • ObjectLayer in object-api-interface.go:246 is the sole gateway between HTTP handlers and storage.
  • erasureServerPools in erasure-server-pool.go:52 aggregates one or more erasureSets, each managing a group of drives as a Reed-Solomon erasure set.
  • All subsystems (IAM, lifecycle, replication, notifications) are initialized once as package-level globals during startup, then accessed via those globals throughout the codebase.

Component diagram (textual)#

┌──────────────────────────────────────────────────────────────────────┐
│                         minio binary                                 │
│                                                                      │
│  ┌──────────────┐  ┌──────────────┐  ┌────────┐  ┌──────────────┐   │
│  │  S3 API      │  │ Admin API    │  │  STS   │  │  Console UI  │   │
│  │  (S3-compat) │  │ (madmin)     │  │  API   │  │  (separate   │   │
│  │  registerAPI │  │ registerAdmin│  │        │  │   process)   │   │
│  └──────┬───────┘  └──────┬───────┘  └───┬───┘  └──────┬───────┘   │
│         │                 │              │              │            │
│         └─────────────────┴──────────────┴──────────────┘           │
│                           │                                          │
│               ┌───────────▼───────────┐                             │
│               │   ObjectLayer         │  ← central interface        │
│               │   (interface)         │                             │
│               └───────────┬───────────┘                             │
│                           │                                          │
│               ┌───────────▼───────────┐                             │
│               │  erasureServerPools   │  ← multi-pool coordinator   │
│               │  (implements          │                             │
│               │   ObjectLayer)        │                             │
│               └───────────┬───────────┘                             │
│                           │                                          │
│         ┌─────────────────┼────────────────────┐                    │
│         │                 │                    │                     │
│  ┌──────▼──────┐  ┌──────▼──────┐   ┌────────▼──────┐             │
│  │ erasureSets │  │ erasureSets │...│  erasureSets  │ ← one/pool  │
│  │ (Pool 0)    │  │ (Pool 1)    │   │  (Pool N)     │             │
│  └──────┬──────┘  └─────────────┘   └───────────────┘             │
│         │                                                            │
│  ┌──────▼──────┐                                                    │
│  │ erasureObjs │ × setCount  ← one per erasure set                 │
│  │ (set 0..N)  │                                                    │
│  └──────┬──────┘                                                    │
│         │                                                            │
│  ┌──────▼───────────────────────────────────────────────┐           │
│  │  StorageAPI[]   ← one per drive (local or remote)   │           │
│  │  xlStorage      ← local disk (POSIX)                 │           │
│  │  storageRESTClient  ← remote disk over HTTP           │           │
│  └──────────────────────────────────────────────────────┘           │
│                                                                      │
│  ──── Side subsystems (initialized as globals) ────                  │
│  globalIAMSys · globalEventNotifier · globalBucketMetadataSys        │
│  globalLifecycleSys · globalBucketTargetSys · globalTierConfigMgr    │
│  globalGrid (intra-cluster RPC) · globalDsync (distributed locking)  │
└──────────────────────────────────────────────────────────────────────┘

Core components#

ObjectLayer (interface)#

  • Package: cmd (cmd/object-api-interface.go)
  • Responsibility: Defines the complete contract for object storage operations — bucket CRUD, object CRUD, multipart uploads, listing, walking, healing, tagging, metadata, transitions, and health. Every handler goes through this interface.
  • Key types: ObjectLayer (interface), ObjectOptions (options bag), MakeBucketOptions, DeleteBucketOptions, WalkOptions
  • Dependencies: Defined in cmd; used by all HTTP handlers in the same package.

erasureServerPools#

  • Package: cmd (cmd/erasure-server-pool.go)
  • Responsibility: Top-level ObjectLayer implementation. Manages a slice of pools ([]*erasureSets). Routes object operations to the correct pool by hashing the object name using a consistent distribution algorithm. Also owns pool-level operations: rebalancing, decommissioning, and cross-pool S3 peer proxying.
  • Key types: erasureServerPools, poolMeta, rebalanceMeta, S3PeerSys
  • Dependencies: erasureSets (per pool), internal/bpool (buffer pool), internal/grid (cluster RPC).

erasureSets#

  • Package: cmd (cmd/erasure-sets.go)
  • Responsibility: Manages a single pool’s worth of erasure sets. Each erasure set is a fixed group of drives (e.g., 16 drives across nodes). Distributes object writes to the correct erasure set using a deterministic hash on the object name. Manages drive connections and reconnections.
  • Key types: erasureSets, erasureObjects (per-set)
  • Dependencies: StorageAPI (per drive), internal/dsync (distributed locking).

erasureObjects (per-set storage engine)#

  • Package: cmd (cmd/erasure-objects.go)
  • Responsibility: Implements Reed-Solomon erasure coding for one set of drives. Encodes object data into N+K shards (data + parity), writes them in parallel across drives, reads and reconstructs from quorum. Owns multipart logic at the erasure level.
  • Key types: erasureObjects, erasureInfo
  • Dependencies: github.com/klauspost/reedsolomon, StorageAPI[].

StorageAPI / xlStorage#

  • Package: cmd (cmd/xl-storage.go, cmd/xl-storage-disk-id-check.go)
  • Responsibility: Lowest storage layer. xlStorage provides POSIX filesystem operations on a single local drive. xlStorageDiskIDCheck wraps it with drive identity validation. storageRESTClient is the remote counterpart, tunneling drive ops over HTTP to peers.
  • Key types: StorageAPI (interface), xlStorage, xlStorageDiskIDCheck, storageRESTClient
  • Dependencies: OS filesystem APIs, internal/ioutil, internal/hash.

HTTP API layer (S3 + Admin + STS)#

  • Package: cmd (cmd/api-router.go, cmd/routers.go, cmd/*-handlers.go)
  • Responsibility: Parses incoming HTTP requests and maps them to ObjectLayer operations. Uses github.com/minio/mux (a fork of gorilla/mux). Three main route groups: S3 API (registerAPIRouter), Admin API (registerAdminRouter), and STS (registerSTSRouter). Middleware chain handles authentication, request ID injection, tracing, and CORS.
  • Key types: No special types — handlers are functions that receive http.ResponseWriter and *http.Request, resolve globalObjectAPI, and delegate.
  • Dependencies: ObjectLayer (via newObjectLayerFn() — a global accessor), globalIAMSys, globalEventNotifier.

IAM subsystem#

  • Package: cmd (cmd/iam*.go)
  • Responsibility: Manages users, service accounts, groups, and policy enforcement. Stores IAM state in the object store itself (in a special .minio.sys/ bucket). Supports LDAP and OpenID Connect identity providers via internal/config/identity.
  • Key types: IAMSys, IAMCache, IAMObjectStore, IAMEtcdStore
  • Dependencies: ObjectLayer, globalEtcdClient (optional), internal/auth, internal/jwt.

internal/grid (cluster RPC)#

  • Package: internal/grid
  • Responsibility: Custom multiplexed RPC layer for intra-cluster communication. Uses WebSocket as the transport with binary-framed messages. Replaces the earlier REST-based storage peer protocol for performance. Typed handler registration with code-generated message types.
  • Key types: Manager, Connection, Handler, typed message wrappers
  • Dependencies: golang.org/x/net/websocket, msgpack codec.

internal/dsync (distributed locking)#

  • Package: internal/dsync
  • Responsibility: Distributed reader-writer mutex using quorum-based consensus. All nodes in a cluster must agree (quorum = N/2+1) before a lock is granted. Used to serialize bucket and object metadata mutations across nodes.
  • Key types: DRWMutex, Locker (interface)
  • Dependencies: internal/grid (for peer lock RPC calls).

Data flow#

Typical S3 PutObject request:

Client HTTP PUT /bucket/object
  │
  ▼
mux.Router (github.com/minio/mux)
  │  globalMiddlewares applied:
  │  addCustomHeadersMiddleware → httpTracerMiddleware → authMiddleware → ...
  ▼
objectAPIHandler.PutObjectHandler  (cmd/object-handlers.go)
  │  1. Authenticate request (IAM + SigV4 verification)
  │  2. Check bucket policy
  │  3. Build ObjectOptions (SSE, versioning, checksums)
  │  4. Wrap request body in hash.Reader + PutObjReader
  ▼
globalObjectAPI.PutObject(ctx, bucket, object, reader, opts)
  │  globalObjectAPI is *erasureServerPools
  ▼
erasureServerPools.PutObject()
  │  1. Hash object name → select pool (by distribution algo)
  │  2. Check if pool is being decommissioned/rebalanced → skip
  ▼
erasureSets.PutObject()
  │  1. Hash object name → select erasure set within pool
  │  2. Acquire distributed namespace lock (dsync)
  ▼
erasureObjects.PutObject()
  │  1. Determine parity count (storage class, max parity flag)
  │  2. Encode data into N+K shards via Reed-Solomon (klauspost/reedsolomon)
  │  3. Write xl.meta (object metadata) + data shards to StorageAPI[] in parallel
  │  4. Wait for write quorum (N/2+1 shards written)
  │  5. Return ObjectInfo
  ▼
StorageAPI.WriteAll() / WriteMetadata()
  │  Local: xlStorage → POSIX write to disk
  │  Remote: storageRESTClient → HTTP PUT to peer → xlStorage on peer
  ▼
Response: 200 OK with ETag + version ID
  │
  ▼ (async post-write)
globalEventNotifier.Publish(EventPut)  → Kafka/NATS/Redis/Webhook targets
globalReplicationState.queueReplication(...)  → background replication goroutine
globalLifecycleSys (scanner) picks up ILM on next cycle

Typical S3 GetObject request:

Client HTTP GET /bucket/object
  ▼
authMiddleware → PutObjectHandler equivalent: GetObjectHandler
  ▼
globalObjectAPI.GetObjectNInfo(ctx, bucket, object, range, headers, opts)
  ▼
erasureServerPools → erasureSets → erasureObjects.GetObjectNInfo()
  │  1. Locate object's xl.meta from quorum of disks
  │  2. Determine best erasure set and drives
  │  3. Read data shards from available drives in parallel (bitrot-verified)
  │  4. Reconstruct via Reed-Solomon if needed (< quorum drives available)
  │  5. Return GetObjectReader (streaming, range-aware)
  ▼
Response streamed back to client

Initialization / Bootstrap#

The startup sequence in serverMain() (cmd/server-main.go:746) is explicitly traced with bootstrapTrace() — every step is instrumented:

  1. Logger init — sets up globalConsoleSys, configures log rotation.
  2. Env loading — loadEnvVarsFromFiles() then serverHandleEarlyEnvVars().
  3. Arg parsing — buildServerCtxt() → serverHandleCmdArgs() parses disk layout (local paths or distributed URIs), TLS certs, and TCP options. Sets globalIsErasure, globalIsDistErasure.
  4. Self-tests — bitrotSelfTest(), erasureSelfTest(), compressSelfTest() verify runtime correctness of cryptographic and erasure primitives.
  5. KMS init — handleKMSConfig() connects to KES (Key Encryption Service) if configured.
  6. Root credentials — loaded from environment or YAML config; node auth token generated for peer-to-peer trust.
  7. Subsystem construction — initAllSubsystems() constructs all global subsystem objects (IAM, events, bucket metadata, lifecycle, SSE, object lock, quotas, versioning, replication, ILM tier, transition state). At this point they are empty shells, not yet loaded from disk.
  8. Grid + Lock Grid init — initGlobalGrid() + initGlobalLockGrid() start the WebSocket-based intra-cluster RPC grid. This is where peers register typed handler endpoints.
  9. HTTP server start — configureServerHandler() builds the mux.Router with all API routes. HTTP server starts listening immediately (before data is loaded) so peers can communicate during boot.
  10. Object layer creation — newObjectLayer() → newErasureServerPools() formats/validates disks, connects to all peer drives, and returns a fully operational ObjectLayer.
  11. Quorum wait — busy-polls newObject.Health() until read quorum is achieved.
  12. Config subsystem init — initServerConfig() reads cluster config from the object store (.minio.sys/config/) with retry logic, then re-initializes all subsystems with live data.
  13. Background services (goroutines):
    • globalIAMSys.Init() — loads users/policies from object store.
    • initConsoleServer() — starts the embedded web console.
    • startFTPServer() / startSFTPServer() — optional protocol gateways.
    • initDataScanner() — background crawler for usage accounting and ILM.
    • initBackgroundReplication() / initBackgroundExpiry() — async replication and lifecycle expiry workers.
    • globalTransitionState.Init() / globalTierConfigMgr.Init() — tiered storage (ILM transition to hot/cold tiers).
    • globalEventNotifier.InitBucketTargets() — connects event notification targets (Kafka, NATS, etc.).
    • globalBucketMetadataSys.Init() — loads all bucket configs from disk.

No DI framework is used. All wiring is manual: subsystem pointers are stored in package-level var global* variables (e.g., globalIAMSys, globalObjectAPI, globalGrid) and accessed directly by all callers. The initialization order is strictly controlled by the ordering of bootstrapTrace() calls in serverMain().

Configuration#

MinIO uses a layered configuration system:

  1. CLI flags — minio server --address :9000 /mnt/data{1...4}. Defines listen address, TLS, FTP/SFTP, idle timeouts, log config.
  2. Environment variables — MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, MINIO_ADDRESS, MINIO_VOLUMES, MINIO_KMS_*, etc. These take precedence and can override almost any behavior.
  3. YAML config file — minio server --config /path/to/config.yaml. A structured format (config.ServerConfigV1 / config.ServerConfig) that can specify server address, credentials, TLS, FTP/SFTP, and pool layout. Added for containerized/Kubernetes deployments.
  4. Object-store config — Per-cluster settings (API rate limits, notification targets, storage class ratios, scanner settings, etc.) are stored as JSON blobs in a special .minio.sys/config/ prefix in the object store itself, loaded by globalConfigSys.Init(newObject) at startup. This allows cluster-wide configuration changes without restarting all nodes.

The internal/config/* packages define strongly-typed config structs for each feature domain (API, batch, compress, DNS, etcd, heal, ILM, notify, storageclass, etc.), each with LookupConfig() functions that consult both the object-store config and environment variables.

Key design decisions#

  1. ObjectLayer as the single seam. Every S3 and Admin API handler accesses storage exclusively through the ObjectLayer interface (cmd/object-api-interface.go:246). This 40-method interface is the one place where the HTTP world meets the storage world. It makes the system testable (a gatewayLayer or stub can be injected) and defines a clean, S3-shaped contract.

  2. Erasure coding as the only storage model. MinIO’s distributed mode uses Reed-Solomon erasure coding everywhere — there is no replication-based mode in the current codebase. The newObjectLayer() function always returns newErasureServerPools(). This is a deliberate simplification: one storage algorithm, one set of consistency semantics, one healing path. The trade-off is write amplification (N+K shards) vs. replication factor savings.

  3. Global variables as the DI pattern. Rather than using a DI framework or passing dependencies down through call stacks, MinIO stores all major subsystem pointers as package-level globals (globalIAMSys, globalObjectAPI, globalEventNotifier, etc.). This makes the initialization order explicit but creates tight coupling — any code in the cmd package can access any subsystem directly. The upside is zero boilerplate; the downside is that testing requires careful global state management.

  4. Custom WebSocket-based grid RPC (internal/grid) instead of gRPC. MinIO built its own multiplexed binary RPC layer on top of WebSocket rather than using gRPC. Rationale (from code comments and design): tighter control over connection pooling, framing, and load distribution; avoids the gRPC HTTP/2 multiplexing overhead for the high-throughput storage path; leverages existing HTTP infrastructure (load balancers, TLS termination).

  5. Config stored in the object store itself. Cluster-wide configuration (notification targets, storage class ratios, bucket policies) is persisted in .minio.sys/config/ — stored using the same erasure-coded object store that MinIO provides to users. This creates a self-referential dependency (the cluster must be healthy to read its own config) but eliminates the need for an external config store (no etcd dependency by default). The retry loop in initServerConfig() handles the chicken-and-egg startup problem.

Backward MinIO — Dependencies MinIO — Patterns Forward