Kubernetes — Architecture#

Architectural style#

Distributed control plane — event-driven reconciliation loops over a shared declarative state store

Kubernetes is not a single monolith, nor a conventional microservices system. It is a set of loosely coupled control loops (controllers) that each watch a shared state store (etcd, via the API server) and continuously drive actual state toward desired state. This is the controller pattern or reconciliation loop, and it pervades every part of the system.

The overarching structure is a federated control plane with:

• A central REST API server (kube-apiserver) as the sole source of truth and the only component that writes directly to etcd.
• Stateless, pluggable controllers (kube-controller-manager) that subscribe to API events and reconcile.
• A plugin-based scheduler (kube-scheduler) that assigns pending pods to nodes.
• A node agent (kubelet) on every host that enforces the desired pod state at the OS/container level.
• A network proxy (kube-proxy) that maintains service load-balancing rules in the kernel.

The components interact only through the API server — they never call each other directly. This makes the system resilient: any component can crash and restart and will pick up where it left off by re-reading current state.

Component diagram (textual)#

┌─────────────────────────────────────────────────────────────────┐
│  Control Plane (typically runs on dedicated master nodes)        │
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  kube-apiserver (aggregation chain)                       │   │
│  │  ┌─────────────────┐  ┌──────────────────┐  ┌─────────┐ │   │
│  │  │  APIAggregator  │→ │  KubeAPIServer   │→ │  CRD    │ │   │
│  │  │  (kube-agg)     │  │  (controlplane)  │  │ APIExt  │ │   │
│  │  └────────┬────────┘  └───────┬──────────┘  └────┬────┘ │   │
│  │           │                   │                   │       │   │
│  │    HTTP handler chain (auth → authz → admission → storage)│   │
│  └───────────────────────────────┼───────────────────────────┘   │
│                                  │ etcd                           │
│                             ┌────▼────┐                           │
│                             │  etcd   │                           │
│                             └─────────┘                           │
│                                                                   │
│  ┌──────────────────────┐   ┌──────────────────────────────┐    │
│  │ kube-controller-mgr  │   │      kube-scheduler           │    │
│  │  ~40 controllers,    │   │  plugin framework:            │    │
│  │  each watching API   │   │  Filter → Score → Bind        │    │
│  │  resources via       │   │  backed by in-memory cache    │    │
│  │  SharedInformers     │   │  of cluster state             │    │
│  └──────────────────────┘   └──────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│  Node (runs on every worker node)                                │
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  kubelet                                                  │   │
│  │  watches API for Pods bound to this node                  │   │
│  │  manages: CRI → container runtime, CSI → volumes,        │   │
│  │           CNI → networking, cAdvisor → monitoring        │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │  kube-proxy                                               │   │
│  │  watches Services/EndpointSlices → iptables/ipvs rules   │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Core components#

kube-apiserver — The Aggregated API Server Chain#

• Package: k8s.io/kubernetes/pkg/controlplane + staging/src/k8s.io/apiserver/pkg/server

• Responsibility: Validates, persists, and serves all cluster state via a versioned REST API. It is the only component that reads/writes etcd. It also enforces authentication, authorization, and admission policies on every request.

• Key types:

  • genericapiserver.GenericAPIServer — the generic REST server framework (auth chain, handler registration, health endpoints)
  • controlplane.Instance — the Kubernetes-specific API server layer that installs all core API groups
  • aggregatorapiserver.APIAggregator — the top-level server that proxies requests to the core server or to aggregated API servers (e.g., metrics-server)

• Architecture detail — delegation chain: The API server is assembled as three nested servers (from CreateServerChain in cmd/kube-apiserver/app/server.go):

  1. CRD extensions server (apiextensions-apiserver) — handles CustomResourceDefinitions
  2. Core API server (controlplane) — handles all built-in resources (Pods, Services, Deployments, …)
  3. Aggregator server (kube-aggregator) — sits at the top; proxies non-local API groups to external API servers

  Each layer delegates unrecognized requests to the layer below it. Requests that fall off the end hit a notFoundHandler.

• Request pipeline: Every HTTP request traverses a handler chain built in staging/src/k8s.io/apiserver/pkg/server/:

  TLS termination → panic recovery → request ID → real client IP
  → auth filters → impersonation → CORS → timeout
  → authN (token, cert, OIDC, …) → authZ (RBAC, Node, Webhook)
  → admission (mutation webhooks → validating webhooks → built-in plugins)
  → REST storage handler → codec (JSON/YAML/Protobuf) → etcd

• Dependencies: etcd (via go.etcd.io/etcd/client/v3), k8s.io/apimachinery, k8s.io/apiserver, all pkg/registry/* packages for per-resource storage.

kube-controller-manager — Reconciliation Engine#

• Package: k8s.io/kubernetes/cmd/kube-controller-manager/app, k8s.io/kubernetes/pkg/controller/*
• Responsibility: Runs ~40 built-in reconciliation controllers in a single process. Each controller watches API objects, computes the delta between desired state and actual state, and issues API calls to close the gap.
• Key types:
  • ControllerDescriptor — a descriptor struct for each controller: its name, InitFunc, and RBAC requirements
  • ControllerContext — shared context passed to all controllers: informer factories, client set, config
  • informers.SharedInformerFactory — produces shared, cached, watch-based iterators over API resources
• Controller pattern: Each controller follows this lifecycle:

  SharedInformer (watches API, populates local cache)
    → work queue (rate-limited, deduplicating)
    → worker goroutines (call reconcile loop)
    → reconcile: read from cache, compare, write to API if needed

• Leader election: Only one instance of the controller manager runs at a time. Leader election uses a Lease resource in the API (etcd-backed) to implement a distributed lock (k8s.io/client-go/tools/leaderelection).
• Dependencies: k8s.io/client-go (typed clients + informers), all pkg/controller/* packages.

kube-scheduler — Declarative Placement Engine#

• Package: k8s.io/kubernetes/pkg/scheduler, k8s.io/kubernetes/pkg/scheduler/framework
• Responsibility: Watches for unscheduled Pods and assigns them to nodes by running a pipeline of plugins. Writes the assignment as a Binding object back to the API server.
• Key types:
  • Scheduler (pkg/scheduler/scheduler.go) — top-level struct; owns a Cache (in-memory cluster state snapshot), a priority queue of unscheduled pods, and a set of profiles
  • framework.Framework (staging/src/k8s.io/kube-scheduler/framework/) — the plugin runner; executes plugins at defined extension points
• Plugin extension points (in execution order):

  PreEnqueue → QueueSort → PreFilter → Filter → PostFilter
  → PreScore → Score → NormalizeScore → Reserve → Permit
  → WaitOnPermit → PreBind → Bind → PostBind

  Each extension point is an interface; multiple plugins can register for each. This replaced the old “predicates and priorities” model in Kubernetes 1.15.
• In-memory cache: The scheduler maintains its own snapshot of node and pod state (separate from the API server) for performance. The cache (pkg/scheduler/backend/cache) is synchronized via informers, not direct etcd reads.
• Dependencies: k8s.io/client-go (read cluster state via informers), k8s.io/kube-scheduler/framework (extension point interfaces).

kubelet — Node Agent#

• Package: k8s.io/kubernetes/pkg/kubelet
• Responsibility: The node-level daemon that enforces pod specs. It watches the API server for Pods scheduled to its node, instructs the container runtime to start/stop containers (via CRI), manages volumes (via CSI), configures networking (via CNI), and reports node/pod status back to the API server.
• Key interfaces (runtime adaptors):
  • CRI (k8s.io/cri-api): gRPC interface to container runtimes (containerd, CRI-O, …) — RuntimeService + ImageService
  • CSI (Container Storage Interface): standardized gRPC for volume provisioning and mounting
  • CNI (Container Network Interface): plugin-based binary protocol for pod networking
• Pod lifecycle management: The kubelet runs a syncPod loop driven by a Pod Lifecycle Event Generator (PLEG) that polls the CRI for container state changes. Changes flow:

  PLEG event → kubelet work queue → syncPod → CRI create/start/stop

• Dependencies: k8s.io/client-go, k8s.io/cri-api, cAdvisor (container resource metrics), opentelemetry (tracing).

client-go — Shared Client Infrastructure#

• Package: staging/src/k8s.io/client-go
• Responsibility: The official Go client for the Kubernetes API, used by all control plane components and external operators. Provides typed clients, a dynamic client, informers (watch + local cache), and work queues.
• Key types:
  • SharedInformerFactory — creates shared, cached watch-based informers for API types; the backbone of all controllers
  • workqueue.RateLimitingInterface — deduplicating, rate-limited queue used by every controller’s reconcile loop
  • Clientset — typed clients for every API group/version/resource
• Significance: client-go is the most widely used component in the Kubernetes ecosystem. Every operator, CRD controller, and custom scheduler written by third parties depends on it.

apimachinery / apiserver — Generic Foundations#

• Package: staging/src/k8s.io/apimachinery, staging/src/k8s.io/apiserver
• Responsibility: apimachinery defines the core type system (runtime.Object, runtime.Scheme, ObjectMeta, codec framework, GVR/GVK). apiserver provides the generic REST server framework (handler chain, storage, admission) that any Kubernetes-style API server (including operators via controller-runtime) can embed.
• Key types:
  • runtime.Scheme — type registry that maps Go types to API group/version/kind
  • runtime.Object — the root interface that all API types implement
  • admission.Interface — the admission plugin interface; all admission plugins (built-in and webhook) implement this
  • storage.Interface — the storage backend interface; the etcd3 implementation is the only production backend

Data flow#

API write (e.g., kubectl apply creating a Deployment)#

kubectl → HTTPS POST /apis/apps/v1/namespaces/default/deployments
  → kube-apiserver aggregator layer (proxy routing)
  → core API server handler chain:
      authN: verify bearer token or client cert
      authZ: RBAC check (can this user create Deployments?)
      admission: mutating webhooks → LimitRanger → ResourceQuota → …
                 validating webhooks → custom validators
      decode: JSON body → internal Go type (via scheme + codec)
      validate: structural validation (field constraints)
      storage: write to etcd via storage.Interface (etcd3 backend)
               set resourceVersion (etcd revision)
      encode: response → JSON → kubectl
  → etcd stores the resource, bumps revision

Controller reconciliation (e.g., Deployment controller)#

etcd resource change
  → kube-apiserver watch stream (chunked HTTP/2)
  → client-go SharedInformer (local cache update)
  → event handler fires → key added to work queue
  → worker goroutine dequeues key
  → Deployment controller reads Deployment + ReplicaSets from local cache
  → computes desired vs actual ReplicaSets
  → issues API call: create/delete/update ReplicaSet
  → etcd updated → triggers further watch events (cascade)

Scheduling a Pod#

Pod created (no nodeName) → appears in scheduler's unscheduled queue
  → scheduler pops pod from queue
  → runs Filter plugins against all nodes (parallel)
  → runs Score plugins on passing nodes → normalize → rank
  → selects best node
  → issues Binding (or writes nodeName via Update) to API server
  → kubelet on target node sees Pod via watch → runs syncPod → CRI

Initialization / Bootstrap#

kube-apiserver startup sequence (from cmd/kube-apiserver/app/server.go):

1. options.NewServerRunOptions() — build default flag-backed config
2. s.Complete(ctx) — resolve defaults, validate, build derived config
3. NewConfig(opts) → builds three configs (aggregator, kubeAPI, apiExtensions)
4. config.Complete() → wire informers, clients, admission plugins
5. CreateServerChain(completed):
   • New(CRD server) — apiextensions-server with empty delegate
   • New(KubeAPI server) — registers all core REST storage installers
   • CreateAggregatorServer(...) — wraps both behind aggregator
6. server.PrepareRun() — install health endpoints, OpenAPI handlers, post-start hooks
7. prepared.Run(ctx) — start HTTPS listener, shared informers, post-start hooks, signal handler

No dependency injection framework is used. Wiring is manual: each New(config) function receives all its dependencies explicitly via the config/options structs. This is the dominant pattern across all five binaries.

Configuration#

All components use the same two-layer configuration pattern:

1. options.*Options — flag-defined, directly populated by cobra/pflag. Holds raw string/int values.
2. *Config / CompletedConfig — derived configuration built by options.Complete(). Holds resolved types (clients, informers, net.IP, tls.Config, etc.).

Flags are organized into named flag sets (cliflag.NamedFlagSets) grouping related flags for better --help output. The flagz endpoint (/flagz) exposes the active flag values over HTTP for debugging.

Feature gates (k8s.io/apiserver/pkg/util/feature, k8s.io/component-base/featuregate) guard experimental functionality and are set via --feature-gates=Alpha1=true,Beta2=false. There is no Viper usage — all configuration comes from flags (in production) or direct struct initialization (in tests).

Key design decisions#

1. API server as the single source of truth. No component stores persistent state locally. All state lives in etcd accessed only via the API server. This makes components stateless (crash-safe) and enables the watch-based event model. The tradeoff is that the API server and etcd are the critical bottleneck for scalability.

2. Level-triggered (not edge-triggered) reconciliation. Controllers do not act on individual events; they act on the current state of the world. A reconcile function always reads the latest API state and computes what needs to change. This means missed events don’t cause permanent drift — the system will self-correct on the next reconcile or full resync. This is the single most important architectural property of Kubernetes.

3. The aggregator delegation chain. The API server is not a single binary but a chain of three cooperating servers assembled at startup (CreateServerChain). This allows CRDs (handled by apiextensions-apiserver) and aggregated API servers (handled by kube-aggregator) to coexist with the core API in a single port/TLS endpoint, while being independently implemented. The pattern is the basis for the entire operator ecosystem.

4. SharedInformer as the universal read path for controllers. No controller reads directly from the API server at query time. All reads go through a SharedInformer-backed local cache. This dramatically reduces load on the API server and provides a consistent, linearizable view of API state for reconciliation logic. The cache is eventually consistent with etcd but controllers are designed to tolerate that.

5. Plugin-first extensibility. The scheduler, admission, and authorization subsystems are all plugin frameworks with well-defined extension points and no hardcoded logic. Built-in policies (RBAC, LimitRanger, default predicates) are just plugins registered at startup. This makes swapping or extending policy possible without forking the binary, and it is the direct ancestor of the CRD + operator pattern that dominates the Kubernetes ecosystem today.