Kubernetes — API Surface#

API types#

Kubernetes exposes functionality through four distinct API surfaces that serve different audiences:

1. REST/HTTP API — the Kubernetes API (kube-apiserver) consumed by clients, controllers, and operators
2. gRPC plugin interfaces — CRI, CSI, CNI, KMS, Device Plugin, DRA, and others consumed by node-level components and external plugins
3. CLI — kubectl consumed by human operators and CI/CD systems; plus kubeadm for cluster lifecycle
4. Library API — client-go, apimachinery, apiserver, kube-scheduler/framework consumed by the operator ecosystem

REST/HTTP API#

Router#

• Router: Custom Go HTTP mux built on go-restful (v3) for versioned API groups + a raw http.ServeMux (NonGoRestfulMux) for non-API endpoints (/healthz, /metrics, /openapi/v3, etc.)
• Route registration: Declarative via APIGroupInfo structs. Each API group registers a set of rest.Storage implementations per resource verb. The genericapiserver.GenericAPIServer.InstallAPIGroups() call iterates these and registers routes automatically.
• URL pattern: All resources follow the pattern /{prefix}/{group}/{version}/namespaces/{ns}/{resource}/{name} for namespaced resources, or /{prefix}/{group}/{version}/{resource}/{name} for cluster-scoped ones. Core API group (v1) uses the legacy prefix /api/v1.

Handler/middleware chain#

The DefaultBuildHandlerChain in staging/src/k8s.io/apiserver/pkg/server/config.go:1036 wraps every request through (outermost to innermost):

WithAuditInit                         → initialize audit context
WithPanicRecovery                     → recover panics, log + 500
WithRequestReceivedTimestamp          → stamp arrival time
WithRequestInfo                       → parse group/version/resource/verb from URL
WithRoutine (optional feature gate)   → run handler in goroutine for stack savings
WithLatencyTrackers                   → start per-filter latency histograms
WithHTTPLogging                       → structured request logging
WithRetryAfter (shutdown mode)        → return 503 + Retry-After during drain
WithHSTS                              → set HSTS headers
WithCacheControl                      → set Cache-Control: no-store
WithProbabilisticGoaway               → random GOAWAY for HTTP/2 connection balancing
WithWatchTerminationDuringShutdown    → drain watch connections gracefully
WithWaitGroup                         → track inflight non-long-running requests
WithRequestDeadline                   → enforce timeout per context deadline
WithTimeoutForNonLongRunningRequests  → 60s default timeout via goroutine + cancel
WithWarningRecorder                   → collect warning headers
WithCORS                              → preflight handling
WithAuthentication                    → token, cert, OIDC, webhook auth
  [WithTracing]                       → OTEL span (after authn, conditional)
WithAudit                             → record audit events for policy-matched requests
WithImpersonation / WithConstrainedImpersonation → honor Impersonate-* headers
WithPriorityAndFairness / WithMaxInFlightLimit    → API Priority & Fairness (APF) or legacy
WithAuthorization                     → RBAC, Node, Webhook authorizers
  → REST storage handler (admission → decode → validate → etcd)

Authentication#

Multiple authenticators composed by unionauth: X.509 client certs, Bearer tokens (static, bootstrap, service account JWTs), OIDC JWTs, Webhook token reviews, Anonymous. Authenticators are tried in order; first success wins.

Key endpoints#

Watch#

GET /api/v1/pods?watch=true opens a long-running HTTP/2 chunked stream. The API server fans out etcd watch events to all connected watchers. This is the primary push mechanism for controllers.

Server-Side Apply#

PATCH with Content-Type: application/apply-patch+yaml invokes the server-side apply merge strategy (declarative ownership model), available since 1.18.

gRPC Plugin APIs#

Kubernetes uses gRPC for the interfaces between the control plane/kubelet and external plugin processes. All plugin APIs use Unix Domain Sockets.

CRI — Container Runtime Interface#

• Proto: staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto
• Services:
  • RuntimeService — ~30 RPCs: RunPodSandbox, StopPodSandbox, CreateContainer, StartContainer, StopContainer, ExecSync, Exec, Attach, PortForward, ContainerStats, UpdateRuntimeConfig, CheckpointContainer, streaming variants (GetContainerEvents, StreamContainerStats, etc.)
  • ImageService — ListImages, PullImage, RemoveImage, ImageStatus, StreamImages
• Consumers: kubelet calls container runtimes (containerd, CRI-O)

Device Plugin API#

• Proto: staging/src/k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1/api.proto
• Services:
  • Registration — Register (plugin tells kubelet it’s ready)
  • DevicePlugin — ListAndWatch (stream device inventory), Allocate, GetPreferredAllocation, PreStartContainer, GetDevicePluginOptions
• Purpose: Hardware accelerators (GPUs, FPGAs, NICs) expose devices to Pods

Dynamic Resource Allocation (DRA)#

• Proto: staging/src/k8s.io/kubelet/pkg/apis/dra/v1/api.proto
• Service: DRAPlugin — NodePrepareResources, NodeUnprepareResources
• Purpose: More flexible resource allocation than Device Plugin; replaces it for complex topologies

Pod Resources API (kubelet)#

• Proto: staging/src/k8s.io/kubelet/pkg/apis/podresources/v1/api.proto
• Service: PodResourcesLister — List, GetAllocatableResources, Get
• Purpose: Out-of-band introspection (for monitoring agents like DCGM) of which devices are allocated to which pods

KMS (Key Management Service)#

• Proto: staging/src/k8s.io/kms/apis/v2/api.proto
• Service: KeyManagementService — Status, Encrypt, Decrypt
• Purpose: Delegate etcd encryption-at-rest key management to external KMS providers

External JWT Signer#

• Proto: staging/src/k8s.io/externaljwt/apis/v1/api.proto
• Service: ExternalJWTSigner — Sign, FetchKeys, Metadata
• Purpose: Delegate service account token signing/verification to an external HSM or KMS

Plugin Registration (kubelet)#

• Proto: staging/src/k8s.io/kubelet/pkg/apis/pluginregistration/v1/api.proto
• Service: Registration — the kubelet-side socket that all plugins (CSI, Device Plugin) connect to in order to announce themselves

CLI#

kubectl#

• Framework: cobra + pflag (staging/src/k8s.io/kubectl/)
• Factory pattern: A util.Factory interface provides lazy access to clients, REST mappers, and schema. Commands receive the factory and IOStreams but no concrete types — this enables complete dependency injection for testing.
• Command structure (from staging/src/k8s.io/kubectl/pkg/cmd/cmd.go):

kubectl
├── Basic Commands (Beginner)
│   ├── create         — create a resource from file or stdin
│   ├── expose         — expose a resource as a Service
│   ├── run            — run an image on the cluster
│   └── set            — set specific features on objects
├── Basic Commands (Intermediate)
│   ├── explain        — get documentation for resource fields
│   ├── get            — display one or many resources
│   ├── edit           — edit a resource in $EDITOR
│   └── delete         — delete resources
├── Deploy Commands
│   ├── rollout        — manage rollouts (history, pause, resume, undo, status)
│   ├── scale          — change replica count
│   └── autoscale      — create HPA
├── Cluster Management
│   ├── certificate    — approve/deny CSRs
│   ├── cluster-info   — display cluster info
│   ├── top            — display resource usage (nodes/pods)
│   ├── cordon/uncordon — mark node (un)schedulable
│   ├── drain          — drain node for maintenance
│   └── taint          — update node taints
├── Troubleshooting
│   ├── describe       — show details of a resource
│   ├── logs           — print container logs
│   ├── attach         — attach to a running container
│   ├── exec           — execute a command in a container
│   ├── port-forward   — forward ports to a pod
│   ├── cp             — copy files to/from container
│   ├── auth           — inspect authorization (can-i, reconcile)
│   └── events         — list events
├── Advanced Commands
│   ├── diff           — diff live vs file
│   ├── apply          — declarative apply (SSA or CSA)
│   ├── patch          — update resource fields
│   ├── replace        — replace a resource
│   ├── wait           — wait for condition
│   └── kustomize      — render kustomization
├── Settings Commands
│   ├── label          — update labels
│   ├── annotate       — update annotations
│   └── completion     — shell completion
└── Other
    ├── alpha          — alpha/experimental subcommands
    ├── config         — modify kubeconfig (contexts, clusters, users)
    ├── plugin         — manage/run kubectl plugins
    ├── version        — print version
    ├── api-versions   — list API versions
    ├── api-resources  — list API resource types
    └── options        — list global flags

• Global flags: --kubeconfig, --context, --server, --namespace, --token, --certificate-authority, --output (-o json|yaml|wide|name|jsonpath|go-template|custom-columns), --dry-run=client|server|none
• Plugin mechanism: kubectl plugin list discovers executables named kubectl-* on $PATH and exposes them as subcommands. No registration needed — pure PATH convention.

kubeadm#

• Framework: cobra
• Top-level commands: init, join, reset, upgrade, token, certs, config, version
• Purpose: Bootstraps and manages cluster lifecycle (PKI, static Pod manifests for control-plane components, node join tokens)

Other binaries (all use cobra)#

• kube-apiserver — flags only, no subcommands; ~200+ flags covering TLS, etcd, auth, admission, feature gates
• kube-controller-manager — flags only; --controllers=* selects which controllers run
• kube-scheduler — flags + --config for KubeSchedulerConfiguration YAML
• kubelet — flags + --config for KubeletConfiguration YAML; --register-node, CRI socket flags
• kube-proxy — flags + --config

Plugin / Extension System#

Kubernetes has one of the most elaborate extension systems in the Go ecosystem, operating at multiple levels:

1. Custom Resource Definitions (CRDs)#

• Mechanism: Declare a new API type as a CRD object. The apiextensions-apiserver automatically serves CRUD + watch for it via the REST API.
• Extension point: Any API group/version/resource not built in to core
• Used by: Every operator in the ecosystem (cert-manager, ArgoCD, Prometheus Operator, …)

2. Aggregated API Servers#

• Mechanism: Deploy a separate binary that implements the genericapiserver framework. Register it via an APIService object. The kube-aggregator proxies matching requests to it.
• Extension point: Custom API semantics that CRDs can’t express (e.g., metrics-server at metrics.k8s.io)

3. Admission Webhooks#

• Mechanism: Register a MutatingWebhookConfiguration or ValidatingWebhookConfiguration object. The API server calls your HTTPS endpoint for matching resource operations.
• Extension point: Policy enforcement, defaulting, validation beyond structural schema
• Interface: admission.Interface (staging/src/k8s.io/apiserver/pkg/admission/)

4. Scheduler Plugin Framework#

• Mechanism: Implement one or more extension-point interfaces (FilterPlugin, ScorePlugin, BindPlugin, etc.) and register them in a KubeSchedulerProfile.
• Extension points (ordered): PreEnqueue → QueueSort → PreFilter → Filter → PostFilter → PreScore → Score → NormalizeScore → Reserve → Permit → WaitOnPermit → PreBind → Bind → PostBind
• Built-in plugins as examples: NodeAffinity, TaintToleration, VolumeBinding, PodTopologySpread, NodeResourcesFit

5. Authorization Webhooks / Node Authorizer / RBAC#

• Mechanism: Implement authorizer.Authorizer interface or use RBAC rules.
• Extension point: Custom policy decisions per request

6. kubelet Plugin APIs (gRPC)#

• Device Plugin: hardware resource management via DevicePlugin gRPC service
• CSI: volume lifecycle via Container Storage Interface gRPC (defined externally by the CSI spec, not inside this repo)
• CNI: network plugin binary protocol (exec-based, not gRPC)
• DRA: Dynamic Resource Allocation for complex hardware topologies

Library API#

The staging directory (staging/src/k8s.io/) contains ~20 independently published modules that form the public library API of the Kubernetes project:

API style#

• client-go: Object-oriented with typed clients per resource. ClientSet gives clientset.CoreV1().Pods(ns).Get(ctx, name, opts). A separate dynamic.Client handles unknown types.
• apimachinery: Functional registration (scheme.AddToGroupVersion()), codec selection via runtime.CodecForVersions()
• apiserver: Builder-style with Config and CompletedConfig — New(config) constructors that verify all required fields are set
• Backward compatibility: Kubernetes maintains strict API compatibility via group/version/kind versioning. The conversion framework in apimachinery handles cross-version translation. The API is versioned at the HTTP URL level; Go types are versioned by package path. client-go follows a compatibility policy: it works with API servers up to one minor version ahead or behind.

Key library extension patterns#

• controller-runtime (separate repo, not in this monorepo) builds on these libraries to provide a higher-level Reconciler interface — the standard way to write operators today
• kubebuilder scaffolds operators using controller-runtime + generated CRD manifests from Go struct tags