etcd — API Surface#

API types#

etcd exposes four distinct API surfaces:

1. gRPC — primary protocol for all client–server communication
2. REST/HTTP (grpc-gateway) — JSON shim over the gRPC services, available on the same port
3. CLI (etcdctl) — administrative and operational CLI wrapping the gRPC client
4. Go library (client/v3) — a first-class Go client library with interface-per-concern design
5. Embedding API (embed.Config) — allows Go programs to run etcd in-process

Additionally, the server exposes a minimal HTTP-only surface for health checks, Prometheus metrics, and peer-to-peer Raft traffic.

gRPC API#

Proto files#

All proto definitions live in api/etcdserverpb/:

Services and RPCs#

KV service — api/etcdserverpb/rpc.proto#

The core key-value store service. Every RPC has a corresponding grpc-gateway REST binding at /v3/kv/....

Watch service — api/etcdserverpb/rpc.proto#

Bidirectional streaming service for change notifications.

Lease service — api/etcdserverpb/rpc.proto#

TTL-based key expiry.

Cluster service — api/etcdserverpb/rpc.proto#

Cluster membership management.

Maintenance service — api/etcdserverpb/rpc.proto#

Operational and observability operations.

Auth service — api/etcdserverpb/rpc.proto#

RBAC access control.

Election service — server/etcdserver/api/v3election/v3electionpb/v3election.proto#

Leader election built on top of leases and the KV transactional model.

Lock service — server/etcdserver/api/v3lock/v3lockpb/v3lock.proto#

Distributed mutual exclusion built on leases and the KV store.

gRPC server setup#

server/etcdserver/api/v3rpc/grpc.go is the single wire-up point. The Server() function creates a grpc.Server and registers all service implementations:

pb.RegisterKVServer(grpcServer, NewQuotaKVServer(s))
pb.RegisterWatchServer(grpcServer, NewWatchServer(s))
pb.RegisterLeaseServer(grpcServer, NewQuotaLeaseServer(s))
pb.RegisterClusterServer(grpcServer, NewClusterServer(s))
pb.RegisterAuthServer(grpcServer, NewAuthServer(s))
pb.RegisterMaintenanceServer(grpcServer, NewMaintenanceServer(s, healthNotifier))
healthpb.RegisterHealthServer(grpcServer, health.NewServer())

Election and Lock services are registered separately in server/embed/serve.go during the HTTP/gRPC mux setup, because they are optional extensions built on client/v3 (they run client-side logic against the embedded server).

Interceptors#

Three interceptor layers are applied in order for all unary calls:

For streaming RPCs, layers 2 and 3 equivalents exist (newStreamInterceptor). OpenTelemetry distributed tracing is added via a gRPC stats handler when EnableDistributedTracing is set.

Quota enforcement is not in the interceptor chain — it is handled at the service object level: NewQuotaKVServer wraps the KV server; NewQuotaLeaseServer wraps the Lease server. Both check the NOSPACE alarm before dispatching.

REST/HTTP API (grpc-gateway)#

The grpc-gateway (grpc-ecosystem/grpc-gateway/v2) is an optional layer that translates HTTP/JSON requests to gRPC calls internally. It is enabled by default (EnableGRPCGateway: true in embed.Config).

Router: gw.ServeMux (grpc-gateway runtime), wrapped in an accessController (auth enforcement for the HTTP path).

Marshaler: Custom protojson marshaler with snake_case field names and emitted default values.

Route registration (server/embed/serve.go):

etcdservergw.RegisterKVHandler(...)
etcdservergw.RegisterWatchHandler(...)
etcdservergw.RegisterLeaseHandler(...)
etcdservergw.RegisterClusterHandler(...)
etcdservergw.RegisterMaintenanceHandler(...)
etcdservergw.RegisterAuthHandler(...)
v3lockgw.RegisterLockHandler(...)
v3electiongw.RegisterElectionHandler(...)

Key REST endpoints (from proto google.api.http annotations):

Notable design choice: All REST endpoints use HTTP POST, even for read operations. This is a deliberate deviation from REST conventions — it avoids URL-length limitations for complex queries and keeps the HTTP layer as a thin transport shim rather than a designed API.

Plain HTTP endpoints (registered separately via etcdhttp):

WebSocket: The tmc/grpc-websocket-proxy middleware wraps the grpc-gateway mux to allow WebSocket connections to streaming endpoints (notably Watch.Watch), enabling browser-based clients.

CLI (etcdctl)#

Framework#

cobra (github.com/spf13/cobra) + pflag (github.com/spf13/pflag). Entry point: etcdctl/ctlv3/ctl.go. Commands added in init() to a global rootCmd.

Command structure#

Commands are organized into 5 groups:

KV Group#

Cluster Maintenance Group#

Concurrency Group#

Authentication Group#

Utility Group#

Flag patterns#

Global persistent flags (hidden, shown via etcdctl options):

Notable pattern: Global flags are deliberately hidden from default help output and shown only via etcdctl options. Per-command flags appear in each command’s --help. This two-tier help system reduces noise for common operations.

Environment variable binding: Every global flag has an ETCD_* env var equivalent (handled at the server level; etcdctl reads flags only).

Go Library API (client/v3)#

Public packages#

The client library is a separate Go module (go.etcd.io/etcd/client/v3 with its own go.mod). This is architecturally significant: applications can depend on the client without pulling in the full server.

Client construction#

client, err := clientv3.New(clientv3.Config{
    Endpoints:   []string{"localhost:2379"},
    DialTimeout: 5 * time.Second,
})
// or
client, err := clientv3.NewFromURL("localhost:2379")

Client is a struct (not an interface) that embeds all service clients. It manages connection pooling and auto-sync of endpoints from the cluster.

Interface-per-concern design#

The client exposes seven interfaces — one per logical capability:

Functional options (OpOption pattern)#

All KV operations accept variadic OpOption functions, enabling a clean composable API without option-struct proliferation:

// Simple get
resp, err := client.Get(ctx, "key")

// Get with options
resp, err := client.Get(ctx, "prefix/",
    clientv3.WithPrefix(),
    clientv3.WithRev(42),
    clientv3.WithLimit(100),
    clientv3.WithSort(clientv3.SortByKey, clientv3.SortAscend),
    clientv3.WithKeysOnly(),
)

// Put with lease and previous value
resp, err := client.Put(ctx, "key", "value",
    clientv3.WithLease(leaseID),
    clientv3.WithPrevKV(),
)

Full set of options includes: WithLease, WithLimit, WithRev, WithSort, WithPrefix, WithRange, WithFromKey, WithSerializable, WithKeysOnly, WithCountOnly, WithMinModRev, WithMaxModRev, WithFirstCreate, WithLastKey, WithProgressNotify, WithCreatedNotify, WithFilterPut, WithFilterDelete, WithPrevKV, WithFragment, WithIgnoreValue, WithIgnoreLease.

STM (Software Transactional Memory) — client/v3/concurrency#

The STM interface provides optimistic concurrency control:

type STM interface {
    Get(key string) string
    Put(key, val string, opts ...OpOption)
    Rev(key string) int64
    Del(key string)
}

concurrency.NewSTM(client, func(stm STM) error {...}) wraps a user function in a Raft-serializable transaction with automatic retry on conflict. Four isolation levels are supported: SerializableSnapshot, Serializable, RepeatableReads, ReadCommitted.

Kubernetes-specific interface — client/v3/kubernetes#

A separate Interface added in 2024 that models the exact contract Kubernetes needs, hiding etcd complexity:

type Interface interface {
    Get(ctx context.Context, key string, opts GetOptions) (GetResponse, error)
    List(ctx context.Context, prefix string, opts ListOptions) (ListResponse, error)
    Count(ctx context.Context, prefix string, opts CountOptions) (int64, error)
    OptimisticPut(ctx context.Context, key string, value []byte, expectedRevision int64, opts PutOptions) (PutResponse, error)
    OptimisticDelete(ctx context.Context, key string, expectedRevision int64, opts DeleteOptions) (DeleteResponse, error)
}

This interface deliberately omits Watch (handled elsewhere in Kubernetes), leases (except as a PutOptions.LeaseID field), and admin operations. It is a rare example of explicitly modeling a consumer’s minimal API surface.

Embedding API (server/embed)#

For Go programs that want to run etcd in-process (e.g., testing, single-binary deployment):

cfg := embed.NewConfig()
cfg.Dir = "/tmp/etcd"
cfg.ListenClientUrls = []url.URL{{Scheme: "http", Host: "localhost:2379"}}
e, err := embed.StartEtcd(cfg)
defer e.Close()
<-e.Server.ReadyNotify() // wait for ready

Extension points in embed.Config:

These two hooks allow embedding users to co-locate custom gRPC/HTTP services alongside etcd without running a separate server.

Peer-to-peer Raft HTTP#

etcd peers communicate via an internal HTTP-based protocol (server/etcdserver/api/rafthttp). This is not a user-facing API but is part of the overall API surface:

Peer traffic is multiplexed with client gRPC traffic on the same TCP port using soheilhy/cmux, with HTTP/1.x identified by connection sniffing.

Key design observations#

1. Proto-first API design#

The entire client API is derived from the proto definitions in api/etcdserverpb/rpc.proto. The proto file is the single source of truth for gRPC stubs, grpc-gateway HTTP bindings, OpenAPI documentation, and Go type aliases in client/v3. This eliminates drift between protocol and implementation.

2. All writes via POST (REST)#

The grpc-gateway REST API uses POST for all endpoints, including reads (/v3/kv/range). This is intentional: it avoids GET query-string length limits for complex range queries, keeps the REST layer as a pure serialization shim, and signals to clients that this is not a “RESTful” API — it is a gRPC API with an HTTP convenience wrapper.

3. Interface segregation in the client#

The client library defines seven narrow interfaces (KV, Watcher, Lease, etc.) rather than one fat Client interface. This means test doubles can be minimal, and library authors building on top of etcd can declare exactly which capabilities they need. The kubernetes.Interface is the extreme example: five methods instead of fifty.

4. Functional options for rich queries#

The OpOption pattern for KV operations handles the combinatorial complexity of etcd’s query model (revision pinning, range queries, sorting, filtering, pagination) without proliferating function overloads or option structs. The same Op type is used internally for Do, Txn, and direct Get/Put/Delete calls — the options are composable.

5. Service extensibility via two hooks#

Embedding users can extend the API surface via embed.Config.ServiceRegister (gRPC) and embed.Config.UserHandlers (HTTP), without forking etcd. This is used by Kubernetes’ embedded etcd tests and by projects like k3s that embed etcd.

6. gRPC health protocol#

etcd registers google.golang.org/grpc/health/grpc_health_v1.HealthServer, enabling standard gRPC health checking alongside the custom /health, /livez, /readyz endpoints. This dual health surface supports both Kubernetes-style probes and gRPC-native service mesh health checks.