sqlc — Architecture#

Architectural style#

Compiler Pipeline with a Language-Agnostic Plugin System

sqlc is architecturally a compiler: it takes SQL files and a schema, produces a typed intermediate representation (IR), and then hands that IR off to code generators. The pipeline stages — parse, catalog-build, type-resolve, IR-emit, codegen — map almost 1:1 to the named packages under internal/. The compiler’s output boundary with code generators is a protobuf-defined GenerateRequest/Response that enables both in-process Go generators and out-of-process plugins (WASM or subprocess) without changing the orchestration layer.

This makes sqlc a layered compiler with an extensible codegen plugin surface, not a typical service or library.

Component diagram (textual)#

┌────────────────────────────────────────────────────────────────┐
│  cmd/sqlc → pkg/cli → internal/cmd/Do()                       │
│  (thin shell: cobra commands; reads sqlc.yaml config)          │
└─────────────────────────────┬──────────────────────────────────┘
                              │
          ┌───────────────────▼──────────────────────┐
          │   internal/cmd/processQuerySets()         │
          │   (errgroup; GOMAXPROCS parallelism)      │
          └───┬───────────────────────────────────────┘
              │                          │
    ┌─────────▼──────────┐    ┌──────────▼──────────┐
    │  parse() pipeline  │    │  codegen() dispatch │
    │                    │    │                     │
    │ compiler.New()     │    │  ┌──────────────┐  │
    │   .ParseCatalog()  │    │  │  Built-in Go  │  │
    │   .ParseQueries()  │    │  │  (golang.Gen) │  │
    │   .Result()        │    │  ├──────────────┤  │
    │                    │    │  │ WASM plugin  │  │
    │  ┌──────────────┐  │    │  │ (wasm.Runner)│  │
    │  │ Engine layer │  │    │  ├──────────────┤  │
    │  │  (dialect-   │  │    │  │ Process plug │  │
    │  │  specific    │  │    │  │(process.Run.)│  │
    │  │  parser +    │  │    │  └──────┬───────┘  │
    │  │  catalog)    │  │    │         │           │
    │  └──────┬───────┘  │    │  plugin.Codegen    │
    │         │ sql/ast  │    │  ServiceClient      │
    │  ┌──────▼───────┐  │    │  (grpc.ClientConn   │
    │  │ sql/catalog  │  │    │   interface)        │
    │  └──────────────┘  │    └─────────────────────┘
    │  ┌──────────────┐  │
    │  │ analyzer     │  │  (optional: live DB type analysis)
    │  │ (optional)   │  │
    │  └──────────────┘  │
    └────────────────────┘
              │
    ┌─────────▼─────────────────────────────────────┐
    │   compiler.Result  →  plugin.GenerateRequest  │
    │   (typed query IR, catalog snapshot)          │
    └───────────────────────────────────────────────┘

There is also an alternate execution path for sqlc.cloud:

internal/cmd/Generate()
  → remoteGenerate()
      → remote.NewClient() + gRPC call to sqlc.cloud
      → returns generated files directly

Core components#

CLI layer#

  • Package: cmd/sqlc, pkg/cli, internal/cmd
  • Responsibility: Entry point. Cobra command registration, flag parsing, I/O wiring. The Do(args, stdin, stdout, stderr) function in internal/cmd is the single top-level dispatcher.
  • Key types: cobra.Command, Env (debug/experiment flags), Options
  • Dependencies: cobra, config, tracer, opts

Config#

  • Package: internal/config, internal/config/convert
  • Responsibility: Parses sqlc.yaml / sqlc.json into typed Go structs. Handles v1/v2 config version migration. Validates config combinations.
  • Key types: config.Config, config.SQL, config.CombinedSettings, config.Plugin
  • Dependencies: gopkg.in/yaml.v3, stdlib encoding/json

Compiler#

  • Package: internal/compiler
  • Responsibility: Orchestrates the parse/type-resolve pipeline for a single SQL queryset. Creates the engine (parser + catalog) for the configured dialect, runs parseCatalog to build the in-memory schema, then parseQueries to type-check each query against the catalog.
  • Key types: Compiler struct, Parser interface (dialect-agnostic parse contract), Result (typed query IR + catalog snapshot)
  • Dependencies: engine/* (dialect-specific parsers), sql/ast, sql/catalog, analyzer, opts

Engine layer (SQL dialects)#

  • Package: internal/engine/postgresql, internal/engine/dolphin (MySQL), internal/engine/sqlite, internal/engine/clickhouse
  • Responsibility: Each dialect sub-package provides a Parser implementation and a NewCatalog() factory. The parser converts the native parse tree (pg_query_go for PostgreSQL, TiDB parser for MySQL, an internal parser for SQLite) into the shared sql/ast node types.
  • Key types: dialect-specific Parser structs; sql/ast.Statement, sql/catalog.Catalog
  • Dependencies: pg_query_go (PostgreSQL), TiDB parser (MySQL); no external dep for SQLite

SQL AST and Catalog#

  • Package: internal/sql/ast, internal/sql/catalog
  • Responsibility: Language-agnostic AST node types that all engines produce. The catalog is an in-memory representation of the database schema (tables, columns, types, functions) built from DDL statements.
  • Key types: ast.Statement, ast.Node, catalog.Catalog, catalog.Schema, catalog.Table
  • Dependencies: none beyond stdlib

Analyzer (optional live-DB type analysis)#

  • Package: internal/analyzer, internal/engine/postgresql/analyzer, internal/engine/sqlite/analyzer
  • Responsibility: When a live database URI is configured, the analyzer sends queries to the actual database and retrieves column type information. Results are disk-cached. Used to supplement or replace static catalog analysis.
  • Key types: Analyzer interface (Analyze, Close, EnsureConn, GetColumnNames), CachedAnalyzer
  • Dependencies: pgx, database/sql, internal/cache

Plugin dispatch (ext layer)#

  • Package: internal/ext, internal/ext/wasm, internal/ext/process
  • Responsibility: Provides three interchangeable backends for code generation, all presenting the same grpc.ClientConnInterface:
    • Built-in Go generators are wrapped with ext.HandleFunc() (in-process function call)
    • WASM plugins are executed via wasm.Runner using the Wazero runtime
    • Process plugins are executed via process.Runner (subprocess with protobuf over stdin/stdout)
  • Key types: ext.Handler interface, wasm.Runner, process.Runner
  • Dependencies: wazero (WASM runtime), google.golang.org/grpc (interface only), internal/plugin

Plugin IR (protobuf)#

  • Package: internal/plugin
  • Responsibility: Protobuf-generated types that form the formal contract between the compiler and all code generators. GenerateRequest carries the full typed query IR; GenerateResponse carries the output files.
  • Key types: plugin.GenerateRequest, plugin.GenerateResponse, plugin.File, plugin.CodegenServiceClient
  • Dependencies: google.golang.org/protobuf

Built-in code generators#

  • Package: internal/codegen/golang, internal/codegen/json
  • Responsibility: Consume a GenerateRequest and produce a GenerateResponse. The Go generator uses text/template with type-mapping tables; the JSON generator emits the IR as JSON. Both are wrapped via ext.HandleFunc to use the same dispatch path as external plugins.
  • Key types: golang.Generate(ctx, *plugin.GenerateRequest), json.Generate(ctx, *plugin.GenerateRequest)
  • Dependencies: text/template, encoding/json, internal/plugin

Vet#

  • Package: internal/vet
  • Responsibility: Implements sqlc vet: loads CEL rule expressions from the config, evaluates them against each query’s metadata (execution plan, row counts, etc.), and reports violations. Uses the analyzer to execute queries against a live database.
  • Key types: VetRule, CEL environment
  • Dependencies: cel-go, internal/analyzer

Data flow#

Typical sqlc generate run:

1. main() → cmd.Do() → cobra.Execute() → genCmd.RunE()
2. Generate(ctx, dir, filename, opts)
   a. readConfig() → config.ParseConfig() → config.Config
   b. config.Validate() + env.Validate()
   c. processQuerySets() [errgroup, GOMAXPROCS workers]
      For each (SQL queryset × generator) pair in parallel:
        i.  parse(ctx, name, dir, sql, combo, parserOpts, stderr)
              → compiler.NewCompiler(sql, combo, parserOpts)
                  → selects engine parser + catalog for dialect
                  → optionally wraps with CachedAnalyzer
              → c.ParseCatalog(schema files)
                  → reads DDL files, calls parser.Parse()
                  → calls catalog.Update(stmt) for each DDL node
              → c.ParseQueries(query files, opts)
                  → reads .sql query files, calls parser.Parse()
                  → for each stmt: parseQuery() → type-resolution
                      → looks up table/column types in catalog
                      → optionally calls analyzer.Analyze() for live DB types
                  → returns compiler.Result{Catalog, Queries}
        ii. codegen(ctx, combo, sql, result)
              → codeGenRequest(result, combo) → plugin.GenerateRequest
              → selects handler: ext.HandleFunc / wasm.Runner / process.Runner
              → plugin.NewCodegenServiceClient(handler).Generate(ctx, req)
              → returns plugin.GenerateResponse{Files}
        iii. g.ProcessResult() → writes filename → source to output map
3. For each filename in output map → os.WriteFile()

Remote path (when cloud.project + --remote flag):

Generate() → remoteGenerate()
  → bundles config + SQL files into remote.GenerateRequest
  → gRPC call to sqlc.cloud
  → receives generated files in response
  → writes to disk

Initialization / Bootstrap#

  1. main() calls cmd.Do(os.Args[1:], stdin, stdout, stderr).
  2. Do() builds a fresh cobra.Command tree on every invocation — there is no global state aside from debug.Debug (set from SQLCDEBUG env var).
  3. If SQLCDEBUG=trace=<path> is set, a runtime trace file is opened before Cobra executes.
  4. Each command reads the config file, constructs a Compiler per query-set, runs the pipeline, and then exits.

There is no dependency injection framework. All wiring is explicit: NewCompiler receives config.SQL + config.CombinedSettings and constructs the correct engine and analyzer based on the engine: field. The codegen() function constructs the correct handler based on which sql.Gen.* field is set.

The processQuerySets function uses errgroup.WithContext with grp.SetLimit(runtime.GOMAXPROCS(0)) to bound parallelism — each (queryset, generator) pair runs in its own goroutine.

Configuration#

  • Format: YAML (sqlc.yaml, sqlc.yml) or JSON (sqlc.json); both supported. YAML takes precedence if both exist.
  • Config versions: v1 and v2 schemas; internal/config/convert handles migration.
  • Key config sections:
    • sql[]: one entry per queryset — specifies engine, schema, queries, gen.go/gen.json, codegen[] (plugins)
    • plugins[]: named plugin definitions with wasm or process backend
    • cloud.project: enables remote execution via sqlc.cloud
    • database.uri / database.managed: enables live-DB analysis
  • No Viper: config is parsed directly with gopkg.in/yaml.v3 and encoding/json. Environment variables are used only for debug/experiment flags (SQLCDEBUG, SQLCEXPERIMENT) read in internal/opts.

Key design decisions#

  1. Protobuf as the plugin contract — The GenerateRequest/Response protobuf IR is the single most important architectural decision. It makes the code-generator boundary language-agnostic: any program that can read a protobuf from stdin and write one to stdout can be a sqlc plugin. WASM plugins add hermetic sandboxing; the same proto is used.

  2. gRPC interface as the universal dispatch abstraction — All three plugin backends (built-in function, WASM, subprocess) implement grpc.ClientConnInterface, so the codegen dispatch loop is a single plugin.NewCodegenServiceClient(handler).Generate() call regardless of plugin type. In-process Go generators are wrapped with ext.HandleFunc to adapt a plain function into this interface — an elegant use of adapter pattern over an interface standard.

  3. Engine isolation via a shared AST — Each SQL dialect has its own parser that converts to internal/sql/ast, a language-agnostic node tree. The compiler and all analysis code operates only on this shared AST, so adding a new dialect never touches the compiler or code generators. ClickHouse is already partially present as evidence of this extensibility.

  4. Optional live-DB analysis with filesystem caching — The Analyzer interface allows sqlc to fall back to a live database connection for type information when static analysis is insufficient. Results are cached on disk (hashed by query + schema) to avoid redundant database round-trips, so iterative development is fast even with database-backed analysis enabled.

  5. pkg/cli as the only public surface — Everything except Do() / Run() is internal/, which makes sqlc safely embeddable as a library while protecting every internal API. The pkg/cli package was originally cmd/sqlc/main.go-equivalent, extracted to allow embedding sqlc in tools like sqlc-gen-* plugins.

  6. Parallel codegen across query-sets — processQuerySets fans out over all (queryset × generator) pairs using an errgroup bounded by GOMAXPROCS. Each pair independently parses its schema and queries, so large configs with many SQL files benefit from multi-core parallelism without any shared mutable state between workers (the generator struct uses a mutex only for the final output-map write).

Backward sqlc — Dependencies sqlc — Patterns Forward