Hugo — Testing#

Test metrics#

• Test files: 376
• Source files (non-test): 514
• Ratio (test / source): ~0.73 — well above average for a project of this complexity
• Test frameworks: stdlib testing + frankban/quicktest (1 506 usages) + rogpeppe/go-internal/testscript (CLI-level tests); no testify, no gomock, no ginkgo

Test organization#

Placement#

Both same-package (package hugolib) and external-test package (package hugolib_test) styles are used. Integration tests universally use the external _test package, which prevents accidental access to unexported internals and keeps test contracts honest.

Helper packages#

Fixtures#

Three distinct fixture styles coexist:

1. Inline txtar archives — the dominant pattern in integration tests. Entire virtual Hugo sites (config, content, layouts) are declared as multi-line strings in -- filename -- txtar format inside the test function.
2. testdata/ directories — used in unit tests for markup (tpl/transform/testdata/), media type parsing, and image pipeline inputs.
3. Golden image files — resources/images/testdata/images_golden/ stores expected PNG/JPEG outputs for image transformation tests. Regenerated by setting WriteGolden: true.

Test patterns#

Integration test builder (txtar-based)#

Hugo’s most architecturally distinctive testing feature is the IntegrationTestBuilder in hugolib/integrationtest_builder.go. It:

• Accepts a txtar-formatted string describing a full virtual Hugo site.
• Materializes the virtual filesystem (via afero.MemMapFs by default, or real OS FS when NeedsOsFS: true).
• Builds the entire Hugo pipeline in-process (HugoSites.Build()).
• Exposes assertion helpers: AssertFileContent(path, ...substrings), AssertFileContentEquals, AssertImage, AssertLogContains.
• Supports incremental rebuild testing: EditFiles(path, old, new) / AddFiles / RemoveFiles followed by .RebuildAllFiles().

The Test(t, files, opts...) and TestE(t, files, opts...) convenience functions reduce boilerplate to two lines for the common case.

Example pattern (from hugolib/hugolib_integration_test.go):

func TestPageTranslationsMap(t *testing.T) {
    t.Parallel()
    files := `
-- hugo.toml --
baseURL = 'https://example.org/'
...
-- layouts/single.html --
{{ .Title }}
`
    b := hugolib.NewIntegrationTestBuilder(
        hugolib.IntegrationTestConfig{T: t, TxtarString: files},
    )
    b.Build()
    b.AssertFileContent("public/en/posts/p1/index.html", "<ul><li>P1-en</li></ul>")
}

This approach makes each test a self-documenting, standalone Hugo site specification.

Table-driven tests#

• Prevalence: Moderate — 250 occurrences of t.Run, tc.name, testCases in *_test.go files.
• Style: Anonymous struct slices with t.Run(tc.name, ...) in unit tests; integration tests more often use named top-level functions per scenario (one txtar site per test function).
• Example: markup/goldmark/codeblocks/codeblocks_integration_test.go:280 — tests slice of struct{name, files, expect string} iterated with t.Run.

Testscript CLI tests#

• Location: testscripts/commands/ (71 .txt files), testscripts/server/, testscripts/withdeploy/, testscripts/unfinished/.
• Framework: rogpeppe/go-internal/testscript — each .txt file is a shell-like script that runs hugo as a subprocess.
• Custom commands registered: checkfile, checkfilecount, cat, ls, lsr, tree, append, replace, httpget (with retry), waitServer (polls for .ready JSON file), stopServer (calls /__stop endpoint).
• TestMain hook: testscript.Main(m, map[string]func(){"hugo": ...}) redirects the hugo command to the in-process commands.Execute() — avoiding subprocess overhead while preserving realistic CLI invocation semantics.
• Example script (testscripts/commands/hugo_build.txt):

  hugo build
  stdout 'Pages.*|1'
  checkfile public/index.html
  grep 'IsServer: false;IsProduction: true'  public/index.html
  -- hugo.toml --
  baseURL = "http://example.org/"
  -- layouts/home.html --
  Home|IsServer: {{ hugo.IsServer }}

Mocking approach#

• Strategy: Interface-based substitution; no mock generation framework.
• Primary mechanism: afero.MemMapFs provides an in-memory filesystem, allowing full Hugo builds without touching disk. The hugofs package’s layered afero wrappers are simply swapped for the memory variant.
• Other fakes: identitytesting provides stubbed Manager implementations. Template rendering is tested against real rendered output (no template engine mocking).
• No gomock/mockery: Zero generated mocks. Dependency seams are real interfaces satisfied by real or minimal-fake implementations.

Integration tests#

• Present: Yes — 76 *_integration_test.go files across the entire codebase.
• How: In-process, using IntegrationTestBuilder and afero.MemMapFs. No Docker, no testcontainers. The “integration” label signals that the test exercises the full Hugo pipeline (config load → content processing → template rendering → output), not a subprocess call.
• Separation: Naming convention (_integration_test.go) in the same directory as the package under test. No build tags.
• Coverage areas: Every major subsystem has integration coverage: markup converters (goldmark, asciidoc), resource transformers (minifier, sass, JS), template system, i18n, modules, image pipeline, cache, server command.

Golden image tests#

• Mechanism: resources/images/imagetesting.GoldenImageTestOpts renders image transformations and compares pixel output against committed PNG files in testdata/images_golden/.
• Update flow: Set WriteGolden: true to regenerate. SkipAssertions: true allows adding new test cases without requiring golden files yet.
• Scope: Covers filter operations, format conversions (webp, animated webp), quality/lossless tradeoffs, and image processing methods.

Test quality observations#

What’s done well#

• Self-documenting tests. The txtar inline site format means every integration test is its own complete spec — no hidden fixture files, no implicit shared state. Coming back to a test after months, the full site context is right there.
• Parallel-safe by design. IntegrationTestBuilder defaults to in-memory FS, making t.Parallel() safe and fast. The 376 test files run with no shared mutable state.
• First-class rebuilds testing. EditFiles / RebuildAllFiles on the same builder instance lets tests verify incremental rebuild correctness — critical for a tool whose watch mode is a core feature.
• Layered CI gating. htesting.SkipSlowTestUnlessCI(t) defers network-dependent and slow tests to CI. htesting.IsGitHubAction() gating for PinnedRunner prevents debug shortcuts from landing in CI.
• Custom checkers extend qt cleanly. hqt.IsSameString (whitespace-normalized) eliminates test fragility from template whitespace differences without hiding real content errors.
• testscript for CLI tests. The .txt testscript files test the real CLI surface (commands, flags, exit codes, stdout/stderr patterns) in a readable, scriptable form. The in-process hugo command registration avoids subprocess overhead while preserving interface fidelity.

What could improve#

• No mock/stub infrastructure for external tools. Tests that need AsciiDoc, Pandoc, or Dart Sass are skipped on most developer machines (htesting.SupportsAll()). There’s no lightweight stub for these external binary dependencies, making that coverage CI-only.
• Integration test build times. With 76 integration test files each building a full Hugo pipeline, test suite duration is significant. The in-memory FS helps, but no parallelism budget is set explicitly — sub-optimal if parallelism is CPU-bound on CI.
• Golden tests not gated by flag. The image golden test update mode (WriteGolden: true) is controlled by a code change, not a test flag (-update). Convention is weaker than flag.Bool("update", ...).

Patterns worth emulating#

1. Txtar inline fixtures — declaring the entire system under test as an inline string is a pattern any CLI or file-processing tool should adopt. It eliminates fixture sprawl and makes tests copyable.
2. Augmented qt with domain checkers — hqt.IsSameString demonstrates that a thin wrapper around a popular assertion library with project-specific normalization is far cheaper than a custom framework.
3. testscript for CLI surface tests — rogpeppe/go-internal/testscript with in-process command registration is the most underused testing approach in the Go ecosystem; Hugo demonstrates its full potential.
4. BailOut for deadlock detection — htesting.BailOut(duration) that prints a full goroutine stack and panics is a pragmatic solution to hanging test debugging.