GitHub CLI (gh) — Testing#

Test metrics#

• Test files: 319
• Source files (non-test): ~487
• Ratio (test files / source files): ~0.65 (roughly 2 test files for every 3 source files)
• Total test functions: 1352
• Table-driven test references: 1406
• Test frameworks: stdlib testing, github.com/stretchr/testify/assert, github.com/stretchr/testify/require

Test organization#

• Placement: Same package (white-box) for command and API tests; _test package (black-box) for acceptance tests. Both styles coexist — command packages use same-package access to test unexported helpers, while the acceptance layer treats the binary as a black box.
• Helper packages:
  • pkg/httpmock — The primary testing infrastructure. A custom http.RoundTripper mock (Registry) that intercepts outbound HTTP calls. Tests register stubs with Register(Matcher, Responder) and call defer http.Verify(t) to assert all stubs were consumed. Matchers cover REST (REST("GET", "user")), GraphQL by query regex (GraphQL("query PullRequestList")), and query-parameter matching. Responders include FileResponse, StringResponse, JSONResponse, StatusJSONResponse, and GraphQLMutation with callback inspection.
  • test/helpers.go — Minimal shared utilities: CmdOut (captures stdout + stderr + browsed URL), OutputStub (fake run.Runnable), ExpectLines (regex matcher, now deprecated in favor of exact assert.Equal).
  • internal/gh/mock — Hand-written mocks for the gh.Config and gh.Migration interfaces.
  • pkg/jsonfieldstest — Generic helper that validates --json field names against a command’s exported fields using the NewCmdFunc[T] type.
  • pkg/cmd/issue/argparsetest — Generic helper for testing argument-parsing logic in issue commands using the same NewCmdFunc[T] pattern.
• Fixtures:
  • Per-command fixtures/ directories containing JSON files (e.g., pkg/cmd/pr/list/fixtures/prList.json) used with httpmock.FileResponse("./fixtures/prList.json").
  • acceptance/testdata/<command>/*.txtar — testscript scenario files for acceptance testing (17+ command domains covered).
  • pkg/cmd/agent-task/shared/testdata/ — JSONL data files for agent-task log parsing tests.

Test patterns#

Table-driven tests#

• Prevalence: Heavy — 1406 occurrences of t.Run, testCases, tt.Run, or tc.name across test files.
• Style: Both named struct (type testCase struct { name string; ... }) and anonymous struct slices ([]struct{ name string; ... }). Tests consistently use t.Run(tc.name, ...) for subtests. The name field is almost universal even for single-assertion subtests, making failure output human-readable.
• Example: pkg/cmd/pr/list/list_test.go — multiple discrete test functions per behavior (TestPRList, TestPRList_nontty, TestPRList_filtering) rather than one monolithic table; this is a deliberate style choice where each scenario gets its own named test instead of rows in a table.

runF injection (primary command test pattern)#

• Strategy: Every command constructor is NewCmdXxx(f *cmdutil.Factory, runF func(*XxxOptions) error) *cobra.Command. In tests, a non-nil runF is injected to intercept the parsed Options struct before execution:

  cmd := NewCmdList(factory, func(opts *ListOptions) error {
      opts.Now = fakeNow
      return listRun(opts)
  })

  This decouples flag parsing (Cobra) from business logic. Tests can assert on opts values (flag resolution, derived values) without triggering any I/O — or call the real run function with deterministic state injected.
• runCommand helper: Each package defines a local runCommand(rt http.RoundTripper, ..., cli string) function that constructs a minimal cmdutil.Factory with a mocked HTTP transport, sets up iostreams.Test(), builds the command, parses shlex.Split(cli), and returns a *test.CmdOut. This pattern is repeated in every command package — it is the canonical unit test entry point.

Mocking approach#

• Strategy: Interface-based hand-written mocks; no gomock or mockery codegen.
• Primary mechanism: pkg/httpmock.Registry as HTTP transport interceptor — almost all command tests mock at the HTTP layer rather than at Go interface boundaries. This tests the full serialization/deserialization path and catches API schema drift.
• Example: pkg/cmd/pr/list/list_test.go:74 — initFakeHTTP() returns a new httpmock.Registry; http.Register(httpmock.GraphQL(...), httpmock.FileResponse("./fixtures/prList.json")) stubs the GraphQL response; defer http.Verify(t) asserts the stub was consumed.
• Secondary mocks: Hand-written Go mocks for specific interfaces (internal/gh/mock/config.go, pkg/search/searcher_mock.go, pkg/cmd/agent-task/capi/client_mock.go) use sync.RWMutex per method for safe concurrent access. These are only used where the interface contract is richer than what HTTP interception can exercise.

Integration tests#

• Present: Yes — two distinct integration layers.
• Go integration tests (build tag integration):
  • 4 files in pkg/cmd/attestation/ (inspect_integration_test.go, verify_integration_test.go, attestation_integration_test.go, sigstore_integration_test.go).
  • Build tag //go:build integration; included by CI with go test -tags=integration ./....
  • Hit real Sigstore infrastructure and/or GitHub API. Use testify/require for assertion.
  • Example: verify_integration_test.go constructs a live SigstoreVerifier, calls runVerify() against public Sigstore bundles for the sigstore/sigstore repo.
• Shell-based integration tests:
  • test/integration/attestation-cmd/ — bash scripts (run-all-tests.sh, verify-*.sh, download.sh) run the built gh binary against live endpoints.
  • Invoked by CI’s integration-tests job after make builds the binary.
• Separation: Build tags (integration, acceptance) keep all three tiers cleanly separated. go test ./... (no tags) runs only unit tests.

Acceptance tests#

• Present: Yes — acceptance/ package with //go:build acceptance.
• How: Uses github.com/cli/go-internal/testscript (a fork of rogpeppe/go-internal/testscript). Test scenarios are written in txtar script format and stored in acceptance/testdata/<command>/*.txtar. Each txtar runs real gh subcommands (registered via testscript.RunMain) against a real GitHub instance configured via GH_ACCEPTANCE_HOST, GH_ACCEPTANCE_ORG, and GH_ACCEPTANCE_TOKEN env vars.
• Example (acceptance/testdata/issue/issue-list.txtar):

  exec gh repo create $ORG/$SCRIPT_NAME-$RANDOM_STRING --add-readme --private
  defer gh repo delete --yes $ORG/$SCRIPT_NAME-$RANDOM_STRING
  exec gh issue create --title 'Feature Request' --body 'Feature Body'
  exec gh issue list
  stdout 'OPEN\tFeature Request'

• Coverage: 17+ command domains (api, auth, issue, label, pr, release, repo, ruleset, search, secret, ssh-key, variable, workflow, extension, gpg-key, org, project).
• Custom commands: defer (cleanup on test exit), env2upper, replace (env var substitution in files), stdout2env, sleep.
• Separation: //go:build acceptance tag; never run in normal CI — requires a live GitHub org.

CI configuration#

• Workflow (go.yml): Runs go test -race -tags=integration ./... on ubuntu-latest, windows-latest, and macos-latest. The -race flag is applied universally — every test run is race-detected.
• Cross-platform: All three OS targets are required to pass. This is intentional — gh supports Windows as a first-class platform and several tests exercise path-handling behavior.
• Attestation integration: Separate integration-tests job builds the binary with make and runs test/integration/attestation-cmd/run-all-tests.sh.
• Additional checks: lint.yml (golangci-lint), codeql.yml (SAST), bump-go.yml (automated Go version updates).

Test quality observations#

• What’s done well:

  • The runF injection pattern creates a clean seam between CLI parsing and business logic, enabling unit tests that don’t shell out to a subprocess. This is architecturally sound and consistently applied across all 35+ commands.
  • httpmock.Registry with defer http.Verify(t) provides both stub injection and consumption assertion in two lines. The Exclude() method (which fails the test if a stubbed URL is called) enables negative-path testing.
  • Three tiers (unit → integration → acceptance) are cleanly separated by build tags. The default go test ./... is fast and hermetic; opt-in tags progressively test more of the real system.
  • Cross-platform CI with -race on every commit reflects mature testing discipline.
  • testscript txtar acceptance tests are human-readable, self-documenting, and produce clear failure output (stdout/stderr captured per step).
  • Per-command fixtures/*.json files keep test data close to the tests that use them and make API contract changes visible as fixture diffs.

• What could improve:

  • Test helpers are somewhat scattered: pkg/httpmock, test/, pkg/jsonfieldstest, and per-package runCommand helpers all serve overlapping purposes. A consolidation pass would reduce the surface area newcomers need to learn.
  • The test/helpers.go ExpectLines function is marked Deprecated but still present; some older tests likely still use it, diluting assertion precision.
  • The acceptance test tier requires a live GitHub org — there is no mid-tier option (e.g., a recording/playback layer) for scenarios that are too complex for httpmock but don’t need a real org.
  • Some test files still use t.Fatal(err) directly after errors rather than require.NoError(t, err), mixing testify and stdlib idioms within the same file.

• Patterns worth emulating:

  • runF injection as a universal CLI test seam — trivially testable commands without process spawning.
  • httpmock.Registry + defer Verify(t) — lightweight HTTP stubbing with automatic unused-stub detection; avoids standing up test servers while still exercising serialization.
  • txtar acceptance scripts for readable, maintainable end-to-end scenarios that serve as living documentation.
  • Per-command fixtures/ JSON for keeping API contract test data versioned alongside the code that consumes it.