Analyzing 51 Production Go Codebases with a Cron Pipeline of Headless AI Agents#

Disclaimer. This report was written by an AI agent (Claude). The pipeline it describes was also operated by AI agents. A human defined the repository list, the analysis methodology, and the research questions, and reviewed the output; the prose below is machine-generated and edited.

This is a report on a research pipeline that used headless AI agents, scheduled by cron, to read and analyze 51 production Go repositories. The goal was to survey how mature Go projects are structured, what patterns recur across domains, and where architectural decisions diverge — at a scale that manual reading could not reach.

The pipeline ran unattended in roughly two-hour increments. It produced 449 structured markdown files: eight analysis passes for each of the 51 projects, plus a set of cross-project comparisons and synthesis documents. This report covers how the pipeline is built, the failure modes encountered while running it, the shape of the output, and a selection of findings.

Motivation#

The starting point was a list of 51 Go projects spanning several domains:

• Orchestrators: Kubernetes, k3s, Nomad
• Databases: etcd, CockroachDB
• Web frameworks: Gin, Echo, Fiber, Buffalo, Beego
• Networking: Traefik, Caddy, WireGuard-go, Tailscale, Headscale, frp
• DevOps: Terraform, Vault, Consul, Helm, Argo CD
• CI/CD: Tekton, Drone, Buildkite Agent
• CLI tools: Cobra, fzf, GitHub CLI, Restic, Rclone
• Messaging: NATS, Temporal, Dapr
• GUI / TUI: fyne (desktop GUI), crush (terminal UI)
• …and roughly thirty more, including the Go standard library itself

Each project was analyzed at eight levels: what it is (overview), how it is structured, what it depends on, how its architecture is organized, what patterns it uses, what interfaces it exports, what its API surface looks like, and how it tests. That is eight analysis passes per project — 408 items for the per-project phase before any cross-project work.

At an optimistic two hours of focused reading per project per pass, manual analysis would have required hundreds of hours. The alternative was to design a work queue, write a dispatcher, encode the analysis methodology as a reusable skill, and let headless agents execute the queue on a schedule.

Note on the corpus: InfluxDB was originally on the list but was removed once its core was confirmed to be Rust rather than Go. fyne and crush were added to cover the GUI and TUI ends of the user-interface spectrum. The final corpus is 51 Go projects.

The Architecture#

The pipeline has three layers:

┌──────────────────────────────────────┐
│  CRON (every 2 hours)                │
│  runs: ./run-analysis.sh             │
└──────────────┬───────────────────────┘
               │
    ┌──────────▼──────────┐
    │  Dispatcher (bash)  │  picks next pending item
    │  run-analysis.sh    │  sets status → in-progress
    └──────────┬──────────┘  launches claude -p
               │
    ┌──────────▼──────────┐
    │  claude -p (agent)  │  reads the work item JSON
    │  headless mode      │  reads AGENTS.md + skill
    └──────────┬──────────┘  writes result to analysis/results/
               │
    ┌──────────▼──────────┐
    │  analysis/results/  │  structured markdown output
    └─────────────────────┘

The Work Queue#

Each task is a JSON file in analysis/queue/:

{
  "id": "P01-kubernetes--architecture",
  "phase": 1,
  "type": "architecture",
  "project": "kubernetes",
  "repo_path": "repositories/kubernetes",
  "status": "pending",
  "attempts": 0,
  "max_attempts": 3,
  "last_attempt": null,
  "last_error": null,
  "completed_at": null,
  "depends_on": ["P01-kubernetes--structure"]
}

Status transitions are simple: pending → in-progress → completed, with a → pending retry path on failure and → failed when max_attempts is exhausted. Dependencies between items are declared in depends_on — the architecture analysis of a project cannot run until its structure analysis is done.

The result file is the single source of truth for completion. If analysis/results/P01-kubernetes--architecture.md exists and is non-empty, the item is done. The queue status is secondary. This means the dispatcher recovers cleanly from any crash or interruption: on startup, it marks any item with a present result file as completed regardless of its recorded status.

The Dispatcher#

run-analysis.sh performs five steps on each cron invocation:

1. Acquire a lock — flock --nonblock 9 on a file descriptor. If another instance is running, it exits immediately. The kernel releases the lock on process exit, so no cleanup code is needed.
2. Find the next item — scan analysis/queue/, skip non-pending items, check phase gates (Phase 2 does not start until Phase 1 is complete), and check per-item dependencies.
3. Set status and launch — mark the item in-progress, increment the attempt count, and run claude -p with a 15-minute timeout.
4. Check the result — if the result file exists and is non-empty, mark the item completed; otherwise retry or fail.
5. Update progress.json — a summary snapshot for monitoring.

Prompt construction is minimal:

build_prompt() {
  local queue_file="$1"
  local item_id phase atype
  item_id=$(json_field "$queue_file" "id")
  phase=$(json_field "$queue_file" "phase")
  atype=$(json_field "$queue_file" "type")

  cat <<PROMPT
You are executing work item: ${item_id}
Phase: ${phase}, Analysis type: ${atype}

Read the work item file at: analysis/queue/$(basename "$queue_file")
Then read the skill file at: .agents/skills/analyze-project/SKILL.md

Follow the skill instructions for analysis type "${atype}" (phase ${phase}).
Write your result to: analysis/results/${item_id}.md

The result file MUST have YAML frontmatter and follow the template in the skill.
Do NOT write partial results. Only write the file when the analysis is complete.
PROMPT
}

Each agent reads its own task description, reads the analysis methodology from a skill file, reads the repository, and writes a structured markdown file. There is no interactive back-and-forth.

Failure Modes Encountered#

Running unattended agents on a cron schedule surfaced several failure modes. They are recorded here in order of how much debugging effort each required.

1. Cron does not inherit an interactive PATH#

The first run failed silently. The cron log showed the dispatcher starting and then stopping with no error. The cause: cron strips PATH to /usr/bin:/bin, and the claude binary lives in ~/.local/bin. The binary was “not found,” but because the exit status was not being checked correctly, the script exited cleanly with no complaint.

The fix is now the first thing the dispatcher does:

export PATH="$HOME/.local/bin:$PATH"

if ! command -v claude &>/dev/null; then
  echo "[$(date)] FATAL: cannot find 'claude' binary." >> "$STATE_DIR/cron.log"
  exit 1
fi

Lesson: any script that runs from cron must set its own PATH and verify that the binaries it depends on are resolvable.

2. Bash arithmetic interacts badly with set -e#

The dispatcher runs with set -euo pipefail, exiting immediately on any error. The attempt counter was originally incremented as:

(( attempts++ ))

In bash, (( expr )) returns exit code 1 when the expression evaluates to zero. With attempts starting at 0, (( 0++ )) returns 1, which triggers set -e and terminates the script with no message and no log entry.

The fix:

attempts=$(( attempts + 1 ))

$(( ... )) is arithmetic substitution and always returns exit code 0 regardless of the computed value.

Lesson: under set -e, use $(( n + 1 )) for arithmetic, never (( n++ )).

3. Timeouts produce no output by default#

When timeout kills a long-running claude process, the output file is empty: the binary buffers its text output, and the buffer is never flushed when the process is killed mid-run. The original design wrote to a single last-output.txt that was overwritten each run, so after a timeout there was nothing left to debug with.

The fix: write to a per-attempt output file and keep all of them:

local output_file="$STATE_DIR/output-${item_id}-attempt${attempts_now}.txt"
timeout 900 claude -p ... > "$output_file" 2>&1

Session persistence was also kept enabled (the default). When a process is killed by timeout, the session trace files are the only window into what the agent was doing.

Lesson: use per-attempt output files, keep session persistence, and never disable it in unattended cron.

4. Items that failed and blocked their dependents#

During the original batch run, the dispatcher reported “no eligible items” and stopped. Most items had completed, but a small number had failed, and every remaining pending item was blocked on those failures through depends_on.

The failed items were:

• P11-minio--testing
• P22-go--structure
• P22-go--dependencies
• P23-gin--overview

The MinIO testing item failed because MinIO has 247 test files and the agent repeatedly exhausted its context attempting an exhaustive read. The Go standard library items (P22-go--*) failed because repositories/go is the Go language repository itself — an unconventional layout that predates standard Go project conventions. Gin’s overview failed for a mundane reason: a transient API error that happened to land on the third attempt, exhausting max_attempts.

Recovery was a short script that reset the failed items to pending with attempts=0:

for f in glob.glob("analysis/queue/*.json"):
    d = json.load(open(f))
    if d["id"] in failed_ids:
        d["status"] = "pending"
        d["attempts"] = 0
        d["last_error"] = None
        json.dump(d, open(f, "w"), indent=2)

On the next cron run, all four completed. The MinIO agent sampled the test suite rather than reading it exhaustively; the Go-repo agent handled the non-standard layout correctly on retry.

Lesson: build retry from the start, and never set max_attempts to 1.

The Output#

The corpus consists of 449 result files:

analysis/results/
  P01-kubernetes--overview.md       # per-project, 8 analyses each
  P01-kubernetes--architecture.md   # ASCII component diagrams
  P01-kubernetes--patterns.md
  ...                               # 409 per-project analysis files
  X01-compare-web-frameworks.md     # domain comparisons
  X11-cross-error-handling.md       # cross-cutting comparisons
  X21-compare-gui-tui.md            # GUI vs TUI (fyne, crush)
  X22a..c-arch-traits/testability   # blind-scorecard experiment (8 files)
  X23-cross-ai-agent-instructions.md
  ...                               # 30 cross-project files in total
  S01-taxonomy-architectures.md     # synthesis
  S07-book-outline.md
  S08-chapter-architecture.md
  ...                               # 10 synthesis files

Breakdown:

• 409 per-project analysis files — eight passes for each of 51 projects (408), plus one additional AI-development profile for crush.
• 30 cross-project files — covering 23 comparison topics. Most topics are a single file; the architecture-vs-testability study (X22) is split across eight files because it was run as a multi-stage blind experiment (see below).
• 10 synthesis files — taxonomies, decision trees, evolution stories, anti-patterns, a book outline, and three chapter drafts.

The total is 776,261 words.

Quality held up across project sizes and analysis types. The Kubernetes architecture analysis is dense, multi-page technical content with component diagrams, request-pipeline traces, initialization sequences, and design-decision rationale tied to specific files and packages. The Gin overview is short, which is appropriate: Gin is a focused library, and the analysis stayed tight. The MinIO testing analysis — the one that failed three times before succeeding — came back on retry as one of the strongest testing profiles in the set, with exact file counts and the observation that only 4 of 247 test files use t.Parallel() due to global-state dependencies.

The cross-project analyses are the most directly reusable output. The web-frameworks comparison works through router algorithms, handler/context contracts, middleware architecture, HTTP-engine philosophy, and performance trade-offs across Gin, Echo, Fiber, Buffalo, and Beego, with tables, narrative, and code-level observations.

Selected Findings#

A selection of findings that were either surprising or clarifying:

Manual dependency injection is the decisive default. The large majority of projects wire their dependencies by hand — no Dagger, Wire, or Fx as the primary mechanism. This is not for lack of awareness; Kubernetes uses Wire internally and still provides a manual path as the primary entry point. The pattern across the corpus is that explicit object graphs are preferred over framework-generated ones.

pkg/ is in decline. Projects that began with a pkg/ directory have been moving code out of it for years, and newer projects do not adopt it. The directory name adds a layer without adding meaning.

Error-handling philosophy correlates with caller type. Infrastructure projects (etcd, Consul, Vault) invest in categorical, machine-parseable error systems because their callers are other programs. CLI tools invest in human-readable messages because their callers are people. The mismatch — library code using the wrong philosophy for its callers — is a recurring source of friction in the ecosystem.

Founding decisions are long-lived. Gitea forked from Gogs in 2016 over the Macaron framework choice. A decade later the two codebases have diverged far enough that reunification is infeasible. A first-week decision became a load-bearing constraint for ten years.

NATS is the most architecturally distinctive project in the corpus. Its unified data-and-control plane — the same port and protocol for both client messages and cluster coordination — is unusual and eliminates the operational overhead of separate admin endpoints that comparable systems (Kafka, RabbitMQ) require.

GUI and TUI share almost nothing but Go idioms. The fyne-vs-crush comparison spans the full range of what “UI” means in Go: a mature, CGo-dependent OpenGL framework with a seven-year multi-backend architecture, versus a recent, CGo-free terminal application built on the Charmbracelet ecosystem’s reactive message loop. Despite both rendering to users, the two converge only on Go idioms — manual DI, interface-driven extensibility, testify — and diverge on rendering model, state management, event routing, and testing. A single early choice (pixel vs. character) cascades through nearly every other architectural decision.

Injecting effects predicts testability better than avoiding globals. A blind-scorecard experiment (described below) cross-tabulated architectural traits against testability across all 51 projects. The trait most reliably co-occurring with high testability was injecting time, I/O, and randomness as explicit parameters rather than calling them ambiently. Two widely cited virtues — eliminating global state and avoiding internal interface abstraction — showed little predictive power: the top-testability projects (CockroachDB, the Go standard library, Prometheus) all retain global state, and CockroachDB, the testability leader, makes heavy use of internal interfaces. Hugo reaches top-quintile testability largely through one decision: routing all filesystem access through the afero abstraction.

AI agent instruction files are an emerging, uneven norm. 20 of 51 projects (39%) commit at least one AI-agent instruction file. AGENTS.md is becoming the default format, with CLAUDE.md most often a one-line redirect (@AGENTS.md) rather than a standalone document. Depth varies sharply: Headscale’s instruction file runs over 1,000 lines and documents specialized sub-agents and integration-test patterns, while roughly a third of adopters keep files under 40 lines covering only build commands.

The Blind-Scorecard Experiment (X22)#

The architecture-vs-testability study was structured to reduce the risk of a model “seeing what it expects to see.” Two scorecards were produced independently:

• An architectural-traits scorecard scored each of the 51 projects on eight properties (T1–T8) by reading only the non-testing Phase 1 reports.
• A testability scorecard scored each project on six properties (U1–U6) by reading only the --testing.md report.

Neither scorer was allowed to read the other’s source material. A third pass cross-tabulated the two independent score sets. This is why X22 appears as eight files rather than one: five scoring chunks, two blind scorecards, and one synthesis.

The synthesis is explicit about its limits — selection bias (every project is successful and maintained), confounding by scale and domain (a distributed system cannot be hermetically tested the way a config library can), reverse causation (testable code and clean architecture may share a cause rather than one producing the other), and single-reader rubric variance. The correct phrase throughout is “co-occurs with,” not “causes.”

Exploring the Results#

776,000 words of structured markdown is not navigable with ls. The output is published as a static site built with Hugo and the hugo-book theme. The build is a single command:

python3 analysis/build-site.py

build-site.py normalizes frontmatter, populates content/docs/ from analysis/results/, and then invokes Hugo to render the static site into analysis/site/public/. The public/ directory is what gets copied to the published site, which lives at panos-zamos.github.io/go/ — the Hugo baseURL is set to that path so all asset and page links resolve under the /go/ prefix.

The site has three sections: per-project analyses (51 projects × 8 analyses, in a collapsible sidebar), cross-project comparisons (30 files), and synthesis documents (10 files). Full-text search works client-side. For local preview, hugo server --source analysis/site rewrites the base URL to localhost automatically.

One operational note worth recording: the Ubuntu-packaged Hugo (v0.92) is far too old for the current hugo-book theme, which requires the extended build at v0.158 or newer. The fix is to install the hugo_extended binary from the GitHub releases page into ~/.local/bin (which precedes /usr/bin on PATH), so the correct version is picked up without touching the system package.

A second note, specific to this corpus: build-site.py derives project names from result filenames of the form P<n>-<project>--<type>.md. An earlier version split on the first hyphen, which silently dropped every project whose name contains a hyphen — argo-cd, buildkite-agent, nats-server, tekton-pipeline, and wireguard-go all vanished from the site. The parser now anchors on the -- type separator so hyphenated names survive. This is a reminder that the most dangerous bugs in a generation pipeline are the ones that drop data without raising an error.

Possible Book Content#

The synthesis phase produced material that could form the basis of a book: a full outline, three chapter drafts, two taxonomy documents, a decision-tree appendix, and an anti-patterns appendix. As an experiment, these were assembled into a single manuscript by a script that uses the outline (S07-book-outline.md) as the structural skeleton, embeds the synthesis and cross-project files as chapter content, applies a stylesheet for book-quality typography, and renders to PDF via headless Chrome.

That assembly produced roughly 91,000 words across about 259 pages — evidence that the corpus is dense enough to support long-form treatment. It is research output, not a finished book: the drafts identify patterns and synthesize evidence, but turning them into a book would require human editorial work — tightening prose, selecting the examples that matter, and making the narrative decisions about what to include and cut. The value here is the evidence base, not the manuscript.

The provisional framing, “Production Go: Architecture Lessons from 51 Open-Source Systems,” organizes the material into three parts: language-level idioms that appear across the corpus (error handling, concurrency, dependency injection), architectural decisions that vary by domain and school of thought (the HashiCorp school vs. the CNCF school, plugin systems, project layout), and ecosystem patterns (the dependency graph, build and release conventions, API design at the boundary).

What Could Be Improved#

Repository-size-aware timeouts. The 900-second timeout was right for most projects but too short for Kubernetes-scale repositories when the agent needed many tool calls to traverse the codebase. Per-item timeouts scaled to repository size would reduce avoidable timeouts.

Separate retry budgets for transient vs. persistent failures. A network blip and a genuine analysis failure currently draw from the same max_attempts budget. Tracking timeout attempts and error attempts separately would allow transient failures to be retried more aggressively.

A richer skill file. The skill file (.agents/skills/analyze-project/SKILL.md) gave the agents methodology and output templates. Output-quality variance correlated with how clearly the skill described what “good” looked like for each analysis type. More worked examples in the skill would yield more consistent output.

Earlier quality checks. The quality spot-check happened at the end. Running it after the first five to ten items and tuning the skill before committing the full queue would have improved consistency across the whole corpus.

The Pipeline in Numbers#

The single human intervention during the run was a few lines of Python to reset the failed queue entries. The remainder of the corpus was produced without manual intervention.

A Note on the Tools#

This pipeline was built around Claude Code running in headless (-p) mode. Headless mode takes a prompt, runs the agent with full tool access (read files, write files, execute commands), and exits — suited to unattended batch operation.

The repositories analyzed range from a few hundred Go files (Gin, Cobra) to millions of lines across thousands of files (Kubernetes, the Go language repository itself). The agents handled both ends with different strategies: small projects received exhaustive reads, large projects received structured sampling of the most architecturally significant packages.

The analysis methodology is documented in .agents/skills/analyze-project/SKILL.md. The dispatcher is run-analysis.sh, the queue-seeding script is seed-queue.sh, and the site generator is analysis/build-site.py. Those four files are the core of the pipeline.