team.management vs autoresearch.
autoresearch is Andrej Karpathy’s overnight-experimentation pattern — the inspiration for the optimize protocols, credited on our landing page. The difference: the loop still explores freely, but the path around it is frozen.
What autoresearch is
autoresearch (≈91k stars as of July 2026, released March 2026) is Karpathy’s demonstration of a research org run by agents: give an AI a small but real LLM training setup and let it experiment autonomously overnight — modify train.py, train for a fixed five minutes, keep the change if val_bpb improved, repeat. You wake up to a log of experiments and, hopefully, a better model. The repo is deliberately small — three files — because the point is the pattern.
The README is clear about how you steer it: you don’t touch the Python — you edit program.md, the Markdown instruction file that sets up your “autonomous research org.” And the training harness, prepare.py, is protected by one line of instructions: “Do not modify prepare.py.”
What team.management is
team.management’s optimize and optimize-unattended protocols take that idea and put an enforced lifecycle around it — for any repo and any scalar metric. Setup asks for the metric, the stopping rules, and the frozen paths. The measurement script is checked for determinism before the loop may start. A baseline is captured. Then the loop runs — one hypothesis, one commit, each measured by the engine, not the model. Completion squashes the baseline-to-best diff and ships the leaderboard with the MR/PR.
Two protocols share that lifecycle. optimize is the interactive twin: checkpoints between batches return control to you for approval. optimize-unattended removes the checkpoints and runs to a termination condition — the autoresearch shape, built for exactly the overnight run.
The core difference — what holds the loop
In autoresearch, everything that holds the loop is prose. program.md tells the agent how to run it. “Don’t touch prepare.py” is a convention the agent honors because it was asked. Even the scoreboard runs on trust: the agent creates and updates results.tsv itself. In a reference repo, that’s a feature — you iterate on the instructions and watch what happens. In a repo you answer for, running unwatched at 4am, it’s a gap: instructions hold until they don’t. team.management moves those load-bearing parts into the engine — the step sequence, the untouchable paths, the measurement, the stopping rules — and leaves the model free where freedom is the point: inside the experiment.
Overnight, that gap has a precise name: nothing stops a fabricated result. autoresearch’s agent runs the training, greps its own run.log for the metric, writes its own row to results.tsv, and marks it keep or discard. A run that hallucinated a better number — or quietly edited the evaluation — produces the same morning log as one that earned it. optimize closes that path. The engine runs the measurement itself on every commit — “LLM-passed metric values are not trusted,” in its own words. The metric script and eval data sit behind frozen paths. And at synthesis, an audit sweeps the whole run for gaming patterns: frozen-path edits, hardcoded best-metric constants, hand-edited leaderboard rows. The audit is a heuristic; the weight sits on the engine-owned measurement.
Could you unfreeze the paths mid-run? You — yes; it’s your config and your repo. The agent — no. Overnight, that difference decides what you wake up to: a leaderboard, or a surprise.
| autoresearch | team.management | |
|---|---|---|
| What it is | A deliberately small reference repo (nanochat training) | Protocols built into the team.management engine |
| The loop is held by | program.md — Markdown instructions the human iterates | Seven enforced steps: setup → metric script → baseline → experimentation → synthesis → review → completion |
| Metric integrity | The agent checks its own result — and maintains results.tsv itself | The engine measures every commit and writes the leaderboard; the model never touches the scoreboard |
| A faked result | Indistinguishable from a real one — the morning log is self-reported | Can’t reach the leaderboard — the engine runs the measurement; a synthesis audit flags gaming patterns |
| Untouchable files | “prepare.py — not modified”: a stated convention | Frozen paths elicited at setup, enforced during the loop |
| Overnight stopping | Let it run; read the log in the morning | max_iterations / max_duration / regression halts / target metric — or interrupt |
| What you wake up to | A log of experiments and (hopefully) a better model | A squashed baseline→best diff + leaderboard, gated by code review |
| Domain | Single-GPU LLM training, val_bpb | Any scalar metric with a measurement script |
| Repository | github.com/karpathy/autoresearch | github.com/TeamManagementPlugin/claude-plugin |
| Price & license | Free to read; no license file as of July 2026 | Free, MIT |
Study the reference — run it on rails
The idea is Karpathy’s, and the credit is on our landing page. To study the pattern in its purest form, read autoresearch — it’s three files, and a joy to read. When the same pattern has to run against a repo you answer for — with a metric nobody can hallucinate and a diff somebody will review — run it as optimize-unattended. Free experimentation, frozen paths.
FAQ
Is autoresearch a product I can point at my own codebase?
By design, no — it’s a deliberately small reference implementation for one domain: an agent iterating on single-GPU nanochat training against a fixed 5-minute budget and a val_bpb metric. Its value is the pattern, not the packaging. Generalizing that pattern to any repo and any scalar metric — with enforcement around it — is exactly what team.management’s optimize and optimize-unattended protocols are.
What did team.management change from autoresearch’s pattern?
Three things moved from prose into the engine. The loop’s structure: program.md instructions became seven enforced protocol steps (setup → metric script → baseline → experimentation → synthesis → code review → completion). The metric: instead of the agent checking whether its own result improved, the engine measures every commit and keeps a leaderboard the model can’t game with a hallucinated number. The ending: instead of a morning log, you get a squashed baseline-to-best-commit diff with the leaderboard attached, gated by code review.
Is it actually safe to leave optimize-unattended running overnight?
That’s what the guardrails are for: frozen paths declared at setup and enforced during the loop, engine-side measurement of every commit (self-reported numbers never reach the leaderboard), termination conditions (max iterations, max duration, halt after N consecutive regressions, or target metric reached), manual interrupt at any time, a metric-gaming audit at synthesis — and a full code-review gate on the cumulative diff before anything ships.
Facts and figures on this page are as of July 2026, verified against the sources linked inline. If you’re reading this much later — check the sources.