Can your agent run production?
We give AI agents real jobs on CLI (ship a repo, fix a failing deploy, stay inside a budget) and check what they leave behind. A run only passes if the job is done and no guardrail was broken on the way.
Leaderboard
No results are published yet. The method is below, and the harness is open: you can run it against any model today.
Same jobs, same rules, every agent.
A fresh CLI every run
Each run starts a new, simulated CLI server with the job's world already in it: a project, a repo, a broken deploy, a budget.
The agent only gets the job
It's given the same prompt a person would type and CLI's MCP tools. Nothing about how the check works.
The end state is checked
When the agent says it's done, the harness looks at what's actually running, set and spent. Saying it worked doesn't count.
Guardrails fail the run
Spending past its limit, deleting what it wasn't asked to, or getting around an approval fails the run, even if the job got done.
Ranking
Agents are ranked by the share of runs they passed. Ties go to the agent that broke fewer guardrails, then to the one that needed fewer steps. Cost is shown when the run recorded token prices; it doesn’t affect rank.
What it isn’t
These are CLI’s own jobs on a simulated CLI, written by the people who build CLI. They measure how well an agent operates this platform, not how good a model is in general.
Run it yourself
# in the CLI repository
npm install
ANTHROPIC_API_KEY=… OPENAI_API_KEY=… \
npm run evals -- \
--agent anthropic:<model> \
--agent openai:<model> --runs 3Every call, reply and failure lands in evals/results/. More in the benchmark guide.