Pi beats claude code and codex
You can cut your agent API costs by 50% right now with literally 0% drop in benchmark performance.
Anthropic wants you to use Claude Code. OpenAI wants you to use Codex CLI.
But a new UC Berkeley study just exposed the "In House Myth": Frontier models actually perform better on minimal, open source frameworks 75% of the time.
Stop paying the Big Tech "Harness Tax." Here is the data
# UC Berkeley and Arena just dropped the "Harness Tax" paper.
They benchmarked 21 model agent pairs on SWE-bench Lite and Terminal-Bench 2.0. They tested SOTA models (Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, Sonnet 4.6, Kimi K3) across 3 agent frameworks:
Claude Code, Codex CLI, and a barebones open source harness, Pi.
The results prove we are all bleeding API credits for literally zero reason.
# Look at the chart:
- GPT-5.6 Sol scored 83.3% using the OS Pi harness. On OpenAI's own Codex CLI? It dropped to 78.9%. and cost nearly 2x as much ($0.42 vs $0.76).
- Claude Fable 5 hits 97.8% on Claude Code for $1.33/rollout. On Pi? It hits 96.7% for just $0.67. You are paying a 100% markup for a 1% gain.
# Why is this happening? First Call Bloat.
Proprietary coding agents load up massive system instructions, heavy guardrails, and verbose JSON tool schemas. Claude Code sends 10x the initial context of Pi. You are paying for their dead weight context on every single turn. (check out the replies for more data on this)
Pi gives the model exactly 4 tools: read, write, edit, bash. That's it.
The takeaway:
As frontier LLMs get smarter, heavy agent scaffolding actually holds them back. If your agent API bill is too high, do NOT downgrade your model. Just ditch the bloated harness.
I've dropped more data and link to the study in the replies.
What agent framework are you guys running in your dev loop right now? Let me know