What is the best harness for each model?
Archived — this story has rotated out of today’s deck. It is kept here in full.
The gist
Runta's FrontierHarness Eval v1.1 ran 27,280 coding-agent runs and found each model had a different best-value harness.
Switching models may mean switching harnesses, and 64.5% of spend went to failed runs.
Background
Runta is a company that builds AI agent tooling; it ran the benchmark on its own runtime with Baseten, which served all five models and priced costs at Baseten list prices times reported tokens. A harness is the software layer that wraps a model and drives it through tasks. Runta's earlier v1.0 ranked nine harnesses on a single model, Kimi K3, and found pass rates within 17 points but median cost per pass differing 17-fold, which it suspected came from the harness-model pairing. V1.1 expands that to 22 harnesses × 5 models × 62 terminal tasks × 4 attempts, 76 times the size of v1.0.
How it unfolded
- Nov 12, 2026A review of 7 agent harnesses said OpenAI's Codex is a snappy harness for ChatGPT models and Cursor's Grok Bot is a great harness for Grok models.
- Nov 15, 2026A Hugging Face harness leaderboard reported that on the same model the best and worst harness are up to 35 points apart.
- Nov 16, 2026A GitHub list of 187 AI agent harnesses, rescored weekly, reported that on SWE-bench Pro swapping only the harness moved GLM-5.2 from 23% to 52% pass@1 and Gemma 4 26B from 15% to 36%, and that harness rankings barely transfer across models.
- Nov 17, 2026Runta published FrontierHarness Eval v1.1, covering 22 harness configurations × 5 models × 62 terminal tasks × 4 attempts (27,280 runs, 83B tokens); each model had a different best-value harness.
- Nov 17, 2026Runta reported that by cost per task the median harness moved 11 of 22 places between models, and that 64.5% of spending went to trials that failed.
Key terms
- harness
- Software testing setup of stubs and drivers that imitates infrastructure to test an application or component.
- pass@1
- A clustering method that partitions observations into k groups, each assigned to the nearest cluster mean.
Source: Wikipedia
Who’s saying what
- Official
- Runta says users should consider switching harnesses when switching models to maximize their experience, and that reruns do not explain the cost-rank moves.
- Expert
- Runta founder Guanlan says the model still sets most of the bill (93% of cost variation) and the harness is the second lever; paying more doesn't buy passes, and output tokens rather than starting prompt size predict cost.
- Caution
- Commenters on daily.dev were intrigued that harness choice matters as much as model choice but pushed back on the benchmark's methodology, task selection transparency, and whether pass/fail metrics reflect real-world coding quality.
Still unverified
Commenters dispute the benchmark's methodology, task selection transparency, and whether pass/fail metrics reflect real-world coding quality; a Hacker News commenter said the founder's posts were flagged, likely for AI patterns in his writing.