What is the best harness for each model?
daily.dev

What is the best harness for each model?

Archived — this story has rotated out of today’s deck. It is kept here in full.

The gist

Runta's FrontierHarness Eval v1.1 ran 27,280 coding-agent runs and found each model had a different best-value harness.
Switching models may mean switching harnesses, and 64.5% of spend went to failed runs.

Background

Runta is a company that builds AI agent tooling; it ran the benchmark on its own runtime with Baseten, which served all five models and priced costs at Baseten list prices times reported tokens. A harness is the software layer that wraps a model and drives it through tasks. Runta's earlier v1.0 ranked nine harnesses on a single model, Kimi K3, and found pass rates within 17 points but median cost per pass differing 17-fold, which it suspected came from the harness-model pairing. V1.1 expands that to 22 harnesses × 5 models × 62 terminal tasks × 4 attempts, 76 times the size of v1.0.

How it unfolded

  1. Nov 12, 2026A review of 7 agent harnesses said OpenAI's Codex is a snappy harness for ChatGPT models and Cursor's Grok Bot is a great harness for Grok models.
  2. Nov 15, 2026A Hugging Face harness leaderboard reported that on the same model the best and worst harness are up to 35 points apart.
  3. Nov 16, 2026A GitHub list of 187 AI agent harnesses, rescored weekly, reported that on SWE-bench Pro swapping only the harness moved GLM-5.2 from 23% to 52% pass@1 and Gemma 4 26B from 15% to 36%, and that harness rankings barely transfer across models.
  4. Nov 17, 2026Runta published FrontierHarness Eval v1.1, covering 22 harness configurations × 5 models × 62 terminal tasks × 4 attempts (27,280 runs, 83B tokens); each model had a different best-value harness.
  5. Nov 17, 2026Runta reported that by cost per task the median harness moved 11 of 22 places between models, and that 64.5% of spending went to trials that failed.

Key terms

harness
Software testing setup of stubs and drivers that imitates infrastructure to test an application or component.
pass@1
A clustering method that partitions observations into k groups, each assigned to the nearest cluster mean.

Source: Wikipedia

Who’s saying what

Official
Runta says users should consider switching harnesses when switching models to maximize their experience, and that reruns do not explain the cost-rank moves.
Expert
Runta founder Guanlan says the model still sets most of the bill (93% of cost variation) and the harness is the second lever; paying more doesn't buy passes, and output tokens rather than starting prompt size predict cost.
Caution
Commenters on daily.dev were intrigued that harness choice matters as much as model choice but pushed back on the benchmark's methodology, task selection transparency, and whether pass/fail metrics reflect real-world coding quality.

Still unverified

Commenters dispute the benchmark's methodology, task selection transparency, and whether pass/fail metrics reflect real-world coding quality; a Hacker News commenter said the founder's posts were flagged, likely for AI patterns in his writing.

Sources

See today’s stories in the app Gistline — free on the App Store