What is the best harness for each model?
Evaluating 22 different “harnesses” (agent runtimes and tool setups) across several open-weight coding models, a new benchmark finds that choice of harness can change cost per successful task by up to 4.4×, and that 64% of inference spend in the tests went to runs that failed the benchmarks. Commenters debate how meaningful these results are, noting limited task coverage, high variance, and the gap between binary pass/fail metrics and real-world workflows where humans steer, review, and iterate with agents. Many argue that harnesses should be tuned both to specific models and to specific domains or projects, and that bespoke or evolving harnesses — rather than “one size fits all” defaults — will matter increasingly as models improve.
Eval setup & headline framing
- The blog evaluates 22 harnesses × 5 open-weight models on 62 Terminal Bench tasks, with 4 attempts each, focusing on cost per pass.
- Some commenters say the original submission title (“64% of inference spend went to runs that failed”) was sensationalized relative to the blog’s neutral title.
- Clarification: “failed” means “didn’t pass all test criteria,” not infrastructure/runtime failure.
Harness vs model importance
- Many agree models dominate performance/cost, but harness choice is a meaningful second lever, especially around step count and tool usage.
- Some argue harness-aware training and self-evolving harnesses are the future; closed, non-editable harnesses are called an “anti-pattern.”
- Others report that, in their experience, the same model in different harnesses often behaves similarly, while switching models changes behavior dramatically.
Methodology concerns & statistical validity
- Critics question:
- Small, selective subset of Terminal Bench tasks (17 TB4, 45 TB2.1) and unclear task selection criteria.
- High variance in long, hard tasks and limited attempts (4 runs per task).
- Whether the reported differences would hold in a larger, repeated sample.
- Some view the study as more marketing than rigorous science.
Coding agents, failure rates, and real-world use
- Several note that humans also fail often; many development attempts and prototypes don’t “pass” on first try.
- Many see high failure spend as expected and acceptable: cost of failure is low, and tokens on failed runs are part of search/exploration.
- Others question whether binary pass/fail on hidden tests is a good proxy for “success” in real workflows where humans guide, review, and refactor.
Optimizing harnesses and models
- Suggestions include:
- Reducing steps per pass instead of just shrinking prompts, since each step resends context.
- Tuning harness prompts per model (some prompts help one model, hurt another).
- Harness-side tuning (prompts, tools, skills) versus model fine-tuning; some say tuning the harness is usually easier.
- There is interest in last-mile fine-tuning for specific harnesses, but also concern about compute cost.
Definitions and scope of “harness”
- Confusion exists over “harness” vs “agent.” One simple framing: harness + model ≈ agent.
- Some see harnesses as mostly simple loops with prompts and tool calls; others emphasize that multi-agent orchestration, budgets, and subagents make harness design crucial.
Limits of benchmarks & user needs
- Commenters stress that single-task benchmarks miss project-level concerns: cross-task memory, evolving requirements, and domain-specific workflows.
- Several want benchmarks including popular proprietary models and non-coding, tool-rich harnesses.
- Takeaway for many: use such evals as rough guidance, but expect to build or tune a harness tailored to your domain.