From 6.7% to 68.3% Task Success: The Harness Made the 10x Difference, Not the Model

Grok Code Fast’s coding benchmark success rate was 6.7%. Swap out a single editing format without touching the model, and it jumps to 68.3%. Not a single bit of model parameters changed.

I ran agents myself over the holidays and had a similar experience. Model releases are coming at a breathless pace, but in practice, the thing that drove extreme performance differences wasn’t the model itself. It was the harness wrapping the model: the combination of system prompt, tool configuration, and middleware.

Same Model, Different Rankings

The LangChain team ran Terminal Bench 2.0 with their own coding agent. They left GPT-5.2-Codex untouched and adjusted only the system prompt, tool configuration, and middleware. The score climbed from 52.8 to 66.5, jumping from outside the top 30 into the top 5 on the leaderboard. Cost of model training: zero.

The key was reasoning budget allocation. Applying xhigh uniformly across all tasks kept results at 53.9%, but splitting by task difficulty into xhigh-high-xhigh pushed it to 66.5%. Problems that were failing due to timeouts got resolved through this allocation strategy. Same model, same token budget, different distribution.

The Editing Format That Was Hiding Capability

An open-source agent developer created an editing approach called hashline. When reading a file, it attaches a 2-3 character hash tag to each line, and the model references only those tags when making edits.

With the old approach, the model had to reproduce the original text character-for-character. One wrong space and it fails. Anyone who has used a coding agent knows the pain of “String not found” errors repeating endlessly. hashline sidesteps this problem structurally.

The results were dramatic. Grok Code Fast went from 6.7% to 68.3%, and Grok 4 Fast cut output tokens by 61%. GPT-4 Turbo went from 26% to 59% through format change alone, and Gemini 3 Flash beat its previous best by 5pp. No model training costs, just a change to the editing interface.

Without a Validation Loop, You Stop at the First Answer

There’s a common failure pattern. The agent writes code, reads back what it wrote, decides it looks fine, and stops there without running a single test.

The LangChain team inserted middleware that forces validation against the task spec right before the agent terminates. A separate middleware also detects “doom loops” where the agent repeatedly edits the same file, nudging it to reconsider its approach. Without these two mechanisms, the score improvement would have been much smaller. Pre-injecting directory structure and available tools into the agent, plus using time budget warnings to push it into a validation phase, also helped.

Cheaper Models Are More Sensitive to the Harness

Models like MiniMax M2.5 and Kimi K2.5 are fast and capable with agentic tool use. Their prices are far lower than large frontier models. The tradeoff is that their baseline knowledge is thinner compared to American large models. MiniMax feels like it was trained specifically for agentic use from the start, a specialization choice made possible by constrained resources, and its low price has driven rapid adoption on platforms like Openclaw.

Looking at hashline benchmark results, weaker models showed dramatically larger performance swings from the format change. MiniMax more than doubled its success rate after applying hashline. Total benchmark cost was around $300.

Benchmarks Are Not Production

One caveat worth keeping in mind. Whether it’s Terminal Bench or the hashline benchmark, these are numbers measured in controlled environments. Real production involves far more variables: codebase size, dependency conflicts, ambiguous requirements. Whether an agent that scores 66.5% on a benchmark performs the same on a 100,000-line legacy project remains unverified. Harness optimization is clearly effective, but directly translating benchmark rankings into production performance expectations is risky.

Still, the direction is clear. There is definitely a range where harness design beats model selection on ROI. A significant portion of the benchmark rankings we see today reflects harness quality, not model capability.