The Model Is Not the Product. The Harness Is.
Every new frontier model launch produces the same argument: which one is best?
GPT-6 Astra or Claude Fable 5.1. Faster or deeper. Better at terminal work or better at reasoning. More capable at browser use or more reliable on a difficult refactor.
It is an attractive question because it has a clean answer. Pick a leaderboard. Circle the bigger number. Declare a winner.
It is also increasingly the wrong question.
In real work, nobody hires a bare model. They hire an agent running inside a system: prompts, tools, permissions, a context strategy, a budget, memory, tests, recovery logic, and a human approval boundary. Change any of those and the same model can look brilliant, average, or unusable.
The unit that matters is not model. It is model × harness × tools × budget × environment × evaluator.
That sounds less satisfying than a benchmark fight. It is much closer to the truth.
A Model Does Not Ship the Work
A language model does not inspect a repository, choose the right files, apply a patch, run the test suite, understand the failure, repair the regression, and ask permission before touching production. An agent stack does.
That distinction matters because the execution layer can determine whether raw capability ever becomes a finished result.
Give a model a vague shell wrapper, a cluttered context window, brittle tool schemas, a ten-turn cap, and no test loop. It may produce a plausible explanation and leave behind a broken change.
Give the same model a clean repository map, reliable patch tools, a stable working memory, access to the relevant tests, room to recover, and a verifier that refuses to accept a merely plausible answer. The result can look like a different model.
This is why one team can swear by a model that another team calls disappointing. They may not be disagreeing about the model at all. They may be describing different harnesses.
The Benchmark Number Has a Configuration Attached
OpenAI reports Astra ahead of Fable 5.1 on its published Terminal-Bench 4.0 configuration: 57.9 percent against 55.8 percent, with a lower estimated task cost. That is useful evidence. It is not a universal law of agent performance.
The score belongs to a specific setup: a particular task set, terminal environment, tool interface, prompt, allowance, verifier, and stopping policy. It tells us how one complete stack performed under those conditions.
Anthropic’s own positioning points in a different but compatible direction. Fable 5.1 is built for demanding reasoning, long-running coding tasks, multistep research, and enormous context. Its API economics make large cached prefixes unusually attractive. If a harness keeps a codebase and working state stable across many turns, cache reads can be much cheaper than repeatedly rebuilding the entire prompt.
Neither claim invalidates the other. Astra may finish a supported tool-heavy task with fewer retries. Fable may be the better economic choice in a persistent loop where the agent must keep a vast repository in view for hours.
The mistake is pretending either result can be separated from the conditions that produced it.
The Harness Is Where Capability Becomes Reliability
The harness has a boring name for something with enormous leverage. It is the layer that decides what the model sees, what it can do, and what happens when it is wrong.
Start with context.
A large context window is not a license to shovel every log file, meeting note, and source file into the prompt. More irrelevant material can make an agent less reliable, not more. A strong harness selects the task-relevant code, the active diff, the failed test output, and a concise summary of what has already been tried. A weak one turns the context window into a junk drawer.
Then come tools.
Models are tuned around particular environments. Shell commands, browser controls, file editors, patch formats, and computer-use actions are not interchangeable. A generic wrapper that returns noisy output or hides essential state can make a capable agent appear incompetent. Conversely, a well-designed native tool interface can remove whole categories of mistakes before the model has to reason about them.
Then comes recovery.
The reliable loop is not “plan, then answer.” It is inspect, modify, test, interpret the failure, repair, and test again. The harness decides whether the agent is allowed to perform that loop, how much it can spend, and when it should stop. A benchmark that rewards a first attempt may flatter an agent that would fail in production. A system that demands a passing verifier may take longer, but it produces work that can survive contact with a codebase.
Finally, permissions change the result.
An agent with unrestricted terminal access is solving a different problem from one in a sandbox with no network, no credentials, and a mandatory approval gate before it can publish. Safety boundaries are not incidental product settings. They change the set of actions the model is capable of taking. Comparing scores without reporting those constraints is like comparing race times without mentioning that one car was allowed on the highway and the other was driven through a warehouse.
The Cheapest Model Is the One That Finishes
The headline API prices for Astra and Fable 5.1 are the same: $10 per million input tokens and $50 per million output tokens. But token price is a poor proxy for cost per completed task.
An agent that completes a task in fewer calls, fewer retries, and fewer human interventions is often cheaper even if its individual tokens cost more. An agent with excellent prompt caching can become cheaper on long-lived work even if it appears expensive on a single, cold-start benchmark.
The actual bill includes more than input and output tokens:
- retry loops;
- failed test cycles;
- context rebuilt from scratch;
- browser and tool latency;
- human review;
- rework after a subtly broken change; and
- the cost of an agent taking an action it should have asked about first.
That last item is easy to ignore until it is not. The best agent is not the one that generates the most output per euro. It is the one that gets accepted work done without creating a cleanup project for someone else.
Stop Buying a Model. Evaluate a Stack.
Teams should stop running prompt shootouts and calling them procurement.
Instead, take a representative set of real tasks: a bug fix, a medium refactor, a browser workflow, a production incident investigation, a research assignment, and a documentation change. Run both complete stacks under the permissions and budgets you will actually use. Keep the task environment and acceptance criteria fixed.
Then measure what matters:
- verified completion rate;
- regression-test pass rate;
- human-review defects and rework;
- wall-clock time;
- total, cached, and output-token use;
- cost per accepted change;
- intervention and recovery count; and
- unsafe or unauthorized attempted actions.
This is slower than comparing a single leaderboard score. It is also the only way to know what you are buying.
There may not be one winner. Astra in Codex may be the better default for tool-heavy terminal work, browser automation, visual QA, and cost-sensitive execution. Fable 5.1 in Claude Code may earn its place on huge repositories, long-running refactors, research-heavy tasks, and loops that can exploit cached context.
That is not indecision. It is routing.
The Next Competitive Edge Is Operations
For the last few years, the industry has treated foundation models as if they were the finished product. That made sense while the gap between model generations was enormous.
The gap is closing in more domains. The differentiator is moving outward: from the weights to the operating system around them.
The organisations that win will not necessarily be those that choose the supposedly smartest model in September. They will be the ones that build the best context discipline, tool interfaces, verification loops, audit trails, permission design, and model-routing policies around the models available in December.
In other words: the model is the engine. The harness is the vehicle, the navigation system, the brakes, and the mechanic who notices the wheel is about to come off.
No serious buyer would judge a fleet of cars by engine specifications alone. It is time to stop judging agent systems that way too.
Sources: OpenAI GPT-6 Astra materials; Anthropic Fable 5.1 documentation; Artificial Analysis Coding Agent Index and Terminal-Bench 4.0; Harness-Bench; Epoch AI SWE-bench methodology; Mara Jade research.
Share this