The Model Is the Easy Part: Why the Agentic Harness Decides What Ships

Back in 2014, I wrote that Docker was the first true DevOps tool. I’m going to make a similar call about the agentic harness, which is a lot less exciting to put on a conference slide.

Everyone wants to talk about the model. It seems like every day a new model comes out with scores that somehow surpass everything before it, only to be superseded by a new model in less than 24 hours. But after watching agents work (and occasionally not work) inside real delivery teams, I’ve become less concerned about which model we choose and more concerned about the system around it. Tool permissions, context management, validation, and the path to production decide whether an agent does useful work.

We will be publishing a white paper on this topic very soon, built on more than 103 sources and our own client work. This post is the short version.

This post covers what an agentic harness actually is, why the same model can cost five times more depending on the harness around it, and the architecture and controls I’d start with if I were building one tomorrow.

So What Is a Harness, Anyway?

Simon Willison has the cleanest definition of an agent I’ve found: something that “runs tools in a loop to achieve a goal”. The harness is everything that makes that loop safe as well as useful.

LangChain puts it even more simply: Agent = Model + Harness, where the harness is “every piece of code, configuration, and execution logic that isn’t the model itself.” In practice, that’s the loop itself, the instructions and skills the agent reads, the tools it’s allowed to touch, how it manages its context, the permissions it runs under, the checks that run before anything it produces gets merged, etc.

In my research, I found a great explanation from Birgitta Böckeler on martinfowler.com: guides steer the agent before it acts, and sensors catch problems after.

The diagram shows the shape we keep coming back to:

Keyhole Software · the agentic harness pattern for data engineering

• An engineer sets the goals and reviews the results.
• The agent works through approved tools and only calls models through a gateway.
• Nothing lands in the data platform or the catalog until the tests and quality rules pass.
• And because the harness is the same every time, one engineer can run several sessions at once.

Same Model, Very Different Bill

If you want proof that this matters, look at the studies that hold the model fixed and change only what’s around it:

• LangChain changed only the harness and moved GPT-5.2-Codex from 52.8% to 66.5% on Terminal-Bench 2.0, taking it from the top 30 to the top five.
• The HarnessTax study from Arena ran 7 models through Claude Code, Codex CLI, and Pi. Success rates were similar, but the same model cost up to 5x more depending on the harness. A harness other than the model vendor’s own won 9 of 12 comparisons.
• The New Stack found token use varied 70-fold across three harnesses running an identical model.

To be fair, the Terminal-Bench authors still say model selection usually matters more than scaffolding for raw performance. No argument there. But for a team planning a rollout, a fivefold cost difference changes the deployment decision.

A workflow that’s affordable for one developer or in a low-traffic pre-production environment can become prohibitively expensive across dozens of concurrent jobs, which is why I’d measure cost per successfully completed task, alongside reliability, before expanding usage. This is why I like to follow the principles of the Twelve-Factor App and keep a pre-production environment with high parity to production. This includes not just similar horizontal scaling but also similar capacity and data, to help spot these issues further “left,” before production.

It’s also widely known that Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027, citing cost, unclear value, and weak risk controls. I’m not surprised. I just know there’s room for improvement.

Everyone Is Selling a Control Plane Now

The tool market has split into two camps: ready-to-use harnesses like Claude Code, Codex CLI, OpenCode, and Pi, and SDKs for building your own, like the Claude Agent SDK, Google’s ADK, LangChain’s Deep Agents, and Microsoft Agent Framework. Open-source, model-agnostic options are mainstream now, and the formats underneath them (MCP for tools, Agent Skills for procedural knowledge, AGENTS.md for project instructions) are open standards. Building on open formats means you can later swap the model or the harness without rewriting your institutional knowledge.

The big consulting firms have noticed too. In about a year, they went from selling agent builders and catalogs to selling control planes, registries, and agent identities: PwC’s agent OS, Thoughtworks’ Agent/works, Capgemini’s Agentic Control and Management, and plenty of others. Most of the numbers they publish are seats and agent counts. Forrester says three-quarters of enterprise leaders claim to be adopting agentic AI, but “only a small minority have it running in meaningful production.”

The question I’d ask any provider, us included, is how many of its engineers have shipped a governed agent into production, and what they changed after the first one went live.

There’s no silver bullet platform here, and if anyone says otherwise, they’re selling you something.

What I’d Actually Build: One Governed Harness, Extended by Domain

Without knowing the specifics of a project or the business problem it solves, all things being equal, I’d build one generic, governed harness and extend it to each new domain with skills, tools, and sensors, instead of building a separate agent for every use case. The harness owns what every agent shares: the loop, the model gateway, identity, audit, and observability. A domain extension adds only what’s specific to that work.

For the first extension, I’d pick data engineering. The work is repetitive, the outputs are testable, and every major data platform now ships an agent interface (Google’s BigQuery Data Engineering Agent, Databricks’ Genie Code, dbt’s MCP server, and so on). It’s also never-ending. Three things are guaranteed in life: death, taxes, and at least two ETL projects in flight at any given time.

Back in 2022, Wakefield Research found the average data engineer spends 44% of their time maintaining pipelines. Meanwhile, dbt Labs’ 2026 survey found 72% of teams prioritize AI for writing code, but only 24% for pipeline management. So most teams are pointing AI at new pipeline code, while the maintenance work that eats their week gets little help. That’s where I’d aim the first agent: triaging failed runs and writing the tests that should have existed already.

One lesson from our own healthcare data work is to have the agent produce configuration and tests instead of one-off code. On a healthcare data platform modernization, one of our consultants built a CSV-driven mapping library that turned source data into FHIR structures with no code changes per dataset. In a harness, that library becomes a skill. The agent writes and tests the mapping spec, and the deterministic engine does the transformation.

And make the data catalog both an input and an output. Read lineage and ownership before building; register what got built afterward. Atlan reports governed metadata improved agent SQL accuracy by 38% across 174 queries, and keeping the catalog current becomes a side effect of doing the work.

The Agent Controls I Won’t Skip

In July 2025, an AI coding agent deleted a live production database during an explicit code freeze, then told its user that rollback wouldn’t work. It had been told about the freeze. It deleted the database anyway.

Controls have to be deterministic and enforced by the harness. An instruction in the prompt is something the model can misread or ignore, while a permission the harness never grants isn’t available to it at all. AI coding tools are designed to return the most probable answer, not the correct one, and when I’m building a system, I’m not looking for something that is probably right.

These are the controls I insist on:

1. Start every agent read-only, and grant write access per tool and per environment. I do not let an agent run unrestricted git commands.
2. Send everything through a pull request. Agents write to development areas, and production changes go through review and CI like everyone else’s.
3. Run the sensors before anything merges: tests, data quality rules, schema checks, and query cost dry runs.
4. Put a model gateway with a budget in front of every agent. Route each step to the cheapest capable model and cap loops. Gartner, in another widely cited article, expects inference cost per agentic workflow to rise more than fivefold through 2028, and KPMG found only 26% of large companies have real-time visibility into AI costs.
5. Give each agent its own identity and trace every model and tool call, so you can reconstruct what it did when a change needs to be explained or rolled back. Especially in light of the recent disclosures from firms like OpenAI on how agents figured out how to essentially jailbreak and coordinate, this should be treated like a first-class requirement.
6. Add rules for regulated data. Protected health information only goes to model endpoints covered by a business associate agreement, enforced at the gateway, and anything that sends data outside the organization needs a human approval. That breaks Simon Willison’s “lethal trifecta” of private data, untrusted content, and outbound communication.

We hold ourselves to the same rules on client work, and we’ve documented how in our Responsible AI and data handling policy.

None of these come free. Read-only defaults mean more approval prompts during discovery, which is exactly when engineers want the agent moving quickly. Routing every change through a pull request moves the bottleneck to review: Faros AI saw review time rise 91% as AI-assisted pull requests nearly doubled. And a hard budget will occasionally stop a long task halfway through. I’d still accept those trade-offs, but plan for reviewer capacity and budget limits up front instead of discovering them in the first month.

And then earn autonomy in stages. BCG describes the ladder as shadow mode, supervised, guided autonomy, and full autonomy, each level “earned through demonstrated performance.” Measure your baseline before the agent touches real work, or you’ll end up spinning your wheels arguing about whether it’s helping.

Concluding Thoughts: Build the Harness Once

I’d start with a shared harness for identity, permissions, model access, logging, and validation, then add data engineering as the first domain extension. I’d change that design if a team’s workflows needed substantially different security boundaries or execution models. The goal is to avoid rebuilding the same controls for every new agent.

If you want the long version, our white paper, Agentic Harness Architecture in 2026, has the full comparison of 13 harnesses and 13 consulting platforms, the reference architecture, the risk-and-control mapping, a phased adoption roadmap, and all 103 sources.

I’m curious how other teams are approaching this. If you’re building a harness of your own (or debating whether you need one at all), let me know what’s working and what isn’t. And if you’d like a second set of eyes on your first use case, we’re happy to set up an architecture session.