The AI race may be entering a strange
new phase. For years, everyone obsessed
over the model itself. But now, some of
the biggest names in AI are starting to
focus on something else entirely. They
are calling it harness engineering.
Because apparently the same AI model can
become up to six times more effective
just by changing the system around it.
Same model, same raw capability,
completely different result. So the
question becomes what is the real
difference? That difference is the
harness. The easiest way to understand
it is this. The model is the
intelligence engine. But the harness is
everything around it that turns that
intelligence into reliable work. It
includes the rules, tools, memory, skill
libraries, verification systems, context
management, permissions, fallback paths,
audit logs, and feedback loops that
guide the model before it acts, while it
acts, and after it gives an answer.
Mitchell Hashimoto, the co-founder of
Hashi Corp and creator of Terraform,
helped push the term into the mainstream
earlier in 2026. His framing was very
direct. When an AI agent makes a
mistake, the answer should not be to
just rerun the same prompt and hope it
works next time. The better answer is to
change the system so that entire class
of mistakes stops coming back. That is
the real shift here. Prompt engineering
was mostly about getting the model to do
something right in one interaction.
Harness engineering is about building an
environment where the model keeps doing
the right thing over time. It is the
difference between correcting an AI once
and designing the system so the same
error becomes much harder to repeat. And
that is why the phrase spread so
quickly. Open AI, Anthropic, Langchain
and other parts of the AI industry have
all been moving in this direction even
when they use slightly different words.
Open AAI published its own essay around
the idea and described how this works
inside large code generation workflows.
According to one article, OpenAI
processed roughly 1 million lines of
code and around 1,500 pull requests in 5
months. With humans moving away from
writing every line manually and towards
shaping the environment around the
agent, Langchain compressed the idea
into a simple message that people could
repeat. Martin Fowler's site gave it a
more formal engineering frame. Anthropic
has often been more practical than
terminological, focusing on the actual
systems and safety layers rather than
the label itself. And that matters
because harness engineering is not some
random new buzzword for old prompting.
Prompting, context, and harness work are
related, but they are not the same
thing. If you change the words the model
directly reads, that is prompt work. If
you change what information the model
receives, that is context work. But if
you change the invisible structure
around the model like the tools it can
call, the checks it must pass, the
memory it can trust, the permissions it
has, and the recovery process when
something goes wrong. That is harness
work. A tool by itself is not the
harness. An MCP server by itself is not
the harness. A skill library by itself
is not the harness. Those are
components. The harness is the assembled
system that decides how all those pieces
work together. And this is where the AI
race starts to look very different. A
Stanford and Singua University joint
study reportedly found that the same
model with different harness designs
could vary in performance by up to six
times. The model stayed the same. The
surrounding scaffold changed. That is a
massive result because it suggests that
as Frontier models become more widely
available and more similar in
capability, the advantage moves to the
team that builds the better system
around them. This also helps explain why
AI adoption in the economy still looks
strange. On one side, Goldman Sachs
argued in April 2023 that generative AI
could raise global GDP by 7% or nearly 7
trillion over a decade. That is a huge
macro claim. But by April 2024, Goldman
said only 4% of US firms had actually
adopted generative AI. Even in
information services where you would
expect adoption to be much higher, the
number was just 16% with 23% expected
within six months. So the promise is
massive but the rollout is still uneven.
That gap is not only about access to
models. Plenty of companies can access
strong models. Now the bigger issue is
that they do not yet have the system
layer that turns AI capability into
repeatable productivity. The model may
be powerful, but without the harness, it
remains fragile. It can answer one
question, generate one file, write one
piece of code, or summarize one
document, but it may struggle to operate
reliably inside a real workflow with
memory, permissions, tools, deadlines,
edge cases, and consequences. This is
especially clear with Agentic AI. A
normal chatbot gives an answer. An agent
has to operate over time. It may need to
open a terminal, search files, read
documentation, write code, test the
result, call an API, update a database,
ask for clarification, store memory,
recover from a failed command, and
decide whether an action is safe before
it touches a live environment. Once an
AI model is embedded inside tools,
browsers, terminals, repositories,
memory stores, and external services,
its behavior is no longer determined by
the model alone. It is determined by the
whole system. That is why a new UC
Berkeley paper argues that for agentic
AI, model scaling alone is no longer the
full story. For normal chatbots, the
model matters the most. But once an AI
becomes an agent, once it starts using
tools, opening files, running commands,
remembering things, and taking actions,
the model is only one part of the
machine. The paper says the next major
bottleneck is system scaling or scaling
the harness. A real agent needs several
layers working together. It needs the
LLM itself, which is the reasoning
engine. It needs memory so it can
remember useful information across
tasks. It needs a context system so it
knows what information to put in front
of the model and what to leave out. It
needs skill routing so it can pick the
right tool or workflow at the right
time. It needs an orchestration loop
which controls the sequence of steps.
And it needs verification and
governance. So the agent cannot just
take risky actions without checks,
permissions, logs or roll back path.
That sounds technical, but this is
already happening in serious AI systems.
Clawed code, open claw and cheetah
clause are different kinds of agent
systems, but they all face the same
basic problem. How do you control what
the AI sees, remembers, uses, checks,
and changes? And the first major problem
is context. A lot of people think a
bigger context window automatically
makes an AI agent better, but the UC
Berkeley paper makes a sharper point.
The hard part is not giving the model
more tokens. The hard part is giving it
the right tokens. A million token
context window does not help much if the
useful detail is buried under old logs,
stale notes, irrelevant files, and
conflicting information. That is where
context rot comes in. The model
technically has the information
somewhere in the window, but the signal
gets drowned in noise. That is why real
systems already fight this aggressively.
Recent analyses of clawed code describe
a five- tier compaction system with
things like micro compact to clean up
old tool results and context collapse to
summarize long conversations. And when a
tool produces a massive output like a
giant server error log, the system does
not just dump everything into the model.
It can write the full file to local disk
and give the model only an 8 kilobyte
preview first. So the agent behaves more
like a developer. Check the top of the
log, understand the shape of the
problem, then dig deeper only when
needed. The second problem is memory.
Memory sounds useful, but bad memory can
be dangerous. An agent might remember an
old note about how a codebase works,
miss the fact that the code was
refactored yesterday, and then
confidently apply the wrong fix. The
paper calls this the stale but confident
problem. The memory is outdated, but the
agent treats it like truth. So, a
serious harness treats memory with
suspicion. Something like a memory MD
file should act more like a hint than a
fact. Before the agent edits files or
takes a risky action, it has to check
the live environment and verify that the
memory is still true. Some systems even
clean memory in the background during
idle time, removing contradictions,
compressing useful lessons, and stopping
the agent from slowly filling up with
old or messy information. The third
problem is skills. Giving an agent more
skills sounds like an obvious upgrade,
but it creates another problem. Choosing
the right one. The agent has to know
which skill to use, when to use it, how
to combine it with other skills, and how
to check the result. A specialized tool
can produce an answer that looks
confident and useful while still being
completely wrong. So the real issue is
not just having skills. It is routing
and checking them. That is where harness
engineering becomes practical. A strong
harness does not just give the model
more tools and hope it behaves. It
connects those tools to checks. Did the
task actually finish? Did the output
match the request? Did the system change
safely? Was the tool result verified? Is
the agent even allowed to continue? And
now researchers are taking this one step
further. They are asking whether AI
agents can improve their own harnesses
from experience. That is where
retrospective harness optimization or RH
comes in. A new paper from Microsoft
Research Asia and City University of
Hong Kong introduces RHO as a way for an
agent to improve its harness by looking
back at its own past work. Instead of
needing a labeled validation set with
correct answers, the system studies old
trajectories, finds difficult and
diverse tasks, reruns them, compares
different attempts, diagnoses what went
wrong, and proposes harness updates. The
key part is that RH does not need ground
truth labels. It uses the agents own
preference over different attempts.
First, it selects a small group of past
tasks that are both hard and diverse.
The paper uses a method called DPP to
balance those two things because
choosing only the hardest tasks can
focus too narrowly on one type of
failure while choosing only for variety
can miss the serious problems. Then it
runs multiple attempts on each task and
looks for two signals. Self- validation
checks whether the agent actually
completed the task properly and catches
things like false assumptions, wrong
tool calls, and stopping too early.
Self-consistency compares different
attempts on the same task and looks for
major disagreements in the plan, the
tools used, or the final answer. Those
signals become instructions for
improving the harness. Then, RHO
generates several candidate harnesses,
tests them against the old one, and
keeps the candidate that performs
better, but only if the score is
actually positive. So, it is not
randomly changing the agent. It is using
past failures to decide what the system
around the model should learn. And the
results are the part that makes this
hard to ignore. Using codecs with GPT
5.5, RH improved S.WEB Pro from 0.59 to
0.78 without external grading. And it
also improved terminal bench 2 and Gaia
2. So the gains showed up across coding,
technical work, and knowledge tasks.
What makes this more interesting is that
RHO was not just giving the agent more
memory. It was changing the actual
system around it. the tools, skills,
instructions, and checks that shape how
the agent works. After optimization, the
agent verified its work more often, used
tools more carefully, and performed
better on long tasks where normal agents
usually start falling apart. That points
to the bigger shift. Future agents may
improve by learning from their own work
history. Every task leaves a trail.
Every failure leaves a clue. And
repeated mistakes can become updates to
the harness itself. Of course, that also
creates risk. If an AI can update
persistent behavior from its own
judgments, it can also reinforce bad
habits or unsafe shortcuts. So, serious
systems still need audit logs, human
approval, and safety checks. And the
next phase of AI may be won by whoever
builds the best harness around the
model. Also, if you want more content
around science, space, and advanced
tech, we've launched a separate channel
for that. Links in the description. Go
check it out. If you think harness
engineering is really the next big AI
advantage, drop your take in the
comments. Hit subscribe if this made you
look at AI agents differently. Thanks
for watching and I'll catch you in the
next one.