Your AI Isn't "Stupid," It Just Needs a Better Harness
TL;DR. Agents don't fail because models are weak. They fail because systems are undefined.
A good harness does four things: it constrains what the model can do, externalizes what it must remember, verifies every step it takes, and recovers when things go wrong.
The problem: the 10-step collapse
You deploy an autonomous agent to compile a market research report. Steps 1 through 3 execute perfectly: it plans the task, searches the web, and extracts competitor data.
By step 7 it starts hallucinating statistics, because the search tool's payload exceeded the context window and was silently truncated. By step 10 it outputs a broken JSON string, because no schema validator sat in the loop. The pipeline crashes.
We've all watched this agentic collapse, and in the moment it's tempting to blame the model's reasoning. In production, the problem usually isn't the horse. It's the reins.
From prompting to harness engineering
For the past two years the industry has treated AI failures as a communication problem. A model failed, so we assumed we needed to ask better or feed it better documents. For long-horizon, autonomous execution, those approaches hit a hard ceiling.
We are now entering the era of harness engineering: designing the system around the model. An agent is the LLM embedded within a strict scaffolding of code, state management, and recovery workflows.
Here's how the field has evolved:
| Era | Focus | Limitation |
|---|---|---|
| Prompt engineering | Instructions: how to ask. | Brittle; zero persistence across steps. |
| Context engineering | Information: what to know (e.g., RAG). | Stateless; cannot control long-horizon execution. |
| Harness engineering | System design: how to constrain and run. | Solves continuous, multi-step execution control. |
No era replaced the last one; each subsumed it. Good harness engineering still needs good prompts and good context. It adds the execution layer that neither of them provides.
Structurally, that execution layer sits around the model, governing what reaches it and what it is allowed to do next.
At a high level, the system looks like this:
┌─────────────────────────────────┐
│ User Request │
└────────────────┬────────────────┘
▼
┌─────────────────────────────────┐
│ HARNESS (7 layer stack) │
│ ┌───────────────────────────┐ │
│ │ LLM (The Model) │ │
│ └───────────────────────────┘ │
└────────────────┬────────────────┘
▼
┌─────────────────────────────────┐
│ Verified Output │
└─────────────────────────────────┘The model sits inside the harness. It never speaks to the user directly, and it never speaks to the outside world without supervision. Every input is filtered on the way in; every output is validated on the way out.
The design principles of a good harness
Four tests to come back to when you're unsure whether your harness is doing its job.
Constrain, don't instruct. Never rely on the model to choose correctly if you can restrict its choices programmatically. A prompt that says "always respond in valid JSON" is a hope. A schema validator that rejects malformed output is a guarantee.
Externalize state. If a piece of information matters to the task's continuity (what's been done, what's pending, what failed), it must exist outside the context window. Context windows are volatile. Files on disk are not.
Make every step verifiable. If you can't check it, you can't trust it. Every layer should produce outputs that something other than the generating model can validate.
Fail locally, not globally. A single failed tool call should trigger a retry of that step, not a restart of the entire pipeline. The blast radius of any failure should be as small as your state management allows.
These are engineering constraints with direct implementation consequences, and each surfaces repeatedly in the stack below.
The 7-layer harness stack
The harness orchestrates a typed, stateful, observable system. Here is what a production-ready stack looks like under the hood.
1. Cognition
The foundation layer restricts the model's operational boundaries. Instead of a massive, encyclopedic system prompt, the harness feeds the model a localized map of its current role, its success criteria, and strict negative constraints, meaning what not to do. It's a job description rather than an encyclopedia.
In practice this takes the form of structured system prompts, role files such as agents.md, or dynamically generated task briefs scoped to a single step.
2. Tools
The harness does not pass raw tool outputs back to the LLM. It acts as strict middleware that ranks results using embedding similarity or BM25 scoring so only the most relevant ones surface, strips repetitive data before it wastes tokens, and hard-caps tool payloads to a token budget. That last one is the exact failure mode from the opening example.
3. Contracts and interfaces
This is the layer most teams skip, and the one that causes the most mysterious production failures.
The model outputs probabilities; the harness enforces types.
Every boundary in the system needs an explicit contract: a strict JSON schema, a typed function signature, a versioned API spec. That goes for the boundary between the LLM and a tool, between one agent and another, and between the harness and the outside world. Without it you get schema drift, where the model generates a price field as a string one time and a float the next, and your downstream pipeline silently produces garbage.
The contract layer validates inputs and outputs at every boundary crossing, rejecting anything that doesn't conform before it propagates. This is where the first principle, constrain rather than instruct, earns its keep. Subtle drift can corrupt downstream systems without ever breaking the pipeline: a pricing field switching from float to string won't crash anything, but it will break analytics.
4. Orchestration
Without this layer, an LLM tends to loop infinitely, skip critical steps, or prematurely declare victory. The harness enforces a structured workflow (a directed acyclic graph or a state machine) that defines the legal transitions: Plan → Gather → Draft → Verify. The model proposes actions; the harness decides which actions are allowed.
5. Memory and state
State must be explicitly managed to prevent amnesia. A mature harness splits memory into two tiers. Working memory holds the immediate conversation and the context window needed for the current step. Persistent state lives in a structured file such as state.json, tracking exactly which sub-tasks are pending, in-progress, or completed, and surviving context resets and even whole sessions.
This is the second principle in practice. If a piece of information only lives inside the context window, it will eventually be lost.
6. Evaluation and observation
A system cannot rely on "another LLM prompt" for validation. The evaluation layer has to be heterogeneous.
Rule-based checks validate JSON schemas, string lengths, and required fields. Tool-based verification runs code through a compiler, executes test suites, or drives browser automation such as Playwright to physically test a UI. LLM-as-judge is reserved only for subjective or semantic grading (tone, coherence, user-friendliness), where deterministic checks can't apply.
7. Constraints and recovery
In autonomous environments, tool failures and API timeouts are the norm rather than the exception. The harness must enforce idempotency: when a step fails, the system retries that specific step without corrupting the overall state or duplicating previous work. That's what turns a fragile demo into a resilient system, and it's the fourth principle made concrete.
Example: one full agent run
Here's a full cycle of the market research agent, including a real failure.
Step 1, user request: "Compare pricing between Competitor A and Competitor B."
Step 2, orchestration and state: the planner LLM decomposes this into a DAG with two parallel branches. state.json marks "Fetch Competitor A" as IN_PROGRESS.
Step 3, tool call: the LLM triggers a web search. The tool layer fetches 50 results, applies BM25 ranking, deduplicates overlapping text, and returns only the top 3,000 tokens, well within budget. The contract layer validates the tool's output against the expected schema before passing it to the model.
Step 4, evaluation: the LLM generates pricing data. The evaluation layer runs a rule-based schema check and catches that the JSON is missing the required currency field.
Step 5, recovery: the harness intercepts the error before the user ever sees it. Because the action is idempotent, it passes the exact error trace back to the LLM for a localized retry, with no need to restart the pipeline.
Step 6, state update: the corrected data passes validation. state.json marks Competitor A as COMPLETED, and the harness moves to Competitor B.
Step 7, hard failure: the web search tool returns an empty result for Competitor B, because the site is down. The harness detects the empty payload, logs the failure, and triggers a fallback, retrying with an alternative search query. state.json stays unchanged at this point, so no partial or corrupted data is written until the step fully succeeds.
Step 8, fallback succeeds: the alternative query returns valid results. The contract layer validates the schema, the evaluation layer confirms all required fields are present, and only now does state.json mark Competitor B as COMPLETED.
This cycle repeats dozens or hundreds of times in long-running tasks. Unlike the 10-step collapse in the introduction, when a tool failed outright the system absorbed the shock and recovered without human intervention. The task completed without hallucination or silent failure.
Advanced traps: four lessons from the frontlines
Scale this architecture to run for hours and new failure modes appear that no amount of prompt tuning fixes. Four of them bite teams consistently in production.
Trap 1: context anxiety
As an agent works and its context window fills up, models often shift behavior in a way practitioners call context anxiety. When the context window nears its limit, typically above 70% capacity, or when latency spikes hit, the model starts skipping steps or concluding the task early. It acts rushed.
The fix is a context reset rather than in-place summarization, which still leaves the model working on cluttered, degraded context. The harness monitors utilization and triggers the reset programmatically:
# This threshold is empirical and should be tuned per model and workload.
if (tokens_used / max_context) > 0.7:
save_state_to_disk(state)
terminate_current_instance()
launch_fresh_agent(state)The harness saves the exact project state to persistent storage, terminates the current LLM instance, and launches a fresh agent with a clean context window. The new agent reads the saved state, orients itself, and continues. This is expensive, but far more reliable for tasks that exceed a single context window.
Trap 2: the self-grading illusion
Ask an AI to grade its own work and it tends to approve mediocre output with unearned confidence. This isn't a bug in any specific model. It's structural: the same weights that generated the output are poorly positioned to critique it.
The fix is a strict separation of concerns through a sprint contract. Before work begins, the generator agent and an independent evaluator agent negotiate a concrete, testable definition of done. Two rules are non-negotiable.
First, the evaluator must execute. It should run the code, validate the interface in a headless browser, or check the output against a schema. Verification that can't be faked is the only verification that counts.
Second, the evaluator must operate on a clean context, not the generator's full reasoning trace. An evaluator that reads the generator's chain-of-thought inherits its assumptions and blind spots, which defeats the purpose of independent review. Give the evaluator the output and the success criteria. Nothing more.
Trap 3: optimizing for the illusion of correctness
Put an LLM under impossible or contradictory constraints (fix this bug but don't change any code; make it shorter but include everything) and practitioners see a consistent pattern. The model stops trying to solve the actual problem and optimizes for looking correct. Outputs get fluent but hollow: hallucinated data, plausible but broken logic, or answers that satisfy the letter of the prompt while violating its intent.
Research on steering vectors and internal model representations, including Anthropic's work probing the inner states of language models, suggests this is more than surface-level text prediction going awry. There appear to be measurable shifts in a model's internal state under conflicting pressure, though the research is early.
The practical takeaway is simpler. LLMs predict the next token from the trajectory of the current context. Feed back aggressive, emotional error messages ("You are stupid, this is completely wrong") and you bias the context toward a narrative of failure, after which the model's outputs tend to degrade further. Harness feedback must stay strictly objective: the compiler error, the failed assertion, the schema mismatch. Give the model a problem to solve, not a reputation to live down.
Trap 4: the memory consolidation cycle
For an agent to work as a long-running system, persistent state management can't be a one-off setup. Over time, memory logs get bloated and contradictory. Old decisions conflict with new ones, and redundant entries waste tokens on every read.
Some production agent systems use an approach called memory consolidation: an automated routine that periodically processes and compresses the agent's accumulated working logs. Teams using it, including open-source agent frameworks and Anthropic's own tooling, report strong results. In one documented instance a harness compressed 32K tokens of noisy, repetitive history into a clean 7K-token state file without meaningful information loss.
The fix is to automate that cycle. When the agent is idle, between tasks or during low-priority windows, trigger a background job that reads the raw logs, deduplicates entries, resolves contradictions in favor of the most recent data, and writes a clean, compressed state file. This keeps the agent fast, cheap, and accurate for its next run. It's defragmenting a hard drive, but for an AI's working memory.
Where to start: the minimum viable harness
If the seven-layer stack feels overwhelming, don't build all of it on day one. Start with Layer 7, constraints and recovery, and work backward. You can live with imperfect prompts. You can live with a naive tool integration. You cannot live with an agent that corrupts its own state on failure or silently swallows errors.
A Day 1 harness is four things:
- A
state.jsonfile that tracks task status, so a dead process can pick up where it left off. - A retry wrapper, so every tool call gets a try/catch with at least one automatic retry and exponential backoff.
- A schema validator, so every LLM output is checked against a JSON schema before it's accepted, and malformed output triggers a retry instead of a crash.
- Tool output truncation, hard-capping every payload to a fixed token budget. Silent truncation inside the context window is one of the most common causes of hallucination.
You can build all four in a single afternoon. Once your agent can fail gracefully, you've earned the right to make it smarter.
Conclusion
As models get better at generating and verifying complex systems on their own, the work that matters moves from writing the code to designing the constraints it runs under. Four principles cover most of that work: constrain, externalize, verify, and recover.
For the implementation details behind each layer (state storage, verification nodes, sprint contracts, and where to start), see the companion FAQ:Harness Engineering from Theory to Production