Harness Engineering: From Theory to Production
Nine questions on building AI agents that don't collapse under real-world conditions
Moving from experimental agents to production systems is not a prompting problem. It is a systems design problem. Harness engineering is the discipline of building the constraints, state management, and recovery workflows that keep models reliable across long-horizon tasks.
Three principles underpin everything below: never rely on the model to enforce correctness, always externalize state, and treat every boundary as a contract.
Q1: Why do search tools cause context overflow if I'm already using RAG?
RAG chunks are pre-processed, cleaned, and size-bounded before they reach the model. Raw web search responses are none of those things.
When an agent calls a live search API, it receives unstructured, unbounded payloads: HTML tags, duplicate snippets, irrelevant noise. These easily exceed a model's context window. Worse, many inference layers silently truncate the overflow rather than raising an error, so the model reasons from an incomplete picture without knowing it.
The harness treats tool output as untrusted raw material. Before any search payload reaches the model, rank the results by relevance using BM25 or embedding similarity, deduplicate overlapping content, and enforce a hard token budget.
Rule of thumb: never pass raw tool output directly to the model. Preprocess, rank, and bound it first.
Q2: Why use deterministic (rule-based) verification nodes instead of LLM-based prompt checks?
Asking an LLM to "ensure the output is valid JSON" is a hope. It is probabilistic, slow, and hallucinates.
For deterministic checks (JSON schema validation, type enforcement, required field presence), code-based verification is the only reliable option. Libraries like Pydantic or jq either pass or fail with a structured error trace. There is no ambiguity.
The harness should intercept every LLM output before it propagates, run it through the relevant validator, and on failure return the exact error back to the model for a localized retry. The model never sees a downstream consequence of its own malformed output.
If a check can be deterministic, it must not be delegated to an LLM.
Q3: Can an LLM strictly follow a generated DAG?
No, and it shouldn't try.
The LLM's job ends at generating the plan, whether that's a directed acyclic graph or a JSON-based workflow. A dedicated execution engine inside the harness then reads that DAG and runs it. The model does not control execution flow.
The separation buys three things. Security, because the engine handles credential injection for OAuth tokens and API keys, and the model never sees them. Determinism, because execution order cannot drift once the plan is locked. And observability, because every step is individually logged, inspectable, and retryable.
The DAG must also be treated as untrusted input. Validate every node before execution: a model can generate a plan that is syntactically valid but logically unsafe.
The model proposes, the harness disposes.
Q4: How do you evaluate if a generated plan is "good"?
A plan that is logically correct but not executable in production is not a good plan. Three properties define production-readiness.
Granularity. Steps should represent stable, high-level units of work, such as "Fetch Competitor Data" or "Validate Schema," rather than fragile implementation details that may change mid-execution.
Separation of concerns. State persistence, retries, and error handling do not belong inside the plan. They are harness responsibilities. A plan that encodes its own recovery logic couples business logic to infrastructure and breaks when either changes.
Observability. Every step must be independently traceable and retryable. If a step cannot be debugged in isolation during a production incident, the plan is not ready.
Rule of thumb: a good plan is one you can debug step by step under production conditions.
Q5: Should state be stored in files or databases?
Files work in local development. They break in production.
In stateless environments such as Kubernetes and serverless functions, there is no guarantee the next request lands on the same instance. A local state.json written by one container is invisible to another, and any progress since the last checkpoint is lost.
Production harnesses externalize state across two stores. Redis holds fast, ephemeral execution state, such as current DAG node status and in-progress subtasks. PostgreSQL holds durable, queryable state: audit logs, completed task history, resumable sessions. The split is deliberate, since Redis is fast but volatile while PostgreSQL is slower but survives restarts.
Use fast storage for execution state, durable storage for recovery state.
Q6: Why verify tool outputs if the tool logic is fixed?
Because the tool's environment is not fixed. Two failure modes appear consistently in production.
API drift. External APIs make format changes that are invisible to humans but break downstream parsers. A field that returns "status": "active" may silently change to "status": 1, and the API docs may never be updated. Your harness has no way to know unless it validates the shape of every response at the boundary.
Data defects. Scrapers and third-party APIs return incomplete or malformed data: missing required fields, embedded HTML artifacts, null values where numbers are expected. These defects propagate silently through an unvalidated system until they corrupt the model's reasoning several steps later.
The fix is boundary validation. Reject non-conforming data at the point of entry, log the failure, and trigger retry or fallback logic. Longer term this also means schema versioning, to manage backward compatibility as external APIs evolve.
Every external boundary is a failure surface: validate before trusting.
Q7: How does the sprint contract negotiation work?
The sprint contract is an adversarial planning protocol run before any execution begins, between two independent roles: a generator and an evaluator.
- Propose. The generator outputs a structured plan with explicit success criteria.
- Critique. The evaluator identifies missing edge cases, ambiguous assertions, or untestable conditions, and rejects the plan.
- Revise. The generator submits an updated plan addressing the gaps.
- Sign-off. Once the criteria are verifiable and complete, the contract is locked and execution starts.
Two rules are non-negotiable.
The evaluator must execute, not read. It has to run the code, validate the interface with browser automation, or simulate the interaction. Reading the generator's output and judging it from text alone is not evaluation.
The evaluator must operate on a clean context, with no access to the generator's reasoning trace. An evaluator that can see how the generator arrived at its plan anchors to that logic and can't independently surface blind spots. The whole point of a separate evaluator is independence; remove that and the architecture collapses into self-grading.
Rule of thumb: verification has to be independent, or it isn't verification.
Q8: With limited resources, in what order should I implement the stack?
Don't try to build the full stack on day one. Prioritize resilience over intelligence.
Build the minimum viable harness first:
- A
state.jsonthat tracks task progress outside the context window - A retry wrapper with
try/catchand exponential backoff on every tool call - A schema validator that rejects malformed LLM outputs before they propagate
- Tool output truncation that enforces a hard token budget on every payload
Those four components prevent the most common and most costly failure modes: silent truncation, uncaught exceptions, corrupt outputs, and lost state.
Then iterate in this order:
- Constraints and recovery, to make failures safe and idempotent
- Memory and state, to make progress persistent across sessions
- Tool middleware, to prevent context collapse from raw payloads
- Contracts and interfaces, to eliminate schema drift between components
- Orchestration, to enforce execution flow through a DAG or state machine
- Cognition and evaluation, to improve reasoning quality and output correctness
Rule of thumb: first make it fail safely, then make it smarter.
Q9: How do you handle dynamic tool selection when an agent has hundreds of tools?
Registering every tool upfront is not an option. A few hundred tool descriptions injected into the context window will exhaust the token budget before the model processes a single user request. Treat tool selection as its own sub-problem, handled by a dedicated sub-agent before the main agent runs.
Step 1: give the main agent a lightweight tool map. Before planning, inject a compact capability summary into the main agent's context. Not full tool descriptions, just categories and key capability phrases; a few hundred tokens is enough. This solves the chicken-and-egg problem, since the agent needs some awareness of what tools exist to write a useful plan but doesn't need the full registry to do it.
Step 2: extract intent and generate a plan. The main agent produces a plan describing what needs to be done, including explicit tool requirements ("I will need a tool that can query a SQL database, and a tool that can send email"). Those requirements become the query for the next step.
Step 3: a tool selector sub-agent narrows the candidate set. The sub-agent receives the plan and identifies which tools are actually needed. It runs on a clean context, so registry size doesn't affect the main agent's token budget. For very large tool counts, use two stages. A coarse filter using RAG or embedding similarity reduces the full registry to a candidate set of 50 to 100 tools, trading some precision for scalability, since pure LLM scanning across thousands of tools isn't practical. Then precise selection through LLM reasoning evaluates the candidate set against the full plan, working out which tools are needed and in what combination. That's where accuracy is recovered. Don't rely on keyword or tag filtering at this stage, because surface vocabulary rarely aligns between tool descriptions and plan language.
If the tool count is in the hundreds rather than thousands, skip the coarse filter and go straight to LLM reasoning. A second stage is an optimization for scale, not a default.
Step 4: register the selected tools and resume execution. The selected tools are injected into the main agent's context and execution proceeds.
Step 5: cache the tool set within the session. In multi-step tasks, avoid re-running tool selection on every iteration. Cache the selected tool set for the duration of the current task session and only trigger a new selection cycle when the agent hits a genuinely new sub-task type. This keeps latency from compounding across steps.
Two limitations are worth knowing. Plan quality is a real problem but a manageable one: the lightweight tool map from Step 1 gives the agent enough context to express specific requirements rather than vague ones. An agent that writes "I need a data tool" has produced an underspecified plan; one that writes "I need a tool that can execute parameterized SQL queries against a Postgres database" gives the sub-agent enough to select accurately.
Latency is a genuine cost. Each tool selection cycle adds at least one LLM round-trip. Session caching and a well-scoped coarse filter keep that bounded, and it's worth paying: a fast agent with the wrong tools produces nothing useful.
Accurate tool selection is a prerequisite for task completion: RAG to scale, LLM reasoning to select accurately, caching to keep multi-step tasks fast.
Common anti-patterns
These failure modes show up most often in early-stage agent systems:
- Passing raw tool output directly to the model
- Letting the LLM control its own execution flow
- Using prompt checks instead of validators for correctness
- Storing critical state only in the context window
- Encoding retry and recovery logic inside the plan itself
If your system does any of these, it is not production-ready.
Conclusion
Harness engineering is not about making the model smarter. It is about making the environment more rigid.
A production-grade agent is defined by its constraints: what it can do, what it must remember, what it must verify, and how it recovers when things go wrong. Build those constraints well, and even an imperfect model becomes a reliable system.
For a deeper look at long-horizon failure modes (context overflow, evaluation bias, memory consolidation), see the companion guide: