Building an Agent Harness: A Step-by-Step Guide

Most AI agent failures I’ve investigated in production share a common root cause: the team built the agent before they built the harness. They wired a language model to a few tools, watched it work in a demo, and shipped it. Six weeks later, they’re drowning in cascading failures, unrecoverable state, and zero insight into what the agent actually did.

A harness is not optional scaffolding. It is the system that makes agents reliable, observable, and safe to run at scale. It governs the execution loop, enforces tool contracts, manages state transitions, and provides the hooks that let you intervene when things go wrong. Without a harness, you don’t have an agent system — you have a language model with a prayer.

This guide walks through building a production-grade agent harness from first principles. We’ll go step by step: from defining the core loop, through tool registration and state management, to observability and failure recovery. The examples are drawn from patterns I’ve seen work at scale — and from the more instructive failures.


What Is an Agent Harness?

Before we build one, let’s be precise. An agent harness is the runtime infrastructure that wraps a language model and manages its agentic execution. It is distinct from the model itself, the tools the model calls, and the application logic that drives the agent’s goal.

Think of it as the operating system for your agent. Just as an OS manages process scheduling, memory, I/O, and fault recovery — independent of any specific application — the harness manages the agent loop, tool dispatch, context windowing, and execution lifecycle — independent of what the agent is trying to accomplish.

The harness sits between three concerns:

  • The model layer: the LLM API, prompt construction, token management
  • The tool layer: external systems, APIs, file I/O, databases
  • The application layer: goals, tasks, user intent, business logic

A well-designed harness means that swapping models, adding tools, or changing business logic each requires touching exactly one layer — not all three simultaneously.


Step 1: Define the Execution Loop

The agent loop is the heartbeat of your harness. Everything else is in service of making this loop reliable.

A minimal loop looks like this:

while not terminal:
    response = model.complete(context)
    if response.has_tool_calls:
        results = dispatch(response.tool_calls)
        context.append(response, results)
    else:
        return response.content

This is deceptively simple. In production, every line of this loop is a potential failure point. Let’s harden each one.

Model Completion

Don’t call the model API directly in your loop. Wrap it in a completion provider abstraction that handles:

  • Retries with exponential backoff on transient failures (429s, 503s)
  • Timeout enforcement — a hung completion call will block your entire loop
  • Response validation — confirm the response has the structure you expect before proceeding
  • Token budget tracking — if you’re approaching context limits, you need to know before the model truncates your history silently
class CompletionProvider:
    def complete(self, context: Context, timeout_ms: int = 30000) -> ModelResponse:
        with enforce_timeout(timeout_ms):
            response = self._call_with_retry(context)
            self._validate_response(response)
            self._track_tokens(context, response)
            return response

Loop Termination

Define explicit terminal conditions before you write a single line of agent logic. At minimum:

  • Natural completion: the model returns content without tool calls
  • Max iterations: a hard ceiling — typically 20–50 steps depending on task complexity
  • Max tokens: stop if cumulative context exceeds your budget
  • Explicit abort signals: a tool or human can signal early termination

The max iterations guard is the most important. Without it, a misbehaving agent will run indefinitely, burning tokens and potentially causing real-world harm through repeated tool invocations.


Step 2: Build the Tool Registry

Tools are the primary way agents interact with the world. The tool registry is the harness component that manages tool definitions, validates calls, and routes dispatch.

Tool Schema and Registration

Each tool in the registry needs three things: a schema (what the model sees), an implementation (what actually runs), and a policy (what constraints govern its execution).

@dataclass
class ToolDefinition:
    name: str
    description: str
    input_schema: JSONSchema
    implementation: Callable
    policy: ToolPolicy

@dataclass  
class ToolPolicy:
    requires_confirmation: bool = False
    max_calls_per_run: int = 100
    timeout_ms: int = 10000
    allowed_in_modes: list[str] = field(default_factory=lambda: ["default"])

The policy layer is where many teams cut corners and later regret it. requires_confirmation enables human-in-the-loop for destructive operations — a database delete, a financial transaction, an email send. max_calls_per_run prevents runaway loops where the agent calls the same tool hundreds of times. Set these constraints at registration time, not scattered across your business logic.

Input Validation Before Dispatch

Never dispatch a tool call without validating the model’s arguments against the schema. Models hallucinate arguments. They invent parameter names. They pass strings where you expect integers. A validation layer catches these before they reach your production systems:

def dispatch(self, tool_call: ToolCall) -> ToolResult:
    tool = self.registry.get(tool_call.name)
    if tool is None:
        return ToolResult.error(f"Unknown tool: {tool_call.name}")

    validation_error = validate_against_schema(tool_call.arguments, tool.input_schema)
    if validation_error:
        return ToolResult.error(f"Invalid arguments: {validation_error}")

    if self._policy_blocked(tool, tool_call):
        return ToolResult.error("Tool call blocked by policy")

    return self._execute_with_timeout(tool, tool_call.arguments, tool.policy.timeout_ms)

Return structured errors back to the model — not exceptions up the call stack. The model can often self-correct from a well-formatted error result. An unhandled exception terminates the loop entirely.


Step 3: Design the Context Manager

Context is your agent’s working memory. The context manager is responsible for maintaining the message history, managing token budgets, and ensuring the model always receives a coherent, within-budget view of the conversation.

The Three Context Problems

Problem 1: Context overflow. Long-running agents accumulate more history than the model’s context window can hold. You need a compression or truncation strategy. The naive approach — dropping the oldest messages — is usually wrong. It discards the system prompt, the original task, and early tool results that informed later decisions.

A better approach: maintain a pinned segment (system prompt, task definition, critical constraints) that never gets dropped, and apply a sliding window or summarization strategy to the middle of the conversation history.

Problem 2: Stale context. Tool results that were accurate at step 3 may be incorrect by step 15 if the underlying data has changed. For long-running tasks, consider expiring time-sensitive tool results and re-fetching when the model references them.

Problem 3: Context manipulation. Tool results can contain prompt injection attempts — adversarial text designed to override the system prompt or hijack the agent’s goal. The context manager should sanitize tool results before inserting them into the model’s context, or at minimum flag them for human review.

A Practical Context Structure

@dataclass
class Context:
    pinned: list[Message]          # Never truncated: system prompt, task
    history: list[Message]         # Sliding window of turn history  
    working_memory: dict[str, Any] # Structured state outside the token budget

    def to_model_messages(self, token_budget: int) -> list[Message]:
        pinned_tokens = count_tokens(self.pinned)
        available = token_budget - pinned_tokens - RESPONSE_RESERVE
        return self.pinned + self._fit_history(available)

The working_memory dict is a pattern I’ve found valuable in complex agents: structured state that lives outside the context window but is available to your harness logic. Tool results, computed values, and intermediate artifacts can live here, with the model receiving only a compact reference rather than the full content.


Step 4: Implement State Management

An agent that can’t be resumed is an agent you can’t trust with long-running tasks. State management is what separates disposable demos from production systems.

Checkpoint Architecture

Design your harness to checkpoint state at every step boundary. A checkpoint contains the complete information needed to resume execution:

@dataclass
class AgentCheckpoint:
    run_id: str
    step_number: int
    context_snapshot: Context
    tool_call_history: list[ToolCallRecord]
    working_memory_snapshot: dict
    created_at: datetime
    status: Literal["running", "paused", "completed", "failed"]

Checkpoint storage can be as simple as serialized JSON in a key-value store. The key requirement is that resuming from checkpoint N produces deterministic behavior — the agent sees exactly the same context it would have seen if it had run continuously.

Idempotent Tool Calls

Checkpointing only solves half the resume problem. The other half is ensuring that resuming from a checkpoint doesn’t re-execute side effects. If your agent sent an email at step 5 and you resume from step 4, it shouldn’t send the email twice.

Design tools with idempotency keys:

def send_email(to: str, subject: str, body: str, idempotency_key: str) -> EmailResult:
    if already_sent(idempotency_key):
        return EmailResult.already_sent()
    return _send_and_record(to, subject, body, idempotency_key)

The harness can generate idempotency keys deterministically from the run ID and step number, so they’re consistent across resume attempts.


Step 5: Add Observability

You cannot operate a production agent system you cannot observe. Observability for agents has three layers: tracing, metrics, and audit logging.

Distributed Tracing

Every agent run should produce a trace that shows the complete execution graph: which tools were called, in what order, with what arguments, and what they returned. This is essential for debugging failures and for the compliance audits that enterprise customers will eventually require.

Instrument your harness at the loop level, not the application level:

class InstrumentedHarness(AgentHarness):
    def run_step(self, context: Context) -> StepResult:
        with tracer.span("agent.step", step=self.step_count) as span:
            response = self.completion_provider.complete(context)
            span.set_attribute("model.tokens_used", response.usage.total_tokens)

            if response.has_tool_calls:
                for call in response.tool_calls:
                    with tracer.span("tool.call", tool=call.name) as tool_span:
                        result = self.dispatch(call)
                        tool_span.set_attribute("tool.success", result.is_success)

            return StepResult(response=response)

Key Metrics to Track

At minimum, instrument these metrics per run:

  • step_count: how many loop iterations did this run take?
  • tokens_consumed: total input + output tokens
  • tool_call_counts: per-tool breakdown of how many times each tool was called
  • tool_error_rate: fraction of tool calls that returned errors
  • run_duration_ms: wall-clock time for the complete run
  • termination_reason: natural, max_steps, timeout, error, or aborted

These metrics will reveal patterns that individual traces cannot: which tool is your most common error source, which task types require the most steps, where your token budget is being consumed.

Audit Logging

Audit logs are different from traces. Traces are for debugging. Audit logs are for accountability — an immutable record of what the agent did and why. Store them separately, with tamper-evident guarantees if your use case requires it.

An audit log entry for a tool call should include the full arguments (or a hash if they contain PII), the full result, the timestamp, and the run context that led to the call.


Step 6: Build the Error Recovery Layer

Errors in agent systems compound. A tool failure at step 3 can cause the model to take a suboptimal action at step 7 that causes a catastrophic error at step 12. The error recovery layer is what prevents individual failures from becoming cascading disasters.

Error Classification

Not all errors are equal. Classify them before deciding how to handle them:

  • Transient errors: network timeouts, rate limits — retry with backoff
  • Tool errors: invalid arguments, permission denied — return to model as error result, let it self-correct
  • Validation errors: model produced structurally invalid output — retry completion with added constraint
  • Budget errors: token or step limit reached — checkpoint and surface to human
  • Safety errors: policy violation detected — abort immediately and alert

The Self-Correction Pattern

For tool errors that result from model mistakes (wrong arguments, wrong tool name), return a structured error result to the model and allow it to retry. Models are surprisingly good at self-correcting from well-formatted error messages. Add a self_correction_attempts counter per step and cap it at 2–3 to prevent infinite retry loops within a single step.

Human Escalation

Design escalation paths before you need them. Some errors should not be automatically resolved. When the agent encounters a decision it cannot make autonomously — a tool requires confirmation, a budget is exhausted, an ambiguity cannot be resolved from context — the harness should pause execution cleanly and surface the issue through a defined channel: a webhook, a Slack message, a task queue entry for human review.


Putting It Together: A Reference Architecture

A production agent harness has these components, each independently testable and deployable:

AgentHarness
├── CompletionProvider     (model API + retry + timeout + token tracking)
├── ToolRegistry           (schema + implementation + policy)
├── ContextManager         (history + compression + injection defense)
├── StateManager           (checkpointing + resume + idempotency)
├── ObservabilityProvider  (tracing + metrics + audit log)
└── ErrorRecoveryLayer     (classification + self-correction + escalation)

The harness itself should be thin — an orchestrator that wires these components together and runs the loop. Business logic lives in the tools and the application layer above. The harness doesn’t know what the agent is trying to do; it only knows how to run it safely.

A team at a mid-sized fintech I advised built their first harness without a state manager. When a 40-step reconciliation run failed at step 38, they had no way to resume — they had to restart from step 1, re-executing 37 tool calls against live financial systems. After implementing checkpointing with idempotency keys, they recovered a failed run in under two minutes with zero duplicate transactions.


Common Mistakes to Avoid

Hardcoding tool lists. Register tools dynamically. Hardcoded lists become maintenance burdens and make it impossible to add tools without touching harness code.

Treating context like a log. Context is a resource with a hard limit. Manage it actively from day one. Adding a context manager after the fact is significantly harder than designing for it upfront.

Skipping the policy layer. It feels like overhead until the agent calls a destructive tool 47 times in a loop. Add tool policies during initial registration.

Building observability last. Instrumentation added after the fact is always incomplete. Build it into the core loop from the start.

Single-threaded assumptions. Production systems run multiple agent instances concurrently. Design your state manager and tool registry to be safe for concurrent access from day one.


Next Steps

A harness is not a one-time build. It evolves with your agents. Start with the execution loop and tool registry — these are the minimum viable harness. Add state management when your tasks exceed a few minutes. Add full observability before you go to production. Add the error recovery layer before you expose the agent to external users.

The investment pays off quickly. Teams that build a solid harness first ship reliable agents faster than teams that bolt reliability on afterward — because they spend their time building agent capabilities, not debugging mysterious production failures.

Ready to go deeper? Explore our companion guides on agent state management and memory persistence and building an observability stack for production agents. If you’re evaluating harness frameworks rather than building from scratch, our agent framework cost-efficiency benchmarking analysis covers the major options with real production data.


Dr. Sarah Chen is a Principal Engineer specializing in production AI agent systems. Her work focuses on the architectural patterns that determine whether agent systems survive contact with production workloads.

Leave a Comment