Most tutorials on building AI agents stop at “call the LLM, parse the output, repeat.” That works for demos. It does not work in production.
After watching dozens of teams ship agents that collapsed under real-world load — hallucinating facts, spinning in infinite loops, burning token budgets on irrelevant context — the patterns that actually survive production become clear. They share three structural properties: a well-designed retrieval layer, a principled agent loop, and disciplined context engineering.
This article is a practitioner’s guide to all three. We’ll build up from first principles, cover the failure modes that aren’t in the documentation, and walk through the architectural decisions that separate agents that scale from agents that don’t.
Why “From Scratch” Still Matters in 2026
Framework proliferation has reached peak noise. LangChain, LlamaIndex, CrewAI, AutoGen, Semantic Kernel — each promises to abstract away the hard parts. And each, under sufficient load, leaks those abstractions back at you at the worst possible moment.
Building from scratch doesn’t mean ignoring frameworks. It means understanding the substrate well enough that when a framework’s opinionated design conflicts with your requirements, you can route around it instead of fighting it.
The three pillars covered here — RAG, agent loops, context engineering — are framework-agnostic. They describe what production agents actually do, regardless of which library wraps them.
Part 1: Retrieval-Augmented Generation (RAG) — Architecture That Holds Under Pressure
What RAG Actually Is (And What It Isn’t)
RAG is not a feature. It’s a system architecture pattern for grounding LLM inference in current, scoped, verifiable information. The canonical description — “retrieve relevant documents, stuff them in the prompt, generate a response” — describes the happy path. Production RAG is mostly everything else.
A production RAG pipeline has five distinct components, each with its own failure budget:
- Ingestion pipeline — chunking, embedding, metadata extraction, index writes
- Query processing — query rewriting, expansion, classification
- Retrieval layer — vector search, keyword search, hybrid ranking
- Reranking — cross-encoder scoring, MMR diversity filtering
- Synthesis — context assembly, citation grounding, answer generation
Most teams build component 3 and call it done. The other four are where reliability actually lives.
Chunking: The Decision That Compounds Everything Downstream
Chunk strategy determines what’s retrievable. Get it wrong and no amount of query engineering fixes it.
The naive approach — fixed-size chunks at 512 tokens — is wrong in two directions simultaneously. Too small and you lose context that crosses chunk boundaries. Too large and retrieval precision degrades because each chunk carries too much semantic signal.
Production patterns that work:
Semantic chunking splits on content boundaries rather than token counts. Paragraph breaks, section headers, and topic shifts are natural split points. Tools like semchunk compute embedding similarity between adjacent sentences and split when similarity drops below a threshold. This costs more at ingestion time but dramatically improves retrieval precision.
Hierarchical chunking stores documents at multiple granularities — sentence, paragraph, section, document — and retrieves at the appropriate level based on query type. A factual lookup (“what is the API rate limit?”) needs sentence-level retrieval. A synthesis task (“summarize the authentication architecture”) needs section-level. Routing between granularities based on query intent is a meaningful quality lever.
Sliding window with overlap is the most pragmatic default for teams that can’t invest in semantic chunking infrastructure. A 256-token window with 64-token overlap catches most cross-boundary context without significant index bloat.
Hybrid Retrieval: Why Vector Search Alone Isn’t Enough
Vector similarity search excels at semantic matching — finding documents that mean the same thing expressed differently. It fails at exact matching: product codes, error message strings, proper nouns with low embedding frequency.
BM25 keyword search has the inverse failure profile. It’s brittle on paraphrasing but reliable on exact terms.
Production retrieval systems use both. The standard pattern:
query → [vector search top-k] + [BM25 top-k]
→ reciprocal rank fusion (RRF) merge
→ reranker cross-encoder scoring
→ top-n to context window
Reciprocal rank fusion is a remarkably robust merge strategy. For a document ranked r in a result list, its RRF score is 1 / (k + r) where k is typically 60. Summing RRF scores across both result lists produces a merged ranking that consistently outperforms either single-retriever baseline.
The reranker is the quality multiplier. A cross-encoder model (BGE Reranker, Cohere Rerank, or a fine-tuned sentence transformer) scores each candidate document against the full query rather than just its embedding. This catches false positives from vector search and recovers relevant documents that keyword search ranked poorly. The cost is latency — a reranker adds 50-200ms depending on candidate count — but the precision gains are consistently worth it for knowledge-intensive tasks.
Retrieval Failure Modes Worth Instrumenting
- Embedding drift: when your LLM is updated but your embedding model isn’t, similarity scores degrade silently. Track embedding model version alongside document metadata.
- Context window stuffing: retrieving top-20 documents when your synthesis model performs best with top-5 is a common trap. Measure answer quality as a function of context document count, not just retrieval recall.
- Query-document distributional mismatch: embedding models trained on general text underperform on domain-specific corpora (legal, medical, code). Fine-tuning the embedding model on domain pairs is almost always worth the investment for specialized agents.
Part 2: Agent Loop Design — Control Flow That Doesn’t Spiral
The Anatomy of a Production Agent Loop
An agent loop is a control flow pattern: observe, reason, act, repeat until done (or until a termination condition fires). The failure modes are almost entirely in the termination logic and the action space definition.
A minimal loop in pseudocode:
def agent_loop(task, tools, context, max_steps=25):
state = AgentState(task=task, history=[], context=context)
for step in range(max_steps):
action = llm_reason(state)
if action.type == "final_answer":
return action.content
if action.type == "tool_call":
result = execute_tool(action.tool, action.args)
state.history.append(ToolResult(action, result))
continue
if action.type == "clarification":
return Clarification(action.question)
return FailureResponse("max_steps_exceeded", state)
This is structurally correct but missing three things every production loop needs: idempotency guards, loop detection, and budget tracking.
Idempotency: Making Repeated Actions Safe
An agent that retries on transient failures (the right behavior) but hits a non-idempotent endpoint twice (destructive) is a production incident waiting to happen. Every tool in the agent’s action space should be classified:
- Safe + idempotent: reads, queries — call freely
- Idempotent writes: PUT-style upserts — safe to retry
- Non-idempotent writes: POST-style creates, financial transactions — require deduplication keys, retry budgets, and explicit human confirmation gates for high-stakes variants
Enforce this at the tool wrapper layer, not in the loop itself. The loop should be able to reason about tool safety from metadata without baking in per-tool logic.
Loop Detection: Catching Agents That Spin
The most pernicious failure mode is an agent that makes apparent progress — calling tools, updating state — but never converges on a final answer. This manifests in three patterns:
Semantic loops: the agent asks the same question in slightly different phrasings, gets slightly different answers, and cycles between them. Detect by embedding each reasoning step and checking cosine similarity against recent history. Similarity above 0.92 for three consecutive steps is a reliable loop signal.
Oscillation: the agent alternates between two conflicting conclusions, applying fix A then reverting to fix B. Track action sequences as n-grams and flag repeated patterns.
Scope creep: the agent expands the task definition at each step until it’s trying to solve a different problem entirely. This is harder to detect algorithmically — it’s best addressed with explicit task scope anchoring in the system prompt and a separate validator that checks whether the current subtask is still traceable to the original goal.
Parallelizing Agent Steps
The synchronous loop above processes one action at a time. Many real tasks have independent subtasks that can run concurrently: retrieve from multiple data sources, call multiple APIs, spawn specialized sub-agents for different task segments.
The production pattern is a fan-out/fan-in loop variant:
def parallel_agent_loop(task, tools, context):
plan = llm_plan(task, context) # decompose into subtasks
subtask_futures = [
execute_subtask_async(subtask, tools)
for subtask in plan.independent_subtasks
]
results = await asyncio.gather(*subtask_futures, return_exceptions=True)
return llm_synthesize(task, results)
Parallelism multiplies throughput but also multiplies failure surface. Implement circuit breakers at the subtask executor level — a subtask that exceeds its latency budget should return a graceful failure result rather than blocking synthesis indefinitely.
Part 3: Context Engineering — The Discipline That Determines Everything
Context Is Your Agent’s Working Memory
The context window is not a prompt. It’s the agent’s entire working memory for a given inference call. What goes in determines what comes out — and most agents waste 40-60% of their context budget on content that degrades rather than improves output quality.
Context engineering is the practice of treating the context window as a constrained resource and allocating it deliberately.
The Five-Zone Context Model
Structure your context window into five functional zones:
Zone 1 — System identity and constraints (2-5%): role definition, behavioral guardrails, output format requirements. Keep this tight. Every token here is overhead on every call.
Zone 2 — Task specification (5-10%): the current task, success criteria, scope boundaries. Be explicit about what the agent should not do — constraint specification reduces hallucinated scope expansion significantly.
Zone 3 — Retrieved knowledge (30-50%): RAG results, tool outputs, reference documents. This is the high-variance zone. Over-fill it and the model loses track of the task; under-fill it and the model hallucinates to compensate. The right fill level is task-specific — calibrate empirically.
Zone 4 — Working memory / history (20-40%): prior reasoning steps, tool call results, intermediate conclusions. Compress aggressively. Replace verbose tool outputs with structured summaries. Maintain a rolling window of the last N steps rather than full history.
Zone 5 — Output scaffolding (5-10%): few-shot examples, format templates, chain-of-thought anchors. Position these immediately before the generation prefix for maximum influence.
Dynamic Context Assembly
Static context assembly — building the same context structure for every call — is the wrong default. Production agents need dynamic assembly based on task type, available budget, and retrieval confidence.
A practical implementation pattern:
class ContextAssembler:
def __init__(self, token_budget: int, model: str):
self.budget = token_budget
self.tokenizer = load_tokenizer(model)
self.zones = ZoneBudgetConfig(model) # per-model zone allocations
def assemble(self, task, retrieved_docs, history, task_type):
context = []
remaining = self.budget
# Zone 1: always include, never compress
system = self.render_system(task_type)
context.append(system)
remaining -= self.count_tokens(system)
# Zone 2: task specification
task_spec = self.render_task(task)
context.append(task_spec)
remaining -= self.count_tokens(task_spec)
# Zone 4: compressed history (fill before retrieved docs on continuation tasks)
history_budget = min(remaining * 0.4, self.zones.history_max)
compressed_history = self.compress_history(history, history_budget)
context.append(compressed_history)
remaining -= self.count_tokens(compressed_history)
# Zone 3: retrieved knowledge — fill remaining budget
docs_budget = remaining * 0.85
selected_docs = self.select_docs(retrieved_docs, docs_budget)
context.append(selected_docs)
return self.render_final(context)
History Compression: The Technique That Extends Effective Context
Long-running agents exhaust their context budget on history long before the task completes. The standard solution — truncate the oldest messages — loses critical early-task context.
Production agents use semantic compression: replace verbose intermediate steps with structured summaries that preserve decision-relevant information while discarding reasoning scaffolding.
The compression cadence matters. Compress every N steps (N=5-10 is typical) rather than waiting until the context is full. Incremental compression preserves fidelity better than bulk compression of a large history block.
A compressed history entry should answer three questions: what was attempted, what was the result, and what was concluded. Everything else is scaffolding that can be discarded.
The Hidden Cost of Over-Specified Prompts
A common mistake in production agent development is treating system prompt length as a proxy for control quality. Long system prompts that specify every edge case behavior tend to produce agents that:
- Miss novel situations the prompt didn’t anticipate (the specification crowded out reasoning capacity)
- Contradict themselves when two rules conflict (no conflict resolution logic)
- Perform worse on core tasks because instruction-following overhead consumes reasoning capacity
The practitioner heuristic: a system prompt should specify what the agent is and what it cannot do, not a procedure for every scenario. Behavioral rules belong in the system prompt. Operational procedures belong in retrieved tool documentation or few-shot examples in Zone 5.
Integrating All Three: A Reference Architecture
The full production pattern connects these three systems into a coherent architecture:
User Input
│
▼
Query Processor (rewrite, classify, expand)
│
▼
Context Assembler ◄─── RAG Retrieval Layer (hybrid search + rerank)
│ │
│ └─── Vector Index + BM25 Index
│
▼
Agent Loop (plan → act → observe → compress history → repeat)
│
├─── Tool Executor (idempotency-classified, circuit-broken)
│
├─── Loop Detector (semantic + pattern-based)
│
└─── Budget Monitor (token, step, latency, cost)
│
▼
Graceful Degradation → Final Response
Every component in this architecture has observable failure modes. The harness — the scaffolding that runs, monitors, and recovers this system — is what separates a working prototype from a production service.
Observability: You Can’t Tune What You Can’t See
Production agent systems require three distinct observability layers:
Trace-level: full span tracing across every loop iteration, tool call, and retrieval operation. LangSmith, Arize Phoenix, and Weights & Biases Weave all provide this. The critical attribute to capture: token budget consumed per zone, per step.
Metric-level: retrieval recall@k, answer faithfulness, tool success rate, loop convergence rate, p95 latency per loop iteration. These are the levers for systematic quality improvement.
Behavioral-level: agent decision pattern analysis — which tools are called in which sequences, where loops tend to form, which task types have the highest step counts. This is where architectural improvements come from.
Teams that instrument only the final response quality miss the 80% of the signal that lives inside the loop.
Where to Go From Here
Building AI agents from scratch requires understanding these three subsystems deeply before reaching for a framework. That understanding shapes every architectural decision downstream — which framework to adopt, where to customize, and where to extend.
The patterns described here represent the current production baseline. Each has failure modes we haven’t covered: adversarial query injection into RAG pipelines, context poisoning via malicious tool outputs, temporal consistency failures in long-running agent sessions. Those are topics for future deep dives.
If you’re architecting a production agent system, start with the context assembler. It forces every upstream decision — chunk size, retrieval k, history window — into explicit, measurable tradeoffs. That discipline propagates upstream into better RAG design and downstream into more stable loop behavior.
If you’re debugging an agent that already exists, start with loop detection and token budget instrumentation. Most production failures are loop failures or context failures, and both are visible in the traces before they manifest as user-facing problems.
The harness — the engineering discipline that holds these systems together under production conditions — is what harness engineering is actually about. The LLM is the engine. You’re building the vehicle.
Dr. Sarah Chen is a Principal Engineer specializing in production AI agent architecture. She writes about the engineering decisions that determine whether AI systems hold up under real-world conditions.
Have a production agent architecture problem? Read our deep dives on agent harness architecture and persistent memory for AI agents for deeper coverage of these topics.