Agentic AI
Complete guide to building AI agents — architecture, tool use, memory, planning patterns, multi-agent systems, frameworks, evaluation, and production best practices.
What Is an AI Agent?
2 topics
Agent vs. Chatbot vs. Pipeline
Understanding what makes something an agent
CHATBOT
Input → LLM → Output
Single call. No memory. No tools. No decisions.
PIPELINE (Chain)
Input → Step1 → Step2 → Step3 → Output
Fixed sequence. LLM at each step. Deterministic path.
AGENT
Input → LLM decides → Tool call? → Observe result
↑________________________|
Dynamic. Self-directed. Loops until goal is met.
AN AGENT HAS:
✅ A goal / task to complete
✅ Tools it can call (functions, APIs, code execution)
✅ Memory (context of what happened)
✅ A reasoning loop (decides what to do next)
✅ The ability to stop when done
KEY DIFFERENCE:
A pipeline follows instructions.
An agent decides its own next steps.💡 Not everything needs to be an agent — pipelines are simpler and more predictable
⚡ Use agents when the number of steps isn't known upfront
The Agent Loop
The core reasoning cycle every agent runs
THE AGENT LOOP (Observe → Think → Act): ┌─────────────────────────────────────────┐ │ 1. PERCEIVE │ │ Receive task + current context │ │ Load memory (past actions, results) │ ├─────────────────────────────────────────┤ │ 2. THINK (LLM call) │ │ Given goal + context + tools: │ │ - Should I call a tool? │ │ - Which tool? With what args? │ │ - Or is the goal complete? │ ├─────────────────────────────────────────┤ │ 3. ACT │ │ Execute tool call │ │ (search, write code, call API…) │ ├─────────────────────────────────────────┤ │ 4. OBSERVE │ │ Receive tool result │ │ Add to context/memory │ ├─────────────────────────────────────────┤ │ 5. REPEAT or STOP │ │ Loop back to step 2 │ │ Stop when: done / error / max steps │ └─────────────────────────────────────────┘
💡 Each loop iteration costs tokens — keep tool outputs concise
⚠️ Always define an exit condition — agents can loop forever without one
Tool Use & Function Calling
3 topics
Defining Tools (OpenAI)
How to define and register tools for an agent
from openai import OpenAI
import json
client = OpenAI()
# Define tools as JSON schema
tools = [
{
"type": "function",
"function": {
"name": "search_web",
"description": "Search the web for current information. Use when you need facts you don't know.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "The search query"
},
"num_results": {
"type": "integer",
"description": "Number of results to return (1-10)",
"default": 5
}
},
"required": ["query"]
}
}
},
{
"type": "function",
"function": {
"name": "run_python",
"description": "Execute Python code and return the output. Use for calculations, data processing.",
"parameters": {
"type": "object",
"properties": {
"code": {"type": "string", "description": "Python code to execute"}
},
"required": ["code"]
}
}
}
]💡 Tool descriptions are part of the prompt — write them clearly and specifically
⚡ Include examples in descriptions for complex tools — it significantly improves accuracy
The Agent Loop in Code
Full implementation of the tool-use loop
import json
from openai import OpenAI
client = OpenAI()
MAX_ITERATIONS = 10 # Safety limit
def run_agent(task: str) -> str:
messages = [
{"role": "system", "content": "You are a helpful research assistant. Use tools to answer questions accurately."},
{"role": "user", "content": task}
]
for iteration in range(MAX_ITERATIONS):
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
tool_choice="auto", # Let model decide
)
msg = response.choices[0].message
messages.append(msg) # Add assistant turn to history
# No tool calls → agent is done
if not msg.tool_calls:
return msg.content
# Execute each tool call
for call in msg.tool_calls:
fn_name = call.function.name
fn_args = json.loads(call.function.arguments)
print(f"[Agent] Calling {fn_name}({fn_args})")
result = dispatch_tool(fn_name, fn_args) # Your tool executor
messages.append({
"role": "tool",
"content": json.dumps(result),
"tool_call_id": call.id
})
return "Max iterations reached without completing the task."
# Run it
answer = run_agent("What are the top 3 Python web frameworks in 2025?")
print(answer)⚠️ Always cap iterations — models can get stuck in loops
💡 Log every tool call and result for debugging and auditing
⚡ tool_choice='required' forces a tool call; 'none' disables tools for that turn
Tool Dispatcher
Safely routing tool calls to real implementations
import subprocess
import requests
def dispatch_tool(name: str, args: dict) -> dict:
"""Route tool calls to actual implementations with error handling."""
try:
if name == "search_web":
return search_web(**args)
elif name == "run_python":
return run_python(**args)
elif name == "read_file":
return read_file(**args)
else:
return {"error": f"Unknown tool: {name}"}
except Exception as e:
return {"error": str(e), "tool": name}
def run_python(code: str) -> dict:
"""Execute Python in a subprocess sandbox."""
try:
result = subprocess.run(
["python3", "-c", code],
capture_output=True, text=True,
timeout=10 # Never run untrusted code without a timeout!
)
return {
"stdout": result.stdout,
"stderr": result.stderr,
"exit_code": result.returncode
}
except subprocess.TimeoutExpired:
return {"error": "Code execution timed out (10s limit)"}
def read_file(path: str) -> dict:
"""Read file with path validation."""
from pathlib import Path
safe_root = Path("/workspace") # Restrict to safe directory
target = (safe_root / path).resolve()
if not str(target).startswith(str(safe_root)):
return {"error": "Path traversal blocked"}
return {"content": target.read_text()}⚠️ NEVER execute code from agents without sandboxing — use Docker, E2B, or subprocess with limits
⚠️ Always validate file paths to prevent directory traversal attacks
💡 Return structured errors so the LLM can self-correct
Memory Systems
2 topics
Types of Agent Memory
The four memory layers every production agent needs
1. IN-CONTEXT MEMORY (Working Memory) What: The messages list passed to the LLM Size: Limited by context window (128k–1M tokens) Speed: Instant — no retrieval needed Use: Current conversation, recent tool results 2. EXTERNAL MEMORY (Long-term / Episodic) What: Vector DB storing past interactions Size: Unlimited Speed: ~50-200ms retrieval Use: User preferences, past sessions, documents Tools: Pinecone, Weaviate, Chroma, pgvector 3. SEMANTIC MEMORY (Knowledge Base) What: Indexed documents, FAQs, product info Size: Unlimited Speed: ~50-200ms retrieval Use: RAG over company knowledge, docs Tools: LlamaIndex, LangChain, Chroma 4. PROCEDURAL MEMORY (Skills) What: Saved prompts, few-shot examples, tool definitions Size: Small — just the patterns Speed: Loaded at startup Use: Teaching agent HOW to do things consistently STRATEGY: - Short session? → in-context only - Multi-session user? → in-context + external - Large knowledge base? → all four types
💡 Don't stuff everything into context — retrieve only what's relevant
⚡ Summarize old turns instead of truncating — preserves key facts
Implementing Memory with pgvector
Store and retrieve agent memory using PostgreSQL + pgvector
from openai import OpenAI
import psycopg2
import json
client = OpenAI()
def embed(text: str) -> list[float]:
"""Create embedding for text."""
res = client.embeddings.create(model="text-embedding-3-small", input=text)
return res.data[0].embedding
def save_memory(conn, user_id: str, content: str, memory_type: str = "episodic"):
"""Save a memory to the vector store."""
vector = embed(content)
with conn.cursor() as cur:
cur.execute(
"INSERT INTO agent_memory (user_id, content, embedding, type) VALUES (%s, %s, %s, %s)",
(user_id, content, json.dumps(vector), memory_type)
)
conn.commit()
def recall_memory(conn, user_id: str, query: str, top_k: int = 5) -> list[str]:
"""Retrieve relevant memories using cosine similarity."""
vector = embed(query)
with conn.cursor() as cur:
cur.execute(
"""
SELECT content
FROM agent_memory
WHERE user_id = %s
ORDER BY embedding <=> %s::vector
LIMIT %s
""",
(user_id, json.dumps(vector), top_k)
)
return [row[0] for row in cur.fetchall()]
# Use in agent
def build_system_prompt(user_id: str, task: str, conn) -> str:
memories = recall_memory(conn, user_id, task)
memory_block = "\n".join(f"- {m}" for m in memories)
return f"""You are a personal assistant.
Relevant memories about this user:
{memory_block}
Use these to personalize your response."""💡 Use text-embedding-3-small — it's 5x cheaper than large with 90% of the quality
⚡ Add a metadata filter (user_id, type) to avoid retrieving irrelevant memories
Agent Patterns
3 topics
ReAct (Reason + Act)
The most common agent pattern — think before acting
# ReAct forces the model to reason before each action
# Paper: https://arxiv.org/abs/2210.03629
REACT_SYSTEM_PROMPT = """
You are a research agent. Solve tasks step by step.
For each step, use this EXACT format:
Thought: [reason about what to do next]
Action: [tool_name]
Action Input: {"param": "value"}
Observation: [tool result will be inserted here]
... repeat Thought/Action/Observation as needed ...
Final Answer: [your complete answer to the original question]
IMPORTANT:
- Only call ONE tool per step
- Always check the Observation before the next Thought
- Stop when you have enough information for a Final Answer
"""
# Example trace:
# Thought: I need to find current Python popularity stats
# Action: search_web
# Action Input: {"query": "Python most popular language 2025"}
# Observation: {"results": [{"title": "Python #1 for 4th year...", ...}]}
# Thought: I have the data I need
# Final Answer: Python remains the #1 language in 2025...💡 ReAct traces are excellent for debugging — you see exactly what the model was thinking
⚡ Works best with GPT-4o, Claude 3.5+ — weaker models struggle to maintain the format
Plan-and-Execute
Separate planning from execution for complex multi-step tasks
from openai import OpenAI
import json
client = OpenAI()
def plan(task: str) -> list[str]:
"""Step 1: Generate a plan as a list of steps."""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a planning assistant. Break complex tasks into ordered, concrete steps. Return JSON: {steps: [str]}."},
{"role": "user", "content": f"Task: {task}"}
],
response_format={"type": "json_object"}
)
return json.loads(response.choices[0].message.content)["steps"]
def execute_step(step: str, context: str) -> str:
"""Step 2: Execute one step with tools."""
# Run single-step agent with ReAct
return run_agent(f"Context so far:\n{context}\n\nCurrent step: {step}")
def replan(original_task: str, steps: list, completed: list, last_result: str) -> list[str]:
"""Step 3: Replan if execution reveals new information."""
response = client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
"content": f"Task: {original_task}\nCompleted: {completed}\nRemaining: {steps}\nLast result: {last_result}\n\nDo remaining steps still make sense? Return updated plan as JSON: {{steps: [str]}}"
}],
response_format={"type": "json_object"}
)
return json.loads(response.choices[0].message.content)["steps"]
# Run the full pattern
task = "Research and write a comparison of the top 3 vector databases"
steps = plan(task)
results = []
for step in steps:
result = execute_step(step, "\n".join(results))
results.append(result)
steps = replan(task, steps[steps.index(step)+1:], steps[:steps.index(step)+1], result)💡 Replanning is the key differentiator — static plans fail when reality differs from expectations
⚡ Use a cheaper model for planning, a stronger model for execution
Reflection & Self-Critique
Agents that review and improve their own outputs
def reflect_and_improve(task: str, initial_output: str, criteria: list[str]) -> str:
"""
Reflexion pattern: agent critiques its own output
and produces an improved version.
Paper: https://arxiv.org/abs/2303.11366
"""
criteria_str = "\n".join(f"- {c}" for c in criteria)
# Step 1: Self-critique
critique = client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
"content": f"""Task: {task}
Output to review:
{initial_output}
Evaluate against these criteria:
{criteria_str}
List specific issues and improvements needed. Be critical."""
}]
).choices[0].message.content
# Step 2: Revise based on critique
improved = client.chat.completions.create(
model="gpt-4o",
messages=[{
"role": "user",
"content": f"""Original task: {task}
First attempt: {initial_output}
Critique: {critique}
Now produce an improved version that addresses all issues."""
}]
).choices[0].message.content
return improved
# Usage
result = reflect_and_improve(
task="Write a Python function to parse CSV files",
initial_output=agent_output,
criteria=["Handles edge cases", "Has error handling", "Includes type hints", "Has docstring"]
)💡 Even one reflection pass significantly improves output quality
⚡ Great for code generation, writing, and reasoning tasks
Multi-Agent Systems
2 topics
Multi-Agent Architectures
Patterns for coordinating multiple specialized agents
1. SUPERVISOR PATTERN
┌──────────────┐
│ Supervisor │ ← decides who does what
└──────┬───────┘
┌────┴─────┬──────────┐
Worker1 Worker2 Worker3
(research) (write) (review)
Best for: Tasks that can be split by role/specialty
Tools: LangGraph with supervisor node
2. PIPELINE PATTERN (Sequential)
Researcher → Analyst → Writer → Editor → Output
Best for: Tasks with a clear handoff sequence
Each agent adds/transforms the previous output
3. PARALLEL PATTERN
Input ──┬── Agent A (research topic 1) ──┐
├── Agent B (research topic 2) ──┤→ Aggregator → Output
└── Agent C (research topic 3) ──┘
Best for: Independent subtasks that can run concurrently
Reduces total latency significantly
4. DEBATE PATTERN
Agent A: Proposes solution
Agent B: Critiques it
Agent A: Defends or revises
Judge Agent: Selects best outcome
Best for: High-stakes decisions, code review, fact-checking
5. HIERARCHICAL
Manager Agent
├── Sub-manager A → Workers
└── Sub-manager B → Workers
Best for: Very complex tasks needing decomposition at multiple levels💡 Start with the simplest architecture that solves the problem
⚡ Multi-agent adds latency and cost — only use when a single agent truly can't do the job
Multi-Agent with LangGraph
Building a supervisor + worker system with LangGraph
from langgraph.graph import StateGraph, END
from langchain_openai import ChatOpenAI
from typing import TypedDict, Literal
import operator
from typing import Annotated
llm = ChatOpenAI(model="gpt-4o")
class State(TypedDict):
task: str
plan: list[str]
results: Annotated[list[str], operator.add] # Accumulate results
final_output: str
next: str
# --- Specialized agents ---
def researcher(state: State) -> State:
"""Searches and collects information."""
result = run_agent_with_tools(state["task"], tools=[search_web, scrape_url])
return {"results": [f"Research: {result}"], "next": "writer"}
def writer(state: State) -> State:
"""Synthesizes research into a draft."""
context = "\n".join(state["results"])
result = llm.invoke(f"Write a report based on:\n{context}\n\nTask: {state['task']}")
return {"results": [f"Draft: {result.content}"], "next": "reviewer"}
def reviewer(state: State) -> State:
"""Reviews and improves the draft."""
draft = state["results"][-1]
reviewed = llm.invoke(f"Review and improve this draft:\n{draft}")
return {"final_output": reviewed.content, "next": END}
def supervisor(state: State) -> Literal["researcher", "writer", "reviewer", "__end__"]:
"""Routes to next agent based on state."""
return state.get("next", "researcher")
# --- Build graph ---
graph = StateGraph(State)
graph.add_node("researcher", researcher)
graph.add_node("writer", writer)
graph.add_node("reviewer", reviewer)
graph.set_entry_point("researcher")
graph.add_conditional_edges("researcher", supervisor)
graph.add_conditional_edges("writer", supervisor)
graph.add_conditional_edges("reviewer", supervisor)
app = graph.compile()
result = app.invoke({"task": "Write a report on the state of AI agents in 2025"})
print(result["final_output"])💡 Use Annotated[list, operator.add] to accumulate results across agents
⚡ Each node should do ONE thing — resist the urge to pack multiple responsibilities into one agent
Frameworks & Tools
2 topics
Framework Comparison
Choosing the right framework for your agent
FRAMEWORK BEST FOR COMPLEXITY ───────────────────────────────────────────────────────────── LangGraph Stateful multi-agent workflows High (github.com/langchain-ai/langgraph) - Explicit graph structure - Built-in checkpointing/memory - Human-in-the-loop support - Production-ready LangChain Agents Quick single-agent prototypes Medium (python.langchain.com/docs/modules/agents) - Large tool ecosystem - Many pre-built agent types - Good for RAG + agent combos OpenAI Assistants Hosted stateful agents Low (platform.openai.com/docs/assistants) - Managed threads + memory - Built-in code interpreter, file search - No infra to manage - Vendor lock-in CrewAI Role-based multi-agent teams Medium (docs.crewai.com) - Intuitive role/goal/backstory model - Good for content pipelines - Less flexible than LangGraph Autogen Conversational multi-agent Medium (microsoft.github.io/autogen) - Agents that chat with each other - Great for debate/critique patterns - Microsoft / Azure native Smolagents Lightweight, code-first Low (huggingface.co/docs/smolagents) - Agents write Python to solve tasks - Minimal abstractions - Good for open-source models Pydantic AI Type-safe, production-first Medium (ai.pydantic.dev) - Strong typing throughout - Great DX for Python developers - Newer but growing fast
💡 LangGraph for production, LangChain Agents for prototyping, OpenAI Assistants for speed
⚡ You can always start with a simple loop and migrate to a framework later
Sandboxed Code Execution
Safe environments for agents that run code
# Option 1: E2B — cloud sandboxes (recommended for production)
# https://e2b.dev
from e2b_code_interpreter import Sandbox
with Sandbox() as sbx:
execution = sbx.run_code("""
import pandas as pd
df = pd.DataFrame({"a": [1,2,3], "b": [4,5,6]})
print(df.describe())
""")
print(execution.text) # stdout
print(execution.error) # stderr
# Option 2: Docker subprocess (self-hosted)
import docker
client = docker.from_env()
def run_in_docker(code: str) -> str:
result = client.containers.run(
image="python:3.12-slim",
command=["python", "-c", code],
mem_limit="128m",
cpu_period=100000,
cpu_quota=50000, # 50% CPU
network_disabled=True, # No network access!
remove=True,
timeout=10
)
return result.decode()
# Option 3: Pyodide (browser/WASM — zero server risk)
# https://pyodide.org
# Run Python in browser — no server execution⚠️ Never let agents execute code directly on your server without sandboxing
💡 E2B is the easiest production option — managed, fast, supports many languages
Popular Agent Tools
2 topics
Web Search & Browsing
Giving agents access to the web
# Tavily — best for agents (structured, fast, cheap)
# https://tavily.com
from tavily import TavilyClient
client = TavilyClient(api_key="tvly-...")
result = client.search("LangGraph tutorial 2025", max_results=5)
# Returns: {results: [{title, url, content, score}]}
# Brave Search API (privacy-focused alternative)
# https://api.search.brave.com
import requests
res = requests.get(
"https://api.search.brave.com/res/v1/web/search",
headers={"X-Subscription-Token": "your-key"},
params={"q": "your query", "count": 5}
)
# Serper (Google results via API)
# https://serper.dev
import requests
res = requests.post(
"https://google.serper.dev/search",
headers={"X-API-KEY": "your-key"},
json={"q": "your query"}
)
# Browser automation — Playwright (for JS-heavy sites)
# https://playwright.dev
from playwright.async_api import async_playwright
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
await page.goto("https://example.com")
content = await page.content()💡 Tavily is purpose-built for agents — returns clean text, not raw HTML
⚡ Use Playwright only when you need to interact with the page (click, fill forms)
File, Database & API Tools
Connecting agents to your data and systems
# File operations tool
def read_file(path: str, start_line: int = 0, end_line: int = 100) -> dict:
"""Read file with line range to avoid dumping large files into context."""
lines = open(path).readlines()
return {
"content": "".join(lines[start_line:end_line]),
"total_lines": len(lines),
"showing": f"{start_line}-{end_line}"
}
def write_file(path: str, content: str) -> dict:
with open(path, "w") as f:
f.write(content)
return {"success": True, "path": path}
# SQL database tool
import psycopg2
def query_db(sql: str, params: list = None) -> dict:
"""Run read-only SQL queries."""
# IMPORTANT: Use read-only DB user in production!
conn = psycopg2.connect(DATABASE_URL)
with conn.cursor() as cur:
cur.execute(sql, params or [])
rows = cur.fetchall()
cols = [d[0] for d in cur.description]
return {"columns": cols, "rows": rows[:50]} # Limit rows
# HTTP / REST API tool
import httpx
async def call_api(method: str, url: str, body: dict = None, headers: dict = None) -> dict:
"""Make HTTP requests. Use for calling external APIs."""
async with httpx.AsyncClient(timeout=30) as client:
response = await client.request(method, url, json=body, headers=headers)
return {"status": response.status_code, "body": response.json()}⚠️ Give the DB tool a read-only database user — never write access unless explicitly needed
💡 Limit rows/lines returned to avoid blowing up the context window
Human-in-the-Loop
2 topics
When & How to Add Human Checkpoints
Pausing agents for human approval on critical actions
# LangGraph interrupt — pause for human approval
from langgraph.graph import StateGraph
from langgraph.checkpoint.memory import MemorySaver
def agent_node(state):
# ... agent logic ...
return state
def action_node(state):
"""Pauses here until human approves."""
# This node will be interrupted before execution
action = state["planned_action"]
execute_action(action) # Only runs after approval
return {**state, "action_done": True}
builder = StateGraph(State)
builder.add_node("agent", agent_node)
builder.add_node("action", action_node)
builder.set_entry_point("agent")
builder.add_edge("agent", "action")
checkpointer = MemorySaver()
# interrupt_before pauses BEFORE executing the node
app = builder.compile(
checkpointer=checkpointer,
interrupt_before=["action"] # Pause before action_node
)
config = {"configurable": {"thread_id": "task-1"}}
# Run until interrupt
app.invoke({"task": "Delete all inactive users"}, config)
# Inspect what the agent planned
state = app.get_state(config)
print("Agent plans to:", state.values["planned_action"])
# Human approves — resume
app.invoke(None, config) # Resume from checkpoint💡 Use interrupt_before for destructive actions: delete, send email, deploy, charge card
⚡ interrupt_after lets the node run but pauses before the next node — good for review
HITL Patterns for Production
Practical human oversight strategies
WHEN TO REQUIRE HUMAN APPROVAL: ✅ Sending emails / messages on behalf of user ✅ Making purchases or financial transactions ✅ Deleting data or records ✅ Deploying code to production ✅ Posting to social media or public channels ✅ Accessing sensitive PII data ✅ Actions with real-world side effects WHEN AGENTS CAN RUN AUTONOMOUSLY: ✅ Read-only research tasks ✅ Drafting content (with human review before send) ✅ Data analysis and summarization ✅ Internal calculations ✅ Test environments (not prod) IMPLEMENTATION OPTIONS: 1. Synchronous (blocking) Agent stops → sends Slack/email to human Human responds → agent resumes Pro: Simple. Con: Slow. 2. Asynchronous with timeout Agent sends request with deadline Human approves before timeout → executes Timeout expires → safe fallback (skip/escalate) 3. Optimistic with rollback Agent executes but logs everything Human can reverse within N hours Pro: Fast. Con: Requires undo capability. 4. Risk scoring Low risk (<30%) → auto-approve Medium risk (30-70%) → queue for review High risk (>70%) → require immediate approval
💡 Ship with more HITL first, then gradually remove as you gain confidence in the agent
⚠️ 'The agent will ask if unsure' is not a safety strategy — define exact trigger conditions
Evaluation & Observability
2 topics
Evaluating Agent Performance
Metrics and methods for measuring agent quality
# LangSmith — best-in-class for LangChain/LangGraph agents
# https://smith.langchain.com
import langsmith
from langsmith import Client
# Auto-tracing — wraps every LLM call, tool call, chain step
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "ls__..."
os.environ["LANGCHAIN_PROJECT"] = "my-agent"
# Create an evaluation dataset
client = Client()
dataset = client.create_dataset("agent-eval-v1")
client.create_examples(
inputs=[{"task": "Summarize this article: ..."}],
outputs=[{"answer": "expected summary..."}],
dataset_id=dataset.id
)
# Define an evaluator
from langchain.smith import RunEvalConfig
eval_config = RunEvalConfig(
evaluators=[
"qa", # LLM-graded correctness
"criteria", # Check specific criteria
"embedding_distance", # Semantic similarity
]
)
# Run evaluation
results = langsmith.aevaluate(
run_agent, # Your agent function
data=dataset,
evaluators=[...],
experiment_prefix="gpt4o-react-v2"
)💡 LangSmith is free for up to 5k traces/month — start using it on day 1
⚡ Build an eval dataset BEFORE you start optimizing — you need a baseline
Key Metrics to Track
What to measure in production agent systems
TASK-LEVEL METRICS: Task success rate — Did the agent complete the goal? Steps to completion — Fewer steps = more efficient Error rate — How often does it fail? Retry rate — How often does it self-correct? QUALITY METRICS: Factual accuracy — Are claims verifiable? (LLM-graded) Instruction following — Did it do what was asked? Format compliance — Did it use the right output format? Hallucination rate — Is it making things up? COST & LATENCY: Tokens per task — Input + output tokens Cost per task — Tokens × price/token Latency p50/p95 — Median and tail latency Tool calls per task — Proxy for complexity SAFETY METRICS: Refusal rate — Declining appropriate requests? Harmful action rate — Taking actions it shouldn't? HITL trigger rate — How often humans get called? OBSERVABILITY TOOLS: LangSmith — langsmith.com (best for LangChain) Langfuse — langfuse.com (open-source, self-host) Arize Phoenix — arize.com/phoenix Helicone — helicone.ai (proxy-based, easy setup)
💡 Track cost per successful task — not just cost per run
⚡ p95 latency matters more than average for user-facing agents
Production Best Practices
2 topics
Production Checklist
Everything you need before shipping an agent to production
PRE-LAUNCH CHECKLIST: SAFETY ☐ Max iterations capped (never infinite loop) ☐ Tool inputs validated (no injection attacks) ☐ Code execution sandboxed (E2B, Docker) ☐ File paths restricted (no directory traversal) ☐ DB access is read-only (unless write is required) ☐ HITL gates on destructive actions ☐ Rate limiting on API tools RELIABILITY ☐ Retry logic with exponential backoff on tool failures ☐ Fallback behavior when tools are unavailable ☐ Timeout on every external call ☐ Graceful degradation (fail safe, not fail loud) ☐ State checkpointed (resume after crash) OBSERVABILITY ☐ Every LLM call logged (prompt + response) ☐ Every tool call logged (name + args + result) ☐ Latency and cost tracked per task ☐ Errors sent to alerting (PagerDuty, Slack) ☐ Eval dataset created and baseline measured COST CONTROL ☐ Per-task token budget defined ☐ Alerts on unexpected cost spikes ☐ Cheap model for planning, strong model for execution ☐ Prompt caching enabled (Anthropic/OpenAI) USER EXPERIENCE ☐ Streaming output so user sees progress ☐ Clear indication agent is working (spinner/status) ☐ User can cancel a running task ☐ Helpful error messages (not raw stack traces)
💡 Ship with all safety checks on, then remove them one by one as trust is earned
⚠️ 'It works in testing' does not mean it's safe in production
Error Handling & Retry Strategy
Making agents resilient to failures
import time
import random
from functools import wraps
def retry_with_backoff(max_retries=3, base_delay=1.0, exceptions=(Exception,)):
"""Decorator for exponential backoff retry."""
def decorator(fn):
@wraps(fn)
def wrapper(*args, **kwargs):
for attempt in range(max_retries):
try:
return fn(*args, **kwargs)
except exceptions as e:
if attempt == max_retries - 1:
raise
delay = base_delay * (2 ** attempt) + random.uniform(0, 0.5)
print(f"[Retry] Attempt {attempt+1} failed: {e}. Retrying in {delay:.1f}s...")
time.sleep(delay)
return wrapper
return decorator
@retry_with_backoff(max_retries=3, base_delay=1.0)
def call_llm(messages):
return client.chat.completions.create(model="gpt-4o", messages=messages)
# Tool-level error handling — return errors for LLM to self-correct
def safe_tool_call(fn, *args, **kwargs) -> dict:
try:
result = fn(*args, **kwargs)
return {"success": True, "result": result}
except ValueError as e:
return {"success": False, "error": f"Invalid input: {e}", "hint": "Check the parameter format"}
except TimeoutError:
return {"success": False, "error": "Tool timed out", "hint": "Try a simpler query"}
except Exception as e:
return {"success": False, "error": str(e)}
# The LLM can read the error and self-correct on the next iteration💡 Return structured errors to the LLM — it will often self-correct without human intervention
⚡ Add hints in error messages to guide the model toward a fix
Related Cheat Sheets
More hands-on references connected to this topic.
LangGraph stateful agents, graph nodes, edges, checkpointing, and multi-agent workflows.
shared topics, same difficulty
AI workflow cheat sheet for building software with AI — planning, incremental development, testing strategies, and codebase management tips.
shared topics
Practical Generative AI playbook for building reliable LLM features: prompting, context design, RAG, evaluation, safety, and production operations.
shared topics
LangChain chains, prompts, memory, RAG pipelines, agents, and tool use patterns.
shared topics
Related Articles
Background reading and deeper explanations for this sheet.
Agent Routing vs Model Routing: Real Production Patterns, Tradeoffs, and Implementation Guide
Choosing between agent routing and model routing can make or break your AI product’s cost, speed, and reliability. In this practical guide, you’ll learn what each pattern is, when to use it, and how production teams combine both to ship robust systems. We’ll walk through real examples, architecture patterns, and code you can adapt immediately.
keyword overlap
AWS AI & App Modernization Guide: SageMaker, Bedrock, ECS/EKS, and Lambda in Practice
AWS offers multiple paths to build intelligent, scalable applications—but choosing the right service can feel overwhelming. In this practical guide, you’ll learn what SageMaker, Bedrock, ECS/EKS, and Lambda are best at, how they fit together, and how to apply them in real-world architectures with beginner-friendly clarity and technical depth.
keyword overlap
Build an Agentic App with FastAPI and Azure AI Foundry: A Practical Beginner-to-Pro Guide
Learn how to design, build, and deploy an agentic application using FastAPI and Azure AI Foundry with practical, real-world examples. This guide walks you from architecture and setup to tool-calling, memory, observability, and production deployment patterns.
keyword overlap
Building Autonomous Agents Without Overengineering: A Practical Guide for Real-World Teams
Autonomous agents can automate meaningful work, but many projects fail because teams overbuild too early. This practical guide shows you how to design, ship, and improve useful agents with simple patterns, clear guardrails, and real-world examples—without turning your stack into a science project.
keyword overlap