Agent Loop Production Hardening: Circuit Breakers, Token Budgets, and Graceful Degradation
An engineering deep dive on production hardening for autonomous agent loops — token-aware rate limiting, circuit breaker state machines, budget-aware runtimes, and empirical failure data from production deployments of LLM agents.
The defining production challenge of LLM agent systems in 2026 is not making agents smarter — it’s making them safe to run unsupervised. Every production agent deployment eventually faces the same failure modes: runaway loops that burn through API budgets in minutes, tools called with invalid arguments that cascade into corrupted state, and degradation patterns that silently eat latency SLAs.
This post maps the engineering patterns that separate production agent systems from prototypes: token-aware rate limiting, circuit breaker state machines, budget-aware runtimes, and graceful degradation strategies backed by real production data from industry deployments.
The Failure Data: What Actually Goes Wrong
A June 2026 analysis of production agent incidents across 23 deployments by Truefoundry found three dominant failure categories 1:
| Failure Mode | Frequency | Avg Cost per Incident | Recovery Time |
|---|---|---|---|
| Runaway loops (retry storms) | 37% | $47.32 | 8.3 min |
| Token budget exhaustion mid-task | 28% | $12.15 | 2.1 min |
| Tool call cascading failures | 22% | $23.80 | 15.7 min |
| Context window overflow | 13% | $3.40 | 0.8 min |
Runaway loops dominate because they compound: a single hallucinated tool call triggers a retry, which fails again, which triggers another retry, each consuming full context windows. One documented case in the study shows an agent generating 46 tool calls in 12 seconds before manual intervention — a $240 bill for a task that should have cost $0.15 1.
Pattern 1: Token-Aware Rate Limiting
Simple request-count rate limiting is insufficient for agent systems because LLM API costs are proportional to token volume, not request count. A single tool call retry with full conversation history attached can cost 50x more than the initial call.
import time
import asyncio
from collections import defaultdict
from dataclasses import dataclass, field
@dataclass
class TokenBucket:
"""Token-aware rate limiter that accounts for input + output tokens."""
max_tokens_per_minute: int = 500_000
max_tokens_per_request: int = 64_000
refill_rate_tokens_per_sec: int = 8_333 # 500K / 60
tokens_remaining: float = 500_000.0
last_refill: float = field(default_factory=time.monotonic)
def _refill(self):
now = time.monotonic()
elapsed = now - self.last_refill
self.tokens_remaining = min(
self.max_tokens_per_minute,
self.tokens_remaining + elapsed * self.refill_rate_tokens_per_sec
)
self.last_refill = now
def try_consume(self, input_tokens: int, output_tokens: int = 4_096) -> tuple[bool, float]:
"""Returns (accepted, wait_time_seconds)."""
self._refill()
total = input_tokens + output_tokens
if total > self.max_tokens_per_request:
return False, float('inf')
if total <= self.tokens_remaining:
self.tokens_remaining -= total
return True, 0.0
deficit = total - self.tokens_remaining
wait_time = deficit / self.refill_rate_tokens_per_sec
return False, wait_time
class TokenAwareAgentRuntime:
"""Agent loop with token-aware rate limiting and request budgeting."""
def __init__(self, budget_tokens: int = 2_000_000):
self.bucket = TokenBucket()
self.budget_remaining = budget_tokens
self.loop_count = 0
self.max_loops = 25
async def run(self, task: str) -> str:
while self.loop_count < self.max_loops:
self.loop_count += 1
# Estimate tokens for next iteration
input_tokens = self._estimate_input()
accepted, wait = self.bucket.try_consume(input_tokens)
if not accepted:
if wait == float('inf'):
return self._degrade("Request exceeds max token limit")
await asyncio.sleep(wait)
continue
if input_tokens > self.budget_remaining:
return self._degrade(f"Token budget exhausted after {self.loop_count} iterations")
# Make the LLM call
result = await self._llm_call(task, input_tokens)
self.budget_remaining -= input_tokens
if result["done"]:
return result["output"]
return self._degrade(f"Max loop count ({self.max_loops}) exceeded")
def _estimate_input(self) -> int:
# Estimate tokens from accumulated conversation history
return len(self._messages) * 4 # rough 4 chars per token
def _degrade(self, reason: str) -> str:
return f"[DEGRADED] {reason} — partial results returned"The Truefoundry team documented that token-aware rate limiting reduced cost overruns by 78% compared to request-count-only limiting in their agent gateway implementation 2. The key insight: track input tokens separately from output tokens — a single retry with full history attached is the most common budget killer.
Pattern 2: Circuit Breaker State Machine for Agent Loops
A circuit breaker for agent loops needs more states than the classic three (closed/open/half-open). Agent failures are rarely binary — they decay gradually as context windows fill and model confidence degrades.
from enum import Enum
import time
from collections import deque
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
WATCHING = "watching" # Elevated failure rate, monitoring
DEGRADED = "degraded" # Reduced functionality mode
OPEN = "open" # Blocked — human intervention needed
class AgentCircuitBreaker:
"""Four-state circuit breaker for agent loops.
States:
- CLOSED: Normal operation, no limits
- WATCHING: Failure rate elevated (>10%), monitoring closely
- DEGRADED: High failure rate (>30%), reduced max iterations
- OPEN: Critical failure rate (>50%), agent disabled
"""
def __init__(self, window_seconds: int = 300, max_failures: int = 5):
self.state = CircuitState.CLOSED
self.failures: deque[float] = deque(maxlen=500)
self.total_calls = 0
self.window = window_seconds
self.state_changed_at = time.monotonic()
self.failure_thresholds = {
CircuitState.CLOSED: 0.10,
CircuitState.WATCHING: 0.30,
CircuitState.DEGRADED: 0.50,
}
# State-dependent agent parameters
self.max_iterations = {
CircuitState.CLOSED: 25,
CircuitState.WATCHING: 15,
CircuitState.DEGRADED: 8,
CircuitState.OPEN: 0,
}
def record_call(self, success: bool):
"""Record an LLM call result and potentially transition state."""
now = time.monotonic()
self.total_calls += 1
if not success:
self.failures.append(now)
# Prune old failures outside window
while self.failures and self.failures[0] < now - self.window:
self.failures.popleft()
self._maybe_transition(now)
def _maybe_transition(self, now: float):
"""Evaluate and apply state transitions."""
rate = self.failure_rate()
current = self.state
if current == CircuitState.CLOSED and rate > self.failure_thresholds[CircuitState.CLOSED]:
self.state = CircuitState.WATCHING
self.state_changed_at = now
elif current == CircuitState.WATCHING and rate > self.failure_thresholds[CircuitState.WATCHING]:
self.state = CircuitState.DEGRADED
self.state_changed_at = now
elif current == CircuitState.DEGRADED and rate > self.failure_thresholds[CircuitState.DEGRADED]:
self.state = CircuitState.OPEN
self.state_changed_at = now
self._alert("Agent circuit OPEN — manual intervention required")
elif current in (CircuitState.WATCHING, CircuitState.DEGRADED) and rate < 0.05:
# Recovery: sustained low failure rate
self.state = CircuitState.CLOSED
self.state_changed_at = now
def failure_rate(self) -> float:
if not self.total_calls:
return 0.0
window_failures = len(self.failures)
return window_failures / max(self.total_calls, 1)
def allowed_iterations(self) -> int:
return self.max_iterations[self.state]This four-state model is adapted from production patterns documented in the Zylos Research March 2026 analysis of graceful degradation in agent systems 3. Their finding: a binary open/closed breaker triggers too late for agent workloads — by the time the failure rate hits 50%, the damage (cost, corrupted state) is already done. The intermediate WATCHING and DEGRADED states allow preemptive intervention: reducing max iterations, switching to cheaper models, or falling back to simpler tool sets.
Pattern 3: Budget-Aware Runtime with Enforceable Limits
The most effective production pattern documented across multiple deployments is the budget-aware runtime — a layer that tracks token consumption, wall-clock time, and tool call count simultaneously, enforcing the strictest limit 4.
@dataclass
class Budget:
"""Multi-dimensional budget for an agent run."""
max_tokens: int = 500_000
max_time_seconds: int = 120
max_tool_calls: int = 30
max_cost_usd: float = 0.50
class BudgetAwareRuntime:
"""Runtime that enforces budget across 4 dimensions simultaneously."""
def __init__(self, budget: Budget):
self.budget = budget
self.tokens_used = 0
self.tool_calls_made = 0
self.cost_incurred = 0.0
self.start_time = time.monotonic()
self.call_log: list[dict] = []
def check(self) -> tuple[bool, str]:
"""Returns (allowed, reason_if_blocked)."""
elapsed = time.monotonic() - self.start_time
checks = [
(self.tokens_used >= self.budget.max_tokens,
f"Token budget exhausted ({self.tokens_used}/{self.budget.max_tokens})"),
(elapsed >= self.budget.max_time_seconds,
f"Time budget exhausted ({elapsed:.0f}s/{self.budget.max_time_seconds}s)"),
(self.tool_calls_made >= self.budget.max_tool_calls,
f"Tool call limit reached ({self.tool_calls_made}/{self.budget.max_tool_calls})"),
(self.cost_incurred >= self.budget.max_cost_usd,
f"Cost budget exhausted (${self.cost_incurred:.2f}/${self.budget.max_cost_usd:.2f})"),
]
for blocked, reason in checks:
if blocked:
return False, reason
return True, "ok"
def record_llm_call(self, input_tokens: int, output_tokens: int, cost: float):
self.tokens_used += input_tokens + output_tokens
self.cost_incurred += cost
self.call_log.append({
"type": "llm",
"tokens": input_tokens + output_tokens,
"cost": cost,
"timestamp": time.monotonic(),
})
def record_tool_call(self, tool_name: str):
self.tool_calls_made += 1
self.call_log.append({
"type": "tool",
"tool": tool_name,
"timestamp": time.monotonic(),
})The key engineering insight from the Reddit production engineering discussion 4 and the Truefoundry blog 2: enforce all four limits simultaneously. Token-only budgets miss runaway time (an agent stuck in a loop generates few tokens but takes forever). Time-only budgets miss cost explosions (a single expensive model call can exceed budget in one shot). The budget check must occur before every LLM call and every tool invocation — checking only at the start of a loop iteration misses mid-tool-call cost.
async def safe_agent_loop(runtime: BudgetAwareRuntime, task: str) -> str:
"""Agent loop with per-call budget enforcement."""
messages = [{"role": "user", "content": task}]
while True:
# CHECK BEFORE LLM CALL
allowed, reason = runtime.check()
if not allowed:
return f"[BUDGET_EXCEEDED] {reason}"
# Make LLM call
response = await llm_call(messages)
runtime.record_llm_call(
input_tokens=response.input_tokens,
output_tokens=response.output_tokens,
cost=response.cost,
)
if response.tool_calls:
for tool_call in response.tool_calls:
# CHECK BEFORE TOOL CALL
allowed, reason = runtime.check()
if not allowed:
return f"[BUDGET_EXCEEDED] {reason}"
result = await execute_tool(tool_call)
runtime.record_tool_call(tool_call.name)
messages.append(tool_call.to_message(result))
else:
return response.contentPattern 4: Graceful Degradation Strategies
When budgets are exhausted or circuits open, the agent should degrade — not crash. Four patterns for graceful degradation in agent systems, ordered by severity and adapted from Zylos Research and Truefoundry analyses 3:
| Strategy | What Changes | When to Apply |
|---|---|---|
| Model downgrade | Switch from flagship model (Claude Sonnet 4.6) to cheaper model (Haiku 4.5) | WATCHING state, or when token budget is 30%+ consumed |
| Tool set reduction | Remove expensive tools (web search, code execution) from available tool list | DEGRADED state, or when tool call budget is 50%+ consumed |
| Max iterations reduction | Reduce from 25 to 8 iterations, forcing earlier partial-result submission | DEGRADED state |
| Human handoff | Stop making decisions, return partial state + request human input | OPEN state, or when all budgets exhausted |
class DegradationStrategy:
"""Progressive degradation based on circuit state and budget remaining."""
def get_config(self, circuit: AgentCircuitBreaker, runtime: BudgetAwareRuntime) -> dict:
config = {
"model": "claude-sonnet-4.6",
"max_iterations": 25,
"available_tools": ["read_file", "write_file", "search", "execute"],
"partial_result_ok": False,
}
state = circuit.state
budget_frac = runtime.tokens_used / max(runtime.budget.max_tokens, 1)
if state == CircuitState.WATCHING or budget_frac > 0.3:
config["model"] = "claude-haiku-4.5"
config["partial_result_ok"] = True
if state == CircuitState.DEGRADED or budget_frac > 0.5:
config["max_iterations"] = 8
config["available_tools"] = ["read_file", "write_file"]
if state == CircuitState.OPEN or budget_frac > 0.8:
config["max_iterations"] = 1
config["available_tools"] = []
config["partial_result_ok"] = True
config["human_handoff"] = True
return configPutting It Together: Hardened Agent Runtime
class HardenedAgentRuntime:
"""Production agent runtime combining circuit breaker, budget enforcement,
token-aware rate limiting, and graceful degradation."""
def __init__(self, task: str):
self.task = task
self.circuit = AgentCircuitBreaker()
self.budget = BudgetAwareRuntime(Budget())
self.rate_limiter = TokenBucket()
self.degrader = DegradationStrategy()
async def run(self) -> str:
config = self.degrader.get_config(self.circuit, self.budget)
messages = [{"role": "user", "content": self.task}]
for iteration in range(config["max_iterations"]):
# Gate 1: Circuit breaker
if self.circuit.allowed_iterations() <= iteration:
return self._handoff("Circuit breaker OPEN")
# Gate 2: Budget check
allowed, reason = self.budget.check()
if not allowed:
return self._handoff(reason)
# Gate 3: Rate limiter
input_tokens = estimate_tokens(messages)
accepted, wait = self.rate_limiter.try_consume(input_tokens)
if not accepted:
if wait == float('inf'):
return self._handoff("Request exceeds rate limit")
await asyncio.sleep(wait)
# Execute
try:
response = await llm_call(messages, model=config["model"])
self.budget.record_llm_call(
response.input_tokens, response.output_tokens, response.cost
)
self.circuit.record_call(success=True)
except Exception as e:
self.circuit.record_call(success=False)
if self.circuit.state == CircuitState.OPEN:
return self._handoff(f"API failure: {e}")
continue # retry with updated config
# Process tool calls
for tool_call in response.tool_calls:
if tool_call.name not in config["available_tools"]:
continue # silently skip tools not in degraded set
allowed, reason = self.budget.check()
if not allowed:
return self._handoff(reason)
try:
result = await execute_tool(tool_call)
self.budget.record_tool_call(tool_call.name)
self.circuit.record_call(success=True)
except Exception:
self.circuit.record_call(success=False)
messages.append(tool_call.to_message(result))
# Completed normally or hit iteration limit
return self._finalize(messages, config)Production Telemetry: What to Monitor
Based on production deployment data from Truefoundry, Zuplo, and Zylos Research 15, track these metrics per agent run:
- Budget utilization ratio — (tokens used / budget). Alert at 0.8 for preemptive action.
- Circuit state transitions — Count of WATCHING/DEGRADED/OPEN transitions. Frequent transitions indicate systemic issues.
- Degradation invocations — How often does the system degrade rather than complete? Above 15% degradation rate indicates insufficient budgets.
- False positive circuit opens — Circuits that opened but the agent was operating correctly. Too many false opens mean adjust thresholds.
- Cost variance — (actual cost / expected cost) per task type. High variance indicates looping or retry storms.
Key Takeaways
-
Request-count rate limiting is insufficient for agent workloads. Token-aware rate limiting that accounts for accumulating conversation history prevents 78% of cost overruns 2.
-
Use a four-state circuit breaker (CLOSED → WATCHING → DEGRADED → OPEN) rather than binary open/closed. Agent failures decay gradually — preemptive intervention beats post-hoc recovery.
-
Enforce budgets on all four dimensions simultaneously: tokens, time, tool calls, cost. Single-dimension budgets always leak.
-
Graceful degradation is a feature, not a fallback. A degraded agent that returns partial results with correct metadata is infinitely more useful than one that crashes silently after burning $50 on retries.
-
Instrument everything. The production telemetry data from monitoring budget utilization and circuit state is the single highest-leverage investment for improving agent reliability.
The engineering lesson from every production agent incident in 2026 is the same: the agent doesn’t need to be smarter — the runtime needs to be safer. These patterns convert that lesson into code.