Agent Evaluation Harness Architecture: Building Systematic Testing Infrastructure for AI Agents

The evaluation harness is the least glamorous but most consequential piece of AI agent infrastructure. Frameworks like LangGraph and CrewAI let you assemble agents in an afternoon, but without a systematic eval harness, you have no way to know if a prompt tweak, tool change, or model swap actually improves behavior. In production, that blind spot leads to regressions that surface in user-facing sessions — and by then it’s too late.

This post covers the architecture of production-grade agent evaluation harnesses: how to structure eval datasets, build LLM-as-judge pipelines, score agent trajectories, implement regression gates, and integrate evals into CI/CD. The patterns described are drawn from published frameworks at Anthropic, Braintrust, Langfuse, and Openlayer, as well as the structural testing methodology from recent academic work.

The Three Layers of Agent Evaluation

Production agent evaluation operates at three distinct layers, each with different data requirements, scoring strategies, and failure modes:

Layer 1 — Component evals test isolated agent internals: tool selection accuracy, argument formatting, retrieval quality, and prompt adherence. These are fast (sub-second per test) and cheap, suitable for pre-commit checks.

Layer 2 — Trajectory evals score the full agent reasoning chain: the sequence of tool calls, intermediate observations, and how the agent recovers from errors. Trajectory scoring is the hardest problem in agent evaluation because there is rarely a single correct path — multiple valid trajectories may reach the same correct result [1].

Layer 3 — Outcome evals measure end-to-end task completion: did the user get what they needed? These are the most expensive (often requiring human review or LLM judges) but provide the ground truth for production quality [1].

Anthropic’s engineering team describes this hierarchy in their eval framework: single-turn evaluations are straightforward (prompt → response → grade), but multi-turn agent evaluations require grading the trajectory itself, including intermediate steps and recovery behavior [2]. The structural testing paper from January 2026 formalizes this further by introducing white-box evaluation that examines internal agent states, not just input/output pairs [3].

Building Eval Datasets for Production

Eval datasets for agents differ fundamentally from traditional ML test sets. An agent eval case must include:

• Task specification: a natural language prompt describing what the agent should accomplish
• Ground truth: either a specific expected output, a set of acceptable outcomes, or a rubric for scoring
• Tool environment: the exact tool configuration (API schemas, available functions, rate limits) the agent operates against
• State precondition: any session state (previous conversation turns, cached context, authenticated user identity)

Braintrust’s agent evaluation framework recommends starting with at least 50-100 production-derived eval cases, then augmenting with adversarial inputs (ambiguous queries, contradictory instructions, tool failure scenarios) [4]. This mirrors the testing methodology used by teams deploying agents at scale — production logs capture real variance; synthetic cases cover edge conditions.

A practical dataset structure looks like:

{
  "task": "Find the latest Q2 earnings report for AAPL and summarize the revenue breakdown",
  "tools_available": ["search_web", "read_file", "query_database"],
  "acceptable_outcomes": [
    "Agent returns revenue figures from the most recent quarterly filing"
  ],
  "failure_modes": [
    "Returns stale Q1 data",
    "Confuses AAPL with another ticker",
    "Hallucinates revenue figures"
  ],
  "state": {
    "session_history": [],
    "user_role": "investor"
  }
}

LLM-as-Judge Pipeline Architecture

LLM-as-judge pipelines are the workhorse of agent evaluation at scale. A production-grade pipeline has four stages:

Stage 1 — Trajectory reconstruction: Replay the agent’s execution log to reconstruct the full decision chain. This includes every LLM call, every tool invocation with arguments and results, and every error or retry. Tools like Langfuse and Phoenix provide OpenTelemetry-based tracing that directly feeds this stage [5].

Stage 2 — Metric computation: Compute structural metrics from the trajectory:

• Tool selection accuracy: did the agent pick the right tool for each step?
• Argument correctness: were function parameters well-formed and semantically appropriate?
• Error recovery rate: when a tool call failed, did the agent retry appropriately or degrade gracefully?
• Step efficiency: how many turns did the agent take vs. a minimal solution?

Stage 3 — LLM judge scoring: Feed the reconstructed trajectory to an evaluator LLM with a structured rubric. The rubric should specify scoring dimensions (correctness, completeness, efficiency, safety) and anchoring examples. Anthropic found that providing 3-5 anchor examples per dimension reduced judge variance by 40% [2].

Stage 4 — Aggregation and regression detection: Roll up per-case scores into a suite-level report. Compare against baselines from the previous evaluation run. Flag any dimension that drops below a configurable threshold — this is the regression gate that blocks deploys.

# Simplified scoring loop
for case in eval_suite:
    trajectory = reconstruct_trajectory(case.session_log)
    metrics = compute_structural_metrics(trajectory)
    judge_score = llm_judge(
        trajectory=trajectory,
        rubric=case.rubric,
        anchor_examples=rubric.anchors
    )
    report[case.id] = {**metrics, "judge_score": judge_score}

regression = detect_regression(report, baseline)
if regression.has_breach():
    block_deploy(regression.dimensions)

Trajectory Scoring: The Hard Problem

Scoring agent trajectories is fundamentally harder than scoring single-turn LLM outputs. The key challenge is path equivalence: two completely different sequences of tool calls can produce equally correct results. A rigid scoring function that penalizes divergence from a canonical path will produce false negatives.

The structural testing paper addresses this by defining behavioral equivalence classes for agent trajectories [3]. Instead of comparing against a single expected path, the evaluator checks whether the trajectory satisfies a set of behavioral properties:

• Functional correctness: does the final state match the expected outcome?
• Tool usage validity: were all tool calls well-formed and authorized?
• Information flow: did the agent correctly propagate state between steps?
• Recovery competence: when errors occurred, were they handled without data loss?

Openlayer’s agent evaluation framework operationalizes this with a trajectory scoring matrix that scores each step on a 3-point scale (correct, acceptable, incorrect) and requires a minimum trajectory score for pass/fail [6]. This is more nuanced than binary pass/fail but simpler than full rubric scoring, making it suitable for CI/CD gating.

Regression Gates and CI/CD Integration

The most important property of an eval harness is that it prevents regressions from reaching production. This requires:

1. A baseline snapshot: run the full eval suite against the current production agent version and record scores per dimension
2. A comparison suite: run the same suite against the candidate version
3. A gate policy: define which metrics must not degrade and by how much

The gate policy should be dimension-specific. A 5% drop in task completion rate is a hard block; a 2% drop in step efficiency is acceptable if completion rate improves. Teams using this approach at Braintrust report catching ~70% of regressions before they reach staging, compared to ~30% with manual review alone [4].

# Regression gate policy
gates:
  task_completion:
    threshold: -0.03  # allow 3% regression max
    action: block_deploy
  tool_accuracy:
    threshold: -0.05
    action: warn
  error_recovery:
    threshold: -0.02
    action: block_deploy
  step_efficiency:
    threshold: +0.10  # allow 10% more steps
    action: warn

Continuous evaluation also means monitoring for drift in production. Openlayer’s approach uses statistical baselines rather than static thresholds: when evaluation metrics deviate beyond a rolling 7-day window, the system automatically flags the change for investigation [6]. This catches model-side regressions (e.g., a Claude update that changes tool-calling behavior) as well as application-side changes.

Cost-Aware Eval Pipeline Design

Running LLM-as-judge evaluations on every commit is expensive. A production harness must tier its evaluation spend:

• Pre-commit tier (30-50 cases, fast structural metrics only, no LLM judge): runs in <30 seconds, costs near zero. Catches tool-calling errors, argument formatting issues, and obvious failures.

• CI tier (200-300 cases, structural metrics + LLM judge on a subset): runs in 5-10 minutes, costs $2-5 per run. Runs on every PR. The LLM judge runs only on trajectories flagged as uncertain by structural metrics [5].

• Nightly tier (1000+ cases, full LLM judge on all trajectories): runs in 1-2 hours, costs $20-50 per run [5]. Full regression suite with statistical analysis. Runs once per day on the main branch.

Langfuse’s agent evaluation guide reports that this tiered approach reduces eval costs by 80% while maintaining 95% regression detection coverage compared to running all cases through LLM judges [5].

Architectural Tradeoffs

The choice of eval harness architecture depends on your agent’s complexity and deployment frequency:

Approach	Latency	Cost	Regression Detection	Best For
Structural metrics only	<1s per case	Negligible	Low (catches ~40%)	Pre-commit, high-frequency changes
LLM judge on trajectory	~5s per case	$0.01-0.05	Medium (~65%)	CI, PR-level gating
Full rubric + anchors	~15s per case	$0.03-0.15	High (~85%)	Nightly, release candidates
Human review + LLM	Hours-days	High	Highest (~95%)	Production releases, compliance

The structural testing paper found that combining structural metrics with LLM judges achieved 91% agreement with human evaluators, compared to 74% for LLM judges alone [3]. The combination catches different failure modes: structural metrics catch tool-calling bugs that LLM judges miss, while LLM judges catch reasoning errors that structural metrics cannot detect.

Putting It Together

A production-grade agent evaluation harness is not a single tool — it’s an architecture with layered defenses. The eval dataset grounds testing in real production patterns. Structural metrics provide fast, cheap signal. LLM judges add depth on uncertain cases. Regression gates enforce quality before deploy. Cost tiering makes it economically sustainable.

Teams that invest in this architecture see measurable returns: faster iteration cycles (because regressions are caught early), higher confidence in agent changes, and production behavior that degrades less over time [4]. The teams that skip eval infrastructure pay that debt in outages and user-facing regressions — the kind that are hard to trace and expensive to fix [1].

---

Sources

[1] Braintrust, “AI agent evaluation: A practical framework for testing multi-step agents”, https://www.braintrust.dev/articles/ai-agent-evaluation-framework

[2] Anthropic, “Demystifying evals for AI agents”, https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

[3] arXiv, “Automated structural testing of LLM-based agents: methods, framework, and case studies”, https://arxiv.org/abs/2601.18827v1

[4] Braintrust, “AI agent evaluation framework — metrics, harnesses, and regression gates”, https://www.braintrust.dev/articles/ai-agent-evaluation-framework

[5] Langfuse, “Agent Evaluation — How to Evaluate LLM Agents”, https://langfuse.com/guides/cookbook/example_pydantic_ai_mcp_agent_evaluation

[6] Openlayer, “Agent Evaluation Guide: Testing AI Agents 2026”, https://www.openlayer.com/blog/post/agent-evaluation-complete-guide-testing-ai-agents

📖 Related Reads

• NiteAgent — AI agent development, frameworks, and production patterns
• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
• Hermes Tutorials — Hermes Agent setup, configuration, and advanced workflows

Cross-links automatically generated from CodeIntel Log.