Eval-Driven Development Lifecycle for Agent Systems

The bottom line: Eval-driven development treats evaluation as the spine of the agent dev lifecycle, not an afterthought. Define success upfront via golden datasets + metrics, run evals on every change as a regression gate, block releases on quality thresholds.

---

What Eval-Driven Development Means

Unit tests catch regressions in deterministic code because the same input always produces the same output. Agent systems violate this assumption at every layer: model, prompt, tool schemas, and context window contents all shift simultaneously. A prompt change that improves one workflow silently degrades three others. A model swap passes every unit test but starts hallucinating tool parameters in edge cases.

Harrison Chase called this out at Interrupt 2025: eval-driven development (EDD) is the single biggest unblocker for moving agents to production. When you can’t write assertions about exact outputs, you need assertions about output distributions — evaluations provide that.

EDD flips the conventional timeline. Instead of building the agent first and writing evals at the end, you define the criteria before writing a prompt. The eval suite becomes the specification. Every code change, prompt edit, or model swap runs against the same golden dataset, and the pass rate determines whether the change ships.

There are three eval phases: development evals (during authoring), regression evals (on every PR), and production monitoring evals (continuous against live traffic). Anthropic’s framework splits evals into capability (“what can it do?”) and reliability (“does it do it every time?”). A reliable agent with limited capability can still ship — an unreliable one can’t. The regression run must catch drift from four simultaneous vectors: model swap, prompt change, tool/schema change, and data/context change. Latitude’s analysis shows why a single-layer test (“test the prompt in isolation”) misses most production failures.

---

Phase 1 — Define Success Upfront

Before writing agent code, define the success metrics. DeepEval (~16k stars, Apache 2.0, pytest-native) provides first-party agent metrics in v3.9.x:

• Task Completion — did the agent accomplish the goal?
• Tool Correctness — did it call the right tools with the right parameters?
• Hallucination — did it fabricate information?
• Answer Relevancy — was the final response on-topic?

DeepEval’s agent evaluation guide documents these with real pytest integration:

import pytest
from deepeval import assert_test
from deepeval.metrics.agents import (
    TaskCompletionMetric,
    ToolCorrectnessMetric,
)
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

def build_trace(input_query, tool_calls, output):
    """Simple trace builder for eval test cases."""
    return {
        "input": input_query,
        "tool_calls": tool_calls,
        "output": output,
    }

agent_traces = [
    build_trace(
        "What's the weather in Berlin?",
        [{"name": "get_weather", "args": {"city": "Berlin"}}],
        "The weather in Berlin is 18°C and partly cloudy.",
    ),
    build_trace(
        "Book a flight to London next Friday",
        [
            {"name": "search_flights", "args": {"destination": "London", "date": "2026-07-31"}},
            {"name": "book_flight", "args": {"flight_id": "BA123", "seat": "aisle"}},
        ],
        "Booked BA123 (aisle seat) to London on July 31.",
    ),
]

@pytest.mark.parametrize("trace", agent_traces)
def test_agent_tool_correctness(trace):
    test_case = LLMTestCase(
        input=trace["input"],
        expected_output=trace["output"],
        actual_output="(captured from agent execution)",
        tools_called=trace["tool_calls"],
        expected_tools=trace["tool_calls"],
    )
    assert_test(
        test_case,
        [
            TaskCompletionMetric(threshold=0.8),
            ToolCorrectnessMetric(threshold=0.9),
        ],
    )

This runs as a standard pytest command. No separate eval framework — just pytest test_agent.py. The metrics use LLM-as-judge internally, which brings caveats discussed below, but the pytest-native integration means your evals compose with your existing test infrastructure.

---

Phase 2 — Build the Golden Dataset

The most important architectural decision is where the golden dataset comes from. The wrong answer: brainstorming prompts in a Google Doc. The right answer: mining production logs.

FutureAGI makes this explicit: “Golden datasets should be distilled from real user logs/production traces, cleaned, labeled, and versioned — not brainstormed.” A brainstormed dataset captures what the team thinks the agent should handle. Production logs capture what users actually send.

Here’s a production log sampling and dedup pattern:

import json
import hashlib
from collections import Counter
from typing import Iterator

def load_production_traces(log_path: str) -> Iterator[dict]:
    with open(log_path) as f:
        for line in f:
            yield json.loads(line)

def deduplicate_traces(traces: list[dict]) -> list[dict]:
    """Fuzzy dedup: hash on normalized input + tool schema signature."""
    seen: set[str] = set()
    deduped: list[dict] = []

    for trace in traces:
        normalized_input = trace["input"].strip().lower()
        # Schema signature captures which tools were involved
        tool_sigs = sorted(
            f"{tc['name']}:{sorted(tc.get('args', {}).keys())}"
            for tc in trace.get("tool_calls", [])
        )
        # Embedding-based dedup would be better, but hash is fast
        key = hashlib.sha256(
            (normalized_input + "|" + "|".join(tool_sigs)).encode()
        ).hexdigest()

        if key not in seen:
            seen.add(key)
            deduped.append(trace)

    return deduped

def sample_diverse_traces(
    traces: list[dict], max_samples: int = 200
) -> list[dict]:
    """Stratified sample: ensure diverse tool combinations."""
    tool_combo_counter: Counter = Counter()
    for t in traces:
        combo = tuple(sorted(c["name"] for c in t.get("tool_calls", [])))
        tool_combo_counter[combo] += 1

    # Oversample rare tool combinations, undersample common ones
    sampled = []
    combo_counts: Counter = Counter()
    for trace in sorted(traces, key=lambda t: tool_combo_counter[
        tuple(sorted(c["name"] for c in t.get("tool_calls", [])))
    ]):
        combo = tuple(sorted(c["name"] for c in trace.get("tool_calls", [])))
        if combo_counts[combo] / tool_combo_counter[combo] < 0.15:
            sampled.append(trace)
            combo_counts[combo] += 1
        if len(sampled) >= max_samples:
            break

    return sampled

# Pipeline:
traces = list(load_production_traces("traces/prod_2026_07.ndjson"))
deduped = deduplicate_traces(traces)
golden = sample_diverse_traces(deduped)

with open("golden_dataset_v3.json", "w") as f:
    json.dump(golden, f, indent=2)

Version the golden dataset like any other artifact — commit it to git, tag releases, track drift between versions. Each version should be human-labeled for ground truth (at least a 10% stratified sample) to calibrate automated metrics. Braintrust emphasizes treating the dataset as infrastructure: the same rigor applied to database schemas applies to the eval suite.

---

Phase 3 — Automate as Regression Tests

Once the golden dataset exists, wrap it in automated regression tests. Three complementary tools:

• DeepEval — pytest-native, best for agent-specific metrics
• promptfoo — YAML-configurable, multi-provider, CLI-first
• Ragas — RAG-specific: faithfulness, answer relevancy, context precision/recall

The MLflow comparison and GenAI.QA’s breakdown both note different sweet spots. DeepEval’s first-party agent metrics are most targeted for agent systems. For multi-provider comparisons, promptfoo’s YAML approach is faster.

Here’s a promptfoo config with assertion thresholds:

# promptfooconfig.yaml
description: "Agent golden eval — Phase 3"
providers:
  - id: openai:gpt-4o
    config:
      tools: "tools/openai_schemas.json"
  - id: anthropic:claude-opus-4
    config:
      tools: "tools/anthropic_schemas.json"

tests: golden_dataset_v3.json

defaultTest:
  options:
    provider: openai:gpt-4o-mini
  asserts:
    - type: contains-any
      value: ["weather", "flights", "error"]
      weight: 1
    - type: latency
      threshold: 10000
      weight: 1

providerOverrides:
  - provider: openai:gpt-4o
    asserts:
      - type: llm-rubric
        value: "Did the agent use the correct tool?"
        threshold: 0.85
        weight: 3
      - type: llm-rubric
        value: "Did the agent include all required parameters?"
        threshold: 0.90
        weight: 3
  - provider: anthropic:claude-opus-4
    asserts:
      - type: llm-rubric
        value: "Did the agent use the correct tool?"
        threshold: 0.80
        weight: 3
      - type: latency
        threshold: 15000
        weight: 2

---

Phase 4 — Gate Releases on Quality Thresholds

A regression test that nobody runs is documentation. The eval suite needs to block CI when scores drop below defined thresholds.

Kunal Ganglani’s production eval guide describes CI gates as the “enforcement layer” of EDD. Without a hard gate, “we’ll fix it next sprint” becomes the default response to regression.

Here’s a GitHub Actions step that blocks on regression:

# .github/workflows/agent-eval-gate.yml
name: Agent Eval Gate
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install deepeval
        run: pip install deepeval

      - name: Run golden eval suite
        id: evals
        run: |
          deepeval test run test_agent_golden.py \
            --xml-output eval_results.xml

      - name: Check regression
        id: check_regression
        run: |
          BASELINE=$(cat baselines/pass_rate_latest.txt)
          CURRENT=$(grep -oP 'Pass Rate: \K[0-9.]+' eval_results.xml)
          REGRESSION=$(echo "$BASELINE - $CURRENT" | bc -l)
          echo "Baseline: $BASELINE  Current: $CURRENT  Delta: $REGRESSION"
          if (( $(echo "$REGRESSION > 0.05" | bc -l) )); then
            echo "FAIL: Regression exceeds 5% threshold"; exit 1
          fi
          if (( $(echo "$CURRENT < 0.70" | bc -l) )); then
            echo "FAIL: Score below 70% minimum"; exit 1
          fi
          echo "$CURRENT" > baselines/pass_rate_latest.txt

      - name: Post PR comment
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const score = '${{ steps.evals.outputs.score }}';
            const passed = parseFloat(score) >= 0.70;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `## 🤖 Eval Gate\n\n**Score**: ${score}\n**Result**: ${passed ? 'PASS' : 'FAIL'}`
            });

The baseline file (pass_rate_latest.txt) should live on the main branch and only update after a merge, so the gate always compares against the last deployed quality level, not the current PR’s branch.

---

Phase 5 — Monitor Production Drift

Even with a perfect eval suite, production creates edge cases the golden dataset never covered. Re-run the golden dataset against the current model version on a schedule, and alert when scores drift.

Anthropic explicitly notes that “automated evals are one layer, not the whole safety net” — combine automated evals + production monitoring + A/B testing + human review. The golden dataset re-run catches drift from all four layers (model, prompt, tools, context) silently.

---

Pitfalls

EDD isn’t free. Three common failure modes:

1. Over-relying on LLM-as-judge. DeepEval, Ragas, and promptfoo all default to LLM-as-judge for semantic metrics. This introduces judge drift (the evaluating model changes behavior) and self-preference bias (GPT-4o judges its own outputs more favorably). Kunal Ganglani’s guide recommends calibrating LLM judges against human-labeled anchors — maintain a 50-sample labeled subset and periodically check if judge scores correlate with human judgments. Correlation below 0.8 means recalibration is needed.

2. Tiny golden datasets. A 10-case eval suite gives confidence about those 10 cases and nothing else. Target 200–500 cases minimum. Below 100, you’re measuring noise.

3. Optimizing for the metric, not the user. When a regression gate blocks shipping, the team tweaks the agent to score higher on the eval suite. If the suite has blind spots (e.g., measures tool selection but not response quality), the agent optimizes tool selection at the expense of everything else. Diversify your eval suite — include Ragas’s context precision/recall, DeepEval’s Step Efficiency, and custom assertion rubrics.

---

Conclusion

Eval-driven development is the discipline that makes agents shippable. The lifecycle — define success upfront, build a production-derived golden dataset, automate as regression tests, gate releases on quality thresholds, monitor for production drift — transforms agent development from prompt tinkering into a measurable engineering practice.

The tooling is mature. DeepEval gives you pytest-native agent metrics. promptfoo handles multi-provider YAML comparisons. Ragas covers RAG-specific evaluation. Pick one, mine your production logs, write your first eval gate. Regressions will be caught at PR time, before they ever reach a user.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
• NiteAgent — AI agent development, frameworks, and production patterns

Cross-links automatically generated from CodeIntel Log.