The Agent Gap: Measuring the Disconnect Between Coding and Agentic Capability Across 126 Models

1. The Blind Spot in Model Selection

When an engineer provisions a model for a coding agent, the instinct is simple: grab the highest coding score and ship it. But production agent systems are not just code generators. They must plan, correct course, maintain state, and recover from failure. Our analysis of 126 models across the Artificial Analysis benchmark suite reveals a systematic blind spot: agentic scores trail coding scores by an average of 23 points. We call this the agent gap.

For anyone building autonomous coding agents, this gap is a hard engineering constraint—not a footnote. If your agent pipeline routes to a model with a 47-point agent gap, you are effectively deploying a code generator that fails at the core competency of an agent: autonomous, multi-step task completion.

2. Methodology

We queried the OpenRouter API /v1/models endpoint on July 29, 2026, filtering for all entries containing complete Artificial Analysis benchmark scores (intelligence_index, coding_index, agentic_index). This yielded 126 qualifying models across providers including OpenAI, Anthropic, Google, Meta, Mistral, and dozens of open-source variants.

Caveats:

• Benchmarks are static snapshots; real-world variance may differ.
• “Agentic score” measures structured agent benchmarks (e.g., SWE-bench, GAIA) and is not a substitute for workload-specific testing.
• Models without all three indices were excluded (roughly 15% of OpenRouter’s catalog).

3. The Agent Gap: Distribution and Central Tendency

The central finding is stark: the average gap (coding_score − agentic_score) across all models is 23.0 points. The distribution is right-skewed—most models cluster in the 15–30 point gap range, with a long tail of catastrophic mismatches.

Gap Distribution (126 models)
Range        Count    Example Model
0–10         14       GPT-5 Mini
11–20        38       Llama 4 70B
21–30        42       Claude Sonnet 4.5
31–40        22       Qwen 3.7 Max
41–50        10       Gemini 3.1 Pro Preview

4. Top Performers: Where Coding and Agency Align

Only six models in the dataset have agentic scores that exceed their coding scores. The standout is GPT-5 Mini (15.6 code, 19.4 agent, gap: −3.8), defying the dominant pattern.

Model	Coding Index	Agentic Index	Gap
GPT-5 Mini	15.6	19.4	−3.8
Claude Opus 5	78.0	76.1	+1.9
Claude Opus 4.5	72.3	70.8	+1.5
GPT-5 Flash	42.1	43.0	−0.9

What these models share: heavy reinforcement learning on agent trajectories. They are not just taught code generation; they are trained on failure recovery, state management, and multi-turn planning.

5. Worst Offenders: The Agent Gap Abyss

Some models are nearly useless for agentic workloads, despite strong coding scores.

Model                  Code    Agent   Gap
Gemini 3.1 Pro Preview 68.8    21.4    **47.4**
KAT-Coder-Pro V2       59.5    15.5    **44.0**
Qwen 3.7 Max           66.0    30.6    **35.4**
DeepSeek-V3-Coder      71.2    36.8    **34.4**
Mistral Large 3        64.0    36.2    **27.8**

Gemini 3.1 Pro Preview is the worst offender: nearly 69 points on coding, but only 21 on agentic tasks. In production, this model would generate correct single-turn code but fail catastrophically on multi-step task decomposition or self-correction loops.

KAT-Coder-Pro V2 (gap: 44.0) is particularly dangerous because its marketing emphasizes “code mastery.” Engineers trusting that framing for agent workloads would be deploying a model that is 74% worse at agency than at coding.

6. Correlation Analysis: Intelligence ≠ Agency

We computed Pearson correlations between all three indices using the OpenRouter model data:

Pair	Correlation	Interpretation
Intelligence ↔ Code	0.980	Almost perfectly linear: coding is near-proxy for general intelligence.
Intelligence ↔ Agent	0.934	Strong, but scatter increases at higher scores.
Code ↔ Agent	0.934	Same—strong but not deterministic.

What this means: Intelligence predicts coding extremely well (r² ≈ 0.96). But the code-to-agent prediction leaves 13% of variance unexplained. That residual is the agent gap—a systematic weakness that no simple scaling law fixes.

Notably, 120 out of 126 models overperform coding relative to intelligence. The market has optimized for code generation benchmarks (HumanEval, MBPP), but that optimization does not transfer to agentic benchmarks (SWE-bench, GAIA). This is a market-wide training artifact.

7. Practical Implications for Production AI Engineering

For Agent Architects

Rule 1: Never proxy-select by coding score. If your agent requires multi-step reasoning or tool use, filter by agentic index first. A model with code=80 and agent=60 will outperform a model with code=90 and agent=20 in realistic agent loops.

Rule 2: Profile your agent workloads. If your system issues single-turn code completion calls (e.g., Copilot-style), the agent gap is irrelevant. If your system involves planning, retries, and state management, require agentic_index ≥ 0.8 * coding_index.

Rule 3: Budget for agentic overhead. Models with large gaps (≥30 points) will require more guardrails, retry logic, and human-in-the-loop fallbacks. That safety engineering often costs more than the model itself.

For Model Selection Teams

Use this table as a quick filter for production agent systems:

Requirement            Min Agentic Index   Max Accepted Gap
High-trust agent       60                  10
Standard agent         40                  20
Code completion only   N/A                 N/A
Experimental/fallback  20                  30

8. Conclusion: The Agent Gap as an Engineering Constraint

The 23-point average agent gap is not a bug in benchmarks—it is a structural property of today’s model training ecosystem. Code generation enjoys massive data availability, synthetic data amplification, and dedicated optimization. Agentic capability requires fundamentally different training: trajectory-level RL, long-context reasoning, and failure recovery.

For the engineering team building production agents, ignoring this gap means shipping systems that look perfect in offline evaluation but break in the multi-turn, ambiguous real world. The smartest move in 2026’s model zoo is not to chase the highest coding score—it’s to measure the agent gap first.

References

1. Artificial Analysis Benchmarks — Industry-standard LLM benchmark suite
2. OpenRouter Models API — Source for model benchmark data
3. SWE-bench Leaderboard — Software engineering benchmark for agents

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
• NiteAgent — AI agent development, frameworks, and production patterns

Cross-links automatically generated from CodeIntel Log.