Production Prompt Caching for LLM APIs: Provider Comparison, Architecture Patterns, and Empirical Hit-Rate Analysis

An engineering deep dive comparing prompt caching across OpenAI, Anthropic, Google, and DeepSeek — KV cache mechanics, cache breakpoint strategies, cache hit rate optimization patterns, and production latency/cost benchmarks from real deployments.

production-architectureinference-optimizationai-engineeringprompt-cachingcost-optimization

Every production LLM deployment above a trivial request volume hits the same economic wall: the cost of reprocessing the same context tokens on every request dominates the inference bill. A 10K-token system prompt repeated across 100K daily requests means a trillion tokens spent on content that never changes — at $3/MTok for GPT-5.5, that’s $3,000/day on static prefix alone [1].

Prompt caching addresses this at the infrastructure level by storing the Key-Value (KV) cache entries for stable prompt prefixes across requests, eliminating the prefill phase for cached tokens. Every major provider now ships some form of it — but the mechanisms, pricing models, cache TTLs, and optimal usage patterns differ substantially.

This essay compares the four dominant provider implementations — OpenAI, Anthropic, Google Gemini, and DeepSeek — against a set of production-relevant dimensions: cache hit mechanics, latency reduction, cost savings, TTL semantics, and architectural constraints. It then maps the optimization patterns that maximize cache hit rates in production.

How Prompt Caching Works at the Infrastructure Level

LLM inference has two phases: prefill (process the prompt, compute KV tensors) and decode (generate tokens one by one). Prefill is compute-bound and dominates first-token latency (TTFT). For a 16K prompt on a dense transformer, prefill can account for 60–80% of end-to-end latency vLLM’s automatic prefix caching documentation [1].

A KV cache stores the key and value matrices computed during the prefill phase for each attention layer. When a new request arrives with a prefix that matches a previously cached prefix — byte-for-byte — the inference engine skips recomputation for those positions and starts decoding from the end of the cached prefix OpenAI’s prompt caching guide [2].

The critical constraint is prefix exactness: cache hits require byte-identical prefixes. A single character change, a trailing whitespace difference, or a reordered tool definition invalidates the cache for everything from that point forward. This constraint drives most of the engineering patterns around prompt caching.

Provider Comparison

OpenAI (Automatic Prefix Caching)

OpenAI applies prompt caching automatically for all requests exceeding 1,024 tokens on GPT-5.5 and newer models OpenAI’s prompt caching guide. There’s no API parameter to toggle it — the provider checks the prompt prefix against an internal LRU cache at the API gateway level.

Dimension Detail
Minimum cache length 1,024 tokens
Granularity 128-token chunks
TTL 5–10 minutes of inactivity per prefix
Cost savings 50% discount on cached input tokens
Cache writes Free (no extra charge)
Latency impact ~40–55% TTFT reduction on cache hits

The automatic nature is a double-edged sword. It requires zero developer effort but provides no visibility into cache state. OpenAI reports cache hit rate via the usage.prompt_tokens_details.cached_tokens field in the response, but there’s no API to pre-warm or inspect the cache.

Production deployments should log cached_tokens vs total_prompt_tokens per request and track cache hit rate as a standard metric. A drop from 70% to 40% signals that dynamic content has leaked into the prompt prefix.

import openai

client = openai.OpenAI()

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},  # long prefix
        {"role": "user", "content": query}
    ]
)

# Track cache hit rate
cached = response.usage.prompt_tokens_details.cached_tokens
total = response.usage.prompt_tokens
print(f"Cache hit ratio: {cached}/{total} = {cached/total*100:.1f}%")

Limitation: OpenAI’s 128-token granularity means the last chunk of the cached prefix is often partial. A prompt of exactly 2,000 tokens will cache 1,920 tokens (15 × 128) and miss on the last 80 tokens. Padding prompts to cache-aligned boundaries can squeeze out an extra 3–5% in cached tokens.

Anthropic (Explicit Cache Breakpoints)

Anthropic’s approach is the opposite of OpenAI’s: caching is fully developer-controlled via cache_control breakpoints in the messages array Anthropic’s prompt caching docs. You decide exactly which content blocks are cached and for how long (5-minute or 1-hour TTL).

Dimension Detail
Minimum cache length 2,048 tokens per breakpoint
Max breakpoints 4 per request
TTL 5 min or 1 hour (configurable)
Cost savings 90% discount on cached input tokens
Cache writes 1.25× input token rate
Latency impact ~80–90% TTFT reduction on cache hits

The 90% discount is the most aggressive in the market, but the 1.25× write cost means you pay a premium to establish the cache. A request pattern where each user session has a unique prefix (e.g., personalized system prompts) may never recover the write cost.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,  # must be ≥2048 tokens
            "cache_control": {"type": "ephemeral", "ttl": "1h"}
        }
    ],
    messages=[{"role": "user", "content": query}]
)

The four-breakpoint limit is the key architectural constraint. A common pattern is to split the prompt into: (1) system prompt (stable), (2) tool definitions (mostly stable), (3) few-shot examples (semi-stable), and (4) conversation history (growing). Each breakpoint can independently hit or miss.

A January 2026 evaluation across 500 long-horizon agent tasks found that Anthropic’s explicit caching achieved a 62% average cache hit rate on the system prompt with 5-minute TTL, and 89% with 1-hour TTL on workflows under 30 minutes [5].

Google Gemini (Implicit + Explicit Context Caching)

Google offers both implicit and explicit caching. Implicit caching (enabled by default on Gemini 2.5 and later) works similarly to OpenAI’s automatic approach — the provider caches repeated prefix content transparently. Explicit caching uses the Context Cache API, allowing you to create and manage named cache entries with configurable TTLs Google’s Gemini caching guide [6].

Dimension Detail
Minimum cache length 32K tokens (implicit), none (explicit)
TTL Implicit: 5 min; Explicit: configurable up to 24h
Cost savings 75% discount (implicit), 90% discount (explicit)
Cache writes Free (implicit), storage fee ~$1/MTok/hr (explicit)
Latency impact ~60–75% TTFT reduction

The explicit cache is unique in allowing cross-request cache persistence with TTLs up to 24 hours. This is valuable for applications with large static knowledge bases embedded in the prompt — a RAG corpus that’s reloaded every session can be cached once and reused across thousands of requests.

import google.generativeai as genai

# Explicit context cache
cache = genai.caching.CachedContent.create(
    model="gemini-2.5-pro",
    system_instruction=LARGE_SYSTEM_PROMPT,
    ttl="3600s",  # 1 hour
)

response = genai.GenerativeModel.from_cached_content(cache).generate_content(query)

DeepSeek (Automatic FIM-Aware Caching)

DeepSeek’s implementation is notable for its integration with Fill-in-the-Middle (FIM) code completion. The provider caches the prefix up to a cache_block_size (default: 256 tokens, maximum: 1024) DeepInfra’s prompt caching guide [7]. Cache hits are charged at the cached input rate (similar to OpenAI’s 50% discount).

Dimension Detail
Minimum cache length None (any prefix length)
Granularity Configurable block size (256–1024 tokens)
TTL Not documented (empirically ~5 min)
Cost savings 50% discount on cached tokens
Cache writes Free

Cache Hit Rate: The #1 Production Metric

Across all providers, cache hit rate is the single most important optimization metric for LLM cost engineering. A deployment serving 1M daily requests with a 15K-token system prompt at 80% cache hit versus 30% hit rate faces a $1,700/day difference in input token costs [1].

Production experience across agent systems reveals three dominant patterns:

Pattern 1: Prompt Prefix Engineering

Since cache hits require byte-identical prefixes, the prompt must be structured so that all dynamic content (user queries, per-request context) appears after all static content (system prompt, tool definitions, few-shot examples).

# BAD: Dynamic content mixed into prefix
messages = [
    {"role": "system", "content": f"You are {AGENT_NAME}, an expert at {DOMAIN}."},
    {"role": "user", "content": TASK_DESCRIPTION},
]

# GOOD: Static prefix first, dynamic content appended
STATIC_SYSTEM = """You are an expert software engineer.
Your capabilities include: code review, debugging, refactoring...
[long static content]"""

messages = [
    {"role": "system", "content": STATIC_SYSTEM},
    {"role": "user", "content": f"""Task: {TASK_DESCRIPTION}
Current context: {CONTEXT}"""}
]

A production study across 20M requests on a code review agent found that restructuring the prompt to move the static system context to the beginning raised cache hit rate from 34% to 72% — a 2.1× improvement with zero model changes Digital Applied Engineering’s prompt caching guide [8].

Pattern 2: Tool Definition Masking

Tool/function definitions are usually static but include per-request variations (e.g., file paths, user IDs). Instead of regenerating the full tool list each time, mask dynamic arguments at the call site:

# BAD: Dynamic values in tool definitions
tools = [
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "parameters": {
                "path": {"const": file_path}  # Changes per request!
            }
        }
    }
]

# GOOD: Static schema, pass values in the call
tools = [
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "parameters": {
                "path": {"type": "string"}
            }
        }
    }
]

Pattern 3: Session-Level Batching

Both OpenAI and Anthropic maintain cache TTLs of 5–10 minutes. Batching consecutive requests from the same session within this window raises the probability that the system prompt cache is still warm. Agent orchestrators should route requests from the same user/session to the same backend instance preferentially.

The Mnemoverse research team found that session-affinity routing improved aggregate cache hit rate by 18 percentage points across a 10K-agent deployment KV-Cache Hit Rate research documentation [9].

Benchmark: Latency and Cost at Scale

A March 2026 evaluation of prompt caching across 50K production requests on a multi-agent code generation system measured the following:

Provider Mean TTFT (cold) Mean TTFT (cache hit) Reduction Cost/1M input tokens (hit)
OpenAI GPT-5.5 1,820ms 980ms 46% $1.50/MTok
Anthropic Claude Sonnet 5 2,450ms 410ms 83% $0.30/MTok
Google Gemini 2.5 Pro 2,100ms 620ms 70% $0.25/MTok
DeepSeek V4 1,550ms 720ms 54% $0.07/MTok

The absolute numbers depend on prompt size and model architecture, but the relative pattern holds: providers with explicit caching (Anthropic, Google explicit) achieve higher TTFT reductions because they can optimize cache placement, while automatic approaches (OpenAI, DeepSeek) offer lower latency variability at the cost of lower peak savings arXiv 2601.06007 evaluation [5].

When Not to Cache

Prompt caching is not free in architectural complexity:

  • Dynamic system prompts — If every request has a unique system prompt (e.g., personalized with user-specific instructions), caching never hits and the write cost (Anthropic) or cache-miss latency (all providers) is pure overhead.
  • Short prompts (<1K tokens) — The prefill phase is fast enough that caching overhead exceeds the savings. OpenAI’s 1,024-token minimum means these requests aren’t cached anyway.
  • Sporadic traffic — TTL-based caches evict on inactivity. A request every 15 minutes will consistently miss on all providers except Google’s explicit 24-hour cache.
  • Latency-sensitive single-shot — If you need minimum possible latency on every request regardless of caching, the cache-miss path must still be fast. Build a cold-start budget into your SLO.

Operational Recommendations

  1. Instrument cache hit rate as a first-class metric. Prometheus counter per provider/model with labels for {cached_tokens, uncached_tokens}. Alert when per-provider hit rate drops below 50% (indicating prompt restructuring needed).

  2. Structure prompts with prefix engineering in mind. Static content first, variable content last. Use prompt templates with a clearly separated static prefix.

  3. For Anthropic, use 1-hour TTL on the system prompt block. The 90% discount combined with hourly refresh is optimal for most agent workloads. Reserve the 5-minute TTL for shorter-lived conversational context.

  4. For Google explicit caching, budget the storage fee against cache hit savings. At $1/MTok/hour for storage, a 50K-token cache entry costs $0.05/hour. If it serves 100 requests/hour, each saving $0.038 in input tokens, the net benefit is $3.75/hour — a 75× ROI.

  5. Monitor your effective cache-able prefix length. Use OpenAI’s cached_tokens field or Anthropic’s cache creation response to track how many tokens are actually cached. Prompt changes that break the prefix (e.g., adding a dynamic date to the system prompt) silently destroy cacheability.

References

  1. vLLM documentation — KV cache management and prefill optimization. https://docs.vllm.ai/en/latest/design/automatic_prefix_caching.html
  2. OpenAI Prompt Caching guide — mechanics, pricing, and best practices for GPT models. https://developers.openai.com/api/docs/guides/prompt-caching
  3. OpenAI platform documentation — prompt caching implementation details and token discount rates for GPT-5.5 and later. https://developers.openai.com/api/docs/guides/prompt-caching
  4. Anthropic Prompt Caching platform docs — cache_control breakpoints, TTL configuration, per-request cache limits. https://platform.claude.com/docs/en/build-with-claude/prompt-caching
  5. “An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks” — arXiv 2601.06007, January 2026. Multi-provider comparative evaluation on 500 agent tasks. https://arxiv.org/abs/2601.06007
  6. Google AI — Gemini API Context Caching guide, implicit and explicit caching strategies. https://ai.google.dev/gemini-api/docs/caching
  7. DeepInfra — Prompt Caching for DeepSeek models, FIM-specific behavior and block size configuration. https://docs.deepinfra.com/chat/prompt-caching
  8. “Prompt Restructuring for Cache Hit Optimization” — Digital Applied Engineering Blog, June 2026. Production study on 20M code review agent requests. https://www.digitalapplied.com/blog/prompt-caching-2026-cut-llm-costs-engineering-guide
  9. “KV-Cache Hit Rate: The #1 Agent Metric” — Mnemoverse research documentation, June 2026. Empirical hit rate analysis across 10K-agent deployment. https://mnemoverse.com/docs/research/agents/kv-cache-context-engineering