LLM Response Caching Architecture: Two-Layer Analysis

An agent harness that re-sends a 20,000-token system prompt on every step of a task is paying repeatedly for tokens the provider has already seen. A sound LLM response caching architecture fixes this with two cooperating layers: provider-side prefix caching and application-controlled semantic caching at the gateway. This article maps how each layer works, when it earns its place, and where it fails.

How This Was Researched

This article is a documentation-driven architecture analysis rather than a benchmark report. It draws on official provider documentation and pricing pages from OpenAI and Anthropic, open-source gateway repositories including GPTCache and LiteLLM, a published case study from ProjectDiscovery, and the SCALM research paper — with no hands-on measurements of our own.

Sources used:

• OpenAI’s prompt caching guide, pricing documentation, and measurement cookbook example
• Anthropic’s prompt caching docs, pricing page, GA announcement, and context engineering essay
• Open-source projects: GPTCache, RedisVL’s LLMCache guide, LiteLLM’s caching docs, and Portkey’s cache documentation
• Published engineering and research: the ProjectDiscovery writeup and the SCALM paper

Methodology: architecture analysis of vendor documentation, pricing pages, repository code, and published engineering reports. This analysis is based on official documentation, pricing pages, and published engineering reports; no benchmarking was performed.

Not covered: inference-latency measurements, independent reproduction of vendor discount claims, and any proprietary internal performance data.

Last researched: September 2026.

What Is LLM Response Caching Architecture?

LLM response caching architecture is the layered design that stores and reuses model inputs and outputs so identical computation is paid for once. Per Anthropic’s prompt caching documentation, it spans provider-managed prefix caching, which discounts repeated prompt prefixes, and application-managed semantic caching, which serves stored answers for similar prompts.

The layers sit at different points in the request path and carry different contracts. Prefix caching is exact token matching inside the provider’s infrastructure: if the leading tokens of a request match a previously seen prefix, the cached portion is billed at a discount. It demands nothing from your code except a stable prefix. Semantic caching runs in your own gateway: the incoming prompt is embedded, compared against stored entries, and — if close enough — answered without a model call. That makes it a probability rather than a guarantee, which is why this article treats the two layers asymmetrically. For where caches, routers, and gateways sit in a full request path, see the AI stack reference.

Layer 1: Provider Prefix Caching (OpenAI and Anthropic)

Provider prefix caching discounts repeated prompt prefixes automatically at the API level. OpenAI caches prefixes of 1,024 tokens or more at a cached-input discount of up to 90 percent per its prompt caching guide, while Anthropic bills cache reads at 0.1× base input price per its documentation.

On OpenAI, caching is automatic for prompts of at least 1,024 tokens, and a single request can produce up to four cache writes, storing the prefix at multiple positions. Cached input bills at up to 90 percent below base input (pricing). On GPT-5.6 and newer models, the prompt_cache_key parameter routes requests with identical prefixes toward the same cache. Measurement is built in: cached_tokens in the response usage exposes your hit rate, and OpenAI’s cookbook example walks through the calculation.

Anthropic is explicit rather than automatic: cache_control breakpoints mark the cacheable blocks. The default TTL is five minutes, refreshed on each hit, with a one-hour option. Reads cost 0.1× base input; writes cost a premium — 1.25× for the five-minute TTL, 2× for the one-hour TTL (pricing).

The ordering rule is identical for both providers: stable content first — system prompt, then tool definitions, then few-shot examples, then user messages — because any edit early in the prefix invalidates everything downstream. Timestamps and retrieved documents belong at the end.

Layer 2: Semantic Response Caching at the Gateway

Semantic response caching at the gateway stores model outputs keyed by embedding similarity rather than exact token match. Options include the open-source GPTCache project, RedisVL’s SemanticCache with distance_threshold and TTL settings, LiteLLM’s proxy semantic backends, and Portkey’s semantic cache with exact-match-first lookup.

Four implementations matter in practice:

• GPTCache — the open-source project that established the pattern: an embedding model maps prompts to vectors; a vector store retrieves the nearest cached response.
• RedisVL SemanticCache — Redis’s production path, configured through distance_threshold plus a TTL; Redis’s semantic-cache use case guide covers setup end to end.
• LiteLLM proxy — semantic caching behind a unified gateway, with Redis, Qdrant, or Valkey backends.
• Portkey — simple and semantic caching combined, checking exact matches first; cached requests are limited to four messages of up to 8,191 tokens each.

The contract differs from Layer 1 on every axis: matching is similarity-based, control is yours — so invalidation is yours — every lookup requires an embedding call, and staleness becomes your bug class rather than the vendor’s. Our tooling comparisons walk these trade-offs in detail.

Decision Framework: When to Add Each Layer

Add caching layers in strict order: exploit provider prefix caching fully first — Anthropic’s context engineering guidance treats a stable, cacheable prefix as a first-class design constraint — then add gateway semantic caching only for idempotent, read-only routes, enforcing exact-match-first lookup, strict thresholds, TTLs, per-tenant namespacing, and cache-hit telemetry.

1. Exhaust Layer 1. Stabilize the prefix, fix the ordering, confirm hits by reading cached_tokens. This is correctness for free — no stored answers, no staleness, no embedding cost.
2. Qualify routes for Layer 2 — idempotent, read-only, and tolerant of a bounded staleness window: FAQ answering, documentation search, classification. Never tool execution or anything with side effects.
3. Hold the non-negotiables: exact-match-first before any similarity hit, a strict similarity threshold, a TTL on every entry, per-tenant namespacing, and hit telemetry that samples false positives. Our agent harness tooling guide shows where these controls live in a production runtime.

Failure Modes and Mitigations

The dominant failure modes are stale responses, poisoned caches, non-determinism at temperature above zero, similarity-threshold false positives, and cache stampedes. Standard mitigations exist for each — TTL caps, per-tenant namespacing with write restrictions, caching only temperature-zero routes, cosine thresholds at 0.95 or above, and request coalescing — as catalogued in Redis’s semantic-cache use case.

• Stale responses. Entries outlive the facts they encode. Cap with TTLs matched to your data’s freshness; Anthropic’s five-minute default is a reasonable conservative anchor.
• Poisoned caches. One tenant’s injected content contaminates another tenant’s hits. Namespaces must be per-tenant, and only validated responses should be writable.
• Non-determinism at temperature > 0. Identical prompts legitimately produce different outputs; a cache collapses that variance silently. Cache only temperature-0 routes.
• Threshold false positives. A loose threshold serves plausible-but-wrong answers, which cost more than fresh tokens. Start at 0.95+ cosine similarity and loosen only on telemetry evidence.
• Cache stampedes. A hot entry expiring everywhere at once sends a thundering herd to the provider. Coalesce: one request repopulates while concurrent callers wait.

Case Study: ProjectDiscovery 59 Percent Cost Reduction

ProjectDiscovery cut LLM costs by 59 percent by fixing how its agent used prompt caching: the cache hit rate rose from 7 to 84 percent across 9.8 billion cached tokens, achieved by making the 20,000-plus-token system prompts — re-sent at every step of 40-step tasks — reliably cacheable, per its engineering writeup.

The details matter: the win came from prefix architecture, not a new vendor. The prompts were already long and repetitive; once ordering and stability were fixed, Layer 1 did the work. Independent research points the same direction — the SCALM paper reports a 63 percent relative improvement in cache-hit ratio over a GPTCache baseline purely through prompt-layout optimization. Hit rate is an architecture property you engineer, not a flag you enable.

FAQ

This FAQ addresses the three operational questions that surface most often when combining provider prefix caching with gateway semantic caching: whether streaming responses are compatible with caching, how to select a similarity threshold, and whether the two layers can run simultaneously, drawing on OpenAI’s and LiteLLM’s documentation.

Does prompt caching work with streaming responses?

Prompt caching works with streaming responses: provider caches apply to the request prefix before generation begins, so cached-input discounts apply equally to streamed and non-streamed calls per OpenAI’s prompt caching guide. Gateway semantic caches behave differently — they store complete responses, so streamed delivery depends on each vendor’s replay implementation.

How do I choose a similarity threshold for semantic caching?

Choose a similarity threshold by starting strict — 0.95 or higher cosine similarity — and loosening only as telemetry shows no false-positive hits. A wrong cache hit costs more than a fresh token because the caller receives a plausible but incorrect answer; Redis’s semantic-cache guide treats distance_threshold tuning as an operational decision.

Can I use both cache layers simultaneously?

Yes — the layers operate at different points in the request path and compose cleanly: the gateway checks its exact and semantic cache first, and on a miss forwards the request so the provider discounts the repeated prefix, per LiteLLM’s caching documentation. A gateway hit avoids the provider round trip; a provider hit still bills discounted input.

The order of operations is the architecture: exploit Layer 1 first, because provider prefix caching is correctness for free, and add Layer 2 only where correctness allows — idempotent routes, strict thresholds, hard TTLs, per-tenant isolation. Teams that invert the order trade a token bill for a correctness debt that is far more expensive to repay.

Related guides

• Our LLM routing strategies analysis covers per-route model selection once caching is in place.
• Our context window management guide pairs with prefix caching — what stays in the prefix determines what can be cached.
• The AI stack reference maps caches, routers, and gateways across a full agent request path.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.