LLM Streaming Architecture Production Patterns: Wire to SLOs

LLM streaming architecture production patterns: why streaming is the default — and what it costs you

LLM streaming architecture production patterns demand treating the stream as a first-class data path with its own contract, failure model, and SLOs — not merely chat completion with stream=true. Streaming shifts cost from end-to-end latency to connection management, partial-token handling, and mid-stream recovery, which is why production systems must define wire formats, backpressure boundaries, and cache-replay rules before the first token leaves the provider. For context on adjacent infrastructure, see the AI stack reference and the tools index.

How this was researched

This analysis is based on official vendor documentation from OpenAI, Anthropic, Google Gemini, NVIDIA NIM, Cloudflare AI Gateway, and Portkey, reviewed against MDN Server-Sent Events guidance. No performance testing or live query execution was performed. Topics not covered include rate-limiting algorithms, inference engine comparisons, and prompt-cache economics — those are addressed in our LLM API rate limiting gateway analysis and LLM caching at scale production architecture posts. Last researched: August 2026.

What are the key LLM streaming architecture production patterns on the wire for OpenAI, Anthropic, and Gemini?

Production systems must parse three distinct SSE wire formats. OpenAI Responses API streams typed semantic events — response.created, response.output_text.delta, response.completed, and error — and is recommended as the preferred streaming interface because it was designed with streaming in mind and offers type safety OpenAI Streaming Guide. Chat Completions, by contrast, streams data-only SSE chunks containing a delta field. Anthropic emits message_start → content_block_start → content_block_delta* → content_block_stop → message_delta → message_stop, with ping events and in-stream error events carrying overloaded_error that corresponds to HTTP 529 Anthropic Streaming Docs. Gemini’s streamGenerateContent uses SSE to push response chunks, while the Live API (BidiGenerateContent) is a stateful WebSocket-based API for bidirectional streaming Gemini API Reference.

# OpenAI Responses API (typed events)
event: response.created
data: {"type":"response.created","response":{"id":"resp_123"}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","delta":{"text":"Hello"}}

event: response.completed
data: {"type":"response.completed","response":{"id":"resp_123"}}

# Anthropic SSE (message flow)
event: message_start
data: {"type":"message","role":"assistant","content":[]}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hel"}}

event: message_stop
data: {"type":"message_stop"}

# Gemini streamGenerateContent (data-only SSE)
data: {"candidates":[{"content":{"parts":[{"text":"Hel"}]}}]}

data: {"candidates":[{"content":{"parts":[{"text":"lo "}]}}]}

data: [DONE]

How do you handle partial tokens and accumulate deltas into coherent output?

Accumulating deltas requires vendor-specific strategies. OpenAI Chat Completions delivers text fragments in delta.content fields that must be concatenated; the Responses API delivers response.output_text.delta events with typed text. Anthropic streams content_block_delta events whose text must be accumulated per content block index, and critically, tool_use inputs arrive as partial JSON strings via input_json_delta — the SDK guidance is to accumulate the string deltas and parse the JSON once a content_block_stop event is received Anthropic Streaming Docs. Moderation scores are structurally post-hoc: OpenAI does not include moderation scores with partial output deltas, and they arrive only after the full generated output is available, making streaming output moderation inherently incomplete during generation OpenAI Streaming Guide. For large outputs, Anthropic SDKs require streaming to avoid HTTP timeouts, using .get_final_message() to accumulate all events.

Where does backpressure belong, and how do you implement token-buffering at the boundary?

Backpressure belongs at the gateway-to-client boundary, not between the provider and gateway. The gateway should buffer tokens and flush based on a dual threshold: flush immediately on the first token (TTFB is sacred), then flush subsequent tokens when either 50 tokens accumulate or 200 milliseconds elapse, whichever comes first. This balances latency for early output against throughput for steady-state delivery. Token buffering at the boundary absorbs downstream jitter without blocking the upstream provider stream, allowing the gateway to maintain a full-speed connection to the LLM while throttling delivery to the client.

What are the failure semantics and fallback strategies for mid-stream errors?

Mid-stream errors in streaming LLM responses have asymmetric recovery semantics. If the first token has not reached the client, the gateway can safely trigger a fallback chain — retrying the same provider or advancing to the next in the chain, as Portkey does on non-2xx responses with on_status_codes overrides for 429/503 Portkey Fallbacks. If the first token has already been delivered, the stream is structurally corrupted: partial output cannot be reconciled, so the gateway must terminate with an error event and cannot fall back transparently. Streaming guardrail verdicts arrive as extra SSE chunks after [DONE] and are informational only — they cannot alter already-sent tokens Portkey Streaming Guardrails.

Provider stream active?
  ├── No → HTTP error before first token → trigger fallback chain
  │         (retry same provider, then next provider in chain)
  │
  ├── Yes → In-stream error event received?
  │         ├── Yes → Was first token delivered to client?
  │         │         ├── Yes → Stream is corrupted; terminate with
  │         │         │           error event, no fallback (client has
  │         │         │           partial output)
  │         │         └── No → Safe to retry; trigger fallback chain
  │         │
  │         └── No → Continue streaming
  │
  └── Provider disconnects silently → Reconnect with
                                      Last-Event-ID (if supported)
                                      or restart stream

How do you cache LLM streams and replay them as valid SSE?

Caching streams requires treating the cache value as a pre-serialized SSE byte stream, not a raw response object. Cloudflare AI Gateway computes cache keys as SHA-256 hashes of provider + endpoint + model + provider auth header + full request body, meaning caching is based on exact match of the entire request Cloudflare AI Gateway Docs. A cache HIT must replay a valid SSE-shaped stream: the stored byte stream is served directly as the response body with Content-Type: text/event-stream, preserving event boundaries and [DONE] terminators. TTL ranges from 60 seconds to one month, and cf-aig-skip-cache bypasses the cache entirely. The replay rule is: on cache hit, stream the stored SSE bytes verbatim — do not re-serialize event objects, as that risks malformed chunk boundaries.

How do you instrument streaming SLOs: TTFT, ITL, and throughput?

Streaming SLOs are defined by three core metrics. Time to First Token (TTFT) is the time from query submission to the first received token, and per NVIDIA NIM benchmarking docs, it generally includes request queuing time, prefill time, and network latency — longer prompts increase TTFT because the attention mechanism uses the full input sequence to create the KV cache before generation begins NVIDIA NIM Benchmarking Docs. Inter-Token Latency (ITL) excludes TTFT and measures the gap between consecutive tokens. End-to-end latency = TTFT + generation time. Tokens Per Second (TPS) = total tokens / (total latency − TTFT). Browser-side, MDN notes that EventSource auto-reconnects by default, with HTTP/1.1 supporting roughly 6 SSE connections per browser per domain and HTTP/2 supporting around 100 MDN SSE Guide.

FAQ

When should you use SSE versus WebSocket for LLM streaming?

Use SSE when the stream is server-to-client only — which covers the vast majority of LLM response streaming — and you want automatic reconnection, HTTP caching, and simple text-based debugging. Use WebSocket when you need bidirectional streaming, such as the Gemini Live API’s BidiGenerateContent for conversational voice or real-time multimodal input, where the client must also push data back to the model mid-stream.

Can you retry a failed LLM stream from where it stopped?

No, standard SSE streaming does not support mid-stream resumption. If the first token has not reached the client, a gateway can safely trigger a fallback chain and restart the stream from scratch. If partial tokens have already been delivered, the stream is structurally corrupted — the client has incomplete output that cannot be reconciled with a retry, so the gateway must terminate with an error event rather than attempt a silent restart.

What is the difference between streaming and non-streaming TTFT measurements?

In non-streaming mode, TTFT is measured from request submission to receiving the complete HTTP response — it conflates prefill, queuing, generation, and network latency into a single opaque value. In streaming mode, TTFT is measured to the first token only, isolating prefill and queuing delay from generation time, which enables separate SLO tracking for TTFT (first-token latency) and ITL (inter-token latency) as defined by NVIDIA NIM benchmarking metrics.

For more on production inference infrastructure, see our Production RAG architecture analysis.