LLM API Rate Limiting for Multi-Tenant Systems

The Multi-Tenant 429 Incident

At 09:14 on a Tuesday, a single tenant’s batch job fires 2,000 concurrent requests through your shared LLM gateway. Your provider’s per-minute token quota is exhausted within seconds — the naive per-tenant counters were checked, passed, and then ignored the provider’s global ceiling. Every other tenant now gets 429 Too Many Requests, including your highest-value customer’s real-time chat. The root cause: no admission-control layer modeling tenant entitlements and external provider quotas as one unified constraint system. This post designs the admission-control system that prevents it.

How This Was Researched

This analysis is based on official provider documentation, published pricing pages, and community engineering reports from production incident write-ups — not hands-on testing. Primary sources include OpenAI’s rate-limit guide, Anthropic’s rate-limit documentation, AWS Bedrock quotas, Google Vertex AI quotas, Azure OpenAI Service quotas, and the canonical Redis rate-limiter pattern. What this post does not cover: real-time benchmarking of limiter implementations, proprietary internal platform details from specific companies, or unpublished provider roadmap information. The architectural patterns described here are derived from documented system-design principles and publicly available provider constraints. Last researched: August 2026.

Model the Quota System as Admission Control

Treat every request as an admission-control decision, not a post-hoc check. The hierarchy is: org → project → user/API key → provider → model → global capacity, each with its own counters; admission requires passing every level from tenant entitlement down to provider capacity.

The lifecycle is check-and-reserve-reconcile. Before dispatch, estimate input tokens from the prompt plus a max output ceiling (from the model’s configured max_tokens). Reserve that estimated total against all relevant counters atomically. After the response returns, reconcile the reservation against actual token usage from the usage object, refunding the over-reserved delta or deducting the overage.

Provider limits differ in dimensionality. OpenAI Rate Limits enforce requests per minute (RPM) and tokens per minute (TPM) as separate dimensions. Anthropic Rate Limits go further, splitting input TPM and output TPM into independent counters. Your reservation must check against every dimension that the target provider enforces, not just a single aggregate token count.

Build Distributed, Token-Aware Limiters

The limiter must be correct under concurrency and partial failure. Three properties matter: atomic check-and-reserve, bounded counter windows, monotonic timestamps.

The Redis Rate Limiter Pattern is the canonical reference: a Lua script atomically increments counters and compares against limits in one round trip, avoiding the race where concurrent requests all pass a read-then-write check.

Counter windows need bounded TTLs. A sliding-window counter stored as a sorted set with per-request timestamps works but grows unboundedly without pruning; a fixed-window counter with a TTL equal to the window length is simpler, combined with a small sliding-window component for the last partial window. Use Redis TIME or a monotonic clock source, never server-local wall clocks that can jump backward. Each request carries an idempotency key so retries and duplicate deliveries don’t double-count.

Compose Tenant Policy with Provider Quotas

Internal tenant quotas are soft constraints you control. Provider quotas are hard constraints you don’t. The gateway sits between them, and the composition rule is simple: tenant policy must never exceed its proportional share of the provider’s quota.

Provider quotas vary by model and region. As covered in our guide to LLM router architecture and engineering, routing decisions and provider failover belong in the same gateway layer that enforces admission control. Bedrock Quotas are defined per foundation model and per AWS region — a quota for Claude on us-east-1 is independent from the same model on us-west-2. Vertex AI Quotas apply per model and per Google Cloud project, with different default limits for different model families. Azure OpenAI Service Quotas operate on provisioned throughput units (PTU) and TPM, with regional variation.

Your tenant policy defines a target share — say 20% of a provider-model-region combination — and the limiter clamps the tenant’s effective ceiling to the minimum of their entitlement and their proportional provider share. When provider quotas change, the gateway must pick up new values without a redeploy: store them in a config service or feature store, not in code.

Add Cost Governance and Usage Reconciliation

Rate limits and spend limits are different control planes. Rate limits answer “how fast can this tenant go?” in requests and tokens per minute; spend limits answer “how much money can this tenant burn?” in dollars per hour, day, or month. Both must be enforced, but with different counters and data sources.

The usage ledger records, for every completed request: tenant, project, user/API key, model, provider, input tokens, output tokens, and the price-book version used for cost calculation. Price books change — providers adjust per-token pricing and you may negotiate custom rates. Without versioning, you cannot reconcile a tenant’s spend against the rate in effect when the request ran.

Prompt caching can meaningfully cut token spend before it reaches the ledger — our analysis of LLM caching at scale covers the cache-hit economics. Configure soft-alert thresholds at 80% of budget: emit a metric, trigger a webhook. At 100%, reject with a clear error code distinguishing budget exhaustion from rate-limit exhaustion. The rejection must be fail-closed — no automatic passthrough to a higher-cost model or different provider without explicit policy review.

Operate Retries, Queues, and Streaming Safely

Retries are the most common way to turn a 429 into a cascade. Implement exponential backoff with full jitter to avoid thundering-herd synchronization, cap retry attempts at three, and honor any Retry-After header the provider returns.

A critical fact: unsuccessful retries still count toward OpenAI per-minute limits, as documented in OpenAI Rate Limits. A request that fails with a 429 and is retried consumes quota on both attempts. Your limiter must account for this — either reserve quota for the retry budget upfront or treat retries as new admissions against the current counter state, which will naturally throttle them as the window fills.

For streaming responses, token counting is approximate mid-stream. Reserve the full output ceiling before dispatch, then reconcile against actuals on stream completion. Do not update tenant counters mid-stream from partial token counts — per-chunk counter updates become a bottleneck and the estimates are unreliable.

Fail-closed for hard limits: if the limiter itself errors (Redis down, config unavailable), reject the request rather than admitting it unchecked. Fallback to a different model only after a full quota and policy recheck against the fallback target — never automatic passthrough.

FAQ

How do you handle provider quota changes without redeploying?

Store provider quotas in a dynamic configuration service with a versioned schema, not in application code. The gateway subscribes to config updates and applies new values to the limiter’s clamp calculation immediately — for increases the next request picks it up, for decreases tenant entitlements clamp proportionally on the next admission check. A short TTL cache (seconds, not minutes) balances latency against freshness.

Should rate limits apply per-token or per-request?

Both, but for different reasons. Per-request limits (RPM) bound the number of round trips regardless of payload size; per-token limits (TPM) bound the actual compute and money consumed. A single tenant could send one 100,000-token prompt and exhaust your entire provider TPM quota, so the admission check evaluates both dimensions simultaneously, reserving against both counters before dispatch.

How do you prevent a single tenant from monopolizing fallback models?

Apply the same admission-control hierarchy to fallback targets as to primary targets. When a request fails on the primary provider and policy allows a fallback, the fallback admission must recheck the tenant’s entitlement on the fallback model, the fallback provider’s quota, and that model’s global capacity. A separate per-tenant fallback budget — a percentage of the fallback model’s total capacity — prevents one tenant’s burst from exhausting the primary and then the fallback, starving every other tenant twice.

📖 Related Reads

• ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
• NoCode Insider — AI workflow automation with no-code tools, agents, and APIs

Cross-links automatically generated from CodeIntel Log.