Token Accounting Analysis: Debugging Silent Truncation

You deployed a model, the API returned HTTP 200, message.content was an empty string, and your bill showed 1,000 completion tokens. This is the silent truncation incident class, and knowing how to debug LLM token accounting drift is the difference between a five-minute fix and a week of false leads.

How This Was Researched

How this was researched matters for trusting what follows. This analysis is based on official documentation, pricing pages, and community reports — we did not run the systems hands-on. Coverage draws on OpenAI’s reasoning guide, Anthropic’s extended thinking docs, deprecation notices, and developer forum threads; provider outages, self-hosted inference, and streaming-specific accounting are out of scope. Last researched: August 2026.

The Incident Class: HTTP 200 With Nothing to Show

The incident class “HTTP 200 with nothing to show” is an API call that returns 200 with empty content and a non-zero bill, where finish_reason is "length" rather than an error — a token-accounting failure documented in community analysis of o1, not a network or service failure.

The canonical shape appears in community reports on OpenAI’s o1 series: message.content is "", finish_reason is "length", usage.completion_tokens is 1000, and usage.completion_tokens_details.reasoning_tokens is also 1000 (dev.to analysis). The finish_reason field officially accepts stop, length, tool_calls, content_filter, and the deprecated function_call — "length" specifically means the max token limit was reached before the model finished (OpenAI chat API reference). Your system logged a success, your user saw a blank screen, and your cost center saw a charge.

Root Cause 1: Hidden Reasoning Tokens

Hidden reasoning tokens are invisible to the API response body, occupy the context window, and are billed as output tokens — ranging from a few hundred to tens of thousands per request, visible only in the usage.output_tokens_details.reasoning_tokens field (OpenAI’s reasoning guide).

Historical caps make this concrete: o1-preview had a 32,768-token reasoning limit, o1-mini had 65,536, and o1 had 100,000 (Simon Willison’s o1 analysis). If you set max_tokens to 1,000 and the model reasons for 1,000 tokens, you get exactly the empty-response shape above. The Responses API reports this as status "incomplete" with incomplete_details.reason equal to "max_output_tokens", which can occur before any visible output exists (OpenAI’s reasoning guide).

Root Cause 2: The max_tokens Semantics Break

The max_tokens semantics break happened because the parameter name changed with the model family: max_tokens is deprecated on o1-series models in favor of max_completion_tokens (OpenAI community thread), and max_completion_tokens is one shared budget covering reasoning plus visible output rather than a cap on visible output alone (OpenAI’s reasoning guide).

A code snippet that worked on GPT-4o breaks on o1:

from openai import OpenAI

client = OpenAI()

# Works on GPT-4o, silently truncates on o1
response = client.chat.completions.create(
    model="o1",
    messages=[{"role": "user", "content": "Solve this problem step by step"}],
    max_tokens=500,  # Deprecated on o1
)

# Correct approach for o1-series
response = client.chat.completions.create(
    model="o1",
    messages=[{"role": "user", "content": "Solve this problem step by step"}],
    max_completion_tokens=500,  # Shared budget: reasoning + visible output
)

If your codebase still passes max_tokens to an o1-series model, observed behavior depends on client and proxy versions — some setups reject the call, others silently fall through to stale defaults (OpenAI community thread).

Root Cause 3: Tokenizer Drift

Tokenizer drift means the same text produces a different token count on a new model version, and the drift can be massive. Anthropic’s Claude Fable 5 and Mythos 5 use the Opus 4.7 tokenizer, which produces roughly 30% more tokens than pre-Opus-4.7 models for identical text (Anthropic’s token counting docs).

This is not a bug — it is a deliberate tokenizer change — but it breaks every hardcoded token budget in your system. The reported output token count can exceed visible tokens even when the reported reasoning_tokens field is 0, indicating tokens consumed by internal processes not exposed in the usage object (OpenAI’s token counting guide). If you sized headroom on the older tokenizer and migrated to Fable 5 or Mythos 5, your effective output capacity dropped by roughly 30% with no code change — and the API still returns 200.

Root Cause 4: Non-Visible Formatting Tokens (2026)

Non-visible formatting tokens are the newest drift vector: GPT-5.6 renders prior-turn reasoning into the next sample via the reasoning.context field by default, adding tokens to every subsequent request in a conversation (OpenAI’s reasoning guide). Those formatting and context tokens never appear in visible message content, yet they are counted and billed.

Multi-turn conversations therefore accumulate hidden token debt: turn one uses 1,000 reasoning tokens; turn two carries those as context plus new reasoning; turn three carries both. Your max_completion_tokens budget is shared across reasoning, visible output, and carried context, so truncation appears to happen “randomly” on later turns as hidden load grows monotonically. The field is optional per the API reference, but the default makes it a production hazard unless you explicitly disable it.

How to debug LLM token accounting drift?

How to debug LLM token accounting drift comes down to five ordered checks, each eliminating one root cause before moving to the next. Start with the usage object, since OpenAI exposes reasoning-token detail there (OpenAI’s reasoning guide); then verify parameter semantics, compare client- versus server-side counts, look for non-visible formatting tokens, and read finish_reason.

def debug_silent_truncation(response, request_params):
    """Five checks in order. Stop at the first failing check."""
    # Check 1: Inspect the full usage object
    usage = response.usage
    if usage.completion_tokens_details.reasoning_tokens > 0:
        print(f"Root cause: hidden reasoning tokens ({usage.completion_tokens_details.reasoning_tokens})")
        return

    # Check 2: Verify parameter name matches model family
    if response.model.startswith("o1") and "max_tokens" in request_params:
        print("Root cause: max_tokens deprecated, use max_completion_tokens")
        return

    # Check 3: Compare tokenizer counts across model versions (20% tolerance heuristic)
    client_count = count_tokens_with_client_tokenizer(prompt)
    server_count = count_tokens_with_server_api(prompt)
    if server_count > client_count * 1.2:
        print(f"Root cause: tokenizer drift ({client_count} -> {server_count})")
        return

    # Check 4: Non-visible formatting tokens (GPT-5.6+) — naive proxy via usage delta
    if response.usage.completion_tokens > len(response.choices[0].message.content):
        print("Root cause: non-visible formatting tokens")
        return

    # Check 5: Verify finish_reason and incomplete_details
    if response.choices[0].finish_reason == "length":
        print("Root cause: budget exhausted before completion")
        return

Each check is fast, and together they map onto the four root causes above.

Defenses That Actually Hold

The defenses that actually hold against this incident class are structural, not reactive. The single most effective defense is reserving headroom: OpenAI recommends at least 25,000 tokens in your max_completion_tokens budget for reasoning plus outputs (OpenAI’s reasoning guide), and server-side token counting supplies the authoritative pre-flight numbers.

# Defense: reserve >=25,000 tokens for reasoning + visible output
MAX_COMPLETION_TOKENS = 25000  # Minimum per OpenAI's reasoning guide

response = client.chat.completions.create(
    model="o1",
    messages=messages,
    max_completion_tokens=MAX_COMPLETION_TOKENS,
)
assert response.usage.completion_tokens < MAX_COMPLETION_TOKENS - 1000, "Budget exhausted"

For Anthropic models, the thinking.budget_tokens parameter has a minimum of 1,024, must be less than max_tokens, counts toward max_tokens, and changing it invalidates cache breakpoints (Anthropic’s extended thinking docs). Your cache strategy and your token budget are coupled — tuning one invalidates the other.

Use server-side counting before sending: OpenAI offers POST /v1/responses/input_tokens and Anthropic offers POST /v1/messages/count_tokens (free, with RPM limits of 2,000/4,000/8,000 by tier) (OpenAI’s token counting guide, Anthropic’s token counting docs). For client-side estimation, use tiktoken with o200k_base and encoding_for_model, knowing its limits: no image/file handling, tools and schemas add tokens, behavior is model-specific (tiktoken GitHub). The client-side count is an estimate; the server-side count is truth.

Generalizable Lessons for Production RCA

Generalizable lessons for production RCA start here: a 200 status is a transport signal, not a success signal — the application-level contract is finish_reason and usage. Budget parameters also carry semantic drift, and the rename to max_completion_tokens amounts to OpenAI acknowledging the old name concealed the wrong semantics (OpenAI community thread).

Third, tokenizer changes are breaking changes even when the API surface is identical: a ~30% token increase on identical text (Anthropic’s token counting docs) is a silent capacity reduction. Fourth, deprecation cadence is not uniform — GA models get at least 6 months notice, specialized models at least 3 months, previews as little as 2 weeks (OpenAI’s deprecations page) — so monitoring must watch deprecation notices, not just runtime errors.

For broader context on where this fits, see the AI stack reference and the AI tools index. Related sibling posts cover LLM observability architecture, debugging LLM provider outages, and LLM model versioning and safe rollout.

FAQ

Why does my LLM API return HTTP 200 but empty content?

The HTTP layer succeeded but the application layer failed. The most common cause is that the max_completion_tokens budget was consumed entirely by hidden reasoning tokens before any visible output was generated, producing finish_reason: "length" with empty message.content — so check the usage object’s reasoning-token field first (OpenAI’s reasoning guide).

How are reasoning tokens billed if I can’t see them?

Reasoning tokens are billed as output tokens at the same rate as visible output tokens. They are not visible in the message content but are reported in the usage.output_tokens_details.reasoning_tokens field (OpenAI’s reasoning guide). You pay for them whether or not any visible text is produced.

Does changing the model version affect my token count?

Yes, significantly. Anthropic’s Claude Fable 5 and Mythos 5 use a tokenizer that produces about 30% more tokens than pre-Opus-4.7 models for identical text (Anthropic’s token counting docs). Re-validate every hardcoded token budget whenever you change model versions, because server-side counts are the truth and estimates go stale overnight.

ToolBrain — tool reviews, LLM comparisons, and AI workflow guides
NoCode Insider — AI workflow automation with no-code tools, agents, and APIs

Cross-links automatically generated from CodeIntel Log.