Structured LLM Output: JSON Mode vs Function Calling

The core distinction in production LLM systems is not a single API setting but a three-tier guarantee ladder for structured LLM output JSON mode vs function calling, each offering different levels of reliability. This is an architectural choice, not a feature toggle.

How This Was Researched

This analysis is based on desk research of official provider documentation, arXiv papers on constrained decoding, and vLLM/Red Hat engineering resources as of August 2026. No hands-on testing or benchmarking was performed; all URLs were verified for HTTP 200 status.

How do you get schema-valid JSON from an LLM?

There are three distinct guarantee levels: JSON mode ensures the output is valid JSON but not that it conforms to your schema; Structured Outputs guarantee schema conformance for a supported subset; function calling is a separate protocol for invoking tools with a structured envelope. This is a critical difference in reliability.

Achieving schema-valid output requires moving beyond simple prompts. OpenAI’s JSON mode “ensures that model output is valid JSON” but explicitly “will not guarantee the output matches any specific schema” OpenAI Structured Outputs docs. True schema adherence demands a different mechanism—grammar-constrained sampling—which underpins the Structured Outputs and strict tool-use features from major providers. Teams that conflate JSON validity with schema conformance ship brittle systems.

Why doesn’t prompting or JSON mode give you a schema contract?

JSON mode is a format guarantee, not a content contract. It prevents syntax errors but does not compel the model to fill in all required fields, use correct enumerations, or avoid extra properties. This leads to a widespread pattern: first generate, then validate.

An OpenAI evaluation showed that while their new gpt-4o-2024-08-06 model with Structured Outputs achieves 100% on complex schema following, the older gpt-4-0613 model scores under 40% on the same task OpenAI Structured Outputs launch post. Without constraints, models frequently hallucinate fields or produce incorrect structures. Consequently, the standard practice remains generating JSON (or using JSON mode) and then catching errors with a validation library. As OpenAI’s own docs advise for JSON-mode users, use a validation library and potentially retries OpenAI Structured Outputs docs. This validate-and-retry loop is the canonical backstop.

How does constrained decoding work at the token level?

Constrained decoding works by masking the model’s logits at each step, permitting only tokens that continue a valid parse of a grammar or finite-state machine (FSM). This reformulates generation as navigating an FSM over the vocabulary Outlines paper (arXiv 2307.09702).

Multiple implementations exist. Outlines represents the schema as an FSM, while XGrammar uses a byte-level pushdown automaton for efficiency XGrammar paper (arXiv 2411.15100). Guidance employs token fast-forwarding with on-the-fly mask computation Guidance repo. Because a practical subset of JSON Schema compiles to a context-free grammar (CFG), it can be compiled into these automata. OpenAI credits Outlines, jsonformer, instructor, guidance, and lark as inspirational projects OpenAI Structured Outputs docs. Self-hosted frameworks like vLLM expose this directly, offering json (JSON Schema), grammar (arbitrary EBNF), structural_tag, choice, and regex modes vLLM structured outputs docs.

Structured LLM output JSON mode vs function calling: which do you reach for?

Function calling connects a model to your system’s tools, while response_format (JSON mode/Structured Outputs) structures its reply. Anthropic separates these as JSON outputs versus strict tool use. The choice is task-dependent: use structured outputs for data extraction, function calling for action execution.

Think of it as a portfolio. For a workload requiring a specific data schema (e.g., a product listing), Structured Outputs are the right guarantee. For triggering an API call (e.g., “search the database”), function calling is the protocol. Anthropic’s documentation illustrates this split Anthropic structured-outputs docs. There is also a cost dimension; Anthropic notes that tool-use definitions add system-prompt token overhead (e.g., 354 tokens on Sonnet 5 with tool_choice: auto/none, or 474 for any/tool) Anthropic tool-use overview. Mixing these concepts—forcing tool calling to get structured data, for example—often leads to inefficient or brittle designs. The tools directory can help evaluate which pattern fits your integration.

Where do provider guarantees break?

Even Structured Outputs have failure modes. The model can still refuse unsafe requests (detected via a refusal field), hit max_tokens length limits, or be truncated by content filters. It prevents schema violations but not semantic errors inside valid JSON fields OpenAI Structured Outputs docs.

Operational gaps also exist. Parallel function calls may be disabled (parallel_tool_calls: false), and schema-supplied requests are often ineligible for Zero Data Retention policies — check ZDR eligibility with your provider. Furthermore, the guarantee only covers structure. If your schema allows an “email” field with a string type, the model might output “banana”. The contract is syntactic, not semantic. Therefore, defense-in-depth with validation is always advised, as outlined in our eval-driven development guide for agent systems.

Your schema is not what you wrote

Providers enforce a supported subset of JSON Schema (2020-12), and SDKs can transform your schema before sending it. OpenAI requires the root to be an object with all properties listed as required; optionality is simulated via union-with-null. They also do not support top-level anyOf, which breaks Zod discriminated unions OpenAI Structured Outputs docs.

Anthropic’s SDKs go further, silently altering schemas to fit their backend: they strip constraints like minimum/maximum and force additionalProperties: false. The SDK then re-validates responses client-side against your original schema Anthropic structured-outputs docs. This means a schema that works in local testing might fail against the API, or worse, pass validation but produce unintended models. Always test with the provider’s API, not just a local JSON Schema validator.

What does grammar-constrained decoding cost in latency and ops?

Compile-on-first-use introduces latency for a schema’s first call, but compiled grammars are cached (e.g., Anthropic caches for 24 hours) Anthropic structured-outputs docs. Backend choice matters: XGrammar caches structures, favoring repeated schemas, while llguidance computes per-token, favoring dynamic schemas Red Hat Developer (vLLM maintainers).

In vLLM, earlier versions (V0) caused system-wide degradation from a single constrained request, while the current version (V1) reports minimal overhead Red Hat Developer (vLLM maintainers). The XGrammar paper claims up to a 100× speedup over prior guided-generation solutions XGrammar paper (arXiv 2411.15100). Ops considerations include cache invalidation (when schemas change) and PHI leakage risk in cached grammar property names XGrammar paper (arXiv 2411.15100). These tradeoffs between first-token latency and throughput are central to choosing between self-hosted solutions and managed APIs.

How do you pick: managed API guarantees or self-hosted constrained decoding?

The decision balances control, dynamism, and compliance. Managed APIs (OpenAI, Anthropic) offer simplicity and provider-backed guarantees but lock you to a schema subset and their latency profile. Self-hosted solutions (vLLM with XGrammar/Outlines) offer maximum control over the schema and backend, ideal for highly dynamic or sensitive workloads.

A practical approach is belt-and-suspenders: use a provider’s Structured Outputs for supported, static schemas, and fall back to a self-hosted decoder for complex or frequent schema changes. Always layer on a validate-and-retry loop using a library like instructor as a final backstop. Consult the AI stack reference for where these components fit in a broader architecture. The portfolio question is: what guarantee does each specific workload truly need, and what operational complexity is the right price for it?

The bottom line

The choice is not about picking a single “best” method but about mapping each workload to the appropriate guarantee level on this portfolio ladder, understanding the underlying grammar-constrained decoding mechanism, and engineering for the inevitable failure modes.

FAQ

Is function calling the same as structured outputs?

No. Function calling is a protocol for a model to indicate it wants to use a tool, providing arguments in a structured format. Structured Outputs are a feature that constrains the model’s general text generation to conform to a specific JSON schema. You can use structured outputs without function calling, and some implementations of function calling don’t use the same underlying grammar-constrained decoding.

Can I use grammar-constrained decoding with any model?

Yes, in principle, if you are self-hosting the model with a compatible inference server like vLLM. The technique works by masking logits during generation, which is model-agnostic. However, provider-managed features like OpenAI’s Structured Outputs or Anthropic’s strict tool use are only available for specific, supported models on their platforms.

Do I still need to validate LLM output if I use Structured Outputs?

Yes. While Structured Outputs guarantee the output matches your schema’s structure, they do not validate the semantic correctness of the values inside. The model can fill a “city_name” string field with “42” or a “boolean” field with “sometimes”. Always perform final validation with a library like Pydantic or Zod to catch these content-level errors.

NiteAgent — AI agent development, frameworks, and production patterns
ToolBrain — tool reviews, LLM comparisons, and AI workflow guides

Cross-links automatically generated from CodeIntel Log.