Published on
Structured Output for LLMs: 3 Methods, Why Valid JSON Still Fails
Start with a narrow JSON Schema and your provider’s native structured-output mode, or constrained decoding if you control inference. Use function or tool calling when the model must request an action, as OpenAI describes. Accept one tradeoff up front: a schema-valid response is not a correct one. Plan for schema versioning, typed parsing, and independent domain validation from day one.
TL;DR:
Use enums for closed labels, regex for short fixed patterns, JSON Schema for ordinary records, and reserve context free grammars for recursive outputs such as SQL.
Prompting is cheapest but needs parsing and repair; constrained decoding guarantees syntax, yet may reduce semantic accuracy on smaller models or complex grammars.
Provider implementations differ in supported schema features and refusal behavior, so test required fields, nulls, extra properties, and refusals before production.
Track syntax validity separately from semantic accuracy, alongside refusal and truncation rates, latency, throughput, and cost; a single success rate can conceal wrong values.
Tool arguments express intent, not permission: require authorization and business rule checks before executing refunds, writes, or other external actions.
BRDGITBuild Reliable AI WorkflowsBRDGIT helps businesses identify AI opportunities, build clear roadmaps, automate workflows, and create custom AI systems for practical execution.Explore BRDGIT’s AI services
Table of Contents
What structured output means and when each mode fits
Three ways to get structured output from an LLM
Picking the right constraint language
Tooling: provider features and constraint engines
Benchmarks: what to measure and what studies show
A production checklist for structured output
Our view on adopting structured outputs pragmatically
How we help you ship structured-output pipelines
FAQ
What structured output means and when each mode fits
Structured output is model text that a program can consume without guessing. The usual formats are JSON, YAML, CSV, SQL fragments, and small domain-specific languages. The point is not tidiness. A pipeline that feeds an LLM response into a database, an API client, or a billing rule cannot tolerate a stray sentence of commentary or a missing bracket.
There are two distinct modes, and mixing them up causes most of the design mistakes we see:
Final-response structured output: the model returns data in a known shape, and your code reads it directly. Think of an extracted invoice, a classification label, or a configuration object.
Function or tool calling: the model returns structured arguments to request an operation. The arguments still need authorization and business-rule checks before anything executes, a distinction OpenAI’s structured outputs announcement draws between returning data and requesting an action.
Where each fits: use the first for API responses and config generation, and the second when the model should trigger a lookup, a write, or another external step.
The practical implication is about trust. A final-response object is data you validate. A tool call is an intent you must approve. When a model proposes “refund this order,” a perfectly valid JSON argument tells you nothing about whether the refund is allowed.
Three use cases cover most of what we build. API responses need a stable contract so downstream services do not break. Config generation needs strict types so a deployment does not fail at 2 a.m. Tool invocation needs both a valid shape and a separate gate before execution. In each case the structure is the easy half of the problem. The hard half is deciding what you do when the structure is right and the content is wrong.
Three ways to get structured output from an LLM
Engineers reach for three methods, and they differ in what they guarantee.
Prompt-based guidance is the cheapest. You describe the format, perhaps add an example, and parse what comes back. It is flexible and works with any model, but it is probabilistic. You need parsing and repair code, and repair can quietly alter data. A Towards Data Science comparison separates prompting, parsing and repair, and constrained decoding for exactly this reason: prompting is cheapest, repair can change values silently, and constrained decoding gives the strongest structural guarantee while still unable to verify facts.
Constrained decoding masks tokens during generation so the model cannot emit anything that breaks the grammar. It is one approach to schema compliance, and engines such as llguidance implement it. The StructureBench study compared Outlines, XGrammar, and Guidance on on-device models. It found that constrained decoding consistently enforces syntactic validity but may not improve semantic accuracy, and can degrade performance on smaller models or complex grammars. A separate analysis of constrained decoding’s hidden cost points the same direction: base models can benefit under structural constraints while instruction-tuned models sometimes lose performance unless prompts and examples are adapted.
Function or tool calling returns structured arguments for an operation you then choose to run. Use it when the model should trigger something and you want an explicit authorization step between intent and action.
Here is how we compare them in practice:
Prompt guidance: low cost, high flexibility, no structural guarantee, needs repair logic.
Constrained decoding: strong syntactic guarantee, possible semantic cost on smaller models or complex grammars, requires either a provider that supports it or control of inference.
Tool calling: clear separation between proposing and doing, but the arguments are still untrusted input.
The decision rule is short. If your code consumes the shape directly, choose final-response structured output. If the model should request or trigger an external action, choose function calling. Reach for plain prompting only when the stakes are low and a parse failure costs you nothing.
Pro Tip: Prototype with the provider’s native schema mode first, then move to constrained decoding only if you hit coverage gaps, cost, or latency limits you cannot tune around.
Picking the right constraint language
The constraint formalism should match the output, not your enthusiasm for grammars. The vLLM documentation lists JSON Schema, regex, choice, grammar, and structural-tag constraints, and the guidance in it reduces to a short ladder. We apply it in this order:
Choice or enum for closed label sets. If a field can only be “approved,” “rejected,” or “needs review,” say so. It is a narrow constraint for closed label sets.
Regex for small, flat patterns. Identifiers, phone numbers, and fixed-format codes fit here. Keep the pattern narrow.
JSON Schema for ordinary records and arrays. This is the practical default for most structured output work. Support is broad, but providers accept different subsets, so test the one you use.
Context-free grammar or EBNF for recursive or language-like output. SQL and nested expressions need this. Expect real cost in compilation, debugging, and runtime overhead.
Grammar engines have become much more efficient. The XGrammar paper shows that fast grammar enforcement depends on tokenizer-aware design, token-class caching, and overlapping CPU mask work with GPU inference. Research on distance-guided decoding adds another wrinkle: ordinary grammar-constrained decoding checks only local prefix feasibility, so a model can follow a prefix that is valid so far but cannot finish within the token budget. Lookahead-aware approaches such as SWYB address that gap.
The advice that follows is deliberately conservative. Start narrow. Prefer enums for closed domains. Avoid broad grammars unless the output truly needs recursion. The vLLM guidance is blunt on the cost side: overusing large grammars raises compilation and debugging cost without improving business correctness. A grammar constrains the output’s structure, but does not guarantee a complete response. It does not tell you the discount percentage is sane.
Pro Tip: Write the narrowest schema that still captures the real data, and widen it only when a concrete failing example forces you to.
Tooling: provider features and constraint engines
Tool choice follows from one question: do you call a hosted API, or do you run inference yourself?
Provider-native structured-output APIs are the simplest way in. The tradeoff is portability. As the OpenAI cookbook notes, providers differ in supported JSON Schema subsets, naming conventions, nullable forms, and refusal behavior. Convergence on JSON Schema is real, but the interfaces are not identical. Before production, test required fields, how nullable values are represented, how additional properties are handled, and what a refusal looks like. The OpenAI structured outputs guide documents distinct refusal and tool-use pathways, which means your error handling has to be provider-aware. Anthropic’s structured outputs documentation and Google’s Gemini announcement cover their own variants of the feature.
Open-source constraint engines matter when you control the serving stack:
Guidance and llguidance: compute token masks during generation. llguidance reports low typical per-token mask cost but documents slower outlier masks, so measure average mask times with your own tokenizer and schemas.
XGrammar: built around context-free grammars, with an adaptive token mask cache and persistent stack that separate context-independent tokens from context-dependent ones. It co-designs the grammar engine with inference so grammar work overlaps with GPU compute, which makes it a fit for recursive grammars and low-overhead serving.
Outlines: an alternative ecosystem that appears in the benchmarks above and supports schema-driven generation.
vLLM: an inference hub that exposes choice, regex, JSON Schema, grammar, and structural-tag constraints across several backends, useful when you want to swap constraint engines without rebuilding serving.
Testing each class needs different attention, but the discipline is shared:
Use real schemas. Toy examples hide compile cost and unsupported keywords.
Check tokenizer compatibility. A mask built for one tokenizer is wrong for another.
Watch cache hit rates and compile time. These often dominate average per-token figures.
Measure end-to-end latency under concurrency. Parser time alone is the wrong unit; the vLLM backend comparison guidance says request latency and throughput are what matter, along with batch size and maximum output length.
Self-hosting changes the economics, and private deployments add their own constraints. If you are weighing on-prem inference, the practical notes in Forge AI Deployment’s guide to LLM performance tuning on energy per token and secure local serving are a useful companion to the latency measurements above.
Benchmarks: what to measure and what studies show
A single “JSON success rate” hides the question you actually care about. The JSONSchemaBench paper argues that benchmarks should report validity, semantic accuracy, refusal and truncation rate, latency, throughput, and cost, because one headline number cannot tell you whether a system produced the right value or merely repaired malformed text. Treat that as your reporting template.
Here is what the three main studies tell you:
JSONSchemaBench: evaluates constrained-decoding frameworks on constraint-compliant generation, coverage of constraint types, and output quality, comparing Guidance, Outlines, llama.cpp, XGrammar, OpenAI, and Gemini. It can help with tool selection because it uses real schemas rather than synthetic ones.
StructureBench: constrained decoding reliably fixes syntax but does not reliably improve semantic accuracy, and results vary by model, task, grammar complexity, and metric definition.
StructEval: systematically tests 18 formats across 44 task types and finds that generation tasks are harder than conversion tasks, with renderable outputs such as HTML and SVG trailing text-only formats.
JSONSchemaBench pairs 10,000 real-world schemas with the official JSON Schema Test Suite. Real schema diversity, not toy examples, is what separates a tool that demos well from one that holds up in production.
Another reminder from StructEval is that even strong models score well below perfect on many structured formats, so do not assume your favorite model is the exception.
The recommendation that falls out of all three is to keep two scoreboards. Report syntactic success separately from semantic correctness. A dashboard that merges them will tell you everything is healthy right up until a wrong value reaches a customer.
A production checklist for structured output
Treat this as a copyable baseline. The OpenAI structured outputs guide describes a production pipeline with schema versioning, a supported-subset check, bounded retries, typed parsing, independent domain validation, observability, and adversarial tests. We have organized it by priority:
Version your schemas and check provider support. Maintain a provider-neutral domain model and generate provider-specific request schemas from it, as the OpenAI cookbook recommends. A schema change should be a deliberate, versioned event.
Parse into typed objects, then validate business rules separately. Schema validity is never your only check. Constrained decoding reduces formatting failures but does not prove extracted values are true. Follow schema checks with deterministic calculations, database or retrieval lookups, and human review for consequential actions, as Anthropic’s guidance suggests.
Bound your retries. Retry on refusals, truncation, and transport failures, but cap the attempts. Add fallback repair logic, and require user confirmation for high-risk actions.
Instrument everything. Track invalid or incomplete responses, refusal and truncation rates, mask compile times, end-to-end latency, and throughput.
Test the ugly cases. Cover long strings, escaping, nested arrays, enums, nulls, extra properties, and max-token termination. A response cut off at the token limit is a locally valid prefix that never finishes, which is precisely the failure that lookahead research targets.
Measure latency honestly if you self-host. Evaluate end-to-end latency with real schemas and concurrency. Compile time and cache hit rate often matter more than average mask time, as the vLLM performance notes point out.
Pro Tip: Log the raw model output next to the parsed object for every failure. When a semantic error surfaces weeks later, that pair is the fastest way to tell a schema gap from a model mistake.
Our view on adopting structured outputs pragmatically
Teams can spend time perfecting a grammar when the real problem is a missing business rule. Our advice is phased and unglamorous. Prototype with provider-native JSON Schema, or a small constrained-decoding proof of concept if you host your own models. Prove the shape works on real inputs. Only then harden it with the checklist above and wire in observability.
Keep two habits throughout. First, separate syntactic checks from semantic ones, in your code and in your dashboards, so a green structural score never masquerades as correctness. Second, keep a provider-neutral domain model so you can change vendors without rewriting your business logic. Portability is not automatic, and the teams that plan for it early are the ones that avoid a painful migration later.
Structure makes a model’s output legible. It does not make it right. The organizations that get value from structured output treat validity as the floor and spend their real effort on everything above it. When teams need help getting from a working prototype to something they can trust, we help with readiness assessments, custom agent builds, schema design, and fractional engineering.
— Team BRDGIT
How we help you ship structured-output pipelines
If your prototype works and production feels risky, we can help close that gap. Our AI Readiness Assessment maps where structured output fits your workflows and where the risks sit. A Custom Agent Build gives you schema-constrained agents that integrate with the systems you already run. Our Fractional Engineer Services put experienced AI engineers on your team for planning, delivery, and the observability work that keeps outputs honest, and Managed AI Support keeps those systems healthy after launch. Current pricing details are on our site.
Start with the AI Readiness Assessment on our main page to see what your use case actually needs, or talk to us about fractional engineers if you already know what you want to build. Bring a real schema and a real failure case, and we will work from there.
FAQ
What is structured output in LLMs?
Structured output means the model returns text in a machine-readable shape such as JSON, YAML, or SQL instead of free prose. It lets downstream code parse the response without guessing. Choose it when a program, not a person, consumes the result.
Is constrained decoding better than prompting for JSON?
Constrained decoding gives a stronger structural guarantee, because tokens that would break the schema are masked during generation. Prompting is cheaper and more flexible but probabilistic, so it needs parsing and repair. Neither verifies that the values are true, as the Towards Data Science comparison notes.
Does schema-valid output mean the answer is correct?
No. A schema-valid response only proves the shape is right. StructureBench found that constrained decoding enforces syntax but may not improve semantic accuracy, so add independent domain validation.
When should I use function calling instead of structured output?
Use function calling when the model should request an action, and final-response structured output when your code consumes the data directly. Tool arguments still need authorization and business-rule checks before anything runs, per OpenAI’s guidance.
Which constraint language should I start with?
Start with the narrowest option that fits: an enum for closed label sets, regex for small flat patterns, and JSON Schema for ordinary records. Reserve grammars for recursive output such as SQL, as the vLLM documentation suggests.
