Structured Output Collapses Answer Diversity Across 44 Language Models
Requesting JSON output significantly reduces answer diversity across 44 LLMs, increasing modal answer concentration from 41% to 64%. The effect is driven by the linguistic register of the format request rather than technical constraints like schema enforcement or constrained decoding. Compression is specific to structured formats like JSON and XML, indicating a link to tool-use post-training behaviors rather than general syntax handling. Model defaults shift dramatically when switching from chat
Analysis
TL;DR
- Requesting JSON output significantly reduces answer diversity across 44 LLMs, increasing modal answer concentration from 41% to 64%.
- The effect is driven by the linguistic register of the format request rather than technical constraints like schema enforcement or constrained decoding.
- Compression is specific to structured formats like JSON and XML, indicating a link to tool-use post-training behaviors rather than general syntax handling.
- Model defaults shift dramatically when switching from chat to structured output, with some models adopting entirely new preferred responses (e.g., "cerulean") only in JSON mode.
Why It Matters
This finding reveals a critical discrepancy between how LLMs behave in conversational evaluations versus how they function in production software pipelines. Practitioners relying on structured outputs for data extraction may inadvertently receive less diverse or more biased results than those observed in standard benchmark tests, necessitating a re-evaluation of evaluation metrics for API-based applications.
Technical Details
- Experimental Setup: Re-ran the "One-Word Census" with 31 prompts across 44 models, comparing unconstrained text responses against requests for JSON-only replies without schema enforcement.
- Quantitative Impact: Mean answer-choice surprisal dropped from 1.80 to 1.58 bits; distinct answers fell from 52 to 36 in the "Pick a word" category.
- Format Specificity: Significant compression was observed for JSON (-0.22 bits) and XML (-0.19 bits), but not for YAML or CSV, and was reversed for arbitrary bracket wrappers, pointing to learned tool-use patterns.
- Decoder Independence: Enforcing schema at the decoder level (
response_format) added negligible additional compression (-0.03 bits), confirming the phenomenon is register-based.
Industry Insight
- Evaluation Bias: Standard chat-based benchmarks may overestimate the diversity and robustness of LLMs in real-world integrations where structured outputs are mandatory.
- Prompt Engineering: Developers should be aware that simple format instructions can act as strong priors, potentially narrowing the model's creative or exploratory range.
- Model Selection: When choosing models for data-intensive tasks, test both chat and structured output modes, as performance characteristics and default behaviors can diverge significantly between the two registers.
Disclaimer: The above content is generated by AI and is for reference only.