PhantomFill: When the Form Demands an Answer, Language Models Invent One
The "PhantomFill" benchmark demonstrates that rigid output schemas (JSON, enums) force language models to hallucinate answers even when the input data is insufficient or missing. Required fields drive fabrication rates to 100% in ten out of thirteen tested models, overriding explicit instructions to refuse answering or state insufficient evidence. Honesty under format pressure is not correlated with model scale; smaller and larger models within the same family may refuse, while mid-sized ones fa
Analysis
TL;DR
- The "PhantomFill" benchmark demonstrates that rigid output schemas (JSON, enums) force language models to hallucinate answers even when the input data is insufficient or missing.
- Required fields drive fabrication rates to 100% in ten out of thirteen tested models, overriding explicit instructions to refuse answering or state insufficient evidence.
- Honesty under format pressure is not correlated with model scale; smaller and larger models within the same family may refuse, while mid-sized ones fabricate.
- The study introduces two metrics: Coerced Fabrication Rate and Escape Utilization Rate, revealing that current training objectives do not adequately measure schema-induced hallucinations.
Why It Matters
This research highlights a critical failure mode in production AI systems where structured data extraction is required. It challenges the assumption that models are inherently truthful, showing that structural constraints can actively induce falsehoods. For practitioners, this means that standard evaluation metrics may miss significant reliability issues in form-filling or API-integration scenarios.
Technical Details
- Experimental Setup: Evaluated thirteen models (including GPT-5.5 and open-weight variants) on inputs designed to be unanswerable (e.g., posts with likes but no replies).
- Schema Pressure: Compared free-text responses against JSON schemas requiring specific fields (e.g., sentiment enums). Free-text allowed honest refusal 98% of the time for GPT-5.5, while required JSON fields caused 100% fabrication in many models.
- Metric Definition: Introduced deterministic scoring via "Coerced Fabrication Rate" (frequency of invented answers under schema pressure) and "Escape Utilization Rate" (frequency of using provided "insufficient evidence" options).
- Scale Anomaly: Found non-linear behavior within model families, where mid-sized models were most prone to fabrication, contradicting the trend that larger models are generally more robust.
Industry Insight
- Schema Design: Avoid mandatory fields for uncertain data; always include an explicit "unknown" or "insufficient evidence" option, though note that open-weight models currently ignore these frequently.
- Evaluation Rigor: Incorporate PhantomFill-style benchmarks into CI/CD pipelines to detect schema-induced hallucinations before deployment.
- Model Selection: Do not assume larger models are safer for structured extraction; test multiple sizes within a family as resistance to fabrication does not scale linearly.
Disclaimer: The above content is generated by AI and is for reference only.