When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents
Introduces OrderBench, a deterministic benchmark evaluating the semantic reliability of LLM agents in restaurant ordering scenarios beyond simple syntax. Demonstrates that while JSON Schema ensures structural validity, it fails to guarantee semantic correctness, with strong models achieving only ~80% semantic success despite 100% schema compliance. Highlights that weaker models exhibit significant rates of "schema-valid unsafe acceptances," where structurally correct outputs contain dangerous or
Analysis
TL;DR
- Introduces OrderBench, a deterministic benchmark evaluating the semantic reliability of LLM agents in restaurant ordering scenarios beyond simple syntax.
- Demonstrates that while JSON Schema ensures structural validity, it fails to guarantee semantic correctness, with strong models achieving only ~80% semantic success despite 100% schema compliance.
- Highlights that weaker models exhibit significant rates of "schema-valid unsafe acceptances," where structurally correct outputs contain dangerous or incorrect business logic.
- Establishes that structured output generation is a necessary interface layer but insufficient for ensuring safe, faithful transactions without additional domain verification.
Why It Matters
This research challenges the common assumption that enforcing JSON schemas eliminates errors in LLM-based transaction systems, revealing a critical gap between syntactic validity and semantic safety. For AI practitioners building agent workflows, it underscores the necessity of implementing robust post-processing verification and fail-closed mechanisms rather than relying solely on model-generated structured outputs. This insight is vital for industries requiring high reliability, such as finance, healthcare, and e-commerce, where semantic errors can lead to significant operational or financial risks.
Technical Details
- Benchmark Design: OrderBench separates evaluation into six distinct dimensions: syntactic validity, schema validity, status decisions, exact item semantics, constraint preservation, and unsafe acceptances.
- Experimental Setup: The study utilized 2,400 calls to four open-source LLMs via the Nebius Token Factory, comparing performance across standard prompt-only modes and JSON-schema constrained modes.
- Key Findings: Even the strongest models achieved 100% schema validity in constrained mode, yet semantic success rates hovered around 80%. Weaker models showed double-digit percentages of unsafe acceptances that were nonetheless schema-compliant.
- Methodology: The approach isolates "transaction compilation" tasks where natural language intents are converted into executable API objects, specifically focusing on the reliability of these conversions in a controlled restaurant ordering context.
Industry Insight
- Implement Domain Verification: Treat LLM structured outputs as untrusted data sources; always implement a verification layer that checks semantic constraints and business logic before executing API calls.
- Adopt Fail-Closed Architectures: Design systems to default to safe, non-execution states when semantic ambiguity or potential violation is detected, rather than proceeding with schema-valid but semantically flawed commands.
- Monitor Beyond Syntax: When evaluating LLM agents for production use, prioritize metrics related to semantic accuracy and constraint preservation over simple parsing success rates, as the latter can mask critical functional errors.
Disclaimer: The above content is generated by AI and is for reference only.