Towards a Deterministic Math Solver for Clinical Language Models
Large language models are unreliable at arithmetic, a critical issue for clinical calculators where numerical errors can change patient recommendations The authors propose a "Program-Solve" interface where the model writes case-specific Python code executed by a restricted local deterministic solver, rather than performing calculations directly Evaluated on MedCalc-Bench Verified (1,100 cases, 55 calculators) using Qwen2.5-7B and Qwen2.5-32B-AWQ, with the 32B model showing a statistically signif
Analysis
TL;DR
- Large language models are unreliable at arithmetic, a critical issue for clinical calculators where numerical errors can change patient recommendations
- The authors propose a "Program-Solve" interface where the model writes case-specific Python code executed by a restricted local deterministic solver, rather than performing calculations directly
- Evaluated on MedCalc-Bench Verified (1,100 cases, 55 calculators) using Qwen2.5-7B and Qwen2.5-32B-AWQ, with the 32B model showing a statistically significant +7.05% accuracy advantage (90.53% vs 83.47%) when using the solver
- A hand-written 22-calculator library achieved exact results on its 440 supported cases but abstained on 60% of cases overall, highlighting coverage limitations
- The approach is not a substitute for verified formulas or reliable variable extraction, and benefits vary significantly by model size
Why It Matters
This research addresses a fundamental reliability gap in deploying LLMs for clinical decision support, where arithmetic precision is non-negotiable. The Program-Solve paradigm offers a practical middle ground between brittle hardcoded calculators and unreliable direct model computation, with implications for any high-stakes domain requiring both natural language understanding and numerical accuracy.
Technical Details
- Program-Solve Interface: The model generates case-specific Python code that runs in a restricted local executor as a deterministic solver, shifting the model's role from computation to orchestration
- Benchmark: MedCalc-Bench Verified contains 1,100 cases across 55 clinical calculators; the authors audited formulas against current clinical guidelines and flagged 16 of 55 for version, use, or coefficient concerns
- Models Evaluated: Qwen2.5-7B and Qwen2.5-32B-AWQ, with both routes (direct arithmetic vs. solver handoff) having access to formulas, gold variables, and full clinical notes
- Key Results: At 7B, solver handoff showed a non-significant +3.29 points (75.31% vs 72.02%, 95% CI [-3.49, 10.38]); at 32B, the advantage was significant at +7.05 points (90.53% vs 83.47%, 95% CI [0.47, 14.60])
- Baseline Comparison: A hand-written library of 22 calculators was exact on 440 supported cases but abstained on 60% of total cases, demonstrating the coverage-reliability tradeoff
Industry Insight
- The Program-Solve approach should be considered for clinical AI systems, but only with larger models (32B+) where the statistical benefit becomes clear; smaller models may not reliably generate correct code
- Auditing benchmark formulas against current guidelines is essential—16 of 55 calculators had concerns, suggesting many existing clinical benchmarks may propagate outdated or incorrect formulas
- Deterministic solvers complement rather than replace the need for robust information extraction; investing in reliable variable extraction from clinical notes remains a critical prerequisite for any calculator system
Disclaimer: The above content is generated by AI and is for reference only.