Your Hallucination Benchmark Is Measuring Your Detector
The author labeled 7,440 model answers across four open-weight LLMs (Phi-4-mini, Mistral-7B, Qwen2.5-7B, Llama-3.1-8B) on HotpotQA and TruthfulQA, discovering that over half of the initial hallucination labels were incorrect, fundamentally changing the results. LLaMA-3.1 appeared to have the lowest hallucination rate (31.2%) under plain prompting, but this was largely because it refused to answer 46.6% of questions — a conservative behavior conflated with accuracy rather than genuine correctness
Analysis
TL;DR
- The author labeled 7,440 model answers across four open-weight LLMs (Phi-4-mini, Mistral-7B, Qwen2.5-7B, Llama-3.1-8B) on HotpotQA and TruthfulQA, discovering that over half of the initial hallucination labels were incorrect, fundamentally changing the results.
- LLaMA-3.1 appeared to have the lowest hallucination rate (31.2%) under plain prompting, but this was largely because it refused to answer 46.6% of questions — a conservative behavior conflated with accuracy rather than genuine correctness.
- Chain-of-thought prompting had opposite effects across models: it reduced hallucinations by ~17-19 percentage points for Qwen and Mistral, but increased LLaMA's rate by 7.4 percentage points by encouraging it to attempt questions it would previously decline.
- Standard metrics (Exact Match, token F1) were effectively useless (0.0–0.8% and 0.016–0.07 respectively), and an NLI-based judge produced a flat ~91% hallucination rate across all models, indicating the evaluator itself was the bottleneck.
- The core thesis: hallucination rate without answer rate is nearly meaningless, and prompt engineering results do not transfer across models even within the same size class.
Why It Matters
This work exposes critical flaws in how hallucination is measured and reported in the LLM community, showing that benchmark numbers can be misleading when they conflate model refusal with accuracy. For AI practitioners, it demonstrates that evaluation methodology — from prompt design to automated judging — can dominate results more than the models themselves, making cross-model comparisons unreliable without standardized reporting of both answer rates and hallucination rates.
Technical Details
- Models & Setup: Four open-weight models (Phi-4-mini 3.8B, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct) tested on 500 HotpotQA hard-split questions and 120 TruthfulQA questions, using three prompts: plain, abstain ("if unsure, say I don't know"), and reasoning ("use step-by-step reasoning").
- Labeling Pipeline: Used a frozen DeBERTa-large-MNLI NLI judge with gold supporting facts as premises and model answers as hypotheses, plus heuristics to classify failures into seven types. The author found that entailment was only 8.9% of verdicts, with neutral at 48.7% and contradiction at 42.3%, leading to an uninformative ~91% hallucination rate under the original classification rule.
- Decoding: Fixed at temperature=0.2, top_p=0.9, 256 max tokens, with do_sample=True and no seed set — meaning results are from a single non-reproducible sampled run.
- Key Metrics Revealed: Exact Match ranged 0.0–0.8% and token F1 ranged 0.016–0.07 across all models and prompts, demonstrating that span-based metrics are inadequate for instruction-tuned models that generate paragraphs rather than short answers.
- Correction Impact: A labeling bug correction shifted Mistral's reasoning prompt improvement from −17.0pp to approximately −25pp, making it the biggest mover, though all directional findings remained stable.
Industry Insight
- Report answer rates alongside hallucination rates: Any benchmark claiming to measure hallucination must also report what fraction of questions the model attempted. A low hallucination rate driven by mass refusal is not a meaningful accuracy improvement and can mislead deployment decisions.
- Prompt engineering is model-specific: Identical prompts can produce opposite effects across models of similar size. Teams should not assume that a chain-of-thought or abstention prompt optimized for one model will generalize, even within the same model family — empirical validation per model is essential.
- Automated NLI judges need calibration: The DeBERTa-based judge produced near-uniform results across models, suggesting that off-the-shelf NLI classifiers may lack the granularity needed for hallucination detection. Practitioners should validate their evaluation pipelines against human-labeled subsets before trusting automated scores at scale.
Disclaimer: The above content is generated by AI and is for reference only.