Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering
Hybrid retrieval consistently outperforms dense and sparse methods in recall and ranking quality for public health QA, though chunk length and topic significantly interact with performance. Providing retrieved context substantially boosts multiple-choice accuracy, allowing smaller open-weight models to match or exceed the performance of larger models without retrieval. A new rubric-based LLM-as-a-judge was validated against human annotations, showing strong agreement on faithfulness and complete
Analysis
TL;DR
- Hybrid retrieval consistently outperforms dense and sparse methods in recall and ranking quality for public health QA, though chunk length and topic significantly interact with performance.
- Providing retrieved context substantially boosts multiple-choice accuracy, allowing smaller open-weight models to match or exceed the performance of larger models without retrieval.
- A new rubric-based LLM-as-a-judge was validated against human annotations, showing strong agreement on faithfulness and completeness but lower reliability for factual consistency and clarity.
- The study extends PubHealthBench to a retrieval-augmented setting, offering practical guidance for building RAG systems grounded in evolving official guidance to mitigate hallucinations.
Why It Matters
This research addresses critical barriers to deploying LLMs in high-stakes public health domains, specifically hallucination and outdated information. By demonstrating that retrieval quality is the primary lever for reliability, it provides actionable strategies for practitioners to enhance model trustworthiness without relying solely on scaling up model size.
Technical Details
- Benchmark Extension: Extended PubHealthBench (7,929 questions from UK Government public health guidance) into a retrieval-augmented setting to evaluate end-to-end RAG performance.
- Retrieval Comparison: Systematically compared dense, sparse, and hybrid retrieval across multiple embedding models and corpus variants, identifying hybrid retrieval as superior for ranking quality.
- Model Performance Analysis: Showed that smaller open-weight models with high-quality retrieval can outperform larger closed-source models without retrieval, highlighting the importance of context selection.
- Evaluation Methodology: Introduced a rubric-based LLM-as-a-judge assessing faithfulness, completeness, clarity, and factual consistency, validated against dual human annotations to identify strengths and weaknesses in automated evaluation.
Industry Insight
- Prioritize hybrid retrieval configurations and rigorous context selection over mere model scaling when building RAG systems for regulated industries like healthcare.
- Exercise caution when using LLM-as-a-judge for evaluating factual consistency and clarity; supplement automated evaluations with targeted human review for these specific dimensions.
- Leverage open-weight models combined with robust retrieval pipelines to achieve cost-effective, high-performance solutions that can be easily updated with new official guidance.
Disclaimer: The above content is generated by AI and is for reference only.