Leveraging Large Language Models for Systematic Literature Review of Disease Spread Models
An LLM pipeline was developed to extract model-relevant information from 536 peer-reviewed agent-based modeling papers on disease spread GPT-4.1 achieved ~77.95% paper-level accuracy and GPT-5.0 achieved ~81.67% paper-level accuracy when compared against human-conducted SLR Field-level accuracy varied widely (32.40% to 100.00%), with complex or subjective fields performing less reliably LLM inter-agreement was identified as a quality indicator: low agreement signals hallucinations, while high ag
Analysis
TL;DR
- An LLM pipeline was developed to extract model-relevant information from 536 peer-reviewed agent-based modeling papers on disease spread
- GPT-4.1 achieved ~77.95% paper-level accuracy and GPT-5.0 achieved ~81.67% paper-level accuracy when compared against human-conducted SLR
- Field-level accuracy varied widely (32.40% to 100.00%), with complex or subjective fields performing less reliably
- LLM inter-agreement was identified as a quality indicator: low agreement signals hallucinations, while high agreement with low accuracy may indicate noise in the human reference dataset
- The study provides practical insights into prompt development and outlines both the potential and limitations of automating full-scale systematic literature reviews in modeling and simulation domains
Why It Matters
This work directly addresses a critical bottleneck in AI and computational research: the time-intensive nature of systematic literature reviews. As LLMs become more capable, demonstrating their reliability (and limitations) for structured information extraction from scientific literature helps practitioners decide when to trust automated pipelines versus human curation. The finding that LLM agreement can serve as a proxy for output quality offers a practical, deployable heuristic for researchers building review automation systems.
Technical Details
- Dataset: 536 peer-reviewed agent-based modeling papers focused on disease spread, with ground truth established by a human-conducted systematic literature review
- Models evaluated: GPT-4.1 and GPT-5.0, tested on an LLM pipeline designed for structured information extraction from scientific papers
- Metrics: Paper-level accuracy (77.95% for GPT-4.1, 81.67% for GPT-5.0) and field-level accuracy (range: 32.40%–100.00%), with performance varying by field complexity and subjectivity
- Key methodological insight: Cross-model agreement was used as a diagnostic signal — low LLM-LLM agreement correlates with hallucinations, while high agreement paired with low accuracy suggests potential errors or noise in the human-annotated reference dataset rather than model failure
- Domain: Applied to agent-based modeling in epidemiology, within the broader context of systematic literature review automation
Industry Insight
- LLMs are approaching human-level reliability for structured literature extraction on well-defined, objective fields, but remain significantly less trustworthy on subjective or complex classification tasks — practitioners should implement human-in-the-loop review for high-stakes fields
- The cross-model agreement diagnostic is a deployable quality-control mechanism: teams building automated review pipelines can use inter-model disagreement as an early warning signal for hallucination-prone outputs without requiring full human re-review
- Prompt engineering and field-level calibration remain essential; the wide accuracy range (32–100%) indicates that one-size-fits-all LLM pipelines will underperform, and domain-specific prompt optimization is necessary before scaling to full literature review automation
Disclaimer: The above content is generated by AI and is for reference only.