Standalone LLM and a Pre-specified Agentic Pipeline for Explaining ICU Mortality Predictions: a Feasibility Study on the eICU Demo Dataset
Standalone LLM outperforms agentic pipeline in SHAP alignment (Jaccard 0.171 vs 0.077) and direction consistency (92.9% vs 78.6%) when explaining ICU mortality predictions Agentic pipeline demonstrates superior guideline grounding (0.762 vs 0.143), value specificity (0.236 vs 0.143), and slightly higher plausibility (0.700 vs 0.671) Standalone LLM produced 1 explanation with explicit outcome leakage on a 38-case subset, while the four-step agentic pipeline produced none XGBoost baseline achieved
Analysis
TL;DR
- Standalone LLM outperforms agentic pipeline in SHAP alignment (Jaccard 0.171 vs 0.077) and direction consistency (92.9% vs 78.6%) when explaining ICU mortality predictions
- Agentic pipeline demonstrates superior guideline grounding (0.762 vs 0.143), value specificity (0.236 vs 0.143), and slightly higher plausibility (0.700 vs 0.671)
- Standalone LLM produced 1 explanation with explicit outcome leakage on a 38-case subset, while the four-step agentic pipeline produced none
- XGBoost baseline achieved AUROC of 0.855 and AUPRC of 0.332 on the eICU Demo dataset (2,353 ICU stays, 8.1% mortality)
- Agentic decomposition improves safety-relevant grounding but should be paired with attribution-based checks for high-stakes clinical risk explanation
Why It Matters
This study directly addresses a critical gap in clinical AI deployment: the disconnect between accurate predictive models and the narrative explanations clinicians need at the bedside. By comparing standalone LLMs against multi-step agentic pipelines, it provides actionable evidence for healthcare AI practitioners deciding between simplicity and structured reasoning in high-stakes medical explanation systems.
Technical Details
- Dataset: Retained local eICU Demo artifact set comprising 2,353 ICU stays with an 8.1% mortality rate; a stratified 38-case explanation subset was used for LLM evaluation, with 14 cases overlapping a SHAP review subset
- Baseline model: XGBoost classifier achieving AUROC 0.855 (95% CI 0.796–0.906) and AUPRC 0.332 (95% CI 0.217–0.494) for ICU mortality prediction
- Standalone LLM: Direct explanation generation without intermediate reasoning steps; evaluated on SHAP alignment, direction consistency, and outcome leakage
- Four-step agentic pipeline: Decomposed pipeline separating data interpretation, guideline checking, and final explanation generation; evaluated on guideline grounding, value specificity, and plausibility metrics
- Evaluation metrics: Jaccard similarity for feature alignment with SHAP, direction consistency percentage, guideline grounding score, value specificity, plausibility rating, and outcome leakage detection
Industry Insight
- Healthcare AI systems should consider hybrid approaches that combine the attribution fidelity of standalone LLMs with the safety grounding of agentic pipelines, rather than treating them as mutually exclusive architectures
- Outcome leakage remains a critical validation concern for standalone LLMs in clinical settings; agentic decomposition with explicit guideline-checking steps can serve as an effective safeguard
- The modest plausibility difference (0.700 vs 0.671) suggests that pipeline complexity does not automatically translate to meaningfully better clinical explanations, warranting cost-benefit analysis before deploying multi-step agents in production healthcare environments
Disclaimer: The above content is generated by AI and is for reference only.