Institution-Specific LLM Prompting Recovers PHI That De-identification Systems and Their Gold Standards Both Miss
LLMs with in-context learning (ICL) outperform purpose-built de-identification systems (F1=0.918 vs TiDE 0.779) on institution-specific PHI detection in pediatric oncology notes A three-stage prompting strategy—HIPAA baseline, adding missed institutional categories, then discouraging over-redaction—recovers 79% of previously missed PHI categories and restores precision Multi-agent and ensemble architectures did not outperform calibrated single-pass prompting (F1 0.906–0.907), demonstrating that
Analysis
TL;DR
- LLMs with in-context learning (ICL) outperform purpose-built de-identification systems (F1=0.918 vs TiDE 0.779) on institution-specific PHI detection in pediatric oncology notes
- A three-stage prompting strategy—HIPAA baseline, adding missed institutional categories, then discouraging over-redaction—recovers 79% of previously missed PHI categories and restores precision
- Multi-agent and ensemble architectures did not outperform calibrated single-pass prompting (F1 0.906–0.907), demonstrating that prompt engineering alone is sufficient
- LLM outputs surfaced 414 candidate annotation gaps; re-annotation confirmed 227 additional PHI spans, enabling the final prompt to reach recall=0.981 (F1=0.907±0.002)
- The study positions LLMs as a legitimate, adaptable alternative to traditional de-identification systems, with institution-specific prompt development as the primary adaptation strategy
Why It Matters
This research addresses a critical gap in healthcare AI: existing de-identification systems systematically miss institutionally situated PHI (hospital abbreviations, building names, internal codes) that is locally determined but essential for HIPAA compliance. For AI practitioners working in clinical settings, it demonstrates that well-calibrated ICL can simultaneously resolve the institutional PHI gap and the precision-recall trade-off in a single LLM call, while also providing a mechanism to audit and improve reference annotation standards.
Technical Details
- Dataset: 100 annotated pediatric oncology notes from Texas Children's Hospital containing 5,322 PHI spans
- Benchmarks: Eight LLMs evaluated against two purpose-built systems (Stanford TiDE, OpenMed PII) and two pattern-based baselines
- Three-stage prompting: (1) HIPAA-aligned baseline prompt, (2) baseline plus institutional PHI categories the model missed, (3) prompt 2 plus instructions to avoid over-redacting clinical content
- Evaluation: 14 multi-agent and ensemble configurations compared against the best single prompt, with recall as the primary safety metric
- Key results: Best LLM F1=0.918±0.001 vs. TiDE 0.779; naming missed categories recovered 48/61 (79%); final prompt achieved recall=0.981 and F1=0.907±0.002 after re-annotation of 227 confirmed PHI spans
Industry Insight
- Healthcare organizations should prioritize institution-specific prompt development over building custom de-identification pipelines, as LLM-based approaches offer superior adaptability and performance for locally situated PHI
- The "audit the reference standard" capability of LLMs—surfacing 414 candidate annotation gaps—represents a novel secondary benefit: de-identification systems can simultaneously improve their own training data quality
- Despite higher computational costs than traditional methods, the single-call-per-note efficiency of calibrated ICL and the elimination of multi-agent complexity make LLM-based de-identification operationally viable for clinical deployment
Disclaimer: The above content is generated by AI and is for reference only.