Extracting Dataset Mentions in Forced Displacement and FCV Documents: A Weakly Supervised Framework with LLM-Based Label Refinement
A weakly supervised framework for extracting dataset mentions in forced displacement and FCV documents without requiring large manually labeled training corpora Uses a two-stage pipeline: a lightweight model generates candidate dataset mentions from unlabeled domain documents, then a frontier LLM validates/rejects candidates and corrects extraction boundaries Synthetic and contrastive examples supplement LLM-refined annotations for fine-tuning the lightweight model Achieves 74.1% precision and 7
Analysis
TL;DR
- A weakly supervised framework for extracting dataset mentions in forced displacement and FCV documents without requiring large manually labeled training corpora
- Uses a two-stage pipeline: a lightweight model generates candidate dataset mentions from unlabeled domain documents, then a frontier LLM validates/rejects candidates and corrects extraction boundaries
- Synthetic and contrastive examples supplement LLM-refined annotations for fine-tuning the lightweight model
- Achieves 74.1% precision and 70.5% recall at mention level, with 89.5% precision among passages containing dataset references
- Demonstrates a practical approach for domain-specific supervision construction when labeled data are scarce
Why It Matters
This work addresses a critical bottleneck in humanitarian and development research: systematically tracing where datasets are referenced across dispersed, unstructured documents. For AI practitioners working in low-resource domains, it demonstrates a scalable weakly supervised paradigm that leverages LLMs as label refinements rather than direct annotators, reducing manual labeling costs while maintaining strong performance. The framework is directly applicable to any domain where domain-specific NLP tasks lack sufficient labeled data.
Technical Details
- Two-stage extraction pipeline: A lightweight model (pre-trained on general research literature) generates candidate dataset mentions from unlabeled forced displacement/FCV documents, followed by a frontier LLM that reviews candidates in context to validate, reject, or correct extraction boundaries
- Label refinement strategy: The LLM acts as an annotator corrector rather than a zero-shot extractor, refining candidate mentions with contextual understanding before annotations are used for fine-tuning
- Data augmentation: Targeted synthetic and contrastive examples are generated from LLM-refined annotations to supplement the training signal and improve model generalization
- Evaluation benchmark: Independent gold-standard benchmark of 1,706 text passages spanning research, humanitarian, and operational documents
- Performance metrics: 74.1% precision / 70.5% recall at mention level; 89.5% precision among positive passages; 88.2% accuracy and 88.6% specificity at passage level for distinguishing referenced vs. non-referenced passages
Industry Insight
- The LLM-as-refiner paradigm offers a cost-effective alternative to full manual annotation for domain adaptation, particularly valuable in humanitarian and policy domains where expert annotators are scarce and expensive
- This approach generalizes beyond the displacement domain: any organization working with specialized corpora (legal, medical, regulatory) can adopt this weakly supervised pipeline to build domain-specific NLP systems without massive labeling investments
- The strong passage-level specificity (88.6%) suggests the framework is well-suited for large-scale screening and prioritization tasks, where identifying relevant documents is the primary bottleneck before deeper analysis
Disclaimer: The above content is generated by AI and is for reference only.