CMNIE: An Information Extraction Benchmark for Chinese Military News
CMNIE introduces a joint information extraction benchmark for Chinese military news, annotating events, entities, and relations under a unified schema The dataset contains 13,000 manually annotated instances covering 7 event types, 10 argument roles, 7 entity types, and 8 relation types Existing resources lacked support for joint extraction modeling across events, arguments, entities, and relations in the military domain Zero-shot LLMs identify relevant semantic units but struggle with exact spa
Analysis
TL;DR
- CMNIE introduces a joint information extraction benchmark for Chinese military news, annotating events, entities, and relations under a unified schema
- The dataset contains 13,000 manually annotated instances covering 7 event types, 10 argument roles, 7 entity types, and 8 relation types
- Existing resources lacked support for joint extraction modeling across events, arguments, entities, and relations in the military domain
- Zero-shot LLMs identify relevant semantic units but struggle with exact span boundary matching against gold annotations
- The benchmark highlights persistent challenges in relation extraction and schema adherence for specialized Chinese news text
Why It Matters
This benchmark addresses a critical gap in domain-specific NLP by providing the first unified schema for joint information extraction in Chinese military news, a domain with high practical value for intelligence analysis and knowledge base construction. It establishes a standardized evaluation framework that reveals the limitations of current LLM-based approaches in exact span matching and relation extraction, guiding future research toward more precise structured extraction methods.
Technical Details
- CMNIE jointly annotates four extraction targets—event triggers, event arguments, named entities, and entity relations—under a single domain-specific schema, moving beyond prior document-level event-only annotations
- The dataset comprises 13,000 instances sourced from public Chinese military news, manually annotated by domain experts
- Evaluation covers supervised IE models, zero-shot large language models, and fine-tuned LLM-based extraction pipelines on a shared test set
- Key schema definitions include 7 event types, 10 argument roles, 7 entity types, and 8 relation types
- Experimental results demonstrate that while LLMs capture semantic relevance, they consistently underperform on exact span matching and relation extraction tasks
Industry Insight
- Domain-specific benchmarks like CMNIE highlight that general-purpose LLMs still require targeted fine-tuning and span-level supervision to achieve production-grade extraction accuracy in specialized fields
- The gap between semantic identification and exact boundary matching suggests future work should prioritize span-aware training objectives and boundary refinement mechanisms
- Military and government domains represent high-value niches where structured extraction benchmarks can drive both academic progress and practical deployment of IE systems
Disclaimer: The above content is generated by AI and is for reference only.