Schema-Constrained Document-Level Event Argument Extraction with Lightweight LLM Fine-Tuning
The study demonstrates that mid-sized open LLMs, specifically Phi-4 (14B), can effectively handle document-level Event Argument Extraction (EAE) under strict schema constraints. A combined approach of role-set injection in prompts, parameter-efficient LoRA fine-tuning, and deterministic decoding with JSON validation achieves state-of-the-art results on the MAVEN-ARG benchmark. Fine-tuned open models outperform previous GPT baselines across mention, entity-coreference, and event-coreference evalu
Analysis
TL;DR
- The study demonstrates that mid-sized open LLMs, specifically Phi-4 (14B), can effectively handle document-level Event Argument Extraction (EAE) under strict schema constraints.
- A combined approach of role-set injection in prompts, parameter-efficient LoRA fine-tuning, and deterministic decoding with JSON validation achieves state-of-the-art results on the MAVEN-ARG benchmark.
- Fine-tuned open models outperform previous GPT baselines across mention, entity-coreference, and event-coreference evaluations, with the best model reaching 42.39% F1 at the event-coreference level.
- The methodology addresses key challenges in EAE such as long-range dependencies, cross-sentence context, and boundary errors through structured inference and post-processing.
Why It Matters
This research is significant for AI practitioners seeking cost-effective, high-performance alternatives to proprietary large models for complex information extraction tasks. By showing that open-weight models can surpass GPT baselines on rigorous benchmarks like MAVEN-ARG, it validates the viability of lightweight, fine-tuned solutions for production environments requiring strict schema compliance and reproducibility.
Technical Details
- Task: Document-level Event Argument Extraction (EAE) on the MAVEN-ARG dataset, focusing on identifying argument spans and assigning schema-defined roles.
- Methodology: Combines three key components: (i) role-set injection in prompts to ensure schema compliance, (ii) parameter-efficient supervised fine-tuning using LoRA with a JSON-only interface, and (iii) deterministic decoding with post-processing steps including JSON validation, invalid role filtering, de-duplication, and span alignment.
- Model: Utilizes mid-sized open LLMs, with Phi-4 (14B) identified as the best performing model.
- Results: Achieves 42.39% F1 at the event-coreference level, outperforming previously reported GPT baselines across all evaluation metrics (mention, entity-coreference, event-coreference).
- Reproducibility: Code to reproduce experiments is publicly available.
Industry Insight
- Organizations should consider fine-tuning mid-sized open models like Phi-4 for specialized extraction tasks rather than relying solely on API-based proprietary models, potentially reducing costs while improving performance on constrained schemas.
- The integration of deterministic decoding and robust post-processing pipelines is critical for ensuring reliability in LLM outputs, particularly when strict structural formats (like JSON) and schema constraints are required.
- This work highlights the growing capability of open-source models to compete with or exceed closed-source giants in specific NLP benchmarks, encouraging further investment in open-model ecosystems and efficient fine-tuning techniques.
Disclaimer: The above content is generated by AI and is for reference only.