Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text
The paper addresses fine-grained inconsistency classification in financial disclosure text, identifying different types of conflicts (numerical, temporal, referential, factual, and normative). A benchmark dataset (SBID-FD) with 5,940 instances and 11 inconsistency labels is used to evaluate various models. Fine-tuned encoders achieve comparable performance to larger language models when provided with gold evidence spans, highlighting the importance of evidence localization. Referential inconsist
Analysis
TL;DR
- The paper addresses fine-grained inconsistency classification in financial disclosure text, identifying different types of conflicts (numerical, temporal, referential, factual, and normative).
- A benchmark dataset (SBID-FD) with 5,940 instances and 11 inconsistency labels is used to evaluate various models.
- Fine-tuned encoders achieve comparable performance to larger language models when provided with gold evidence spans, highlighting the importance of evidence localization.
- Referential inconsistencies are particularly sensitive to localization quality, while Factual and Logical inconsistencies remain challenging even with correct evidence.
- Progress requires improvements in both evidence extraction and reasoning over closely related inconsistency categories.
Why It Matters
This research is relevant for AI practitioners and researchers working on financial document analysis, as it highlights the challenges in detecting and classifying subtle inconsistencies that can have significant implications for compliance and risk management. The findings suggest that while large language models show promise, smaller, specialized models can be highly effective when paired with robust evidence extraction methods.
Technical Details
- Dataset: SBID-FD, a synthetic financial-disclosure benchmark with 5,940 instances and 11 inconsistency labels, including paired reference evidence spans.
- Models Compared: Frozen embedding classifiers, fine-tuned encoders, evidence-augmented classifiers, prompted large language models (e.g., GPT-5.4), and LoRA-adapted generative models (e.g., Qwen3.5-9B).
- Performance: A fine-tuned 300M encoder achieves 61.9% accuracy, compared to 61.5% for a LoRA-adapted Qwen3.5-9B model and 61.3% for GPT-5.4. Providing gold evidence spans improves the fine-tuned encoder's accuracy to 65.3%.
- Analysis: Class-level analyses reveal that Referential inconsistencies are especially sensitive to localization quality, while Factual and Logical inconsistencies remain difficult even with correct evidence. Oracle, distractor, and per-class analyses separate localization errors from residual type-discrimination errors.
Industry Insight
- Efficiency vs. Scale: Compact supervised encoders can achieve performance comparable to much larger models when equipped with high-quality evidence, suggesting potential cost savings in deployment.
- Evidence Localization: Improving evidence extraction mechanisms could significantly enhance the accuracy of inconsistency detection systems, particularly for Referential inconsistencies.
- Reasoning Challenges: Developing better reasoning capabilities for distinguishing between closely related inconsistency types (e.g., Factual vs. Logical) remains a critical area for future research and development.
Disclaimer: The above content is generated by AI and is for reference only.