Reliable Financial Named Entity Recognition under Domain Shift
Financial NER models trained on one textual register (e.g., SEC filings) degrade significantly when deployed on out-of-domain text like social media, and standard F1 scores fail to indicate which predictions remain safe to automate Confidence rankings shift under distribution shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, while entity-span probability and self-consistency prove more robust and better calibrated without post-hoc fitting S
Analysis
TL;DR
- Financial NER models trained on one textual register (e.g., SEC filings) degrade significantly when deployed on out-of-domain text like social media, and standard F1 scores fail to indicate which predictions remain safe to automate
- Confidence rankings shift under distribution shift: whole-output probability is the strongest in-domain error detector but deteriorates out of domain, while entity-span probability and self-consistency prove more robust and better calibrated without post-hoc fitting
- Selective prediction via abstention reduces sentence error from 34.3% to below 2% on the highest-confidence 40% of in-domain inputs and remains useful on financial news, but recovers no meaningfully large clean subset under extreme social-media shift
- The study evaluates BERT taggers and LoRA-tuned Qwen2.5-0.5B/1.5B models across a three-tier stress test (SEC filings → financial news → general social media) using five inference-time confidence signals, three training seeds, and bootstrap intervals
- Results motivate a staged deployment strategy that detects severe distribution shift upstream before applying prediction-level confidence gating
Why It Matters
This research directly addresses a critical gap in production AI systems: standard accuracy metrics like F1 scores are insufficient for determining when predictions are safe to automate under domain shift, which is a common real-world scenario in financial NLP pipelines. The findings provide actionable guidance for practitioners building financial information extraction systems, showing that confidence estimation strategies must be carefully chosen based on the expected deployment domain and that upstream domain detection is essential before relying on selective prediction.
Technical Details
- Models evaluated: BERT-based NER tagger and LoRA-tuned Qwen2.5-0.5B and Qwen2.5-1.5B language models, tested across three training seeds with bootstrap confidence intervals
- Three-tier domain stress test: SEC filings (in-domain) → financial news (mild shift) → general-topic social media (extreme out-of-domain), creating a progressive distribution shift gradient
- Five inference-time confidence signals: whole-output probability, entity-span probability, self-consistency, and two additional signals; self-consistency emerged as better calibrated without requiring post-hoc fitting
- Selective prediction results: Abstention on highest-confidence 40% of in-domain inputs reduced sentence error from 34.3% to below 2%; performance on financial news remained useful but collapsed under extreme social-media shift with no recoverable clean subset
- Key finding on confidence ranking instability: The relative effectiveness of confidence signals changes under distribution shift, with whole-output probability losing its in-domain advantage while entity-span probability and self-consistency maintain robustness
Industry Insight
- Organizations deploying financial NER systems should implement upstream domain classification or drift detection as a gate before applying confidence-based selective prediction, since no single confidence signal remains reliable across all domains
- Self-consistency-based confidence estimation is a strong candidate for production financial NLP pipelines due to its calibration properties and robustness to domain shift without requiring expensive post-hoc calibration procedures
- The dramatic performance collapse on social media suggests that financial AI systems should explicitly define their operational domain scope and avoid overgeneralizing extraction capabilities to user-generated content without dedicated domain adaptation or separate model pipelines
Disclaimer: The above content is generated by AI and is for reference only.