Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal
Random train-test splits inflate Matthews Correlation Coefficient (MCC) by 1.1× to 6.5× compared to chronological splits in financial-news direction prediction, with the inflation scaling with model capacity and feature richness End-to-end fine-tuning of FinBERT amplifies rather than closes the leakage gap, yielding a size-matched ratio of 1.75× between random and chronological splits M&A articles are the only event category showing a statistically significant positive signal under near-temporal
Analysis
TL;DR
- Random train-test splits inflate Matthews Correlation Coefficient (MCC) by 1.1× to 6.5× compared to chronological splits in financial-news direction prediction, with the inflation scaling with model capacity and feature richness
- End-to-end fine-tuning of FinBERT amplifies rather than closes the leakage gap, yielding a size-matched ratio of 1.75× between random and chronological splits
- M&A articles are the only event category showing a statistically significant positive signal under near-temporal chronological evaluation (TF-IDF MCC = 0.138 train-only vs 0.068 under train∪val refit; p < 10⁻³), but this signal is localized to 2024-2025 European-tilted semantics and does not transfer to FNSPID's 2009-2020 U.S. corpus
- Three independent role labellers converge on acquirer-tagged articles as the signal locus, suggesting the leakage is concentrated in specific entity roles rather than being uniformly distributed
- Chronological splitting acts as a "characteristics-purging" mechanism for financial NLP, stripping predictable stale signal and leaving only a small, event-localized, lexically shallow residual
Why It Matters
This audit exposes a critical validity problem in financial NLP benchmarking: many reported performance gains may reflect temporal leakage rather than genuine predictive ability. For practitioners building production systems, the findings imply that models validated on randomly split financial news data may severely overestimate real-world performance, especially in time-sensitive trading or risk applications.
Technical Details
- Corpus and scope: 49,799 financial news articles evaluated across 16 feature-model combinations, including classical (TF-IDF), embedding-based (MiniLM, FinBERT, RoBERTa-large, DeBERTa-v3-large), and LLM-based approaches (zero/few-shot and LoRA probes on Llama-3 and Qwen2.5)
- Leakage quantification: Random splits produce MCC inflation ranging from 1.1× to 6.5× relative to chronological splits, with the magnitude tracking both model capacity and feature richness; FinBERT fine-tuning re-amplifies the gap rather than mitigating it (size-matched ratio 1.75×)
- M&A signal localization: Under near-temporal chronological evaluation, M&A is the sole audited category with a significant locked-test signal; permutation testing (10,000 iterations) confirms p < 10⁻³, but cross-corpus validation against FNSPID's 2009-2020 U.S. data shows zero transferability
- Role-level analysis: Three independent role labellers independently identify acquirer-tagged articles as the primary signal locus, though the authors characterize this as a power-limited qualitative convergence rather than a formally hypothesis-tested asymmetry
Industry Insight
- Benchmark designers and reviewers should mandate temporal leakage audits as a standard disclosure requirement for any financial-NLP paper, analogous to how characteristics-purging is standard in asset pricing research
- Practitioners should treat chronologically split performance as the ground-truth baseline for deployment readiness; random-split results should be flagged as upper-bound estimates with potentially severe overfitting to temporal artifacts
- The non-transferability of the M&A signal across corpora and regions suggests that apparent predictive signals in financial NLP may be highly context-specific, warning against overgeneralizing findings from one market or time period to another
Disclaimer: The above content is generated by AI and is for reference only.