The Granularity Paradox: How Temporal Disaggregation Inflates In-Sample Fit and Compounds Out-of-Sample Error
The "Granularity Paradox" reveals that finer temporal disaggregation improves in-sample fit but degrades out-of-sample accuracy due to recursive error compounding over longer horizons. Standard pointwise metrics like RMSE and MAE systematically mask cumulative error propagation, leading to misleading assessments of model adequacy. Model performance varies non-monotonically with granularity; for instance, Holt-Winters fails catastrophically at daily frequencies, while LSTM exhibits a U-shaped err
Analysis
TL;DR
- The "Granularity Paradox" reveals that finer temporal disaggregation improves in-sample fit but degrades out-of-sample accuracy due to recursive error compounding over longer horizons.
- Standard pointwise metrics like RMSE and MAE systematically mask cumulative error propagation, leading to misleading assessments of model adequacy.
- Model performance varies non-monotonically with granularity; for instance, Holt-Winters fails catastrophically at daily frequencies, while LSTM exhibits a U-shaped error curve.
- Linear Regression demonstrates stability across all granularities, indicating the paradox is driven by recursive feedback topology rather than model complexity.
- A new "consensus-dissensus" diagnostic is introduced to compare pointwise metrics against cumulative Total Percentage Forecast Error (TPFE) to identify masking errors.
Why It Matters
This research challenges conventional time-series evaluation practices by demonstrating that standard metrics can hide severe long-horizon failures, particularly in recursive forecasting models. For AI practitioners and data scientists, it highlights the critical need for goal-dependent cumulative metrics when deploying models for multi-step ahead predictions, ensuring that in-sample improvements do not come at the cost of out-of-sample reliability.
Technical Details
- Core Phenomenon: The study formalizes the trade-off where increasing dataset size (N) via disaggregation inflates in-sample diagnostics but compounds out-of-sample error (H) through recursive feedback loops.
- Experimental Setup: Benchmarked 10 models (naïve, statistical, ML, DL) across six temporal granularities (Annual to Daily) using a 13-year public procurement dataset.
- Key Findings:
- Holt-Winters showed extreme degradation at Daily granularity (Test R-squared: -151, TPFE: 425.85%).
- LSTM displayed a U-shaped error trajectory, worsening at Bi-Weekly (35.94%) before recovering at Daily (TPFE: 4.35%, R-squared: 0.66).
- Linear Regression remained stable (TPFE: 16.3-17.0%), isolating the issue to recursive architectures.
- Diagnostic Tool: Introduced the consensus-dissensus diagnostic to detect discrepancies between standard pointwise metrics and cumulative TPFE, flagging models with masked systematic error propagation.
Industry Insight
- Metric Selection: Organizations relying on RMSE or MAE for model selection in supply chain or financial forecasting should adopt cumulative error metrics (like TPFE) to evaluate long-horizon performance accurately.
- Model Architecture Choice: When high-frequency forecasting is required, non-recursive models (like Linear Regression) or specific deep learning architectures (like LSTM with careful horizon tuning) may outperform traditional statistical methods prone to error compounding.
- Evaluation Protocol: Implement a dual-metric evaluation strategy that contrasts pointwise accuracy with cumulative forecast error to prevent deploying models that appear robust in-sample but fail in real-world, multi-step applications.
Disclaimer: The above content is generated by AI and is for reference only.