When to Retrain: An Empirical Study of Retraining Policies for Streaming ML Under Concept Drift, Budget, and Latency Constraints
The most consequential design decision in streaming ML systems is whether the deployed model supports incremental per-sample updates, not which retraining policy is chosen Without incremental learning, retraining policy choice creates 15-55 percentage point differences in post-drift accuracy across abrupt, gradual, and recurring drift regimes Simple periodic retraining significantly outperforms reactive policies (error-threshold and ADWIN-based drift detection) under abrupt and gradual drift Rea
Analysis
TL;DR
- The most consequential design decision in streaming ML systems is whether the deployed model supports incremental per-sample updates, not which retraining policy is chosen
- Without incremental learning, retraining policy choice creates 15-55 percentage point differences in post-drift accuracy across abrupt, gradual, and recurring drift regimes
- Simple periodic retraining significantly outperforms reactive policies (error-threshold and ADWIN-based drift detection) under abrupt and gradual drift
- Reactive policies retain an advantage only under recurring drift conditions
- A latency-budget queueing interaction was discovered that silently halves effective retraining budgets
Why It Matters
This study provides much-needed empirical guidance for ML engineers who must make retraining decisions under real-world constraints of finite budgets and deployment latency. It challenges the common assumption that sophisticated reactive retraining policies are inherently superior, showing that for many practical scenarios, simple periodic retraining combined with incremental learning is the dominant factor in maintaining model performance.
Technical Details
- Evaluated three retraining policies—periodic retraining, error-threshold triggering, and ADWIN-based statistical drift detection—against a no-retrain baseline in a unified system model that explicitly accounts for retraining budgets and training-plus-deployment latency
- Conducted 3,933 experiment runs across three drift regimes (abrupt, gradual, recurring), three budget levels, up to five latency levels, three datasets, and two learning modes (with and without incremental updates)
- Used a linear online learner with immediate labels as the base model, enabling clean isolation of the retraining policy effect from learning algorithm complexity
- Documented systematic failure modes of reactive policies and identified a latency-budget queueing interaction that effectively halves retraining budgets without explicit awareness
- Released the full simulator, dataset pipelines, and per-run artifacts for reproducibility
Industry Insight
- Organizations should prioritize enabling incremental per-sample model updates in their ML infrastructure before investing in complex retraining orchestration, as this single decision dwarfs the impact of retraining policy choice
- For systems without incremental learning, periodic retraining should be the default strategy unless the data exhibits clearly recurring drift patterns, where reactive policies may justify their added complexity
- Teams should audit their retraining pipelines for the latency-budget queueing interaction, as undetected queueing delays can silently consume up to 50% of allocated retraining budgets, leading to underfunded model refresh cycles
Disclaimer: The above content is generated by AI and is for reference only.