HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning
HERO is a new benchmark library for Federated Continual Learning (FCL) designed to decouple task splits, client data distributions, and task sequences to enable fairer comparisons. The library introduces HERO-Core, which uses parameters alpha ($\alpha$) for client data skew and rho ($\rho$) for task-order mismatch to systematically control heterogeneity. Evaluations on CIFAR-100 and TinyImageNet reveal that average accuracy metrics can mask poor performance among the weakest clients (bottom-10%)
Analysis
TL;DR
- HERO is a new benchmark library for Federated Continual Learning (FCL) designed to decouple task splits, client data distributions, and task sequences to enable fairer comparisons.
- The library introduces HERO-Core, which uses parameters alpha ($\alpha$) for client data skew and rho ($\rho$) for task-order mismatch to systematically control heterogeneity.
- Evaluations on CIFAR-100 and TinyImageNet reveal that average accuracy metrics can mask poor performance among the weakest clients (bottom-10%).
- The framework demonstrates portability beyond image data through a graph-based Domain-IL case study on OGB-MolPCBA, showing consistent exposure of domain-shift difficulties.
- HERO provides open-source benchmark streams, configurations, and reporting scripts to standardize reproducible and setting-aware FCL research.
Why It Matters
This work addresses a critical reproducibility crisis in Federated Continual Learning, where inconsistent evaluation protocols make it difficult to compare existing methods. By providing a standardized, modular benchmarking tool, HERO enables researchers to isolate specific sources of heterogeneity, leading to more robust algorithm development and clearer insights into model fairness and generalization in distributed settings.
Technical Details
- Decoupled Benchmark Construction: HERO separates three traditionally coupled variables: task split, client data split, and client task sequence, allowing independent manipulation of these factors.
- HERO-Core Parameters: The core benchmark utilizes $\alpha$ to regulate the degree of non-IID client data skew and $\rho$ to control the mismatch in task ordering across clients.
- Evaluation Metrics: Beyond standard final average accuracy and average forgetting, the library emphasizes bottom-10% client accuracy to highlight equity and robustness issues often hidden by aggregate metrics.
- Cross-Domain Validation: Includes a graph-based Domain-Incremental Learning (Domain-IL) portability test on the OGB-MolPCBA dataset, verifying that the interface effectively captures domain-shift challenges in molecular property prediction.
- Open Implementation: Releases complete benchmark streams, configuration files, method implementations, and reporting scripts to ensure full transparency and reproducibility.
Industry Insight
- Researchers should adopt standardized heterogeneity parameters like $\alpha$ and $\rho$ when evaluating new FCL algorithms to ensure their results are comparable across studies.
- Practitioners must look beyond average accuracy metrics; monitoring tail-client performance (e.g., bottom-10%) is essential for deploying equitable federated systems in real-world, heterogeneous environments.
- The demonstrated portability of the HERO interface to graph data suggests that similar modular benchmarking frameworks could be valuable for other modalities like NLP or time-series forecasting in federated settings.
Disclaimer: The above content is generated by AI and is for reference only.