GLOBE: Trajectory-Aligned Gradient Matching with Structured Sparse Optimization for Coreset Selection
GLOBE introduces trajectory-aligned gradient matching for coreset selection, moving beyond single-snapshot gradients to capture evolving optimization dynamics across multiple training checkpoints A multi-order matching objective jointly aligns first-order mean and projected uncentered second-order moments of gradient trajectories to preserve full-dataset training behavior Structured sparsity via Group LASSO, Elastic Net regularization, and nonnegative budget constraints enables simultaneous grou
Analysis
TL;DR
- GLOBE introduces trajectory-aligned gradient matching for coreset selection, moving beyond single-snapshot gradients to capture evolving optimization dynamics across multiple training checkpoints
- A multi-order matching objective jointly aligns first-order mean and projected uncentered second-order moments of gradient trajectories to preserve full-dataset training behavior
- Structured sparsity via Group LASSO, Elastic Net regularization, and nonnegative budget constraints enables simultaneous group- and sample-level sparsity while stabilizing weights of correlated trajectories
- Class-balanced Top-K selection ensures adequate category coverage under limited sampling budgets
- GLOBE consistently outperforms existing coreset selection methods across six benchmarks and five architectures, especially at low retention ratios
Why It Matters
Coreset selection is critical for on-device and resource-constrained deep learning, where storing and processing full training datasets is infeasible. GLOBE addresses a key limitation of prior gradient-based methods—reliance on static, single-point gradients—by incorporating dynamic optimization trajectories, making it directly relevant to practitioners building efficient, data-efficient training pipelines for edge devices and large-scale distributed systems.
Technical Details
- Gradient Trajectory Representation: Each training sample is represented by a gradient trajectory constructed across multiple training checkpoints, capturing its influence throughout different optimization stages rather than at a single snapshot
- Multi-Order Matching Objective: GLOBE jointly aligns the first-order mean and projected uncentered second-order moments of gradient trajectories, preserving both directional and variance structure of the full dataset's training dynamics
- Structured Sparsity Framework: Combines Group LASSO (group-level sparsity), Elastic Net regularization (sample-level sparsity with correlation handling), and nonnegative budget constraints to produce stable, interpretable sample weights
- Class-Balanced Top-K Selection: After sparse weight optimization, a class-balanced Top-K procedure selects the final coreset, ensuring representative category coverage even at aggressive retention ratios
- Empirical Validation: Evaluated across six benchmarks and five architectures, demonstrating consistent superiority over existing methods, with particularly strong gains at low retention ratios
Industry Insight
- The shift from static to trajectory-aligned gradient matching represents a paradigm improvement for coreset selection; practitioners should consider dynamic gradient information when designing data subset strategies for on-device or federated learning
- The combination of multi-order moment matching with structured sparsity offers a reusable template for other data selection and subset optimization problems beyond coreset selection
- As on-device AI deployment accelerates, methods like GLOBE that enable training with significantly reduced data while preserving accuracy will become increasingly valuable for edge computing and privacy-sensitive applications
Disclaimer: The above content is generated by AI and is for reference only.