Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-training
MASS addresses the challenge of selecting high-value subsets from large supervised fine-tuning data pools by formulating data selection as a coarse-to-fine hierarchical coverage problem The method uses a dense autoencoder to learn low-dimensional principal manifold coordinates for coarse semantic grouping, disentangling dominant semantic directions from fine-grained supervision differences and local noise Within each semantic group, a TopK sparse autoencoder performs quality-aware sparse feature
Analysis
TL;DR
- MASS addresses the challenge of selecting high-value subsets from large supervised fine-tuning data pools by formulating data selection as a coarse-to-fine hierarchical coverage problem
- The method uses a dense autoencoder to learn low-dimensional principal manifold coordinates for coarse semantic grouping, disentangling dominant semantic directions from fine-grained supervision differences and local noise
- Within each semantic group, a TopK sparse autoencoder performs quality-aware sparse feature coverage to capture fine-grained diversity
- Experiments on Vision Flan and LLaVA-CoT benchmarks demonstrate that MASS consistently outperforms strong data selection baselines across multiple data budgets
- In several settings, MASS matches or surpasses full data training using only a small subset of data, significantly reducing training costs
Why It Matters
This work addresses a critical bottleneck in LLM post-training: as supervised fine-tuning datasets continue to scale, the cost of training on full data becomes prohibitive, making intelligent data selection essential. The hierarchical approach of separating coarse semantic grouping from fine-grained feature coverage offers a principled framework that could become a standard technique for efficient model training across the industry.
Technical Details
- Dense Autoencoder for Manifold Learning: MASS first employs a dense autoencoder to learn low-dimensional principal manifold coordinates from the original embedding space, enabling effective coarse semantic grouping while disentangling dominant semantic directions from noise
- TopK Sparse Autoencoder for Feature Coverage: Within each semantic group, a TopK sparse autoencoder captures quality-aware sparse features, ensuring fine-grained diversity is preserved during data selection
- Hierarchical Coverage Formulation: The method frames data selection as a two-level coverage problem—first covering the semantic manifold broadly, then ensuring sparse feature coverage within each group
- Benchmarks: Evaluated on Vision Flan and LLaVA-CoT datasets across multiple data budgets, comparing against strong data selection baselines
- Performance: Achieves competitive or superior results to full-data training while using only a fraction of the data
Industry Insight
- The hierarchical coverage framework provides a scalable blueprint for data curation that can be adapted beyond LLMs to other domains where training data is abundant but expensive to process
- As model training costs continue to rise, methods like MASS that enable significant data reduction without performance loss will become increasingly valuable for both large labs and resource-constrained researchers
- The separation of semantic and sparse feature coverage suggests that future data selection methods should explicitly model multiple levels of structure rather than relying on flat embedding-space metrics
Disclaimer: The above content is generated by AI and is for reference only.