Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training
Data-DPO introduces a target model-oriented approach to SFT data selection, addressing the gap in existing methods that treat data value as static rather than model-compatible The method uses one-step probing to observe local training feedback, converting activation differences into pairwise data preferences for a lightweight reward model Data-DPO combines target model preferences, external quality scores, and marginal diversity to construct stable and effective training subsets Experiments on V
Analysis
TL;DR
- Data-DPO introduces a target model-oriented approach to SFT data selection, addressing the gap in existing methods that treat data value as static rather than model-compatible
- The method uses one-step probing to observe local training feedback, converting activation differences into pairwise data preferences for a lightweight reward model
- Data-DPO combines target model preferences, external quality scores, and marginal diversity to construct stable and effective training subsets
- Experiments on Vision-Flan and LLaVA-CoT demonstrate consistent outperformance over existing baselines across multiple data budgets
- Data-DPO stably surpasses full data training performance, suggesting that intelligent data selection can exceed the utility of training on all available data
Why It Matters
This work addresses a critical bottleneck in LLM post-training: the inefficiency of using all available data for supervised fine-tuning. By making data selection adaptive to the target model's capability distribution, Data-DPO offers a practical pathway to reduce training costs while improving or maintaining performance. For AI practitioners managing large-scale fine-tuning pipelines, this approach could significantly lower compute requirements without sacrificing model quality.
Technical Details
- One-step probing mechanism: Data-DPO performs a single training step on candidate samples to observe the target model's local training feedback, capturing how different data points affect the model's activations
- Pairwise preference construction: Activation differences across samples are transformed into pairwise data preferences, enabling a preference-based ranking rather than absolute scoring
- Lightweight reward model: A compact reward model is trained to learn target-model-aware data preferences, making the selection process computationally efficient
- Multi-factor selection strategy: The final training subset is constructed by combining three signals: target model preference, external quality scores, and marginal diversity, balancing performance, quality, and coverage
- Benchmarks: Evaluated on Vision-Flan and LLaVA-CoT datasets, demonstrating robustness across multiple data budgets and consistent superiority over full-data training
Industry Insight
- The shift from static to dynamic data valuation represents a paradigm change in data curation; practitioners should consider model-aware selection strategies rather than relying solely on heuristic or quality-based filtering
- The combination of preference learning with diversity optimization offers a template for future data selection systems, particularly in resource-constrained environments where full-data training is infeasible
- As LLM post-training increasingly dominates compute budgets, methods like Data-DPO that can match or exceed full-data performance with subsets will become essential for scaling efficient training pipelines
Disclaimer: The above content is generated by AI and is for reference only.