DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
DataPrep-Bench is introduced as the first unified benchmark evaluating LLMs on both data construction and data quality evaluation, grounded in downstream training utility rather than surface-level text properties. The benchmark assesses two complementary capabilities: transforming raw sources into supervised training data (data construction) and predicting the training value of candidate datasets before fine-tuning (quality evaluation). A new skill-guided agent, Data-Construction-Skill, signific
Analysis
TL;DR
- DataPrep-Bench is introduced as the first unified benchmark evaluating LLMs on both data construction and data quality evaluation, grounded in downstream training utility rather than surface-level text properties.
- The benchmark assesses two complementary capabilities: transforming raw sources into supervised training data (data construction) and predicting the training value of candidate datasets before fine-tuning (quality evaluation).
- A new skill-guided agent, Data-Construction-Skill, significantly outperforms baselines, lifting performance by nearly 20 points absolute on Llama-3.1-8B Finance tasks compared to using only Dolly-15k.
- The Distributional Alignment Score (DAS), a metric based on Maximum Mean Discrepancy (MMD) between candidate datasets and domain proxies, achieves the highest cross-model correlation in four of six domains and exceeds r > 0.70 in Math, Science, and Medical fields.
Why It Matters
This research addresses a critical gap in the AI lifecycle by providing a standardized way to measure how effectively LLMs can prepare their own training data, shifting focus from model architecture to data-centric workflows. For practitioners, it offers validated tools and metrics to optimize data curation processes, ensuring that high-quality, downstream-utility-focused data is used for fine-tuning. This is essential for improving model performance while reducing costs associated with ineffective or noisy training datasets.
Technical Details
- Benchmark Structure: DataPrep-Bench jointly evaluates data construction and data quality evaluation across six domains and multiple base models using a shared downstream-grounded protocol.
- Data Construction Evaluation: Methods are scored by fine-tuning a base model on their outputs combined with the Dolly-15k dataset. The release includes "Data-Construction-Skill," an agent that uses specific skills to guide data transformation, demonstrating superior performance in knowledge-extraction-dense domains.
- Data Quality Evaluation Metric: Introduces the Distributional Alignment Score (DAS), which calculates the MMD between a candidate dataset and a domain proxy to predict downstream performance. DAS is evaluated via Pearson correlation with actual downstream performance on a shared candidate pool.
- Performance Results: DAS outperforms existing quality-, diversity-, and heuristic-based evaluators, being the only metric to simultaneously achieve a correlation coefficient greater than 0.70 in Math, Science, and Medical domains.
Industry Insight
- Shift to Data-Centric AI: Organizations should prioritize investing in automated data preparation pipelines and agents, as the quality of training data is now quantifiable and optimizable through benchmarks like DataPrep-Bench.
- Adoption of Downstream-Grounded Metrics: Relying on surface-level textual heuristics for data quality is insufficient. Practitioners should adopt distribution-based metrics like DAS that correlate strongly with actual model performance improvements in specialized domains.
- Standardization of Evaluation: As LLMs increasingly participate in their own training data creation, adopting unified benchmarks will be crucial for comparing different data preparation strategies and ensuring reproducibility in model development.
Disclaimer: The above content is generated by AI and is for reference only.