Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation 超越沙普利：基于影响力的LLM对齐与评估数据审计管道

TL;DR

• The paper introduces a scalable, inference-only data valuation pipeline for LLM alignment that approximates Shapley values without iterative model retraining.
• It maps semantic k-NN neighborhoods into a directed graph to evaluate data utility using zero-shot and one-shot conditional log-likelihood shifts from a reference LLM.
• The pipeline isolates gradient-conflicting records by translating predictive influence scores into localized advantage metrics, enabling efficient auditing of large-scale alignment datasets.
• Applied to HelpSteer2, it reduced manual audit search space by 99.1% and uncovered falsely-labeled records across diverse failure modes.
• When applied to Anthropic’s HH-RLHF training and evaluation splits, it revealed thousands of hidden safety and factual preference inversions, exposing vulnerabilities in benchmark integrity due to flawed human ground-truth labels.

Why It Matters

This work addresses a critical bottleneck in LLM alignment: the degradation of dataset quality at scale, which undermines model performance and safety guarantees. By providing an efficient, mathematically grounded method to detect subtle but harmful data inconsistencies—especially those invisible to standard deduplication or judge-based audits—it offers practitioners a powerful tool to sanitize both training and evaluation datasets, ensuring more reliable and trustworthy alignment processes.

Technical Details

• The framework uses a reference LLM to compute conditional log-likelihood shifts under zero-shot and one-shot settings, measuring how each data point influences the model’s output distribution.
• Semantic k-NN neighborhoods are constructed based on embedding similarities and mapped into a directed graph where edges represent directional influence between samples.
• Shapley value approximation is achieved via Monte Carlo sampling over subsets of the neighborhood graph, avoiding costly retraining while preserving theoretical fairness properties.
• Localized advantage metrics are derived from these influence scores to identify records that contradict dominant patterns or introduce conflicting gradients during fine-tuning.
• The method was validated on two major alignment benchmarks: HelpSteer2 (instruction tuning) and HH-RLHF (preference learning), demonstrating high precision in detecting label errors and preference inversions.

Industry Insight

AI teams should integrate automated influence-based auditing pipelines like this one into their standard data curation workflows, especially when scaling up instruction-tuning or RLHF datasets. Relying solely on human annotation or surface-level deduplication risks propagating latent contradictions that can degrade model behavior post-deployment. Furthermore, evaluation benchmarks must be regularly audited using such methods to ensure they reflect true capability rather than artifacts of biased or erroneous labeling—otherwise, reported improvements may be illusory.

TL;DR

• 提出一种基于影响力的LLM数据审计管道，无需迭代重训练即可近似Shapley值。
• 通过语义k-NN邻域映射到有向图，利用参考LLM的概率分布评估数据效用。
• 在HelpSteer2和HH-RLHF数据集上验证了该管道的有效性，揭示了隐藏的安全性和事实偏好反转问题。
• 暴露了当前基准测试中的严重漏洞：高质量模型可能因客观错误的人类标签而受到惩罚。
• 提供了一种数学基础且高效的诊断工具，用于发现人类标签错误、清理评估基准并确保LLM对齐数据的完整性。

为什么值得看

这篇文章对AI从业者具有重要意义，因为它提供了一种高效的方法来检测和纠正大规模数据集中的潜在问题，这些问题可能会严重影响LLM的对齐和性能。通过揭示现有评估基准中的缺陷，它促使研究人员重新审视数据质量和评估标准的重要性。

技术解析

1. 数据估值管道：引入了一种可扩展的、仅推理的数据估值方法，能够近似Shapley值而不需要反复进行模型重训练。这种方法大大降低了计算成本和时间消耗。
2. 语义k-NN邻域与有向图：将语义相似的样本组织成k-NN邻域，并将其转换为有向图结构，以便更有效地分析数据之间的关系和影响。
3. 条件对数似然偏移：使用零样本或少量样本的条件对数似然变化来直接衡量单个数据点对于模型预测的影响程度。
4. 局部优势指标：将上述预测影响力转化为具体的局部优势指标，从而识别出那些可能导致梯度冲突的数据记录。
5. 实验结果：在两个知名数据集上的应用展示了该方法的有效性，不仅大幅减少了人工审核的工作量，还发现了大量未被注意到的质量问题。

行业启示

1. 重视数据质量：随着LLM规模的不断扩大，数据质量问题变得越来越突出。企业应加强对训练数据和评估数据的严格审查机制，确保其准确性和一致性。
2. 改进评估体系：现有的某些评估基准可能存在偏差或不准确之处，需要定期更新和完善，以更好地反映真实世界的应用场景和需求。
3. 自动化审计工具的发展：未来可以开发更多类似的自动化审计工具，帮助团队快速定位并解决数据中的潜在问题，提高整体研发效率和质量控制水平。