Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation 评估联邦预训练：下游微调与内在评估的可靠性研究

TL;DR

• Federated pre-training evaluation is challenging due to varying client participation and local data availability across distributed settings
• The study uses a controlled 16M parameter transformer trained on identical client data in both centralized and federated settings to compare evaluation protocols
• Downstream fine-tuning on GLUE (full, head-only, and reduced-data variants) does NOT reliably preserve the pre-training quality ranking
• Direct next-token prediction on GLUE text as an intrinsic evaluation signal shows strong correspondence with pre-training test perplexity
• The findings suggest evaluation signals closer to the original pre-training objective should be prioritized over downstream fine-tuning for comparing federated models

Why It Matters

This research directly addresses a critical gap in federated learning evaluation: practitioners currently rely heavily on downstream fine-tuning benchmarks to assess pre-trained model quality, but this study demonstrates that such benchmarks can produce misleading rankings. For AI engineers and researchers working with federated foundation models, this means evaluation protocols need rethinking to ensure fair and accurate model comparison.

Technical Details

• Model: 16M parameter transformer, trained on identical client data in both centralized and federated settings for controlled comparison
• Evaluation protocols compared: Downstream fine-tuning on GLUE (full fine-tuning, head-only fine-tuning, reduced-data variants) versus intrinsic evaluation via next-token prediction on GLUE text
• Methodology: Reference ranking established on pre-training testset; each evaluation protocol assessed by whether it preserves this ranking
• Key finding: Downstream fine-tuning fails to preserve pre-training ranking, while next-token prediction exhibits strong correspondence with pre-training test perplexity
• Dataset: GLUE benchmark used for both downstream fine-tuning and intrinsic next-token prediction evaluation

Industry Insight

• Federated learning teams should adopt intrinsic evaluation metrics (like next-token prediction) alongside or instead of downstream fine-tuning when comparing pre-trained models, as the latter may produce unreliable rankings
• The gap between pre-training perplexity and downstream performance suggests that federated pre-training quality cannot be accurately inferred from task-specific adaptation alone, necessitating new evaluation standards for the federated foundation model ecosystem
• Researchers and practitioners should be cautious when claiming superiority of one federated pre-training approach over another based solely on downstream benchmark performance, as these results may not faithfully reflect true pre-training quality

摘要

由于分布式环境中客户端参与度和本地数据可用性各不相同，联邦预训练评估具有挑战性
该研究使用一个受控的16M参数Transformer，在集中式和联邦式设置下使用相同的客户端数据进行训练，以比较评估协议
在GLUE上的下游微调（完整、仅头部和减少数据变体）并不能可靠地保持预训练质量排名
直接在GLUE文本上进行下一个token预测作为内在评估信号，与预训练测试困惑度显示出强相关性
研究结果表明，在比较联邦模型时，应优先考虑更接近原始预训练目标的评估信号，而非下游微调

深度分析

太长不看版

• 由于分布式环境中客户端参与度和本地数据可用性各不相同，联邦预训练评估具有挑战性
• 该研究使用一个受控的16M参数Transformer，在集中式和联邦式设置下使用相同的客户端数据进行训练，以比较评估协议
• 在GLUE上的下游微调（完整、仅头部和减少数据变体）并不能可靠地保持预训练质量排名
• 直接在GLUE文本上进行下一个token预测作为内在评估信号，与预训练测试困惑度显示出强相关性
• 研究结果表明，在比较联邦模型时，应优先考虑更接近原始预训练目标的评估信号，而非下游微调

为什么重要

这项研究直接解决了联邦学习评估中的一个关键空白：从业者目前严重依赖下游微调基准来评估预训练模型质量，但本研究证明此类基准可能产生误导性排名。对于从事联邦基础模型工作的AI工程师和研究人员来说，这意味着需要重新思考评估协议，以确保公平准确的模型比较。

技术细节

• 模型：16M参数Transformer，在集中式和联邦式设置下使用相同的客户端数据进行训练，以实现受控比较
• 比较的评估协议：在GLUE上的下游微调（完整微调、仅头部微调、减少数据变体）与通过在GLUE文本上进行下一个token预测的内在评估
• 方法论：在预训练测试集上建立参考排名；通过每个评估协议是否保持这一排名来评估