Evaluating the Impact of Reviewer Guideline Design on LLM-Based Automated Peer Review 评估审稿人指南设计对基于LLM的自动同行评审的影响

TL;DR

• Official conference guidelines yield the most human-consistent results for LLM-based automated peer review.
• Reviewer-imitating guidelines generated from high-quality human reviews are generally less effective than official guidelines.
• Strict rubric-style scoring consistently degrades performance, emphasizing the need for subjective and holistic evaluation approaches.

Why It Matters

This research provides critical insights into optimizing automated peer review systems by highlighting the importance of guideline design over imitation or rigid scoring frameworks. For AI practitioners and researchers, it underscores that leveraging established, practice-refined evaluation criteria can significantly enhance the reliability and alignment of AI-driven review processes with human judgment.

Technical Details

• The study compares two types of reviewer guidelines: official conference guidelines and reviewer-imitating guidelines generated using LLMs from high-quality human reviews.
• Experiments evaluate how these guidelines impact the consistency of automated review results with human judgments.
• Results show that official guidelines outperform imitative ones in producing human-like review outcomes.
• Enforcing strict rubric-style scoring negatively affects performance, suggesting flexibility in scoring is crucial for accurate automated reviewing.

Industry Insight

AI professionals should prioritize incorporating well-established, practice-refined guidelines when designing automated peer review systems rather than relying solely on imitation models or overly structured scoring methods. This approach can improve system accuracy and trustworthiness in real-world applications where nuanced, subjective evaluations are necessary.

TL;DR

• 研究分析了不同评审指南（官方会议指南 vs LLM生成的人为模仿指南）对基于大语言模型的自动同行评审的影响。
• 实验结果表明，官方会议指南产生的评审结果与人类判断最一致，表明经过实践检验的评价标准能有效指导自动化评审。
• LLM生成的模仿人评审指南效果普遍不如官方指南，且强制使用严格的量表式评分会持续降低性能。
• 研究强调了在自动化评审中允许主观性和整体性评分的重要性。
• 该工作为优化AI辅助学术评审系统提供了实证依据和方向建议。

为什么值得看

本文揭示了在设计LLM驱动的自动评审系统时，采用经过实际学术会议验证的正式指南比单纯模仿人类评审风格更有效，这对构建可靠、可落地的学术AI工具具有直接指导意义。同时，它警示过度结构化评分可能损害评审质量，提醒开发者在自动化过程中保留必要的人文弹性。

技术解析

• 研究对比了两类核心评审指南：一是各学术会议发布的官方评审准则，二是利用高质量人类评审数据通过LLM生成的“模仿人类”风格的评审提示词。
• 评估指标聚焦于自动化评审输出与真实人类评审的一致性程度，未明确说明具体相似度计算方法或使用的基准数据集规模。
• 关键发现是“严格 rubric-style scoring”（即高度结构化的打分表） consistently degraded performance，暗示模型在受限格式下难以捕捉 nuanced 的学术判断。
• 方法论隐含了多轮迭代测试不同prompt策略的过程，但未披露模型版本、训练细节或消融实验设计。
• 结论支持将领域内成熟的人工评审经验转化为机器指令，而非试图让AI完全复现个体评审者的主观偏好。

行业启示

• 学术出版机构在引入AI评审助手时，应优先整合自身长期积累的标准化评审框架，而非依赖通用LLM凭空生成评审逻辑。
• 系统设计需避免将评审过程过度量化，应保留空间让模型进行综合判断，以匹配人类专家的 holistic evaluation 能力。
• 未来方向可探索混合模式：结合官方指南的结构稳定性与少量人工微调后的灵活性，平衡效率与深度理解需求。