Evidence Integration in Large Language Models 大语言模型中的证据整合

TL;DR

• LLMs integrate external evidence through a distributional mechanism where evidence shifts the receiver's distribution of initial answers, governed by receiver prior weight and candidate evidence tilt
• Three core predictions confirmed: (1) candidates more probable to the receiver are more persuasive, (2) receivers integrate their own characteristic errors more readily than foreign errors, (3) identical evidence can improve weaker models while harming stronger ones
• Evidence integration occurs late in the network as a structured sequence: admitting external candidates, promoting them, and transporting them into the answer state
• Verification representations are decodable but have minimal causal impact on final answers; the states underlying verification and candidate integration are fully dissociable
• LLMs integrate candidates even after internally verifying their invalidity (93-100% with propositional constraints; up to 99.4% on held-out scientific reasoning)

Why It Matters

This research fundamentally reframes how we understand LLM reasoning with external evidence, revealing that integration is a receiver-specific control policy rather than a simple trust mechanism. For AI practitioners building RAG systems, agentic workflows, or tool-augmented models, these findings expose critical vulnerabilities where models may incorporate flawed evidence due to receiver-congruent biases. The discovery that verification and integration operate through dissociable neural states has direct implications for designing more robust evidence-handling architectures.

Technical Details

• Distributional theory: Evidence integration modeled as a shift in the receiver's answer distribution, parameterized by receiver prior weight and candidate evidence tilt, yielding testable predictions about persuasion dynamics
• Empirical validation: Over 10 million trials across 12 LLMs from 4 families, spanning 8 domains including 4 scientific discovery tasks (quantum mechanics, physics, genetics, molecular biology)
• Receiver-relative reliability frontier: A novel metric showing receiver-congruent errors depress performance more steeply than random errors of equivalent rate, quantifying domain-specific vulnerability
• Causal interventions: Demonstrated that candidate integration is implemented late in the network through a structured sequence—admitting external candidates, promoting them, and transporting them into the answer state
• J-lens decomposition: Revealed full dissociability between the neural state underlying verbalized verification and the state underlying candidate integration, with verification representations having little causal impact on answers

Industry Insight

• RAG and agentic system designers should account for receiver-specific integration biases rather than assuming evidence is processed uniformly; models may systematically over-integrate evidence that aligns with their existing error patterns
• The finding that identical evidence can help weaker models while harming stronger ones suggests that evidence integration quality is non-monotonic with model capability, requiring domain-specific calibration rather than one-size-fits-all prompting strategies
• Since verification and integration are neurally dissociable, building safeguards that only strengthen verification without addressing the late-stage integration pathway will be insufficient; interventions must target the candidate promotion and transport mechanisms directly

摘要

大语言模型通过分布机制整合外部证据，其中证据会改变接收者初始答案的分布，受接收者先验权重和候选证据倾斜的调控
三项核心预测得到证实：（1）对接收者而言更可能的候选答案更具说服力，（2）接收者更容易整合自身特征性错误而非外部错误，（3）相同证据可能提升较弱模型的表现，却损害较强模型
证据整合发生在网络后期，以结构化序列进行：接纳外部候选答案、提升其地位，并将其传输至答案状态
验证表征可被解码，但对最终答案的因果影响极小；支撑验证和候选整合的状态完全可分离
大语言模型即使在内部验证证据无效后仍会整合候选答案（命题约束下93-100%；在保留的科学推理任务中高达99.4%）

深度分析

简而言之

• 大语言模型通过分布机制整合外部证据，其中证据会改变接收者初始答案的分布，受接收者先验权重和候选证据倾斜的调控
• 三项核心预测得到证实：（1）对接收者而言更可能的候选答案更具说服力，（2）接收者更容易整合自身特征性错误而非外部错误，（3）相同证据可能提升较弱模型的表现，却损害较强模型
• 证据整合发生在网络后期，以结构化序列进行：接纳外部候选答案、提升其地位，并将其传输至答案状态
• 验证表征可被解码，但对最终答案的因果影响极小；支撑验证和候选整合的状态完全可分离
• 大语言模型即使在内部验证证据无效后仍会整合候选答案（命题约束下93-100%；在保留的科学推理任务中高达99.4%）

研究意义

这项研究从根本上重塑了我们对大语言模型整合外部证据的理解，揭示整合是一种与接收者相关的控制策略，而非简单的信任机制。对于构建RAG系统、代理工作流或工具增强模型的人工智能从业者而言，这些发现暴露了关键漏洞——模型可能因与接收者一致的偏差而纳入有缺陷的证据。研究发现验证和整合通过可分离的神经状态运作，这一发现对设计更可靠的人工智能系统具有重要意义。