AI models' written reasoning steps correspond to distinct internal patterns, a new study finds 研究发现：AI模型的文字推理步骤对应不同的内部模式

TL;DR

• Researchers at KAIST and Naver AI Lab demonstrated that distinct reasoning steps (extraction, decomposition, formula recall, deduction, computation, etc.) are separable within language models' internal numerical representations, not just in their text outputs.
• The signal for reasoning operations is strongest in the middle layers of transformer models, with the same words acquiring different internal representations depending on the reasoning step they belong to.
• Reasoning steps are context-dependent: blocking attention to preceding tokens weakens the internal signal, showing that steps build on prior context rather than emerging in isolation.
• Even incorrect reasoning steps remain internally identifiable by type, suggesting internal representations encode the nature of computation independently of output correctness.
• The findings have implications for AI safety and interpretability, particularly as newer architectures like OpenAI's Astra shift reasoning into internal numerical states.

Why It Matters

This research bridges the gap between observable chain-of-thought outputs and the hidden internal computations that drive them, offering a potential pathway for monitoring and steering model reasoning in real time. As AI systems increasingly rely on internal reasoning beyond their text outputs (e.g., Recurrent Depth architectures), the ability to read and interpret those internal states becomes critical for safety oversight, especially given that models disclose only a fraction of their actual reasoning process.

Technical Details

• Eight reasoning operations were defined and used as labels: extraction, decomposition, formula recall, deduction, computation, and three others, applied to solution paths from math problems.
• Three base models were tested: Qwen2.5-7B, Qwen3-8B, and Gemma4-31B, with replication on Llama-3-8B. Solution segments were labeled using GPT-5.
• Layer-wise analysis revealed that reasoning operation separability peaks in middle layers; early layers show overlapping representations for common function words, while middle and later layers disambiguate them by operation type.
• Control experiments ruled out surface-level explanations: a token-only classifier underperformed the internal-representation classifier, and position within the solution path did not account for the observed separability.
• Intervention study showed that blocking attention to the preceding 30 tokens weakened the reasoning-step signal, confirming context-dependence. Transferability was demonstrated on GPQA-Diamond and MATH-500 benchmarks for Qwen3-8B.

Industry Insight

• As next-generation architectures increasingly internalize reasoning (e.g., Recurrent Depth in OpenAI's Astra), interpretability tools that read internal states will become essential for oversight—textual chain-of-thought alone is insufficient as a safety mechanism.
• The finding that flawed reasoning steps remain identifiable internally suggests potential for real-time error detection and intervention during model generation, though this remains unproven and is a promising direction for applied research.
• The context-dependence of reasoning signals implies that any interpretability framework must account for sequential dependencies; isolating a single step without its preceding context will yield incomplete or degraded signals.

摘要

韩国科学技术院（KAIST）与Naver AI Lab的研究人员证明，语言模型内部数值表示中可分离出不同的推理步骤（如提取、分解、公式回忆、演绎、计算等），而不仅限于文本输出层面。
推理操作的信号在Transformer模型的中层最强，相同的词语会根据其所属的推理步骤获得不同的内部表示。
推理步骤具有上下文依赖性：阻断对前置token的注意力会削弱内部信号，表明推理步骤建立在先前上下文之上，而非孤立产生。
即使错误的推理步骤在内部仍可被识别其类型，说明内部表示编码了计算本身的性质，与输出正确性无关。
该发现对AI安全与可解释性具有启示意义，尤其随着OpenAI的Astra等新型架构将推理过程转向内部数值状态。

深度分析

极简总结

• 韩国科学技术院（KAIST）与Naver AI Lab的研究人员证明，语言模型内部数值表示中可分离出不同的推理步骤（如提取、分解、公式回忆、演绎、计算等），而不仅限于文本输出层面。
• 推理操作的信号在Transformer模型的中层最强，相同的词语会根据其所属的推理步骤获得不同的内部表示。
• 推理步骤具有上下文依赖性：阻断对前置token的注意力会削弱内部信号，表明推理步骤建立在先前上下文之上，而非孤立产生。
• 即使错误的推理步骤在内部仍可被识别其类型，说明内部表示编码了计算本身的性质，与输出正确性无关。
• 该发现对AI安全与可解释性具有启示意义，尤其随着OpenAI的Astra等新型架构将推理过程转向内部数值状态。

为何重要

这项研究弥合了可观察的链式思维输出与其背后驱动推理的隐藏内部计算之间的鸿沟，为实时监测和引导模型推理提供了潜在路径。随着AI系统越来越多地依赖文本输出之外的内部推理（如循环深度架构），解读和解释这些内部状态的能力对安全监督变得至关重要，尤其