Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models

TL;DR

• Researchers identify "magnetic vectors" in LLMs—special token vectors that organize surrounding tokens through attraction (elongation) or repulsion (compression)
• Function words consistently act as repelling magnets in early layers, with magnets reorganizing their polarities uniquely in deeper layers
• During fine-tuning, task-functional tokens emerge as magnets; in QA, answer-span tokens become repelling magnets that geometrically carve answers from context
• Causal evidence: removing early-layer repelling magnets devastates syntactic tasks (POS tagging drops from 91% to below 10%) while sparing semantics, and removing late-layer attracting magnets has the reverse effect
• This pattern is consistent across different LLM architectures, sizes, and layer configurations, offering a probe-free path to understanding geometric linguistic organization

Why It Matters

This research provides a novel, probe-free methodology for interpreting how LLMs organically structure linguistic computation across layers, bridging the gap between geometric representations and linguistic phenomena. For AI practitioners, it offers actionable insights into which layers and token types are critical for specific task types, enabling more targeted fine-tuning and model editing strategies.

Technical Details

• Magnetic vectors are identified as token vectors that exert attractive or repulsive forces on neighboring tokens, with attracting magnets elongating co-directional tokens and repelling magnets compressing them
• Layer-wise linguistic organization: Function words serve as repelling magnets in early layers; magnets reorganize polarities in distinct patterns deeper in the model, suggesting deliberate layer-wise processing
• Task-specific magnet emergence: After fine-tuning, task-functional tokens become magnets—e.g., in question answering, answer-span tokens become uniquely repelling magnets in the final layer, geometrically isolating answers from surrounding context
• Causal validation: Ablation studies demonstrate that removing early-layer repelling magnets collapses syntactic task performance (POS tagging: 91% → <10%) while preserving semantic tasks; removing late-layer attracting magnets produces the opposite effect
• Generalizability: The magnetic phenomenon is consistent across different LLM architectures, model sizes, and layer configurations

Industry Insight

• Model interpretability efforts can leverage magnetic vector analysis as a probe-free diagnostic tool, reducing reliance on external probing methods and enabling direct geometric analysis of internal representations
• Fine-tuning strategies should account for magnet reorganization—task-specific magnets emerge naturally during adaptation, suggesting that targeted magnet manipulation could enable efficient, minimal-editing fine-tuning
• The causal dissociation between syntactic (early-layer) and semantic (late-layer) processing implies that model compression or pruning should preserve early-layer repelling magnets for syntax-dependent applications and late-layer attracting magnets for semantic reasoning tasks

TL;DR

• 提出“磁向量”(magnetic vectors)概念，发现LLM内部存在能吸引或排斥周围token的特殊向量组。
• 吸引型磁向量会使同向token几何拉伸，排斥型则使其压缩，形成类似物理磁铁的对立空间组织模式。
• 功能词在模型浅层稳定充当排斥磁，深层磁向量会随任务需求重新组织极性分布。
• 下游任务微调后，任务关键token会自发涌现为磁向量（如QA任务中答案span在末层成为排斥磁以几何切割上下文）。
• 干预实验证实因果性：移除浅层排斥磁严重破坏句法任务（POS标注准确率从91%跌至<10%），移除深层吸引磁则主要损害语义任务。

为什么值得看

该研究首次从几何向量空间视角揭示了LLM逐层处理语言的内在组织规律，为理解大模型“黑盒”提供了无需探针的直观路径。对AI从业者而言，磁向量机制为模型可解释性研究、任务导向的微调策略以及关键token的精准干预提供了全新的理论框架与实验依据。

技术解析

• 核心几何机制：通过向量空间分析识别磁向量，发现吸引极性导致同向token特征空间拉伸，排斥极性导致压缩，形成对立的空间组织模式。
• 分层语言规律：浅层中功能词统计显著地充当排斥磁；随着层数加深，磁向量的极性