How are linear representations learned? Exact solutions to the dynamics of abstraction
The paper introduces a dynamical theory of "abstraction," analyzing how linear concept representations emerge during training rather than just existing post-training. Exact solutions for linear networks reveal that final abstraction is determined by joint data/target geometry, improves with depth, and is capped by initialization scale. A universal "attenuation law" is proven for nonlinear networks (ReLU and erf), showing that activations weaken abstraction compared to preactivations. Empirical v
Analysis
TL;DR
- The paper introduces a dynamical theory of "abstraction," analyzing how linear concept representations emerge during training rather than just existing post-training.
- Exact solutions for linear networks reveal that final abstraction is determined by joint data/target geometry, improves with depth, and is capped by initialization scale.
- A universal "attenuation law" is proven for nonlinear networks (ReLU and erf), showing that activations weaken abstraction compared to preactivations.
- Empirical validation on open models (DINOv3, Gemma 4) confirms the attenuation law, and the theory is applied to enhance linear probe generalization in LLMs.
Why It Matters
This research bridges a critical gap in mechanistic interpretability by explaining the process of feature learning, providing theoretical grounding for why linear probes work and where they fail. For practitioners, understanding the attenuation law and initialization effects offers actionable levers to improve model interpretability and control mechanisms without retraining entire models.
Technical Details
- Linear Network Analysis: Derives exact analytical solutions for the full trajectory of abstraction in minimal linear networks, identifying three governing principles: joint geometry dependence, depth benefit, and initialization scaling limits.
- Nonlinear Extension: Compares erf and ReLU networks, finding that erf networks closely approximate linear theory, whereas ReLU networks rely more heavily on input geometry than target geometry.
- Attenuation Law: Proves that both ReLU and erf nonlinearities systematically weaken the alignment of concept directions in hidden activations relative to preactivations.
- Empirical Validation: Demonstrates the attenuation law using representations from DINOv3 and Gemma 4, and shows practical application by improving linear probe generalization performance in Large Language Models.
Industry Insight
- Probe Design: Practitioners should account for the attenuation law when designing interpretability pipelines; relying solely on hidden layer activations may underestimate concept strength compared to preactivations or output layers.
- Initialization Sensitivity: Since initialization scale controls maximum abstraction, careful tuning of weight initialization strategies could be used to optimize the emergence of interpretable features during training.
- Interpretability Robustness: The finding that ReLU networks depend more on input geometry suggests that interpretability methods may need to be adapted differently for ReLU-based architectures versus other nonlinearities to maintain accuracy.
Disclaimer: The above content is generated by AI and is for reference only.