Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens
The paper introduces Sparse Readout Prism (SRP), a method that decomposes the readout (unembedding) matrix of language models into sparse features, enabling analysis independent of any fitting corpus The authors demonstrate "corpus conditionality": two lenses fit on different corpora can assign different tokens to the same hidden states, revealing a fundamental limitation of existing lens methods SRP expresses any token logit or logit difference as a sum of contributions from sparse readout feat
Analysis
TL;DR
- The paper introduces Sparse Readout Prism (SRP), a method that decomposes the readout (unembedding) matrix of language models into sparse features, enabling analysis independent of any fitting corpus
- The authors demonstrate "corpus conditionality": two lenses fit on different corpora can assign different tokens to the same hidden states, revealing a fundamental limitation of existing lens methods
- SRP expresses any token logit or logit difference as a sum of contributions from sparse readout features, establishing features as a new unit of analysis for interpretability
- SRP's sparse approximation reconstructs 8.9–17.3 percentage points more of tested logit differences than the strongest of six geometric baselines
- While token-level readings vary with fitting corpus, the dominant readout features remain stable, providing a corpus-independent control for mechanistic interpretability research
Why It Matters
This work addresses a critical confound in mechanistic interpretability: logit-lens readings conflate hidden state representations with the readout matrix, making it difficult to determine whether observed patterns reflect genuine internal representations or artifacts of the decoding process. By isolating readout structure from corpus-dependent fitting, SRP provides researchers with a more reliable tool for comparing interpretations across models, layers, and training conditions.
Technical Details
- Sparse Readout Prism (SRP) decomposes the unembedding matrix using only its weights (no corpus required), expressing token logits and logit differences as additive contributions from sparse readout features
- The paper identifies and formalizes corpus conditionality — the phenomenon where lenses fitted on different corpora produce divergent token assignments for identical hidden states, undermining cross-study comparability
- Evaluation: SRP outperforms six geometric baselines (based on readout row relationships) by 8.9–17.3 percentage points in logit difference reconstruction accuracy
- Ablation studies confirm that shifting SRP features shifts logit differences proportionally to their computed contributions, validating the decomposition's causal interpretability
- The dominant readout features prove stable across corpora, unlike token-level readings, making SRP a corpus-independent control for lens-based analyses
Industry Insight
- Interpretability tooling should move beyond token-level explanations; feature-level decompositions like SRP offer more stable and comparable insights across model variants and training regimes
- Researchers using logit-lens or related methods should account for corpus conditionality when comparing results across studies, as differing fitting corpora may produce misleading apparent discrepancies
- SRP's corpus-free construction makes it particularly valuable for benchmarking and auditing models where training data is proprietary or unavailable, enabling readout analysis without data access
Disclaimer: The above content is generated by AI and is for reference only.