Steering the Language Axis: From Linear Decodability to Causal Control
Language identity in LLMs is not merely linearly decodable from hidden states but can be causally controlled through compact PCA-derived "language axes" Steering along these geometric directions reliably forces language switching across both cross-script (English-Chinese) and same-script (English-Spanish) settings Language commitment is highly localized and layer-specific, with English-Chinese switching requiring later-layer intervention while English-Spanish transitions exhibit bimodal sensitiv
Analysis
TL;DR
- Language identity in LLMs is not merely linearly decodable from hidden states but can be causally controlled through compact PCA-derived "language axes"
- Steering along these geometric directions reliably forces language switching across both cross-script (English-Chinese) and same-script (English-Spanish) settings
- Language commitment is highly localized and layer-specific, with English-Chinese switching requiring later-layer intervention while English-Spanish transitions exhibit bimodal sensitivity at earlier layers
- Targeted ablation reveals a fundamental English reversion bias: removing the language signal causes the model to fall back to English regardless of input prompt
- The study conducted exhaustive causal intervention analysis across 1.26 million generations on FLORES-200 using Qwen 3.5-2B and Llama-3.2-1B-Instruct
Why It Matters
This research fundamentally advances our understanding of multilingual LLM internals by demonstrating that language selection is governed by causally active, direction-dependent features rather than being an emergent property distributed across hidden states. For AI practitioners building multilingual systems, these findings enable precise, lightweight language steering mechanisms without retraining, while revealing a concerning English-centric bias that could affect deployment in low-resource language contexts.
Technical Details
- Methodology: PCA-derived "language axes" isolated from hidden states, followed by exhaustive causal intervention analysis including steering and ablation experiments across 1.26 million generations on FLORES-200
- Models evaluated: Qwen 3.5-2B and Llama-3.2-1B-Instruct, representing different model families and architectures
- Key finding on intervention locality: English-to-Chinese switching resists early-layer intervention and steers effectively only in later layers, while English-to-Spanish transitions show earlier, bimodal sensitivity patterns
- English reversion bias: Ablation experiments demonstrated that once the language signal is removed, models consistently revert to English generation regardless of input prompt language
- Control validation: Equal-magnitude random perturbations yielded virtually no language switching effect, confirming the specificity of the identified language axes
Industry Insight
- Multilingual LLM deployments should account for inherent English-centric bias when designing language-switching interfaces, as ablated models default to English rather than respecting input language cues
- The layer-specific nature of language commitment suggests that lightweight intervention modules could be inserted at optimal layers for real-time language steering without full model retraining
- The bimodal sensitivity pattern in same-script transitions (English-Spanish) versus unimodal cross-script patterns (English-Chinese) indicates that script distance fundamentally shapes where language decisions are made in the network, informing future interpretability research and multilingual model architecture design
Disclaimer: The above content is generated by AI and is for reference only.