Discovering Cross-Language Reasoning Invariance in LLMs with Geometry-Invariant Sparse Autoencoders
Researchers investigated whether multilingual LLMs rely on shared cross-language features or language-specific computations when solving the same math problems A novel Geometry-Invariant Sparse Autoencoder (GI-SAE) was introduced, combining reconstruction loss with InfoNCE contrastive loss to encourage language-invariant activations GI-SAE successfully amplifies cross-language feature alignment (higher CKA and Jaccard similarity), but geometric similarity does not guarantee functional interchang
Analysis
TL;DR
- Researchers investigated whether multilingual LLMs rely on shared cross-language features or language-specific computations when solving the same math problems
- A novel Geometry-Invariant Sparse Autoencoder (GI-SAE) was introduced, combining reconstruction loss with InfoNCE contrastive loss to encourage language-invariant activations
- GI-SAE successfully amplifies cross-language feature alignment (higher CKA and Jaccard similarity), but geometric similarity does not guarantee functional interchangeability
- Cross-language reasoning invariance is model- and architecture-dependent, with strong effects in Qwen, no benefit in Gemma, and mixed results in Llama and Phi
- Feature swapping experiments using KL divergence revealed that shared geometric structure alone is insufficient for functional equivalence across languages
Why It Matters
This research directly addresses a fundamental question in multilingual AI: whether models truly share reasoning mechanisms across languages or merely produce similar outputs through divergent internal computations. For AI practitioners building multilingual systems, understanding whether cross-language feature sharing exists—and which architectures support it—has implications for model efficiency, transfer learning, and interpretability. The GI-SAE methodology also offers a new tool for probing multilingual representation alignment beyond surface-level metrics.
Technical Details
- Dataset and Setup: Evaluated five models from four families (Qwen, Gemma, Llama, Phi) on the Multilingual Grade School Math (MGSM) dataset across six languages (English, German, French, Spanish, Russian, Chinese), retaining only problems with valid reasoning traces in all languages
- CKA Analysis: Used Centered Kernel Alignment to identify layers exhibiting cross-language representational alignment before applying SAEs
- GI-SAE Architecture: Introduced a contrastive sparse autoencoder that supplements standard reconstruction loss with an Information Noise-Contrastive Estimation (InfoNCE) loss, training the encoder to produce similar activations for the same problem across different languages and token positions
- Functional Interchangeability Test: Swapped SAE feature values between languages during forward passes and measured output changes via KL divergence per feature, establishing a direct test of whether geometrically similar features are functionally equivalent
- Key Finding: Higher geometric similarity (CKA/Jaccard) from GI-SAE did not consistently translate to functional interchangeability, revealing a critical gap between representational alignment and operational equivalence
Industry Insight
- Model architecture choices significantly impact cross-language reasoning sharing; practitioners should evaluate GI-SAE-style probing when selecting or fine-tuning multilingual models for reasoning-heavy tasks
- The gap between geometric similarity and functional interchangeability suggests that interpretability tools based solely on representational alignment may overstate cross-lingual transfer potential—functional validation is essential
- GI-SAE's contrastive approach could be adapted beyond multilingual settings to study invariance along other dimensions (e.g., domain, style, or prompt framing), offering a generalizable framework for probing feature robustness in LLMs
Disclaimer: The above content is generated by AI and is for reference only.