Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection

Background

Sparse autoencoders (SAEs) have been employed to enhance feature-level interpretability in large language models (LLMs). However, their effectiveness in multilingual settings has been limited due to biases from English-only training data and heuristic layer selection methods. This article addresses these issues by developing a principled approach for multilingual SAE steering.

Key Points

The study demonstrates that training SAEs on multilingual data consistently improves cross-lingual representations, leading to more reliable language control across different layers and model families. An \emph{a priori} steering layer-selection rule is introduced based on the intersection of multilingual alignment and language separability, predicting effective intervention depths without extensive search.

Training Multilingual SAEs

The authors train SAEs on multilingual datasets to leverage cross-lingual representations. This approach ensures that the autoencoders learn more generalized features, which are beneficial for various languages. The results indicate a consistent enhancement in quality-preserving language control across different layers and model sizes.

Layer Selection Rule

A key innovation is an \emph{a priori} steering layer-selection rule derived from the intersection of multilingual alignment and language separability metrics. This rule helps predict effective intervention depths, reducing the need for exhaustive layerwise searches. The rule is based on the idea that certain layers are more sensitive to linguistic differences, making them better candidates for steering.

Evaluation

The proposed approach is evaluated on LLaMA-3.1-8B and Gemma-2-9B models across machine translation and cross-lingual summarization (CrossSumm) tasks using metrics such as SpBLEU, ROUGE-L, COMET, and LaSE. The results show that multilingual SAEs combined with intersection-selected layers stabilize the trade-off between language identification accuracy and generation quality.

Significance

The study provides a significant advancement in multilingual SAE steering by leveraging cross-lingual data and introducing a principled layer-selection method. This work not only enhances the reliability of SAE-based language control but also offers a predictive framework for future research, making it easier to fine-tune LLMs for diverse linguistic contexts without extensive experimentation.

Key Insights:

1. Enhanced Cross-Lingual Representations: Training SAEs on multilingual data improves cross-lingual performance.
2. Predictive Layer Selection: An \emph{a priori} rule based on multilingual alignment and separability metrics stabilizes language control trade-offs.
3. Generalizability: The proposed methods can be applied to different model families, making them versatile for various LLMs.

These insights contribute substantially to the field of multilingual natural language processing, offering practical solutions for improving the interpretability and reliability of large language models.