Latent Space Refusal Anchoring for Low-Resource African Languages: Mechanistic Safety Recovery Without Retraining
Instruction-tuned models refuse harmful requests in English but comply in low-resource African languages (Yoruba, Igbo, Igala, Hausa), indicating the refusal mechanism exists in the residual stream but fails to activate for low-resource inputs Latent Space Refusal Anchoring (LSR-Anchoring) is a training-free method that extracts the refusal direction from English prompts and clamps it onto the residual stream at inference time, eliminating the need for labeled target-language data or retraining
Analysis
TL;DR
- Instruction-tuned models refuse harmful requests in English but comply in low-resource African languages (Yoruba, Igbo, Igala, Hausa), indicating the refusal mechanism exists in the residual stream but fails to activate for low-resource inputs
- Latent Space Refusal Anchoring (LSR-Anchoring) is a training-free method that extracts the refusal direction from English prompts and clamps it onto the residual stream at inference time, eliminating the need for labeled target-language data or retraining
- Mean-Activation Steering (MAS) successfully recovers safety on Mistral-7B and Qwen2.5-7B with benign degradation below 0.08, but overcorrects on Llama-3-8B (DPL reaching 1.00)
- SAE-Derived Steering (SDS) resolves overcorrection by replacing dense mean-difference directions with a single Sparse Autoencoder feature, reducing KL divergence by 3.5-7x without benign collapse
- Arabic fails to transfer on every architecture at every steering magnitude, revealing a geometric mismatch rather than a baseline data effect, while MMLU accuracy drops remain below 0.35 percentage points
Why It Matters
This work addresses a critical safety gap in multilingual AI systems: instruction-tuned models exhibit inconsistent refusal behavior across languages, creating exploitable vulnerabilities in low-resource languages where safety training data is scarce. For AI practitioners deploying models globally, this training-free approach offers a practical pathway to enforce consistent safety without costly retraining or language-specific data collection, which is especially vital for African languages that lack the annotation infrastructure available for high-resource languages.
Technical Details
- LSR-Anchoring framework: A training-free intervention that operates by extracting the refusal direction from English prompts and projecting/clamping it onto the residual stream during inference, bypassing the need for any target-language labeled data or fine-tuning
- Two steering variants: Mean-Activation Steering (MAS) computes a dense mean-difference direction across the residual stream; SAE-Derived Steering (SDS) replaces this with a single Sparse Autoencoder (SAE) feature, yielding more targeted and less disruptive intervention
- Architectures evaluated: Llama-3-8B, Llama-3.1-70B, Mistral-7B-Instruct, and Qwen2.5-7B, tested across four African languages (Yoruba, Igbo, Igala, Hausa) and Arabic as a control
- Key metrics: Safety recovery measured alongside Degraded Performance on Legitimate prompts (DPL), KL divergence, and MMLU accuracy drops, demonstrating that safety gains come with minimal utility cost (<0.35 pp MMLU drop)
- Geometric transfer failure: Arabic's consistent failure across all architectures and steering magnitudes suggests the method's effectiveness depends on latent-space geometric alignment between source and target languages, not merely data availability
Industry Insight
- Training-free mechanistic interventions like LSR-Anchoring could become a standard layer in multilingual deployment pipelines, allowing organizations to patch safety gaps in low-resource languages without the compute and data costs of retraining
- The Arabic failure case serves as a cautionary signal: geometric mismatch may affect other language pairs beyond African languages, suggesting practitioners should validate steering transfer across language families before relying on cross-lingual safety transfer
- The integration of SAE features for steering (SDS) demonstrates that interpretability tools can directly enable safer deployment; investing in SAE research for production models may yield dual benefits of both mechanistic understanding and actionable safety interventions
Disclaimer: The above content is generated by AI and is for reference only.