Brain-LLM Alignment Tracks Training Data, Not Typology

Background

The research explores whether brain-LLM (large language model) alignment generalizes across languages beyond the current focus on English. Using functional magnetic resonance imaging (fMRI) data from participants and multiple language models, the study aims to understand the factors governing cross-linguistic differences in alignment.

Key Points

• Training-Language Dominance: The central finding is that training-language dominance drives the alignment pattern more than any inherent property of English. A Chinese-dominant model (Baichuan2-7B) aligned best with Chinese brains and worst with English.
• Formal Typological Distance: Formal typological distance independently correlates with alignment degradation, suggesting that models trained on languages further from the target language in terms of linguistic structure show poorer alignment.
• Regional Variations: Syntax-associated brain regions (IFG) showed a steeper typological gradient ($2.3\times$ times greater than lexico-semantic regions like PTL), indicating that syntactic processing is more sensitive to cross-linguistic variations.

Significance

1. Revealing Artifacts in Alignment: The study highlights that the "English advantage" observed in brain-LLM alignment is an artifact of training data composition rather than a universal characteristic.
2. Understanding Typological Structure: By identifying formal typological distance as a significant factor, the research suggests that models need to account for linguistic structure more comprehensively to achieve better cross-linguistic performance.
3. Implications for Model Development: These findings imply that future language model development should consider not only training data but also the linguistic properties of different languages, potentially leading to more universally effective models.

Key Insights

• The alignment patterns are driven by the training environment and less by the inherent characteristics of English or any other language.
• Syntax plays a crucial role in brain-LLM alignment, highlighting the importance of syntactic processing in model design.
• Tokenization fertility significantly influences optimal encoding layers, suggesting that text preprocessing techniques can impact cross-linguistic performance.

These results underscore the need for more nuanced and linguistically informed approaches to aligning language models with human brains across different languages.