Disentangling Language Roles in Multilingual LLM Task Execution

This paper is a welcome corrective to a lazy intuition in the multilingual AI space. We’ve tended to assume that a mismatch between instruction and response languages, or a soup of three different languages, creates a uniform difficulty gradient. The thinking goes: more mismatches, worse performance. This work dismantles that assumption with elegant precision. By constructing MTM-Bench with its fully crossed 27-triplet design, the researchers isolate the signal from the noise. What they found is that the model’s struggle isn’t a generic "confusion" but a structured vulnerability. When a model is asked to think in Spanish based on Chinese content but produce an English response, the bottleneck isn’t the multilingual juggling act itself—it’s the specific, high-stakes task of formulating and adhering to the output language constraint. A single mismatch in the response slot wreaks more havoc than a model grappling with two mismatches that don't involve the final output language.

This has profound implications for how we build and evaluate these systems. Current multilingual benchmarks often treat the task as a monolithic "follow the instruction in this language." MTM-Bench argues that we must dissect the task anatomy. The response language is not just another variable; it’s the execution target. When that target language conflicts with the model’s "internal" reasoning language (likely shaped by its dominant training data), the system faces a core tension: optimizing for semantic accuracy versus obeying the stylistic and linguistic format constraint. The study shows these can decouple. A model might correctly understand a semantic reversal task (get the logic right) while utterly failing to deliver the answer in the requested language, thus failing the joint success metric. This reveals a critical weakness: many models are competent translators of thought into a lingua franca (often English) but poor at executing the final, crucial step of constrained, monolingual output when that constraint conflicts with their default mode.

The finding that mismatch count is not a monotonic predictor of difficulty is particularly insightful. It suggests that simply increasing the number of mismatched languages in a test set, as some benchmarks do, doesn’t linearly scale difficulty. The specific configuration of mismatches creates discrete challenge profiles. This means a model could appear robust on a mixed-language test while harboring a critical, silent failure mode when pushed to output in a less-resourced language. It also explains why model leaderboards can shuffle when benchmarks change—the models aren’t uniformly better or worse; they have different fault lines along these structural roles.

Finally, the observation that "task families fail through distinct channels" is a call to move beyond aggregate scores. Semantic correctness alone is a hollow victory if the output is in the wrong language. Conversely, perfect language adherence with flawed logic is useless. The reliable execution this benchmark measures—success that requires both correct understanding and correct formatting—is the real target for usable, global AI systems. This paper pushes us to see multilingual capability not as a blanket proficiency but as a series of specific, interlocking competencies, where the final act of speaking correctly is where the performance most often, and most critically, breaks down.