Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation
The paper evaluates prompt scope and demonstration selection as key variables in local LLM machine translation, focusing on English-to-Romance and English-to-Germanic translations. It compares three local instruction-tuned LLMs (llama3.2:3b, mistral:latest, qwen2.5:14b) against dedicated MT baselines using zero-shot and few-shot prompting with different demonstration selection strategies. Results show that dedicated MT systems remain stronger overall, especially for Germanic languages, while few
Analysis
TL;DR
- The paper evaluates prompt scope and demonstration selection as key variables in local LLM machine translation, focusing on English-to-Romance and English-to-Germanic translations.
- It compares three local instruction-tuned LLMs (llama3.2:3b, mistral:latest, qwen2.5:14b) against dedicated MT baselines using zero-shot and few-shot prompting with different demonstration selection strategies.
- Results show that dedicated MT systems remain stronger overall, especially for Germanic languages, while few-shot prompting benefits some LLMs but harms others.
- Family-scope prompting is feasible for larger LLMs but reveals structured-output failures in smaller models.
- The study emphasizes evaluating LLM translation by prompt scope, retrieval strategy, and multi-target compliance beyond traditional metrics.
Why It Matters
This research is crucial for AI practitioners and researchers working on localized or resource-constrained machine translation systems, as it provides insights into how prompt design and demonstration selection impact performance. It highlights the trade-offs between general-purpose LLMs and specialized MT models, guiding best practices for deploying LLMs in real-world translation scenarios where users may request multiple languages or example-based conditioning.
Technical Details
- Models Evaluated: llama3.2:3b, mistral:latest, qwen2.5:14b (local instruction-tuned LLMs) vs. OPUS-MT and NLLB-200 (dedicated MT baselines).
- Task: English-to-Romance and English-to-Germanic translation on the FLORES devtest split covering nine EU languages.
- Prompting Strategies: Zero-shot prompting, k=5 few-shot prompting with random, lexical-similarity, and embedding-similarity demonstration selection.
- Scope Comparison: Single-target prompts vs. JSON-formatted family-scope prompts requesting all languages in a family simultaneously.
- Key Findings: Embedding retrieval slightly outperforms other methods for stronger LLMs; family-scope prompting works well for larger models but causes output structure issues in smaller ones.
Industry Insight
- Organizations deploying LLMs for multilingual translation should prioritize model size and architecture when choosing between single-target and family-scope prompting to avoid structural errors.
- For cost-sensitive applications using smaller LLMs, dedicated MT systems like OPUS-MT or NLLB-200 may still offer superior accuracy, particularly for Germanic languages.
- Practitioners should experiment with embedding-based demonstration selection for stronger LLMs to marginally improve few-shot performance, though gains over simpler methods are modest.
Disclaimer: The above content is generated by AI and is for reference only.