TalkFa: A Unified Benchmark for Farsi Dialogue Generation and Understanding
TalkFa is introduced as the first unified benchmark for Farsi dialogue generation and understanding, addressing a critical gap for a language spoken by over 120 million people The benchmark comprises three datasets: Wiki-FaDial (4.2K knowledge-grounded dialogues), DailyDialog-Fa (6.6K dialogues with dialogue act and emotion annotations), and PlayDial-Fa (2.1K theatrical dialogues with sentiment labels) All dialogues undergo multi-stage review by native Farsi speakers, ensuring high-quality human
Analysis
TL;DR
- TalkFa is introduced as the first unified benchmark for Farsi dialogue generation and understanding, addressing a critical gap for a language spoken by over 120 million people
- The benchmark comprises three datasets: Wiki-FaDial (4.2K knowledge-grounded dialogues), DailyDialog-Fa (6.6K dialogues with dialogue act and emotion annotations), and PlayDial-Fa (2.1K theatrical dialogues with sentiment labels)
- All dialogues undergo multi-stage review by native Farsi speakers, ensuring high-quality human-approved data despite LLM-assisted construction
- LoRA fine-tuning on LLaMA and Mistral models substantially improves dialogue generation, achieving over 90% of final performance gains with only 25-50% of training data
- Automatic metrics substantially overestimate dialogue quality compared to human evaluation, and zero-shot frontier LLMs still struggle, confirming TalkFa as a challenging benchmark
Why It Matters
This benchmark addresses a significant resource gap in low-resource NLP by providing high-quality, human-validated Farsi dialogue data, enabling the research community to develop and evaluate models for one of the world's most spoken languages. The finding that automatic metrics overestimate dialogue quality serves as a critical caution for practitioners relying solely on automated evaluation. The efficient fine-tuning results with LoRA demonstrate that resource-constrained settings can still achieve strong performance, which is highly relevant for low-resource language communities.
Technical Details
- Dataset Composition: Three complementary datasets totaling 12.9K dialogues—Wiki-FaDial (4.2K Wikipedia-grounded dialogues for knowledge-grounded generation), DailyDialog-Fa (6.6K daily conversations annotated for dialogue acts and emotions), and PlayDial-Fa (2.1K theatrical dialogues with sentiment labels)
- Data Quality Pipeline: LLMs assist in initial data construction, but every dialogue undergoes multi-stage review and revision by native Farsi speakers; only human-approved final versions are released
- Model Experiments: Six LLaMA and Mistral models evaluated; LoRA fine-tuning shows substantial improvements in dialogue generation with only 25-50% of training data recovering over 90% of final performance gains
- Task-Specific Best Performers: Fabert achieves best dialogue-act classification, LoRA-Mistral-7B leads emotion recognition, and Mistral-24B achieves highest sentiment classification scores
- Evaluation Insights: Human evaluation and independent external validation confirm benchmark reliability; GPT-4.1 as LLM judge reveals automatic metrics substantially overestimate dialogue quality; zero-shot evaluation with frontier LLMs demonstrates the benchmark remains challenging
Industry Insight
- The significant overestimation by automatic metrics when compared to human judgment and LLM judges highlights the urgent need for human-in-the-loop evaluation in low-resource language settings, where automated tools may be less reliable
- The LoRA efficiency findings suggest that organizations working with low-resource languages can achieve strong dialogue performance with minimal data and computational resources, lowering barriers to entry for multilingual AI development
- The release of all datasets, annotation guidelines, code, and checkpoints sets a transparency standard that the community should follow, particularly for underrepresented language projects aiming to build trust and enable reproducibility
Disclaimer: The above content is generated by AI and is for reference only.