GLAN-QnA-KR: A Seedless Taxonomy-Driven Korean Instruction Corpus GLAN-QnA-KR：一种无种子分类驱动的韩语指令语料库

TL;DR

• GLAN-QnA-KR is a 303,581-row synthetic Korean instruction-QA corpus generated using Microsoft's Phi-3.5-MoE-instruct model via a seedless taxonomy-driven pipeline.
• The dataset covers 1,084 disciplines with English labels and Korean text, featuring a difficulty scale of 100-900 and high uniqueness with near-zero duplicate clusters.
• Rigorous contamination audits against major benchmarks (KMMLU, KoBEST, HAE-RAE-Bench) show minimal overlap, ensuring data integrity for training.
• It stands as the largest single-pipeline synthetic Korean instruction corpus verifiable on Hugging Face Hub under a seedless protocol.
• Released under OpenRAIL license, it provides an openly redistributable resource for advancing Korean NLP capabilities.

Why It Matters

This corpus addresses a critical gap in high-quality, large-scale synthetic data for Korean language models, which often lag behind English resources. By providing a rigorously audited, contamination-free dataset, it enables researchers to train more robust and accurate Korean LLMs without the risk of data leakage from evaluation sets. This facilitates better performance in downstream tasks such as instruction tuning and domain-specific knowledge acquisition.

Technical Details

• Generation Pipeline: Utilizes the GLAN synthesis pipeline with Microsoft's Phi-3.5-MoE-instruct as the producer model, generating data in December 2024.
• Dataset Structure: Contains 303,581 rows spanning 1,084 English-labeled disciplines, with Korean Q&A pairs, a median question length of 313 characters, and answer length of 1,098 characters.
• Uniqueness Metrics: Exact duplicates are extremely rare (1 in 303,581), and character-trigram near-duplicate clusters (Jaccard >= 0.9) are zero in a 5,000-sample probe.
• Contamination Audit: Tested against KMMLU, KoBEST, and HAE-RAE-Bench; maximum character-trigram Jaccard similarity was 0.163, with no test items exceeding 0.7. Multilingual-E5 cosine similarity showed only one item at >= 0.90 and none at >= 0.95.
• Licensing: Distributed under the OpenRAIL license, allowing for open redistribution and research use.

Industry Insight

• Data Quality over Quantity: The emphasis on low contamination and high uniqueness highlights the importance of rigorous auditing in synthetic data generation, setting a new standard for Korean NLP datasets.
• Scalability of Seedless Methods: The success of the seedless taxonomy-driven approach suggests that automated, scalable methods can produce high-quality instructional data without manual seed curation, reducing costs and increasing coverage.
• Resource Availability: As the largest verified synthetic Korean corpus, GLAN-QnA-KR will likely become a foundational resource for pre-training and fine-tuning Korean LLMs, accelerating progress in low-resource language AI.

摘要

GLAN-QnA-KR 是一个包含 303,581 行的合成韩语指令-问答语料库，通过无种子分类法驱动管道，利用微软的 Phi-3.5-MoE-instruct 模型生成。
该数据集涵盖 1,084 个学科，标签为英文，文本为韩语，难度等级在 100-900 之间，具有极高的唯一性，重复簇接近于零。
针对主要基准测试（KMMLU、KoBEST、HAE-RAE-Bench）进行的严格污染审计显示重叠极少，确保了训练数据的完整性。
它是 Hugging Face Hub 上可验证的最大单一管道合成韩语指令语料库，且遵循无种子协议。
该数据集以 OpenRAIL 许可证发布，为提升韩语 NLP 能力提供了可公开重新分发的资源。

深度分析

TL;DR

• GLAN-QnA-KR 是一个包含 303,581 行的合成韩语指令-问答语料库，通过无种子分类法驱动管道，利用微软的 Phi-3.5-MoE-instruct 模型生成。
• 该数据集涵盖 1,084 个学科，标签为英文，文本为韩语，难度等级在 100-900 之间，具有极高的唯一性，重复簇接近于零。
• 针对主要基准测试（KMMLU、KoBEST、HAE-RAE-Bench）进行的严格污染审计显示重叠极少，确保了训练数据的完整性。
• 它是 Hugging Face Hub 上可验证的最大单一管道合成韩语指令语料库，且遵循无种子协议。
• 该数据集以 OpenRAIL 许可证发布，为提升韩语 NLP 能力提供了可公开重新分发的资源。

重要性

该语料库解决了韩语语言模型在高规模、高质量合成数据方面的关键缺口，而韩语资源通常落后于英语资源。通过提供经过严格审计且无污染的数据集，它使研究人员能够训练更稳健、更准确的韩语大语言模型，而无需担心从评估集中发生数据泄漏。这有助于改善指令微调和领域特定知识获取等下游任务的性能。

技术细节

• 生成管道：使用 GLAN 合成管道，以微软的 Phi-3.5-MoE-instruct 作为生产者模型，于 2024 年 12 月生成数据。
• 数据集结构：包含 303,581 行，跨越 1,084 个带有英文标签的学科，包括韩语问答对，问题中位长度为 313 个字符，答案中位长度为 1,098 个字符。
• 唯一性指标：完全重复的情况极为罕见（303,581 行中仅 1 例），字符三元组近重复簇（