The Morphological Core of Dungan: A Two-Dialect Finite-State Model and a Multi-Genre Evaluation
A finite-state morphological analyzer for Dungan (a Sinitic language of Central Asia) was built using HFST, covering both the Gansu and Shaanxi dialect varieties Overt inflection is extremely rare: only 9.3% of recognized tokens in the encyclopaedic register carry an overt morphological marker, with just ten grammatical categories and near-absence of degree marking Ambiguity is sharply localized to two clitics (-di and -ni), with 78.1% of tokens receiving a single unambiguous analysis The gramma
Analysis
TL;DR
- A finite-state morphological analyzer for Dungan (a Sinitic language of Central Asia) was built using HFST, covering both the Gansu and Shaanxi dialect varieties
- Overt inflection is extremely rare: only 9.3% of recognized tokens in the encyclopaedic register carry an overt morphological marker, with just ten grammatical categories and near-absence of degree marking
- Ambiguity is sharply localized to two clitics (-di and -ni), with 78.1% of tokens receiving a single unambiguous analysis
- The grammatical core is effectively closed: 78–95% of analyzer failures are due to lexical gaps rather than missing morphological rules (at most 4.5% of failures)
- Held-out coverage (80–85%) matches or exceeds development coverage (73%), and morphology contributes only 5.2 percentage points beyond a bare stem list (67.4%), confirming that the open frontier of Dungan is lexical, not morphological
Why It Matters
This work demonstrates a rigorous, data-driven approach to quantifying the morphological complexity of a low-resource language, providing a template for how computational tools can validate and refine linguistic descriptions. For NLP practitioners working with morphologically sparse or under-resourced languages, it offers a concrete methodology for separating lexical from morphological coverage gaps—a distinction critical for prioritizing resource development efforts.
Technical Details
- The analyzer is implemented with HFST (Helsinki Finite-State Technology) and covers both major Dungan dialect groups: the Gansu variety (literary standard) and the Shaanxi variety
- Evaluation was conducted across three genres, with the encyclopaedic register serving as the primary benchmark for quantifying overt inflection frequency
- The model deliberately formalizes existing grammatical knowledge rather than proposing new descriptions, enabling systematic measurement of morphological properties on real corpora
- Coverage metrics show held-out performance (80–85%) is no lower than development performance (73%), indicating no significant overfitting
- A bare stem list without any morphological rules achieves 67.4% coverage, meaning the full morphological model adds only 5.2 percentage points, quantifying the actual contribution of morphology to token recognition
Industry Insight
- For low-resource language NLP, this study reinforces that investing in lexical coverage (comprehensive word lists, dictionaries) often yields higher returns than building complex morphological analyzers, especially for analytic languages
- The open release of the analyzer, sources, and evaluation scripts sets a reproducibility standard that other computational linguists can follow when developing resources for under-resourced languages
- The methodology of using a morphological analyzer as a "measuring instrument" to distinguish lexical gaps from morphological underspecification could be adapted as a diagnostic framework for evaluating and improving NLP pipelines across diverse language types
Disclaimer: The above content is generated by AI and is for reference only.