What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts
Human-annotated dataset of 100 Quran recitation recordings with 162 localized error events across ten annotation labels, addressing the challenge of distinguishing mistakes from repetitions, repairs, and accepted spelling variations Executable evaluator jointly scores label correctness and word-position localization, achieving label-aware F1 of 0.525 and localization F1 of 0.826 on a plain diff baseline Pilot evaluation across three coding agents and eight models shows wide performance variance
Analysis
TL;DR
- Human-annotated dataset of 100 Quran recitation recordings with 162 localized error events across ten annotation labels, addressing the challenge of distinguishing mistakes from repetitions, repairs, and accepted spelling variations
- Executable evaluator jointly scores label correctness and word-position localization, achieving label-aware F1 of 0.525 and localization F1 of 0.826 on a plain diff baseline
- Pilot evaluation across three coding agents and eight models shows wide performance variance (label-aware F1 0.143 to 0.892), with seven of eight runs significantly outperforming baselines
- Core finding: detection is largely solved; the remaining challenge is conventional—span boundaries and label assignments are stipulated by adjudication rather than visible in the text
- Seven of 162 events defeat all six same-day runs, five due to a single orthographic rule, revealing that annotation interfaces must be validated before interpreting baseline failures
Why It Matters
This work addresses a critical gap in domain-specific ASR evaluation: how to systematically annotate and measure error types in specialized recitation tasks where accepted variations (spelling, repetition, repairs) must be distinguished from genuine mistakes. For AI practitioners building evaluation pipelines for niche linguistic domains, it demonstrates that baseline failures may reflect annotation interface issues rather than algorithmic limitations, and that what appears to be a detection problem may actually be a convention problem requiring human adjudication.
Technical Details
- Dataset: 100 production Quran recitation recordings, human-annotated with 348 scored units and 162 localized events across ten combined labels (mistakes, repetitions, repairs, opening formulas, accepted spelling differences)
- Evaluator: Executable joint scorer that evaluates both label correctness and word-position localization simultaneously, reporting label-aware F1, localization F1, and exact-span F1 metrics
- Baselines: Plain diff achieves label-aware F1 0.525 and localization F1 0.826; adapted production cleaner/alignment components reach 0.518 and 0.786 respectively, with exact-span F1 0.505 for both
- Pilot: Eight 20-minute runs across three coding agents and eight models, with label-aware F1 ranging from 0.143 to 0.892; seven runs significantly outperform baselines while one collapses below naive diff due to missing normalization
- Key insight: 970 of 972 gold-event instances draw overlapping predictions across runs, indicating detection is solved; the remaining 972-970=2 discrepancy and all span-boundary disagreements are conventional, not algorithmic
Industry Insight
- Annotation validation is prerequisite: Before interpreting baseline failures in specialized NLP tasks, validate that annotation interfaces correctly capture the ground truth—five of the seven unbeatable events were defeated by a single orthographic rule, suggesting interface issues rather than model limitations
- Convention over detection: In domain-specific evaluation tasks, once detection capability reaches a threshold, further gains require resolving conventional disagreements (span boundaries, label assignments) through human adjudication, not algorithmic improvements
- Pilot design matters: The wide performance variance (0.143 to 0.892 F1) across eight runs demonstrates that even with the same task definition, implementation choices (normalization steps, alignment components) dramatically affect results—standardized pilot protocols are essential for fair comparison
Disclaimer: The above content is generated by AI and is for reference only.