AVERT: Audio-Verified Adjudication for Spoken Dialogue State Tracking
AVERT introduces an audio-verified adjudication framework for spoken dialogue state tracking that corrects ASR errors without retraining base models Three targeted operators—vote, add, and swap—address inconsistent cross-turn values, omitted slots, and audio-inconsistent predictions respectively On SpokenWOZ, AVERT achieves 40.13 JGA, outperforming both a base speech-LLM (33.04) and a text-only editor (38.34), matching a 1B end-to-end system (39.32) The audio-conditioned verifier provides a stat
Analysis
TL;DR
- AVERT introduces an audio-verified adjudication framework for spoken dialogue state tracking that corrects ASR errors without retraining base models
- Three targeted operators—vote, add, and swap—address inconsistent cross-turn values, omitted slots, and audio-inconsistent predictions respectively
- On SpokenWOZ, AVERT achieves 40.13 JGA, outperforming both a base speech-LLM (33.04) and a text-only editor (38.34), matching a 1B end-to-end system (39.32)
- The audio-conditioned verifier provides a statistically significant gain, and restricting operators to error-prone slot subsets prevents overwriting correct categorical values
Why It Matters
Spoken dialogue systems remain bottlenecked by ASR errors that propagate across conversation turns, and this work demonstrates that post-hoc adjudication with audio verification can close a significant performance gap without expensive retraining. For practitioners building conversational AI, AVERT offers a plug-and-play improvement layer that approaches end-to-end system performance using modular components, making it a practical pathway for incremental deployment.
Technical Details
- AVERT operates as a post-processing adjudication layer on top of existing speech-LLM outputs, combining cross-turn agreement scoring with a trained audio-conditioned verifier to evaluate candidate slot values
- Three error-specific operators are deployed: "vote" resolves values predicted inconsistently across turns, "add" recovers omitted slots, and "swap" corrects values unsupported by the audio signal
- Each operator is restricted to a selected subset of slots where its corresponding error type is most common, preventing unintended overwrites of correct categorical values
- Evaluated on SpokenWOZ, AVERT uses two 1B-parameter decoders and achieves 40.13 JGA compared to 33.04 for the base speech-LLM and 38.34 for a text-only editor, without any retraining of the underlying models
Industry Insight
- Modular adjudication layers like AVERT provide a cost-effective alternative to training massive end-to-end spoken dialogue systems, enabling teams to incrementally improve existing pipelines without full retraining
- The finding that operator restriction to error-prone slot subsets is critical suggests that future systems should incorporate error-profile-aware gating rather than applying corrections uniformly across all slots
- Audio-verified correction represents a broader paradigm shift: rather than treating ASR errors as irrecoverable noise, spoken dialogue systems can leverage raw audio signals as an independent verification source to resolve ambiguities that text-only approaches cannot
Disclaimer: The above content is generated by AI and is for reference only.