Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does
Hear2Act introduces a unified evaluation protocol with 480 persona-grounded scenarios to test whether prosodic cues in speech alter downstream task-oriented dialogue decisions Audio-capable LLMs show minimal improvement (14.6% to 15.3% optimal-solution rate) when relying solely on prosody-mediated feedback without explicit lexical signals Explicitly inferring concern status from audio and representing it in text before action selection boosts performance to 39.6%, approaching the 40.7% ceiling w
Analysis
TL;DR
- Hear2Act introduces a unified evaluation protocol with 480 persona-grounded scenarios to test whether prosodic cues in speech alter downstream task-oriented dialogue decisions
- Audio-capable LLMs show minimal improvement (14.6% to 15.3% optimal-solution rate) when relying solely on prosody-mediated feedback without explicit lexical signals
- Explicitly inferring concern status from audio and representing it in text before action selection boosts performance to 39.6%, approaching the 40.7% ceiling with ground-truth state
- Prosody matters primarily when lexical evidence is insufficient; raw audio input alone is unreliable for driving assistant actions without an intermediate textual representation
Why It Matters
This benchmark addresses a critical gap in evaluating spoken AI assistants by isolating the impact of prosody on decision-making, rather than treating perception and action as separate problems. For practitioners building voice-based agents, it reveals that simply adding audio input is insufficient—structured reasoning over prosodic signals is essential for reliable task performance.
Technical Details
- Hear2Act Benchmark: 480 persona-grounded scenarios with hidden user concerns and objectively verifiable outcomes; each scenario varies whether concerns are conveyed explicitly in words or primarily through prosody
- Three evaluation conditions: transcript-only access, audio access, and concern-state access, enabling direct comparison of how prosodic information influences downstream decisions
- Experimental setup: Two audio-capable LLMs evaluated under prosody-mediated feedback versus explicit concern inference with textual representation
- Key metric: Optimal-solution rate, measuring whether the assistant selects the correct next action given the user's hidden concern
- Findings: Prosody alone yields negligible gains (14.6% → 15.3%), but explicit inference from audio to text representation recovers most of the performance gap (39.6% vs. 40.7% ground truth)
Industry Insight
- Voice assistant developers should invest in intermediate reasoning layers that explicitly extract and represent prosodic information in text before action selection, rather than relying on end-to-end audio-to-action pipelines
- Benchmark design should decouple prosodic perception from task execution to accurately diagnose where failures occur in spoken dialogue systems
- The results suggest a practical architecture: audio encoder → prosodic feature extraction → textual concern representation → decision module, which can bridge the gap between perception and action in spoken AI assistants
Disclaimer: The above content is generated by AI and is for reference only.