DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents
Introduces DuplexSpeechBench-IFEval (DSB-IFEval), a benchmark with 1,038 test cases across eight assistant roles to evaluate implicit instruction-following in real-time full-duplex voice agents Proposes two evaluation metrics: Instruction Adherence Score (IAS) for real-time floor management and Persona Adherence Score (PAS) for persona-consistent content judged by LLMs Full-duplex models (F-Actor, PersonaPlex) show significant adherence drops under persona-only conditioning (9.7% and 4.5%), whil
Analysis
TL;DR
- Introduces DuplexSpeechBench-IFEval (DSB-IFEval), a benchmark with 1,038 test cases across eight assistant roles to evaluate implicit instruction-following in real-time full-duplex voice agents
- Proposes two evaluation metrics: Instruction Adherence Score (IAS) for real-time floor management and Persona Adherence Score (PAS) for persona-consistent content judged by LLMs
- Full-duplex models (F-Actor, PersonaPlex) show significant adherence drops under persona-only conditioning (9.7% and 4.5%), while GPT-Realtime, MiniCPM-o, and Fun-Audio-Chat maintain persona consistency but lack adaptive floor behavior
- Systems reliably follow conflicting directives aligned with their persona but struggle to override them under safety conflicts
- Inferring role-implied behavior, executing it at the right conversational moment, and resolving competing instructions remain three distinct, unsolved challenges for full-duplex voice agents
Why It Matters
This benchmark addresses a critical gap in evaluating real-time spoken dialogue systems, where most existing tests rely on explicit turn-management instructions rather than the persona-driven configurations used in production. For AI practitioners building voice agents, the findings reveal that architectural choices create fundamentally different trade-offs between explicit instruction following and implicit persona adherence, directly impacting how systems should be designed and evaluated for real-world deployment.
Technical Details
- DSB-IFEval comprises 1,038 test cases spanning eight diverse assistant roles, evaluating five conditioning protocols: default behavior, explicit behavioral instructions, persona-implied behavior, combined persona-rule conditioning, and instruction conflict
- Two core metrics: Instruction Adherence Score (IAS), a deterministic measure of real-time floor management (listening, backchanneling, interrupting, yielding), and Persona Adherence Score (PAS), an LLM-judged measure of persona-consistent content generation
- Six real-time speech systems evaluated: F-Actor, PersonaPlex, GPT-Realtime, MiniCPM-o, Fun-Audio-Chat, and one additional system
- Full-duplex architectures (F-Actor, PersonaPlex) are more sensitive to conditioning protocol changes, with measurable adherence drops when behavior must be inferred from persona rather than explicitly stated
- Systems show a safety override gap: even when they follow conflicting persona-aligned directives, they fail to override those directives when safety conflicts arise
Industry Insight
- Benchmark designers and agent developers should move beyond explicit instruction-based evaluation; real-world agents are configured through personas, and the 4.5–9.7% adherence drop under persona-only conditioning represents a significant deployment risk that current benchmarks fail to capture
- Architecture selection should be guided by the intended conditioning strategy: full-duplex models excel at real-time floor management but require explicit behavioral instructions, while other architectures maintain persona consistency but lack adaptive proactive behaviors—hybrid approaches may be necessary
- Safety override mechanisms remain a critical unresolved gap; systems that faithfully follow persona directives may propagate harmful behavior when those directives conflict with safety constraints, suggesting that safety guardrails must operate independently of persona conditioning rather than relying on the model to self-correct
Disclaimer: The above content is generated by AI and is for reference only.