Robust Critics: Defending LLMs Against Multi-Turn Attacks
Introduces Dialogue Critic Guided Sampling (DCGS), a framework that infers user intent dynamically across multi-turn dialogues rather than relying on static safety rules. Models adversarial dialogue as a Markov Decision Process, employing value and regret-based critics at both token and utterance levels to score candidate responses. Provides theoretical proof that inference-time reweighting approximates exponential tilting of the base policy, guaranteeing improved expected return over group-rela
Analysis
TL;DR
- Introduces Dialogue Critic Guided Sampling (DCGS), a framework that infers user intent dynamically across multi-turn dialogues rather than relying on static safety rules.
- Models adversarial dialogue as a Markov Decision Process, employing value and regret-based critics at both token and utterance levels to score candidate responses.
- Provides theoretical proof that inference-time reweighting approximates exponential tilting of the base policy, guaranteeing improved expected return over group-relative objectives.
- Demonstrates superior robustness against jailbreaks on benchmarks like CARES-18k, WildJailbreak, Redbench, and Harmbench compared to frontier models.
- Achieves transferability to existing frontier models without requiring fine-tuning, enhancing their resilience to multi-turn attacks.
Why It Matters
This research addresses a critical gap in LLM safety by moving beyond single-turn contextual bandit approaches to handle the complexity of evolving intents in multi-turn conversations. For practitioners, it offers a method to distinguish between malicious exploitation and genuine misunderstanding, reducing false positives for legitimate users while blocking sophisticated attacks. The ability to improve robustness in frontier models without fine-tuning provides a cost-effective pathway for deploying safer AI systems in production environments.
Technical Details
- Framework: Dialogue Critic Guided Sampling (DCGS) dynamically infers user intent at every turn based on full conversational history.
- Modeling: Adversarial dialogue is formalized as a Markov Decision Process (MDP).
- Critics: The system learns value and regret-based critics at two granularities: individual tokens and full utterances (responses).
- Mechanism: Candidate responses are scored via an action-value critic, and inference-time reweighting is used to approximate exponential tilting of the base policy.
- Evaluation: Tested on CARES-18k, WildJailbreak, Redbench, and Harmbench, showing state-of-the-art performance in adversarial dialogue tasks.
Industry Insight
- Shift from Static to Dynamic Safety: Organizations should move away from rigid keyword or rule-based filters for multi-turn interactions, adopting intent-aware frameworks that consider conversation trajectory.
- Zero-Shot Robustness Enhancement: The transferability of DCGS to frontier models suggests that safety improvements can be applied as a layer or sampling strategy without the computational overhead of retraining large models.
- Theoretical Guarantees in Practice: The proof regarding exponential tilting offers a stronger mathematical foundation for safety interventions compared to heuristic group-relative objectives, encouraging deeper integration of reinforcement learning theory into safety pipelines.
Disclaimer: The above content is generated by AI and is for reference only.