DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance
DUET introduces a token-selective on-policy distillation method that addresses runtime-injected prohibition compliance in LLM deployments, where policies vary per request and per tenant The method pairs two identical-weight teachers—one that sees the prohibition (positive) and one that doesn't (negative)—using their per-token disagreement to isolate the causal effect of the prohibition Two complementary mechanisms drive DUET: signal cleaning (discarding agreement tokens as redundant or prefix-co
Analysis
TL;DR
- DUET introduces a token-selective on-policy distillation method that addresses runtime-injected prohibition compliance in LLM deployments, where policies vary per request and per tenant
- The method pairs two identical-weight teachers—one that sees the prohibition (positive) and one that doesn't (negative)—using their per-token disagreement to isolate the causal effect of the prohibition
- Two complementary mechanisms drive DUET: signal cleaning (discarding agreement tokens as redundant or prefix-corrupted) and preference-directed learning (pushing the student toward the positive teacher at token granularity, embedding DPO-style optimization directly into on-policy distillation)
- Evaluated across 1.5B-8B Qwen variants, DUET achieves 72.3-85.2% violation compliance while preserving 88-93% normal utility, outperforming both teacher models and other distillation baselines
- External evaluation on SysBench confirms improved safety alignment with minimal degradation on GSM8K and MATH-500
Why It Matters
Conventional post-training approaches like SFT and DPO are structurally ill-suited for runtime-injected prohibitions because they either bury violation signals in compliant labels or mismatch sequence-level preferences against token-localized violations. DUET provides a practical pathway for enterprises to enforce dynamic, per-tenant compliance policies without retraining from scratch, directly addressing a critical gap between research-grade alignment and production-grade governance.
Technical Details
- Dual-Teacher Architecture: DUET uses two identical-weight teacher models that differ only in prohibition visibility—one receives the prohibition context (positive teacher) and one does not (negative teacher). This controlled difference ensures that per-token disagreement isolates the prohibition's causal effect without contamination from model capacity or architectural mismatch.
- Signal Cleaning Mechanism: Tokens where both teachers agree are discarded as either redundant or corrupted by prefix effects, leaving only the disagreement tokens that carry the prohibition-specific signal for distillation.
- Preference-Directed Learning: At token granularity, the student is pushed away from the negative teacher's distribution and toward the positive teacher's, effectively embedding DPO-style pairwise optimization directly into on-policy distillation without requiring offline preference datasets.
- Benchmark: The authors construct an industrial Prohibition-Compliance benchmark spanning five task families covering explicit-refusal, paraphrase robustness, and over-refusal scenarios.
- Results: Across Qwen 1.5B-8B variants, DUET achieves 72.3-85.2% violation compliance with 88-93% normal utility preservation. SysBench evaluation shows improved safety alignment with minimal performance drop on GSM8K and MATH-500.
Industry Insight
- Runtime-injected prohibitions are becoming a deployment reality for enterprise LLMs; methods like DUET that enable compliance without full retraining will be essential for multi-tenant platforms where policies change frequently across customers.
- The token-level granularity of DUET's approach suggests a broader shift from sequence-level alignment (DPO, RLHF) toward finer-grained, context-aware compliance mechanisms that can adapt to dynamic constraints at inference time.
- The strong utility preservation (88-93%) demonstrates that prohibition compliance need not come at the cost of general capability, making this approach viable for production systems where both safety and performance are non-negotiable.
Disclaimer: The above content is generated by AI and is for reference only.