Behavior Leverage Imbalance in Multi-Teacher On-Policy Distillation
Multi-teacher on-policy distillation for agentic LLMs can cause "behavior leverage imbalance," leading to over-calling of tools despite aggregate loss metrics appearing stable. Local token-level signals at structural entry points (e.g., tags, function names) exert disproportionate influence on global generation modes, a factor often missed by standard divergence monitoring. The proposed "Soft Clamp" method dynamically compresses extreme per-token Jensen-Shannon divergence while preserving gradie
Analysis
TL;DR
- Multi-teacher on-policy distillation for agentic LLMs can cause "behavior leverage imbalance," leading to over-calling of tools despite aggregate loss metrics appearing stable.
- Local token-level signals at structural entry points (e.g., tags, function names) exert disproportionate influence on global generation modes, a factor often missed by standard divergence monitoring.
- The proposed "Soft Clamp" method dynamically compresses extreme per-token Jensen-Shannon divergence while preserving gradients, effectively calibrating teacher signals.
- Empirical results on APIGen-MT show Soft Clamp reduces tool over-calling from 13.7% to 9.0% without sacrificing decision accuracy compared to vanilla Generalized Knowledge Distillation.
- The study highlights the necessity of monitoring the spatial distribution of teacher signals rather than relying solely on aggregate loss values during distillation training.
Why It Matters
This research addresses a critical failure mode in training agentic language models that rely on multi-teacher distillation, revealing that standard aggregate metrics can mask significant behavioral shifts like excessive tool usage. For AI practitioners, it provides a concrete solution (Soft Clamp) to improve the reliability and efficiency of tool-use agents by preventing redundant or incorrect tool calls, which is essential for deploying robust autonomous systems.
Technical Details
- Problem Analysis: The authors identify "behavior leverage imbalance" where vanilla generalized knowledge distillation (GKD) improves tool-call recall but induces over-calling. They demonstrate that aggregate losses and total token exposure do not adequately explain this shift.
- Mechanism: The core issue lies in local token-level signals at mode-entry and structural positions (such as
<tool>tags or function names) having disproportionate control over the global generation mode. - Proposed Solution: "Soft Clamp," a per-token divergence calibration method that dynamically compresses extreme token-level Jensen-Shannon divergence. Crucially, it preserves non-zero gradients to ensure learning continues.
- Evaluation Benchmarks: The method was tested on APIGen-MT and BFCL (Big Function Call Benchmark).
- Performance Metrics: On APIGen-MT, Soft Clamp reduced over-calling rates from 13.7% to 9.0% relative to vanilla GKD while maintaining equivalent decision accuracy. In BFCL multi-turn diagnostics, it also reduced tool-call loops and repeated calls.
Industry Insight
- Monitoring Granularity: AI teams implementing multi-teacher distillation should move beyond aggregate loss tracking and implement fine-grained, token-level divergence monitoring, particularly around structural tokens and mode-entry points.
- Agent Reliability: Reducing over-calling is vital for cost efficiency and latency in agentic workflows; techniques like Soft Clamp offer a path to more stable and predictable agent behavior without retraining from scratch.
- Distillation Strategy: When combining specialized teachers (e.g., one for tool use, one for direct response), explicit calibration of signal influence is necessary to prevent one teacher from dominating the policy in unintended ways.
Disclaimer: The above content is generated by AI and is for reference only.