Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation
Knowledge distillation (KD) gains below 5 percentage points in small-model function routing are statistically indistinguishable from seed variance, as per-seed standard deviations range from 2.8 to 48.7 percentage points Three of seven KD variants exhibit bimodal collapse, with 1 in 3–5 seeds dropping below 55% accuracy while others train normally; a fourth shows elevated variance Collapse manifests in distinct failure modes: wrong-function selection (ce_kd, ce_paraphrase) and a novel output-tru
Analysis
TL;DR
- Knowledge distillation (KD) gains below 5 percentage points in small-model function routing are statistically indistinguishable from seed variance, as per-seed standard deviations range from 2.8 to 48.7 percentage points
- Three of seven KD variants exhibit bimodal collapse, with 1 in 3–5 seeds dropping below 55% accuracy while others train normally; a fourth shows elevated variance
- Collapse manifests in distinct failure modes: wrong-function selection (ce_kd, ce_paraphrase) and a novel output-truncation mode (reasoning_kd), where the model generates reasoning but terminates before emitting a function name, achieving only 0.9% accuracy
- Only progressive_kd and rank_kd avoid collapse across observed seeds, maintaining sigma ≤ 3.9 percentage points
- Single-seed evaluation is fundamentally inadequate for detecting central failure modes in small-model KD; input enrichment gains also reverse under controlled multi-seed testing
Why It Matters
This work exposes a critical validity gap in small-model knowledge distillation research: single-seed evaluations routinely overstate KD gains and miss catastrophic bimodal failure modes entirely. For practitioners deploying distilled models in production function-routing scenarios, these findings mean that reported improvements may be noise rather than signal, and that certain KD variants carry hidden collapse risk that only multi-seed evaluation can surface.
Technical Details
- Setup: 740-instance healthcare API routing task using a 1.5B Qwen student model distilled from a 20B teacher, comparing eight KD variants against supervised cross-entropy baseline with 3–6 seeds per key configuration
- Bimodal collapse: Three KD variants (ce_kd, ce_paraphrase, reasoning_kd) show bimodal performance distributions; ce_kd and ce_paraphrase fail via wrong-function selection, while reasoning_kd exhibits a previously undocumented output-truncation failure mode (model emits reasoning tokens but terminates before function name output, yielding 0.9% accuracy)
- Stable variants: progressive_kd and rank_kd are the only configurations avoiding collapse across all observed seeds, with standard deviations capped at 3.9 percentage points
- Input enrichment reversal: A naive cross-split input enrichment strategy showed a +3.78 pp gain but reversed to −2.70 pp under controlled within-split multi-seed re-testing, demonstrating evaluation artifact susceptibility
- Variance magnitude: Per-seed standard deviations of 2.8–48.7 pp completely engulf any claimed KD gain below 5 pp, rendering most published small gains statistically meaningless
Industry Insight
- Multi-seed evaluation should be treated as a minimum standard for any small-model KD paper or production deployment claim; single-seed results are insufficient to distinguish signal from variance and may conceal catastrophic failure modes
- When selecting KD strategies for production function-routing systems, progressive_kd and rank_kd offer the only empirically verified robustness against bimodal collapse among the variants tested
- Input enrichment and other preprocessing tricks should be validated under within-split multi-seed protocols rather than cross-split designs, as evaluation methodology alone can reverse apparent performance gains
Disclaimer: The above content is generated by AI and is for reference only.