Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Converted 21 of 28 full-attention layers in Qwen3-0.6B-Base to KDA (Kimi Delta Attention) linear-attention layers on a single consumer GPU, revealing a critical "interface injury" where perplexity improved but multiple-choice accuracy collapsed to near-random (25-29% vs. 50.6%) Diagnosed the failure using a four-permutation diagnostic that rotates answer options while holding content fixed, discovering the model predict "A" 81% of the time rather than engaging with answer content Repaired the in
Analysis
TL;DR
- Converted 21 of 28 full-attention layers in Qwen3-0.6B-Base to KDA (Kimi Delta Attention) linear-attention layers on a single consumer GPU, revealing a critical "interface injury" where perplexity improved but multiple-choice accuracy collapsed to near-random (25-29% vs. 50.6%)
- Diagnosed the failure using a four-permutation diagnostic that rotates answer options while holding content fixed, discovering the model predict "A" 81% of the time rather than engaging with answer content
- Repaired the interface with a 1,000-step format-targeted completion-only KL distillation stage, recovering +12.48 points on C-Eval and halving label-stickiness, followed by persona SFT and on-policy DPO to preserve gains
- Identified a silent FP32-master failure mode where bf16 optimizer updates were swallowed, highlighting precision-mismatch pitfalls in low-budget distillation pipelines
- Released full code, weights, recipes, and audit trail, demonstrating that standard distillation metrics can mask catastrophic interface-level failures
Why It Matters
This work exposes a dangerous blind spot in knowledge distillation: models can appear healthy by perplexity while suffering from interface injuries that destroy functional performance. For practitioners converting attention mechanisms or compressing models, it demonstrates that standard evaluation metrics are insufficient and that diagnostic techniques like option-rotation permutation tests are essential for catching latent failure modes before deployment.
Technical Details
- Architecture conversion: Replaced 21 of 28 full-attention layers in Qwen3-0.6B-Base with KDA (Kimi Delta Attention) linear-attention layers, targeting a single consumer-grade GPU budget
- Distillation pipeline: Used hidden-state alignment and end-to-end KL distillation to train the student, which successfully reduced perplexity gap but failed to transfer multiple-choice reasoning capability
- Four-permutation diagnostic: Rotated answer options (A/B/C/D) across four permutations while keeping answer content constant, revealing that 106 of 161 questions maintained the same label under all rotations and the model predicted "A" in 81% of cases
- Repair strategy: Applied a 1,000-step format-targeted completion-only KL distillation stage focused on the answer interface, followed by persona SFT and one round of on-policy DPO to stabilize and preserve benchmark performance
- Engineering failure mode: Discovered that FP32 master weights with bf16 optimizer updates caused silent swallowing of gradient updates, a precision-mismatch bug that threatened convergence
Industry Insight
- When compressing or modifying transformer architectures, always pair perplexity metrics with interface-level diagnostics; a model can distill well on next-token prediction while completely failing at task-specific reasoning
- Option-rotation permutation tests offer a low-cost, high-signal diagnostic for detecting label-dependence failures in multiple-choice benchmarks, and should become standard practice in model evaluation pipelines
- Precision mixing (FP32 master weights with bf16 optimizers) requires explicit validation in low-budget training setups, as silent update swallowing can go undetected until functional benchmarks reveal catastrophic failure
Disclaimer: The above content is generated by AI and is for reference only.