Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning
APM (Activation-Prune-Merge) enables training-free cross-scale model fusion by using activation-guided pruning to extract salient components from a large donor model and inject them into a smaller recipient The method eliminates the need for explicit neuron-wise semantic alignment by treating the donor as a source of concentrated functional components rather than requiring precise structural transplantation APM improved a 3B recipient model's average accuracy from 55.5% to 60.6% across 16 benchm
Analysis
TL;DR
- APM (Activation-Prune-Merge) enables training-free cross-scale model fusion by using activation-guided pruning to extract salient components from a large donor model and inject them into a smaller recipient
- The method eliminates the need for explicit neuron-wise semantic alignment by treating the donor as a source of concentrated functional components rather than requiring precise structural transplantation
- APM improved a 3B recipient model's average accuracy from 55.5% to 60.6% across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification
- Significant per-benchmark gains were observed: RTE accuracy jumped from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%
- Analysis of injection ratios and sequential multi-stage fusion demonstrates that activation-guided extraction preserves the small-ratio fusion regime while improving transfer quality
Why It Matters
This work addresses a critical gap in heterogeneous model fusion by demonstrating that cross-scale knowledge transfer is viable without explicit semantic alignment, reducing the computational and engineering overhead typically required for model merging. For AI practitioners working with resource-constrained deployments, APM offers a practical pathway to enhance small models using larger counterparts without expensive fine-tuning or architectural reengineering.
Technical Details
- APM Framework: Constructs task-conditioned activation maps on the donor model, then selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune the donor down to match the recipient architecture before injecting it via a micro interpolation coefficient
- Cross-Scale Fusion: The donor is treated as a source of concentrated functional components; truncation to smaller architecture combined with tiny mixing weights already yields improvements over the baseline recipient
- Evaluation: Tested across 16 benchmarks covering reasoning, mathematics, code generation, instruction following, and classification tasks, with the recipient being a 3B parameter model
- Multi-Stage Fusion: Sequential multi-stage fusion analysis showed that activation-guided extraction improves the quality of transferable donor slices while maintaining small injection ratios
- Key Finding: Cross-scale heterogeneous fusion succeeds without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected through activation-based pruning
Industry Insight
- The training-free nature of APM makes it highly attractive for production environments where fine-tuning large models is computationally prohibitive, enabling organizations to upgrade smaller deployed models using existing larger variants without additional training costs
- The activation-guided pruning approach could become a standard technique for model compression and knowledge distillation pipelines, particularly for edge and mobile deployments where model size constraints are critical
- The success of micro-interpolation coefficients suggests that future work should explore optimal injection strategies and multi-donor fusion scenarios, potentially enabling ensemble-like improvements without the inference latency of running multiple models
Disclaimer: The above content is generated by AI and is for reference only.