Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't
Representations align strongly (ridge R²=0.84) across Transformer sizes while raw parameters align weakly, indicating transfer value resides in initialization rather than weight structure Dense weight projection for size conversion is functionally destructive because basis mixing breaks rotary embeddings, per-head attention, GELU, and LayerNorm structure Conversion decomposes into two independent levers: least-squares compensation (optimizing zero-shot function) and variance-preserving rescale (
Analysis
TL;DR
- Representations align strongly (ridge R²=0.84) across Transformer sizes while raw parameters align weakly, indicating transfer value resides in initialization rather than weight structure
- Dense weight projection for size conversion is functionally destructive because basis mixing breaks rotary embeddings, per-head attention, GELU, and LayerNorm structure
- Conversion decomposes into two independent levers: least-squares compensation (optimizing zero-shot function) and variance-preserving rescale (optimizing training dynamics)
- At low continued pre-training budgets (30M tokens), the method significantly outperforms subcloning variants and from-scratch training, but converges to parity with subcloning at ~33x larger budgets
- The approach breaks down at ~5x donor scale (6.9B→1.4B) due to ill-conditioning of the compensation solve, suggesting dimension-aware regularization as a remedy
Why It Matters
This work directly challenges the industry assumption that model families must train every size from scratch, offering a token-efficient pathway to derive smaller models from existing large pretrained weights. For practitioners constrained by compute budgets, the finding that conversion initialization beats from-scratch training by up to 18x at low budgets provides a practical strategy for deploying smaller variants without full retraining.
Technical Details
- Representation vs. parameter alignment: Ridge regression across Pythia sizes shows R²=0.84 for hidden representations but weak parameter-space alignment, confirming that semantic content transfers even when weights do not directly correspond.
- Dense weight projection failure: A provable result shows that naively projecting weights between sizes breaks structural invariants (rotary position embeddings, per-head attention, GELU activation, LayerNorm), making it functionally destructive rather than an artifact of model assembly.
- Two-lever conversion framework: (1) Least-squares compensation maps donor weights to minimize zero-shot loss on the target architecture; (2) variance-preserving rescale maintains activation statistics for stable continued pre-training dynamics.
- Empirical results on Pythia 1.4B→410M: At 30M tokens, compensation beats subcloning on width-reduced pairs (84.0±1.8 vs. 89.7±3.7) and depth-reduced pairs (109.3 vs. 117.9), both across 3/3 seeds. At 33× budget, both methods reach parity (40.0 vs. 40.0), still far ahead of from-scratch.
- Scaling boundary: At ~5× donor scale (6.9B→1.4B), stacking both levers causes over-correction traced to ill-conditioning of the least-squares solve at large width, pointing to dimension-aware regularization as the fix.
Industry Insight
- Organizations maintaining model families should consider conversion-based initialization for smaller variants rather than training each size independently, especially when compute for continued pre-training is limited—low-budget conversion delivers disproportionate gains.
- The ill-conditioning observed at large scale gaps (~5×) suggests that direct weight conversion has practical limits; investing in dimension-aware regularization could extend the approach to more aggressive size reductions and larger donor models.
- The decomposition into functional (compensation) and dynamical (rescale) levers provides a modular framework that can be independently improved, offering a clear research roadmap for enhancing transfer efficiency between arbitrarily sized Transformers.
Disclaimer: The above content is generated by AI and is for reference only.