Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training
Transformer weight magnitudes follow a stable two-parameter Weibull distribution (shape k ≈ 1.2), with the scale parameter λ capturing nearly all training-induced variation A training-free, pre-training statistic—bigram conditional entropy D = H(next | prev)—predicts how much λ grows during training, governed by the law λ² − λ₀² = C₀(η) + C₁(η)(H_r − D)^0.59 After normalizing out per-learning-rate coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto a single curve
Analysis
TL;DR
- Transformer weight magnitudes follow a stable two-parameter Weibull distribution (shape k ≈ 1.2), with the scale parameter λ capturing nearly all training-induced variation
- A training-free, pre-training statistic—bigram conditional entropy D = H(next | prev)—predicts how much λ grows during training, governed by the law λ² − λ₀² = C₀(η) + C₁(η)(H_r − D)^0.59
- After normalizing out per-learning-rate coefficients, 23 runs spanning an order of magnitude in learning rate collapse onto a single curve with unit slope (R² = 0.941), outperforming direct per-η fits (R² ≈ 0.82)
- The convex exponent 0.59 is inherited from an independently measured data-side saturation relation, not directly fitted to the weight-growth curve
- Cross-corpus prediction fails for code (over-predicts), suggesting redundancy R is a second axis in a broader data-to-weight framework Φ(D, R, A, H)
Why It Matters
This work provides a rare pre-training, training-free predictor of post-training weight-scale behavior, enabling practitioners to estimate model capacity utilization and training dynamics before committing computational resources. The finding that diverse learning rates and architectures collapse onto a single predictive law suggests deep, universal structure in how data predictability shapes learned representations, offering a new lens for dataset selection and curriculum design.
Technical Details
- Weibull weight-scale model: Trained transformer weight magnitudes are summarized by a two-parameter Weibull distribution; the shape parameter k ≈ 1.2 is stable across layers and models, making the scale parameter λ the primary carrier of training-induced change.
- Bigram conditional entropy as predictor: The pre-training statistic D = H(next | prev) quantifies corpus predictability. The growth law λ² − λ₀² = C₀(η) + C₁(η)(H_r − D)^0.59 relates weight-scale expansion to the gap between a matched-budget shuffle baseline H_r and the corpus entropy D.
- Universal collapse across learning rates: Normalizing by per-η coefficients causes 23 runs across an order of magnitude in learning rate to collapse onto (H_r − D)^0.59 with unit slope (R² = 0.941), demonstrating that the functional form is learning-rate-invariant.
- Forward prediction accuracy: End-to-end self-validation recovers held-out within-family weight growth with 5.7% relative error. The law holds at both model-level and per-layer resolutions and across two tested architectures, with only coefficients changing.
- Boundary and extension: Cross-corpus prediction over-predicts on code corpora, implicating data redundancy as a second axis and motivating a broader framework Φ(D, R, A, H) where R = redundancy and A = architecture.
Industry Insight
- Pre-training corpus entropy can serve as a cheap proxy for estimating training intensity and weight-scale outcomes, enabling better budget allocation and early stopping decisions without running full training trials.
- The universal collapse across learning rates suggests that hyperparameter tuning may be decoupled from fundamental data-to-weight scaling laws, pointing toward more robust, transferable training strategies.
- The failure of cross-corpus prediction for code highlights that redundancy-aware dataset curation is essential; future work extending Φ(D, R, A, H) could enable more accurate cross-domain generalization estimates and guide mixed-corpus training decisions.
Disclaimer: The above content is generated by AI and is for reference only.