The Symmetry That Breaks Neural Network Averaging
Naive weight averaging of independently trained neural networks often produces worse results than either model alone due to permutation symmetry Permutation symmetry creates m! equivalent solutions per hidden layer (e.g., ~10^1166 for a 512-unit layer), each representing a different column ordering of the same functional report Non-convexity is not an artifact of ReLU activations but a fundamental consequence of learning adaptive basis functions rather than fixed ones Model merging succeeds when
Analysis
TL;DR
- Naive weight averaging of independently trained neural networks often produces worse results than either model alone due to permutation symmetry
- Permutation symmetry creates m! equivalent solutions per hidden layer (e.g., ~10^1166 for a 512-unit layer), each representing a different column ordering of the same functional report
- Non-convexity is not an artifact of ReLU activations but a fundamental consequence of learning adaptive basis functions rather than fixed ones
- Model merging succeeds when weights are properly aligned (matching neuron permutations) and fails when naive element-wise averaging mixes incompatible parameter orderings
- The loss landscape contains multiple isolated basins separated by barriers, with the midpoint between symmetric minima exhibiting significantly higher loss (6.4× in the two-neuron example)
Why It Matters
This explains a fundamental limitation that directly impacts practical LLM engineering workflows including model soups, federated averaging, and ensemble methods. Understanding permutation symmetry helps practitioners recognize why seemingly simple weight averaging strategies fail and guides the development of alignment-aware merging techniques that can unlock the full potential of model combination.
Technical Details
- Permutation Symmetry: Swapping neuron parameters within a hidden layer produces identical network functions and loss values, creating m! equivalent minima for each m-unit layer
- Adaptive Basis Functions: Unlike polynomial regression with fixed bases (x, x², x³), neural networks learn both the basis functions and coefficients simultaneously, making the optimization landscape inherently non-convex
- Two-Neuron Example: Training on data generated from ground truth w=(2,6) with ReLU activations shows minima at (2,6) and (6,2) with MSE=0.093, while their midpoint (4,4) yields MSE=0.594 (6.4× worse)
- Weight Alignment: Naive averaging of Model A [(1.938, 6.081)] and Model B [(6.081, 1.938)] produces [(4.009, 4.009)] with MSE=0.600, but aligning B's permutations to A before averaging restores the original performance (MSE=0.092)
- Linear Mode Connectivity: The barrier between symmetric minima represents the same phenomenon studied in linear mode connectivity literature, quantified by the loss increase along the straight path between solutions
Industry Insight
- practitioners should implement permutation-aware alignment (e.g., optimal transport or greedy matching) before performing any weight averaging or model merging operations to avoid catastrophic performance degradation
- the exponential growth of equivalent solutions (m! per layer) means that even modest layer widths create astronomically large solution spaces, making naive ensemble methods fundamentally flawed without alignment
- model soup and federated averaging techniques should incorporate symmetry-breaking constraints or alignment protocols as standard preprocessing steps rather than treating weight averaging as a trivial operation
Disclaimer: The above content is generated by AI and is for reference only.