When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
Continual learning (catastrophic forgetting) and model merging (weight-disentanglement error) are unified under a single concept: task interference, where a parameter update for one task shifts outputs on another. Task interference is formalized as a layer-wise Frobenius inner product ⟨ΔW_ℓ, J_ℓ(x)⟩_F, with a theoretical upper bound isolating the spectral norm ||ΔW_ℓ||_2 as an optimizer-controllable factor. The Muon optimizer is identified as a mechanism that intrinsically regulates spectral nor
Analysis
TL;DR
- Continual learning (catastrophic forgetting) and model merging (weight-disentanglement error) are unified under a single concept: task interference, where a parameter update for one task shifts outputs on another.
- Task interference is formalized as a layer-wise Frobenius inner product ⟨ΔW_ℓ, J_ℓ(x)⟩_F, with a theoretical upper bound isolating the spectral norm ||ΔW_ℓ||_2 as an optimizer-controllable factor.
- The Muon optimizer is identified as a mechanism that intrinsically regulates spectral norm, thereby tightening the task interference bound for both CL and MM.
- Replacing AdamW with Muon yields up to +5.02 accuracy points on an eight-task model-merging benchmark across three CLIP backbones.
- Muon delivers uniformly positive gains across ten class-incremental protocols, three task-incremental protocols, and the 11-task MTIL benchmark for continual learning.
Why It Matters
This work reframes two major challenges in multi-task AI—catastrophic forgetting and model merging incompatibility—as manifestations of the same underlying phenomenon, offering a unified theoretical lens. For practitioners, it demonstrates that optimizer choice is not merely an implementation detail but a principled lever for controlling task interference, with the Muon optimizer providing measurable gains across diverse benchmarks.
Technical Details
- Task Interference Formalization: The paper defines task interference as a layer-wise Frobenius inner product between the weight update ΔW_ℓ and the Jacobian J_ℓ(x) of layer outputs, capturing how updates for one task perturb performance on another.
- Spectral Norm Bound: A theoretical upper bound is derived that isolates the spectral norm ||ΔW_ℓ||_2 as the key optimizer-controllable factor governing task interference; per-mode analysis confirms this bound tracks the dominant empirical interference.
- Muon as Spectral Regulator: Muon is shown to regulate spectral norm by construction, making it a principled optimizer-centric solution that tightens the interference bound without requiring task-specific modifications.
- Experimental Validation: Benchmarks include an eight-task model-merging setup across three CLIP backbones (up to +5.02 accuracy over AdamW), plus continual learning evaluations across ten class-incremental protocols, three task-incremental protocols, and the 11-task MTIL benchmark with uniformly positive gains.
Industry Insight
- Optimizer selection should be treated as a strategic decision in multi-task and continual learning pipelines, not a hyperparameter afterthought; Muon's spectral norm control offers a plug-and-play improvement over AdamW.
- The unified task interference framework suggests that future research on model merging and catastrophic forgetting can share theoretical tools and solutions, accelerating progress across both subfields.
- For production systems deploying merged models or incremental learning, adopting Muon could yield immediate accuracy gains with minimal engineering overhead, particularly in vision-language settings like CLIP.
Disclaimer: The above content is generated by AI and is for reference only.