Hierarchical Copula-Gumbel-Top-K Routing: Two-Sided Dependence Control for Frozen Mixture-of-Experts at Fixed Per-Token Routing Laws
Introduces Hierarchical Copula-Gumbel-Top-K (CGA) routing, a two-sided dependence control mechanism for frozen Mixture-of-Experts (MoE) models that preserves individual token routing laws while enabling joint distribution control across tokens Uses an exchangeable Gaussian copula to positively correlate Gumbel perturbations within token groups, increasing within-group expert-set coherence Employs a tunable antithetic construction across disjoint group pairs to introduce negative dependence, redu
Analysis
TL;DR
- Introduces Hierarchical Copula-Gumbel-Top-K (CGA) routing, a two-sided dependence control mechanism for frozen Mixture-of-Experts (MoE) models that preserves individual token routing laws while enabling joint distribution control across tokens
- Uses an exchangeable Gaussian copula to positively correlate Gumbel perturbations within token groups, increasing within-group expert-set coherence
- Employs a tunable antithetic construction across disjoint group pairs to introduce negative dependence, reducing expert load variance
- Proves that both operations preserve each token's ordered Top-K sample, mixture weights, and inclusion probabilities identically to independent routing conditioned on pre-routing logits
- Validates the mechanism via a small-scale pilot; the frozen base model is evaluated only in the forward direction with gradients confined to a small trainable controller using a score-function estimator
Why It Matters
This work addresses a critical practical challenge in MoE models: balancing expert load dispersion against routing coherence without retraining the entire base model. For AI practitioners deploying large MoE systems, the ability to tune dependence structures via a lightweight controller offers a parameter-efficient path to managing expert utilization and reducing routing instability.
Technical Details
- Core Problem: Given fixed per-token routing laws in a stochastic Gumbel-Top-K router, characterizes the space of reachable joint distributions over routing choices across tokens
- Within-Group Positive Dependence: An exchangeable Gaussian copula positively correlates Gumbel perturbations at each expert coordinate for related tokens, increasing coherence in expert-set selection within groups
- Cross-Group Negative Dependence: A tunable antithetic construction introduces selectable negative dependence across disjoint token groups, counterbalancing load concentration
- Invariance Guarantee: Both operations provably preserve individual token marginal distributions (Top-K samples, mixture weights, inclusion probabilities) conditioned on pre-routing logits, ensuring conditional expected expert traffic remains unchanged
- Training Architecture: A small controller driven by frozen features adjusts the two dependence dials; the frozen network runs only in the forward pass, with gradients confined to the controller via score-function estimation
Industry Insight
- The two-dial dependence control framework offers a principled, training-efficient alternative to full fine-tuning for stabilizing MoE routing, particularly valuable for large-scale models where retraining is prohibitively expensive
- The theoretical trade-off characterization (positive coupling inflates load variance, cross-group opposition reduces it) provides actionable guidance for practitioners tuning expert load balancing in production MoE deployments
- While the pilot validates the mechanism, the lack of demonstrated task-level gains suggests further research is needed to connect routing coherence improvements to downstream performance; this remains an open opportunity for the community
Disclaimer: The above content is generated by AI and is for reference only.