Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections
Mixture-of-Experts (MoE) design fails when directly applied to convolutional networks because parallel convolutional experts reading the same input channels learn nearly identical filters MoCE shifts the expert axis from operator duplication to channel selection, where each expert is a single output channel with a learned sparse support of k << C input channels The method uses an input-adaptive softmax temperature (predicted per input) enabling dynamic interpolation between mean-like and max-lik
Analysis
TL;DR
- Mixture-of-Experts (MoE) design fails when directly applied to convolutional networks because parallel convolutional experts reading the same input channels learn nearly identical filters
- MoCE shifts the expert axis from operator duplication to channel selection, where each expert is a single output channel with a learned sparse support of k << C input channels
- The method uses an input-adaptive softmax temperature (predicted per input) enabling dynamic interpolation between mean-like and max-like aggregation
- MoCE replaces dense pointwise projections with O(k/C) relative cost, achieving 16.7% MAC reduction and lower end-to-end latency while matching or exceeding dense baselines
Why It Matters
This work addresses a fundamental architectural question about how sparse routing paradigms from language models can be meaningfully adapted to vision backbones, offering a principled alternative to dense channel-mixing layers. For practitioners building efficient vision models, MoCE provides a drop-in replacement for 1x1 convolutions that delivers measurable latency and compute savings without accuracy degradation.
Technical Details
- Core innovation: Each expert in MoCE corresponds to a single output channel with a sparse support over k << C input channels, replacing the dense pointwise (1x1) projection used in standard channel-mixing layers
- Input-adaptive aggregation: A softmax over selected channels uses a per-input predicted temperature, allowing each expert to dynamically shift between mean-like (high temperature) and max-like (low temperature) aggregation behavior
- Residual expert: A dedicated residual expert summarizes the unselected channels, ensuring no information is lost from channels outside the sparse support
- Load-balancing loss: A regularization term enforces complete channel coverage across experts, preventing collapse to a subset of channels
- Evaluation: Tested across ResNet backbones on ImageNet-1K and CIFAR-100, transfer learning settings, and EfficientViT architectures, achieving 16.7% MAC reduction and improved end-to-end latency while matching or exceeding dense baselines
Industry Insight
- The failure mode identified—parallel convolutional experts learning identical filters—suggests that naive MoE-to-CNN transfers may require structural modifications beyond simple architectural substitution; channel-level sparsity should be considered a first-class design choice
- MoCE's input-adaptive temperature mechanism offers a generalizable pattern for dynamic computation in vision transformers and CNNs, potentially applicable beyond channel-mixing layers to other pointwise operations
- The demonstrated wall-clock savings alongside theoretical MAC reductions indicate that sparse channel selection is practically viable for deployment, making it attractive for edge and mobile vision applications where latency is critical
Disclaimer: The above content is generated by AI and is for reference only.