ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
ExFold is a training-free expert-folding framework that jointly accelerates both prefill and decode phases of MoE inference without requiring model retraining It reformulates both phases as a single budgeted output-approximation problem, using calibrated scalar projectors to fold excluded expert contributions into retained experts The key insight is that many expert outputs are directionally aligned but differ in magnitude, enabling a pairwise scalar-projector matrix calibrated on unlabeled data
Analysis
TL;DR
- ExFold is a training-free expert-folding framework that jointly accelerates both prefill and decode phases of MoE inference without requiring model retraining
- It reformulates both phases as a single budgeted output-approximation problem, using calibrated scalar projectors to fold excluded expert contributions into retained experts
- The key insight is that many expert outputs are directionally aligned but differ in magnitude, enabling a pairwise scalar-projector matrix calibrated on unlabeled data
- Prefill acceleration operates as token-level Top-K folding while decode acceleration operates as batch-level expert-pool folding, sharing one unified folding mechanism
- Implemented as a plug-and-play vLLM plugin with a custom CUDA kernel, achieving up to 1.41x TTFT and 2.45x TPOT speedups while retaining ~99% of original model quality
Why It Matters
MoE models are increasingly popular for their quality-efficiency tradeoff, but low-latency serving remains a critical bottleneck due to fundamentally different prefill and decode constraints. ExFold addresses a significant gap by unifying acceleration across both phases in a training-free manner, making it immediately deployable without retraining overhead. This is particularly relevant for practitioners deploying large-scale MoE models in production where both throughput and latency matter.
Technical Details
- Core mechanism: ExFold casts prefill and decode as a unified budgeted output-approximation problem. A constrained expert set is executed per phase, while excluded experts' contributions are projected onto retained experts via calibrated scalar projectors
- Scalar projector calibration: A pairwise scalar-projector matrix is learned on unlabeled data, exploiting the observation that expert outputs are often directionally aligned but vary in magnitude
- Phase-specific folding: Prefill uses token-level Top-K expert folding (optimizing per-token computation), while decode uses batch-level expert-pool folding (optimizing memory traffic from batch-wise activated experts)
- Implementation: Deployed as a plug-and-play vLLM plugin with a lightweight expert-folding CUDA kernel, requiring no model modifications
- Performance: Up to 1.41x TTFT (time-to-first-token) and 2.45x TPOT (time-per-output-token) speedups with approximately 99% quality retention
Industry Insight
- Training-free acceleration methods like ExFold lower the barrier to deploying MoE models at scale, eliminating the need for costly retraining pipelines that many production teams cannot afford
- The unified prefill-decode approach addresses a real fragmentation in current optimization tools, which typically target only one phase—practitioners should evaluate whether existing single-phase tools are leaving performance on the table
- The directional-alignment insight underlying scalar projection could generalize beyond MoE models to other sparse-activation architectures, suggesting a broader research direction for training-free approximation techniques
Disclaimer: The above content is generated by AI and is for reference only.