ESTS at WMT26: Routing-Informed Expert Pruning for Model Compression
ESTS team submitted six compression variants to WMT26 Model Compression Shared Task for En-Zh and En-Arabic translation, all derived from GPT-OSS-20B Novel routing-informed expert pruning approach uses task-specific routing mass to rank experts and cross-lingual routing divergence to allocate retained capacity across layers Pruned models undergo recovery tuning on GPT-5.1-generated synthetic data followed by MXFP4 quantization of retained expert projection weights Final models range from 4.186B
Analysis
TL;DR
- ESTS team submitted six compression variants to WMT26 Model Compression Shared Task for En-Zh and En-Arabic translation, all derived from GPT-OSS-20B
- Novel routing-informed expert pruning approach uses task-specific routing mass to rank experts and cross-lingual routing divergence to allocate retained capacity across layers
- Pruned models undergo recovery tuning on GPT-5.1-generated synthetic data followed by MXFP4 quantization of retained expert projection weights
- Final models range from 4.186B to 7.770B parameters with packed artifact sizes of 4.55 to 6.33 GiB
- Robust inference pipeline includes category inference, output validation, retries, segmented fallback, and JSON reconstruction for instruction-conditioned settings
Why It Matters
This work demonstrates a practical, scalable approach to compressing large open-source language models for production machine translation, achieving significant parameter reduction while maintaining competitive quality. The routing-informed pruning methodology offers a principled alternative to uniform pruning strategies, potentially applicable to any MoE-based architecture. The full pipeline—from expert selection through quantization to robust inference—provides a reproducible blueprint for practitioners looking to deploy compressed models in resource-constrained environments.
Technical Details
- Expert Pruning via Routing Analysis: Uses task-specific routing mass to rank experts by importance and cross-lingual routing divergence to determine how retained capacity should be distributed across transformer layers, then physically removes low-importance experts rather than merely masking them
- Recovery Tuning and Quantization: Post-pruning specialists are recovery-tuned on GPT-5.1-generated synthetic translation data, then further compressed via MXFP4 quantization applied specifically to retained expert projection weights
- Three Compression Operating Points per Direction: Six total submissions covering En-Zh and En-Arabic, each with three compression levels, enabling analysis of the quality-compression tradeoff curve
- Robust Inference System: Implements category inference, output validation, retries, segmented fallback, and source-owned JSON reconstruction to handle the instruction-conditioned WMT26 evaluation setting reliably
- Evaluation: Internal xCOMET-XL evaluation using GPT-5.1 pseudo-references for cross-operating-point comparison; parameter counts span 4.186B–7.770B with artifact sizes of 4.55–6.33 GiB
Industry Insight
- Routing-based expert pruning represents a significant advancement over heuristic or uniform compression methods; practitioners should consider analyzing routing patterns in their own MoE models before applying pruning to maximize quality retention
- The combination of synthetic data recovery tuning (using GPT-5.1) with aggressive quantization (MXFP4) demonstrates that high-quality compressed models can be produced without access to large parallel corpora, lowering the barrier for specialized translation system development
- The full inference pipeline—including validation, retries, and fallback—highlights that model compression is only one component of production readiness; robust serving infrastructure is equally critical for meeting competition and real-world quality requirements
Disclaimer: The above content is generated by AI and is for reference only.