Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings
Giga-Embeddings introduces a sparse 10B-parameter Mixture-of-Experts encoder with ~1.8B active parameters per token, achieving top retrieval quality across English, Russian, multilingual, and code benchmarks The model processes 114.5k tokens/second with 1024-token inputs in vLLM, delivering 25% higher throughput than a dense 3B model and 1.56–2.65x the throughput of evaluated external systems The family includes a dense 3B encoder and a distilled 480M compact model trained with a dimension-agnos
Analysis
TL;DR
- Giga-Embeddings introduces a sparse 10B-parameter Mixture-of-Experts encoder with ~1.8B active parameters per token, achieving top retrieval quality across English, Russian, multilingual, and code benchmarks
- The model processes 114.5k tokens/second with 1024-token inputs in vLLM, delivering 25% higher throughput than a dense 3B model and 1.56–2.65x the throughput of evaluated external systems
- The family includes a dense 3B encoder and a distilled 480M compact model trained with a dimension-agnostic objective that aligns teacher-student similarity distributions
- The 480M model scores 70.98 on Russian MTEB, surpassing FRIDA while using 42% fewer parameters
- All three model checkpoints are publicly released
Why It Matters
This work demonstrates that sparse Mixture-of-Experts architectures can break the traditional tradeoff between embedding quality and serving throughput, which is critical for production RAG and retrieval systems. The release of a compact distilled variant that outperforms larger competitors with significantly fewer parameters provides a practical path for resource-constrained deployments.
Technical Details
- Sparse 10B MoE encoder: ~1.8B active parameters per token, achieving the strongest aggregate performance across English, Russian, multilingual, and code MTEB benchmark suites
- Throughput performance: 114.5k tokens/second at 1024-token input length using vLLM, outperforming a dense 3B model by 25% and external systems by 1.56–2.65x
- Model family: Three checkpoints — sparse 10B MoE, dense 3B, and distilled 480M — covering a range of compute and memory budgets
- Distillation method: A dimension-agnostic objective that aligns teacher and student similarity distributions, enabling the 480M model to achieve 70.98 on Russian MTEB with 42% fewer parameters than FRIDA
- Benchmarks: Evaluated across four MTEB suites (English, Russian, multilingual, code)
Industry Insight
- Sparse MoE encoders are becoming a viable production strategy for high-throughput embedding serving, offering a compelling alternative to dense models that scale linearly in both parameters and inference cost
- The dimension-agnostic distillation objective presents a reusable technique for compressing large embedding models without sacrificing retrieval quality, particularly valuable for edge and cost-sensitive deployments
- With all checkpoints released, practitioners can adopt a tiered strategy: the 10B MoE for maximum retrieval accuracy, the 3B dense model as a balanced middle ground, and the 480M model for low-latency, resource-constrained environments
Disclaimer: The above content is generated by AI and is for reference only.