GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta's GEM (Generative Ads Recommendation Model) now trains at LLM scale on thousands of GPUs, doubling E2E training efficiency to 20-25% MFU while scaling training FLOPs 4x in 12 months Custom kernel library (Jagged Flash Attention, GDPA, BlockAttention) and mixed ultra-low precision training (MXFP8) were co-designed specifically for recommendation workloads Topology-aware 5D parallelism with SM-free collectives combines 2D FSDP + Expert Parallelism for dense parameters with Fully Sharded 2D Mo
Analysis
TL;DR
- Meta's GEM (Generative Ads Recommendation Model) now trains at LLM scale on thousands of GPUs, doubling E2E training efficiency to 20-25% MFU while scaling training FLOPs 4x in 12 months
- Custom kernel library (Jagged Flash Attention, GDPA, BlockAttention) and mixed ultra-low precision training (MXFP8) were co-designed specifically for recommendation workloads
- Topology-aware 5D parallelism with SM-free collectives combines 2D FSDP + Expert Parallelism for dense parameters with Fully Sharded 2D Model Parallelism for sparse parameters
- Recommendation model training faces fundamentally different challenges than LLMs: jagged inputs, asymmetric attention patterns, memory-bound operations, and numerical sensitivity
- Standard LLM-optimized infrastructure does not transfer directly to recommendation systems, requiring significant hardware/software co-design innovation
Why It Matters
This work represents a critical milestone in extending LLM-scale training techniques to recommendation systems, which power some of the most commercially impactful AI products (Meta's ad platform). The co-design approach demonstrates that one-size-fits-all infrastructure solutions are insufficient — different AI workloads require purpose-built optimization strategies. For practitioners, this highlights the growing importance of understanding the unique computational characteristics of non-LLM workloads as models continue to scale.
Technical Details
- Hybrid Architecture: GEM combines trillions of sparse embedding parameters with billions of dense parameters, processing both sequence features (user activity history) and non-sequence features (location, ad creative) with customized attention mechanisms and cross-feature learning
- Compute Efficiency Innovations: Custom kernel library including Jagged Flash Attention (JFA) for variable-length sequences, Generalized Dot-Product Attention (GDPA), BlockAttention, and mixed ultra-low precision training (MXFP8 attention and MLP) optimized for recommendation-specific patterns
- Scaling Efficiency: Topology-aware 5D parallelism with SM-free collectives — 2D FSDP + Expert Parallelism for dense parameters, Fully Sharded 2D Model Parallelism for sparse parameters — co-designed with Meta's multi-tiered network hierarchy
- Key Challenges Addressed: Jagged inputs (variable sequence lengths wasting up to 50% compute if padded), asymmetric attention patterns (self-attention: long sequences/short window; cross-attention: long queries/short KV; PMA: short queries/long KV), memory-bound operations with small embedding dimensions, and numerical sensitivity of CTR/CVR prediction tasks
- Efficiency Framework: E2E MFU = Local MFU (compute efficiency) × Scaling Ratio (scaling efficiency), decomposing the optimization into two distinct but related problems
Industry Insight
- The boundary between recommendation systems and LLMs is blurring — as rec models scale to foundation model sizes, they require fundamentally different infrastructure than text-based LLMs, creating a new specialization opportunity in AI systems engineering
- Hardware/software co-design is becoming essential at scale; Meta's approach of building purpose-built kernels and precision schemes for specific workload characteristics suggests that generic frameworks will increasingly hit diminishing returns for non-standard AI applications
- The 4x FLOPs scaling with doubled efficiency in 12 months demonstrates that significant gains are still achievable through systems-level optimization, not just model architecture improvements — a reminder that infrastructure investment remains a high-leverage strategy for organizations training large models
Disclaimer: The above content is generated by AI and is for reference only.