Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler
Meta utilized sched_ext, a BPF-based extensible scheduling framework, to resolve latency regressions caused by Linux kernel upgrades in their ad serving fleet. The custom scheduling policy achieved a 28% reduction in p99 latency for the ads retrieval stage, resulting in a 1.1% increase in ads ranked and 3.28 MW of power savings. By soft-partitioning CPUs into latency-critical and non-critical pools, the solution improved L3 cache locality and reduced expensive DRAM accesses. Implementing the sch
Analysis
TL;DR
- Meta utilized sched_ext, a BPF-based extensible scheduling framework, to resolve latency regressions caused by Linux kernel upgrades in their ad serving fleet.
- The custom scheduling policy achieved a 28% reduction in p99 latency for the ads retrieval stage, resulting in a 1.1% increase in ads ranked and 3.28 MW of power savings.
- By soft-partitioning CPUs into latency-critical and non-critical pools, the solution improved L3 cache locality and reduced expensive DRAM accesses.
- Implementing the scheduler as a user-space BPF program enabled rapid iteration, allowing subsequent updates to deliver an additional 60% latency reduction and 18% fewer timeouts within days.
Why It Matters
This case study demonstrates how low-level infrastructure optimizations, specifically workload-aware scheduling, can directly translate to significant business value in high-scale distributed systems. It highlights the practical viability of BPF-based kernel extensions for decoupling application-specific performance tuning from upstream kernel release cycles, offering a blueprint for other tech giants managing massive, latency-sensitive workloads.
Technical Details
- Framework: Utilized
sched_ext, an upstream, BPF-based extensible scheduling framework integrated into Linux kernel v6.12, developed in partnership with Google’s ghOSt team. - Architecture: The policy soft-partitions CPUs into two dynamic pools based on load-based heuristics: one for latency-critical request path threads and another for less sensitive background work.
- Implementation: The scheduling logic is implemented as a BPF program loaded via a user-space binary, allowing the kernel to call into the scheduler via event-driven callbacks for thread wake-ups, enqueueing, dispatching, and idle transitions.
- Performance Metrics: Initial deployment on kernel v6.9 reduced p99 latency by 28%; subsequent user-space-only updates added a further 60% latency reduction and cut timeout errors by 18%.
- Optimization Mechanism: By keeping related work on the same CPUs over time, the approach maximizes last-level cache (L3) locality, thereby minimizing costly DRAM access patterns.
Industry Insight
- Decouple Optimization from Kernel Releases: Organizations should consider BPF-based scheduling frameworks to maintain independent optimization paths, avoiding the operational friction and technical debt associated with waiting for upstream kernel updates.
- User-Space Iteration Speed: Implementing complex system-level policies in user-space (via BPF) drastically reduces the feedback loop for experimentation, enabling daily or weekly performance tweaks instead of multi-month release cycles.
- Workload-Aware Infrastructure: General-purpose schedulers like CFS or EEVDF may not suffice for extreme-scale, latency-sensitive applications; custom, domain-specific scheduling policies can yield substantial ROI improvements through better resource utilization and cache efficiency.
Disclaimer: The above content is generated by AI and is for reference only.