Pinterest Engineers Eliminate CPU Zombie Processes to Resolve Production Bottlenecks

TL;DR

• Pinterest resolved intermittent CPU starvation causing ML training crashes by identifying "zombie" memory cgroups leaked by a crash-looping Amazon ECS agent.
• The issue caused a >25% drop in training success rates due to Elastic Network Adapter (ENA) resets and packet loss on their PinCompute platform.
• A default-enabled ECS agent on AWS Deep Learning AMIs leaked ~70,000 inactive memcgs while only ~240 remained active, saturating individual CPU cores.
• Engineers used mpstat and NetflixFlamescope to visualize performance data, revealing the kubelet process spiking to 6.5% CPU due to traversing the bloated cgroup list.
• The fix involved disabling the ECS agent's systemd unit in the base image and rebooting nodes to clear accumulated cgroups, restoring network stability.

Why It Matters

This incident highlights a critical gap in observability for large-scale Kubernetes environments, where aggregate metrics can mask severe underlying kernel-level bottlenecks. It underscores the importance of auditing default configurations in base images, especially in high-frequency ML workloads where micro-latency issues in network drivers can cascade into total task failure. For AI infrastructure teams, it serves as a case study in how user-space daemon mismanagement can silently degrade kernel performance, impacting distributed computing stability.

Key Data

• 25%+ decrease in training task success rate in some use cases due to ENA device resets.
• ~70,000 "zombie" memcgs accumulated vs. ~240 actively used memcgs.
• Individual CPU cores hit 100% system utilization for several seconds continuously.
• Kubelet process CPU usage spiked from <1% to ~6.5% during incident windows.
• Tens of thousands of Ray clusters provisioned monthly on PinCompute.
• ENA device reset triggered when transaction completion stalls for >5 seconds.
• Performance captures taken every 2 minutes over a 12-hour reproduction window.

Technical Details

• Root Cause Identification: The investigation utilized mpstat for per-core analysis because standard dashboards showed normal aggregate CPU utilization. This revealed that individual cores dedicated to handling ENA network interrupts were saturating, blocking the NAPI polling thread and triggering self-healing ENA resets.
• Profiling Tooling: The team employed continuous performance profiling within a 12-hour window, visualizing the data using NetflixFlamescope. This allowed them to pinpoint the exact moments of network resets and correlate them with the mem_cgroup_nr_lru_pages kernel function consuming significant CPU time.
• Cgroup Leak Mechanism: The default Amazon ECS agent on the AWS Deep Learning AMI entered a crash loop because it was not intended for use on Pinterest’s platform. Each restart leaked memory cgroups (memcgs), causing the kubelet to traverse a massive list of ~70,000 entries during state synchronization, monopolizing a CPU core for seconds.
• Remediation Strategy: Engineers disabled the ECS agent's systemd unit in the base image to prevent further leaks and rebooted affected machines to clear the accumulated kernel state. This stabilized the memcg count and eliminated network restarts.
• Future Observability: The team is exploring continuous, time-indexed profiling tools like gProfiler (in collaboration with Intel), Parca, and Grafana Pyroscope to provide cluster-wide visibility and shorten the path from symptom to root cause.

Industry Insight

• Base Image Hygiene: AI practitioners should rigorously audit default configurations and services in base images (e.g., AWS DL AMIs), as unused daemons can introduce significant performance overheads that are difficult to detect via high-level metrics.
• Granular Monitoring: Aggregate CPU metrics are insufficient for debugging network-intensive ML workloads. Implementing per-core and kernel-level observability (via eBPF tools) is essential to catch subtle resource contention that leads to network adapter resets.
• Tooling for Diagnostics: The shift towards continuous profiling platforms (like Parca or Pyroscope) indicates a move away from reactive, manual debugging. Investing in these tools can significantly reduce Mean Time to Resolution (MTTR) for complex infrastructure issues in distributed systems.

zation, which appeared normal. The actual issue was the saturation of specific individual cores handling network interrupts, which masked the underlying bottleneck when looking at the overall system average.

Q: What is the role of the Amazon ECS agent in this incident?
A: The Amazon ECS agent was included by default in the AWS Deep Learning AMI used by Pinterest. Since Pinterest did not use ECS, the agent entered a crash loop, continuously leaking memory cgroups and causing the kubelet to spend excessive time traversing the bloated cgroup list.