Why Is PyTorch Compile So Fast: Kernel Fusion

How Fusion Rewrites the Execution Plan

This technical article demonstrates a compiler optimization that fundamentally alters the runtime behavior of PyTorch code. The core transformation is the conversion of a developer's high-level, modular code into a low-level, monolithic computation. The presented example—three elemental operations (multiply, add, sigmoid)—would naively require three separate GPU kernels in eager mode. Each kernel suffers from two systemic costs:

• Kernel Launch Overhead: The CPU must initiate each kernel, a non-trivial latency.
• Memory Traffic: Each kernel's result must be written to and then read from slower global memory.

The Inductor compiler performs vertical fusion, tracing the data dependency (the "chain" of operations) to generate a single Triton kernel that executes all three steps. The critical insight is that the compiler's primary goal is not to make each mathematical operation faster, but to minimize the movement of data. By keeping the intermediate values (tmp in the code) in the GPU's fastest registers between computation steps, it attacks the dominant bottleneck in many workloads: memory bandwidth.

The Principle of Locality Over Arithmetic

The article subtly highlights a paradigm shift in performance optimization for massively parallel processors. The programming abstraction (calling separate PyTorch functions) encourages clear, readable code but is misaligned with hardware efficiency. The compiler bridges this gap by prioritizing data locality—a principle where keeping data close to the compute unit is more important than reducing arithmetic instructions. The fused kernel performs exactly the same mathematical operations (a multiplication, an addition, a sigmoid) as the three original kernels. The optimization isn't about doing fewer calculations; it's about ensuring the data for the next calculation is already present where it's needed.

Trade-offs and the Role of the Compiler

This optimization is not manual but automated by torch.compile, suggesting a trade-off between developer productivity and peak performance. The article implies that relying on the compiler allows developers to write clean, abstract code while the system handles hardware-specific tiling and fusion strategies. This shifts the programmer's role: instead of manually optimizing memory access patterns (as in CUDA C++), one must structure code with clear data dependencies to enable compiler analysis. The "vertical" fusion pattern is especially common because it mirrors the sequential dataflow of neural network layers (normalization -> linear -> activation), making this a broadly applicable and powerful default optimization.