What a Kernel Is, and Why Everyone Is Writing New Ones
GPU kernels are the critical layer between silicon and software, where the real performance bottlenecks of LLM inference are solved through memory optimization rather than arithmetic FlashAttention eliminates the O(S²) memory footprint of attention by processing tiles in shared memory with online softmax, achieving exact results while reducing held memory by ~250x at 64K tokens The GPU memory hierarchy has one dramatic step—a 1,600-fold gap between L2 cache (50 MB) and HBM (80 GB)—making shared
Analysis
TL;DR
- GPU kernels are the critical layer between silicon and software, where the real performance bottlenecks of LLM inference are solved through memory optimization rather than arithmetic
- FlashAttention eliminates the O(S²) memory footprint of attention by processing tiles in shared memory with online softmax, achieving exact results while reducing held memory by ~250x at 64K tokens
- The GPU memory hierarchy has one dramatic step—a 1,600-fold gap between L2 cache (50 MB) and HBM (80 GB)—making shared memory (228 KiB per SM) the key optimization target
- FlashInfer unifies multiple cache layouts into a single block-sparse format and performs dynamic batch planning per forward pass, making it the default for serving on Blackwell GPUs
- The two "Tritons" (kernel language vs. inference server) share only a name; NVIDIA renamed Triton Inference Server to Dynamo Triton in March 2025 to reduce confusion
Why It Matters
Understanding this kernel layer is essential for anyone deploying LLMs in production, as it explains why attention was the bottleneck and how modern libraries achieve order-of-magnitude memory reductions without sacrificing correctness. For AI practitioners, knowing which kernel stack their engine uses—and why—directly impacts throughput, memory efficiency, and hardware utilization across different GPU generations.
Technical Details
- GPU Memory Hierarchy: Four tiers on a log scale—registers (33 MiB), shared memory (29 MiB), L2 cache (50 MB), and HBM3 (80 GB)—with a 1,600-fold bandwidth gap between L2 and HBM being the critical boundary; each SM has only 228 KiB of shared memory, with 48 KiB per block unless explicitly opted in
- FlashAttention Mechanism: Processes attention in tiles loaded into shared memory, using online softmax (Milakov & Gimelshein, 2018) to maintain running max and sum without materializing the full S×S score matrix; computes exact attention (not approximate), trading extra FLOPs in rescaling for massive memory savings
- FlashInfer Architecture: Treats every KV cache layout (paged, shared prefixes, sliding windows) as a single block-sparse matrix format, generating kernels from templates; performs dynamic batch planning by inspecting ragged serving batches and rebuilding schedules every step, spreading planning cost across 36 layers on CPU while GPU computes
- Triton Language: A Python-like kernel language from Harvard/OpenAI that compiles to GPU code via PyTorch's torch.compile; users control block sizes, thread lockstep, and pipeline depth, producing roughly a tenth the code of traditional CUDA with most of the performance
- CUTLASS 3.x: NVIDIA's template library for GEMM and variants, including CuTe layout system used by FlashAttention-3; critical for grouped GEMM operations powering mixture-of-experts layers, with gaps over naive implementations far exceeding marginal optimizations
Industry Insight
- Kernel selection is now a primary deployment decision: FlashInfer is becoming the default for serving on Blackwell while FlashAttention remains preferred on H100, meaning infrastructure choices must be GPU-generation-aware rather than assuming one-size-fits-all
- The shift from bandwidth-bound to compute-bound attention means performance tuning should now focus on tensor core utilization and tiling strategies rather than memory bandwidth optimization, fundamentally changing how teams approach inference profiling
- The convergence of libraries (TensorRT-LLM shipping kernels into FlashInfer, vLLM's Blackwell-first-choice switch) suggests the ecosystem is consolidating around a few high-quality kernel implementations rather than fragmenting, making it increasingly important to understand which backend your stack actually uses
Disclaimer: The above content is generated by AI and is for reference only.