TLX Block Attention: A Warp-Specialized Blackwell Kernel for Fixed-Block Sparse Self-Attention

Background

Self-attention is the foundational mechanism in Transformers, enabling models to weigh the relevance of every element in a sequence against every other element. Block-diagonal self-attention—where sequences are partitioned into fixed-size groups that attend only within themselves—is a widely-used pattern in recommendation and feature-interaction models (e.g., BlockBERT). In Meta's ads ranking production stack, workloads typically involve batch sizes of 1152, sequences up to ~4k tokens, head dimensions of 64 or 128, and ~70% sparsity in the attention structure. As these models scale, attention cost becomes the dominant bottleneck.

General-purpose kernels like Flash Attention v2 (with block masking) or FlexAttention (FA4) are currently used but are suboptimal. FlexAttention operates at a minimum tile size of 256, which is incompatible with the 64-token blocks these production models require. Flash Attention v2 remains the strongest available baseline at this tile size but leaves significant performance on the table because its design caters to arbitrary-length causal attention patterns.

Key Points

• The core thesis: When the attention pattern is known at compile time, you can build something dramatically faster. The fixed constraint that every Q tile attends to exactly one K/V tile can be propagated through the entire algorithm.

• Cascade of simplifications from the fixed-block constraint:

  • Eliminates multi-iteration accumulators: Standard Flash Attention iterates a Q tile over multiple K/V tiles, maintaining running statistics (row-wise max and log-sum-exp). With block-diagonal attention, this collapses into a single GEMM.
  • Removes online softmax correction: Flash Attention applies a correction factor at each step (alpha = exp(m_old - m_new)) to preserve numerical stability across tiles. When there's only one K/V tile per Q tile, this entire mechanism is unnecessary overhead.
  • Eliminates auxiliary kernel launches: Bookkeeping operations like logsumexp that are essential for arbitrary-length attention become pure overhead in the block-diagonal setting.

• Performance results on NVIDIA B200 GPUs:

  • ~1.85× forward speedup over Flash Attention v2
  • ~2.50× backward speedup over Flash Attention v2
  • ~3.5× speedup for the combined attention-and-rotary backward pass when rotary embeddings are fused into the attention epilogue

• Built on TLX (Triton Language Extensions): A set of low-level extensions to the Triton compiler exposing hardware-native control over:

  • Warp specialization
  • Asynchronous tensor core operations
  • Memory hierarchy management on Blackwell GPUs
  • TLX bridges the gap between Triton's Python-level productivity and the fine-grained hardware control traditionally requiring raw CUDA or CUTLASS

Significance

This work demonstrates a powerful architectural principle: domain-specific compile-time knowledge can yield multiplicative speedups over general-purpose implementations. Rather than building a universal attention kernel and masking out irrelevant computations, encoding the known structure directly into the algorithm eliminates entire categories of overhead—multi-tile iteration, numerical correction stages, and auxiliary launches. The ~3.5× speedup for fused attention-plus-rotary backward passes is particularly notable, showing that kernel fusion combined with structural awareness compounds performance gains. For production recommendation systems at Meta's scale, where attention is the dominant cost and patterns are fixed, this approach translates directly into meaningful infrastructure savings and latency reductions.