Training Hybrid Block Diffusion Language Models with Partial Bidirectionality 使用部分双向性训练混合块扩散语言模型

TL;DR

• The paper introduces BDLM Mamba-H, a hybrid architecture combining Mamba state-space models with attention to enable high-throughput long-context generation via block diffusion.
• It solves the caching inefficiency of prior bidirectional Mamba hybrids by restricting the reverse Mamba scan to the active denoising block, allowing exact KV cache reuse.
• In parameter sweeps, the 87M model achieves superior C4-en validation perplexity compared to pure attention baselines, remaining competitive at 350M parameters.
• The architecture delivers massive throughput gains, reaching 19.7x speedup over full-sequence DiffuMamba-H and 3.7x over BDLM attention at long context lengths (65K-262K tokens).

Why It Matters

This research addresses the critical bottleneck of memory bandwidth in long-context LLM generation, offering a viable path to scale context windows without prohibitive latency costs. By enabling efficient block-wise diffusion with precise caching, it demonstrates that hybrid state-space models can outperform traditional attention mechanisms in both speed and quality for specific generative tasks.

Technical Details

• Architecture: A Block Diffusion Language Model (BDLM) hybrid that integrates Mamba linear-time mixers with attention mechanisms, specifically designed for block-wise token generation.
• Key Innovation: Restricts the reverse-direction Mamba scan to the active denoising block rather than the entire sequence, ensuring states are prefix-only and enabling exact caching across diffusion steps.
• Benchmarking: Evaluated on the C4-en dataset with parameter counts of 87M and 350M, measuring validation perplexity against BDLM attention and full-sequence baselines like DiffuMamba-H.
• Performance Metrics: Achieved 19.7x throughput improvement over DiffuMamba-H at 65K tokens and 3.7x over BDLM attention at 262K tokens, highlighting significant efficiency gains in long-context inference.

Industry Insight

Hybrid architectures combining state-space models with attention offer a promising direction for optimizing long-context generation, particularly where memory bandwidth is the primary constraint. Practitioners should consider block-diffusion strategies with localized bidirectional scanning to reduce computational overhead in large-scale text generation pipelines. This approach may become standard for applications requiring real-time processing of extremely long documents or codebases.

TL;DR

• 提出BDLM Mamba-Hybrid架构，通过限制反向Mamba扫描范围至当前去噪块，解决了混合扩散模型中状态无法精确缓存的技术难题。
• 在87M参数规模的DCLM搜索中，该模型在C4-en验证集上取得了优于纯注意力基线和全序列基线的最佳困惑度。
• 长上下文推理性能显著提升，在65K token长度下吞吐量达到全序列DiffuMamba-H的19.7倍，在262K token长度下达到BDLM注意力的3.7倍。
• 证明了Mamba混合架构在长上下文扩散生成任务中的潜力，有效平衡了内存带宽瓶颈与并行计算需求。

为什么值得看

本文针对大语言模型长上下文生成中内存带宽受限的核心痛点，提出了结合Mamba线性复杂度与扩散模型并行生成优势的混合方案。对于致力于优化LLM推理效率、降低显存占用以及探索非Transformer架构的研究者而言，该工作提供了关键的架构改进思路和数据支撑。

技术解析

• 核心创新：针对早期混合扩散模型（如DiffuMamba）因反向扫描需遍历整个序列导致状态非前缀依赖、无法复用KV缓存的问题，提出将反向Mamba扫描严格限制在“活跃去噪块”内，实现了跨块的精确缓存复用。
• 模型架构：采用BDLM（Block Diffusion Language Model）与Mamba-Attention混合结构，结合了Mamba的高效序列建模能力和注意力机制的表达能力，同时利用块级扩散策略提升并行度。
• 实验规模与基准：进行了87M和350M参数规模的DCLM（Data Compiling Language Models）超参数搜索，使用C4-en作为验证基准评估困惑度（Perplexity）。
• 性能指标：在长上下文场景下，BDLM Mamba-H展现了极高的吞吐量优势，特别是在65K和262K token长度下的对比实验结果，验证了其在处理长序列时的效率优势。

行业启示

• 架构融合趋势：单一架构难以兼顾效率与能力，Mamba等线性注意力变体与扩散模型或传统Attention的混合架构将成为解决长上下文生成瓶颈的重要方向。
• 缓存优化是关键：在长上下文生成中，能否实现精确的状态缓存复用直接决定推理吞吐量，未来模型设计需优先考虑计算图的结构以支持高效的增量式或块式缓存管理。
• 长上下文推理成本降低：通过提高吞吐量而非单纯增加算力，可以显著降低长文本生成的边际成本，为大规模部署长上下文应用提供可行的技术路径。