Alibaba's Qwen Team Releases Qwen3.8-Flash-Next: A 125B Multimodal MoE With 6B Active Parameters Previewing the Qwen4 Architecture 阿里巴巴Qwen团队发布Qwen3.8-Flash-Next：125B多模态MoE，6B激活参数预览Qwen4架构

TL;DR

• Qwen3.8-Flash-Next is a 125B multimodal MoE model with only 6B active parameters per token, combining a 51B N-gram embedding table and a 4B multi-token prediction module for a total of 180B parameters on disk.
• The architecture introduces a hybrid attention mechanism: three of every four layers use Gated DeltaNet (linear attention with fixed-size recurrent state), while the fourth uses Qwen Sparse Attention (QSA) operating at micro-block granularity with a budget of 512 blocks/2048 tokens.
• Training cost is reported at roughly one-ninth that of Qwen3.7-Plus, with native 262K token context extensible to 1M via YaRN, and significant serving speedups (up to 10.2× prefill and 6.6× decode at 1M tokens).
• The model achieves strong results on coding and agentic benchmarks (91.9 on LiveCodeBench v6, 73.9 on CoWorkBench, 95.7 on MathVision with code interpreter) but trails frontier reasoning models like Claude Opus 4.6 on HLE.
• Released under the qwen-community-1.0 license (not Apache-2.0), with FP8 checkpoint at 172.78 GiB requiring multi-GPU deployment (TP2 minimum on GB300, TEP8 on 8×H200).

Why It Matters

This release represents a significant architectural shift toward extreme sparsity and hybrid attention, demonstrating that large-scale MoE models can maintain competitive performance while activating only a fraction of their parameters per token — a critical step toward cost-effective deployment at scale. The reported 9× training cost reduction and multi-modal agentic capabilities signal Alibaba's strategy to make frontier-level models more accessible, though the non-Apache license and hardware requirements temper immediate self-hosting viability for smaller organizations.

Technical Details

• Architecture: 48-layer backbone with 12 × (3 × Gated DeltaNet → 1 × Qwen Sparse Attention) layout. The MoE layer contains 512 experts, activating 10 routed plus 1 shared expert per token, with expert intermediate dimension of 640. Gated Residual uses a bottleneck rank of 320 with element-wise read gates and per-branch scalar write gates across 4 parallel branches.
• N-gram Embedding: A 20,000,000-entry bigram/trigram lookup table at layer 2 provides deterministic capacity expansion; supports asynchronous prefetch to host memory (NVIDIA devices only currently).
• Training: Muon optimizer applied alongside AdamW to specific weight categories, batch-size warmup eliminated, and scaling laws refitted. The 4B multi-token prediction (MTP) module aids training efficiency.
• Deployment: FP8 checkpoint is 172.78 GiB; BF16 is 335.28 GiB. vLLM (TP2 min on GB300, TP4 recommended), SGLang, TokenSpeed, transformers serve, and llama.cpp (GGUF) are supported. Fine-tuning via Unsloth, Swift, and LLaMA-Factory.
• Context & Modes: 262,144 tokens native, 1,000,000 with YaRN. Thinking mode enabled by default with reasoning_effort (xhigh/medium/low); recommended settings are temperature 1.0/top_p 0.95 for thinking and 0.7/0.80 for instruct.

Industry Insight

• The hybrid Gated DeltaNet + Sparse Attention design offers a compelling template for balancing long-context efficiency with computational tractability; expect similar architectures to appear in subsequent open-weight releases as the community validates the approach.
• The 1/9 training cost claim, if independently verified, could redefine cost-performance expectations for large MoE models and accelerate the open-weight race — but practitioners should treat vendor-reported speedup figures cautiously until third-party benchmarks are available.
• The qwen-community-1.0 license (as opposed to Apache-2.0) and the multi-GPU deployment requirement mean this model is best suited for well-resourced teams and API integrations rather than individual researchers or small startups; commercial users must carefully review licensing terms before production use.

TL;DR

• Qwen3.8-Flash-Next是阿里巴巴Qwen团队发布的125B多模态MoE模型，总参数量180B（含51B N-gram嵌入和4B多token预测模块），每token仅激活6B参数
• 采用Gated DeltaNet与Qwen Sparse Attention混合架构（48层中3/4为GDN，1/4为QSA），配合Gated Residual、N-gram Embedding和Muon优化器四大创新
• 训练成本约为Qwen3.7-Plus的1/9，推理性能显著提升（QSA kernel在1M tokens下prefill加速7.6×-10.2×，decode加速4.9×-6.6×）
• 原生上下文262K tokens，通过YaRN可扩展至100万tokens；FP8权重172.78 GiB，需多GPU节点部署（TP2最低，TP4推荐）
• 定位为Qwen4的架构预览，类似Qwen3-Next对Qwen3.5的角色；许可证为qwen-community-1.0（非Apache-2.0），商用需验证条款

为什么值得看

本文揭示了开源模型在成本效率与架构创新上的新突破——通过混合注意力机制和稀疏激活技术，在保持125B规模的同时将每token计算量降至6B，训练成本压缩至1/9，为大规模模型的高效部署提供了可复用的技术路径。对AI从业者而言，该模型展示了MoE架构与线性注意力结合的工程可行性，以及N-gram嵌入等确定性容量扩展方式的实用价值。

技术解析

混合注意力架构：模型采用48层结构，布局为12×(3×GDN→1×QSA)。Gated DeltaNet（GDN）作为线性注意力层，将历史压缩为固定大小的循环状态；Qwen Sparse Attention（QSA）以微块粒度（512块/2048 tokens）通过轻量级索引器选择上下文，而非逐token处理。这种设计在长上下文场景下显著降低计算开销。

MoE与参数激活：MoE层包含512个专家，每token路由激活10个专家加1个共享专家，专家中间维度为640。结合51B N-gram嵌入表（2000万条bigram/trigram，位于第2层）和4B多token预测模块，总参数量180B但每token仅激活6B，实现"大模型容量、小模型计算"。

训练与优化创新：引入Muon优化器与AdamW配合使用，针对特定权重类别；消除batch-size warmup阶段并重新拟合scaling laws。Gated Residual将残差流扩展为4个并行分支，配备element-wise读门和per-branch标量写门，瓶颈秩为320，增强信息流动控制。

部署与性能基准：FP8权重172.78 GiB，BF16为335.28 GiB，需多GPU节点（GB300上TP2最低/TP4推荐，8×H200使用TEP8）。SWE-bench Pro达62.5，LiveCodeBench v6达91.9，MathVision（含代码解释器）达95.7；但在HLE（35.9 vs Claude Opus 4.6的40.0）和NL2Repo-Bench（48.1 vs DeepSeek-V4的54.2）上仍有差距，前沿推理能力尚存瓶颈。

行业启示

成本效率成为开源模型竞争新维度：训练成本压缩至1/9表明，通过架构创新（稀疏激活、混合注意力）而非单纯堆参数量，可实现更经济的大模型训练路径。这为资源受限的研究机构和中小企业提供了可行的技术选择，可能加速开源生态的多元化发展。

混合注意力架构的实用化拐点：GDN+QSA的混合设计验证了线性注意力与稀疏注意力结合的工程可行性——既保留长上下文处理能力，又通过固定状态压缩降低计算复杂度。这一架构范式可能被后续模型（如Qwen4）广泛采用，推动注意力机制从"全量计算"向"选择性计算"演进。

部署门槛与许可证策略影响商业化路径：FP8权重172GiB的规模要求多GPU节点，限制了单机部署场景；而qwen-community-1.0许可证（非Apache-2.0）意味着商用需额外验证条款。这提示企业在采用开源模型时，需综合评估基础设施成本与法律合规风险，避免技术选型与商业目标脱节。