German AI consortium releases Soofi S, an open 30B model that tops benchmarks in both English and German 德国AI联盟发布Soofi S，开源30B模型在英语和德语基准测试中均名列前茅

TL;DR

• Soofi S is an open-source 30B-parameter language model developed by a German consortium, trained entirely on Deutsche Telekom’s Industrial AI Cloud.
• It utilizes a hybrid Mamba-Transformer architecture that activates only 3.2 billion parameters per token, enabling consistent throughput even with extremely long contexts.
• The model prioritizes German-language data, increasing its share from 7.2% to 15.3% across training phases, resulting in top-tier performance on German benchmarks.
• Soofi S outperforms other fully open models like OLMo 3 32B and Apertus 70B in English, German, and coding tasks, though it shows weaknesses in long-context word extraction.

Why It Matters

This release demonstrates a viable path for sovereign AI infrastructure within Europe, proving that high-performance models can be trained on domestic cloud resources without relying on foreign hyperscalers. For practitioners, it highlights the efficiency gains of hybrid Mamba-Transformer architectures, offering a scalable solution for applications requiring long-context processing with lower computational overhead.

Technical Details

• Architecture: A mixture-of-experts model based on Nvidia’s Nemotron 3 Nano design, combining Mamba-2 layers with standard attention layers. It has 31.6 billion total parameters but activates only ~3.2 billion per token.
• Memory Efficiency: Only 6 of its 52 layers maintain a KV cache, preventing the linear memory growth typical of dense transformers and allowing stable throughput at context lengths up to 256,000 tokens.
• Training Data: Processed 27 trillion tokens across three phases, heavily weighting German sources such as HPLT, German Commons, FinePDFs, and the Genios newspaper corpus.
• Performance Metrics: Achieved 73.8% on HumanEval and 84.2% on German MBPP. It leads all fully open models on aggregate German and English benchmarks, surpassing larger competitors like Apertus 70B.

Industry Insight

The success of Soofi S suggests that targeted data curation for specific languages can significantly boost model performance in those domains without compromising multilingual capabilities, challenging the dominance of English-centric training recipes. Additionally, the architectural choice validates hybrid state-space models as a practical alternative to dense transformers for enterprise deployments where long-context latency and cost are critical constraints.

TL;DR

• 德国AI联盟发布开源模型Soofi S，总参数量316亿，激活参数仅32亿，在英语和德语基准测试中超越Olmo 3 32B和Apertus 70B等全开源模型。
• 采用混合Mamba-Transformer架构，KV缓存增长极小，在长上下文（4万至25.6万token）下吞吐量保持平稳，显著优于传统稠密模型。
• 训练数据刻意增加德语比例（从7.2%升至15.3%），并整合HPLT、Genios等高质量德语语料，强化了对德语及欧洲本土知识的理解能力。
• 模型完全在德国电信工业AI云上训练，体现了主权基础设施的重要性，但在长文本高频词提取任务上存在特定弱点。

为什么值得看

本文展示了非英语语言模型如何通过特定的架构优化和数据配比实现性能突破，为多语言大模型的本地化部署提供了重要参考。其混合架构在长上下文场景下的高吞吐量表现，揭示了未来高效能推理引擎的技术演进方向。

技术解析

• 混合架构设计：Soofi S采用类似Nvidia Nemotron 3 Nano的混合Mamba-Transformer结构，总参数量31.6B，但每token仅激活约3.2B参数。这种稀疏激活机制使其计算成本接近3B模型，同时保持了30B模型的表达能力。
• 长上下文效率：传统Transformer的KV缓存随上下文线性增长，而Soofi S仅在52层中的6层维护KV缓存。这使得其在40k token上下文、32并发请求下，每GPU生成速度比14-24B稠密模型快8倍，且在4k至256k token范围内吞吐量几乎无衰减。
• 分阶段训练策略：共处理27万亿token，分为三阶段。第一阶段学习基础语言（20T token，含代码/数学/网页）；第二阶段使用更高质量数据精炼模式（6T token）；第三阶段通过百万级长文档扩展上下文窗口。
• 德语数据增强：相比Nemotron参考配方中非英语语言仅占5%，Soofi S在第一阶段将德语占比设为7.2%，第二阶段提升至15.3%。数据来源包括HPLT、German Commons、FinePDFs/Wiki及商业授权的Genios报纸语料。

行业启示

• 主权AI与基础设施自主可控：该模型完全在德国电信云上训练，证明了建立独立于美国云厂商的主权AI基础设施是可行的，对于追求数据安全和语言主权的欧洲机构具有示范意义。
• 混合架构成为长文本推理新标准：Soofi S在长上下文下的稳定吞吐量表现，表明Mamba-2等混合架构可能取代纯Transformer成为处理超长文档场景的主流选择，有助于降低推理成本。
• 垂直领域数据配比决定语言表现：通过刻意调整非英语（特别是德语）数据的训练比例，模型在特定语言基准上取得了显著优势，这提示开发者在构建多语言模型时，应根据目标市场动态调整数据权重而非简单依赖通用预训练。