Meet Needle 2: An Open 45M-Parameter Tool-Calling Model That Ships as a 14MB Binary and Runs a Full Session in 28MB of RAM 认识 Needle 2：一款开源的 45M 参数工具调用模型，以 14MB 二进制文件发布，仅需 28MB 内存即可运行完整会话

TL;DR

• Needle 2 is a 45M-parameter open model for tool calling, device use, and structured extraction, shipping as a single 14MB binary with ~28MB RAM usage
• It achieves 500 tokens/sec on Raspberry Pi 5 and 400–1,500 tokens/sec on AR/VR headsets and sub-$200 phones, targeting hardware without GPUs or NPUs
• The model uses a "Simple Attention Network" architecture with Hadamard MLP, GQA attention, engram key-value memory from hashed n-gram tables, and multi-lane hyper-connections across 27 layers and 512 width
• It leads on Seal-Tools in-domain (32.6) and OOD (28.7) benchmarks, with 98.3% function-name accuracy on Mobile Actions, though trails on BFCL v4 (42.6 vs 60.8 for LFM2.5)
• The system uses CQ2-bit quantization with weights never decompressing into RAM, byte-level grammar constrained decoding, contrastive retrieval for tool selection (top-5 from larger catalogs), and a confidence scoring mechanism for escalation-based failure handling

Why It Matters

Needle 2 represents a paradigm shift in on-device AI: proving that highly specialized, tiny models can outperform much larger general-purpose models on narrow tasks when the design premise is tightly scoped. For AI practitioners building for constrained hardware—wearables, IoT, automotive, robotics—this provides a practical, deployable solution that eliminates cloud dependency, reduces latency, and addresses privacy concerns where audio or data cannot leave the device.

Technical Details

• Architecture: Simple Attention Network with 27 layers, 512 width, Hadamard MLP replacing standard FFN, GQA attention, engram key-value memory from hashed n-gram tables (8M parameters as gathered memory), and multi-lane hyper-connections
• Quantization & Runtime: CQ2-bit quantization keeps weights compressed; 2-bit codes expand inside vector registers into integer dot products (int8 arithmetic path). The engine probes CPU at startup and selects optimal kernel tier (SDOT, NEON, AVX2, RISC-V vectors, wasm SIMD, or scalar)
• Memory Management: 256-token sliding window attention with system turn and tool declarations pinned as KV sinks, ensuring critical context is never evicted. Memory stays flat at ~28MB regardless of conversation length
• Tool Retrieval & Grammar: Above five tools, a contrastive retrieval head embeds each schema once and admits only top-5 per turn. Byte-level grammar compiled from JSON schemas constrains every token, skipping up to 98% of vocabulary projection on structural tokens
• Confidence & Escalation: Every response carries a confidence value (minimum of calibrated post-hoc head and decoding probability). Off-topic requests return empty call []. Users set thresholds to act, re-ask, or escalate to larger models
• Training: 115B-token proprietary pretraining corpus with 38B tokens of post-training. Only 35M of 45M parameters are matmul-active, yielding 70 MFLOPs/token—dramatically lower than LFM2.5 (460), FunctionGemma (540), or Apple FM (~6,000)

Industry Insight

• The "narrow and stated plainly" design philosophy—mapping messy sentences to typed function signatures requires no world knowledge—should guide other teams building vertical AI solutions; specialization at small scale can outcompete generalization at large scale for well-defined tasks
• The retrieval + grammar + confidence triad offers a blueprint for production tool-calling systems: retrieve relevant tools to reduce context, constrain output with grammar to eliminate invalid generations, and use calibrated confidence to handle uncertainty safely rather than hallucinating
• Companies shipping firmware, apps, or AI features on constrained hardware (wearables, IoT, automotive, robotics, POS) should evaluate Needle 2 as an offline fallback or primary model, while cloud-first SaaS teams will see limited direct benefit—this is infrastructure for the edge, not the data center

摘要

Needle 2 是一个45M参数的开源模型，用于工具调用、设备控制和结构化提取，以单个14MB二进制文件发布，内存占用约28MB
它在树莓派5上达到500 tokens/sec，在AR/VR头显和200美元以下手机上达到400–1,500 tokens/sec，针对没有GPU或NPU的硬件
该模型采用"简单注意力网络"架构，包含Hadamard MLP、GQA注意力、来自哈希n-gram表的engram键值记忆，以及跨27层和512宽度的多车道超连接
它在Seal-Tools域内（32.6）和域外（28.7）基准测试中领先，Mobile Actions函数名准确率达98.3%，但在BFCL v4上落后（42.6对比LFM2.5的60.8）
系统采用CQ2-bit量化，权重从不解压到RAM，字节级语法约束解码，对比检索用于工具选择（从更大目录中取前5个），以及基于置信度评分的升级式故障处理机制

深度分析

简而言之

• Needle 2 是一个45M参数的开源模型，用于工具调用、设备控制和结构化提取，以单个14MB二进制文件发布，内存占用约28MB
• 它在树莓派5上达到500 tokens/sec，在AR/VR头显和200美元以下手机上达到400–1,500 tokens/sec，针对没有GPU或NPU的硬件
• 该模型采用"简单注意力网络"架构，包含Hadamard MLP、GQA注意力、来自哈希n-gram表的engram键值记忆，以及跨27层和512宽度的多车道超连接
• 它在Seal-Tools域内（32.6）和域外（28.7）基准测试中领先，Mobile Actions函数名准确率达98.3%，但在BFCL v4上落后（42.6对比LFM2.5的60.8）
• 系统采用CQ2-bit量化，权重从不解压到RAM，字节级语法约束解码，对比检索用于工具选择（从更大目录中取前5个），以及基于置信度评分的升级式故障处理机制

为什么重要

Needle 2 代表了设备端AI的范式转变：证明了高度专业化的小型模型在狭窄任务上可以超越更大的通用模型，前提是设计前提高度聚焦。对于为受限硬件（可穿戴设备、物联网、汽车、机器人）构建AI的从业者来说，这提供了一个实用、可部署的解决方案，消除了对云端的依赖，降低了延迟，并解决了音频或数据不能离开设备的隐私问题。

技术细节

• 架构：简单注意力网络，27层