Perplexity Open Sources Lily: A Rust + Metal Inference Engine for Qwen3.6-35B-A3B on Apple Silicon Perplexity开源Lily：面向Apple Silicon的Qwen3.6-35B-A3B Rust+Metal推理引擎

TL;DR

• Perplexity open-sourced Lily, a Rust + Metal inference engine for Qwen3.6-35B-A3B on Apple silicon, deliberately excluding PyTorch and MLX from the execution path
• The engine achieves 1.23x faster prefill and 1.35x faster decode compared to MLX-LM on a 40-core, 128 GB M5 Max at batch 1
• Key prefill optimizations include GPU-resident expert routing (+89%) and dequantization fused into grouped GEMM (+77.4%), avoiding unified memory bandwidth bottlenecks
• Decode performance gains stem from GQA packing (+23.8% at 32K context) and fixed-block attention layout (+40.2% at 128K context)
• The 4-bit checkpoint is 19.4 GB, requiring 32 GB+ unified memory on Apple Silicon, with negligible accuracy loss (0.04% perplexity increase, 96.35% top-token agreement)

Why It Matters

Lily demonstrates that narrow, hand-optimized inference engines can significantly outperform general-purpose frameworks like MLX-LM by eliminating abstraction overhead and tailoring execution to a single model architecture. This approach is particularly relevant for practitioners deploying large MoE models on Apple Silicon, where memory bandwidth and kernel efficiency are critical bottlenecks. The open-source release provides a reference implementation for researchers exploring custom inference runtimes beyond mainstream frameworks.

Technical Details

• Architecture: Single-process Rust runtime driving generation, with hand-written Metal kernels; OpenAI-compatible chat-completions API for token streaming; no PyTorch or MLX in the execution path
• Model: Qwen3.6-35B-A3B (35B total parameters, ~3B activated per token), featuring a router selecting 8 of 256 experts plus one shared expert, 10 full-attention layers with grouped-query attention (16 query heads, 2 KV heads), and 30 Gated DeltaNet layers
• Prefill optimizations: Groupwise affine 4-bit quantization (64 weights per group) with dequantization fused into grouped GEMM using threadgroup memory and FP32 accumulation; GPU-resident routing histogram, prefix scan, scatter, and block map within a single command buffer eliminating CPU synchronization; 32-row tiles with four simdgroups; register-resident Gated DeltaNet scan
• Decode optimizations: Concurrent Metal pass recording real kernel dependencies to overlap 795 independent kernels; token written directly into GPU-resident input slot eliminating per-token CPU round trips; four fused kernel chains keeping intermediates in registers; coalesced cache reads improving key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s
• Benchmarks: Tested on M5 Max (40-core, 128 GB) across 10 lengths (256 to 128K tokens); Lily averaged 4,156 prefill tokens/s vs 3,388 (MLX-LM) and 170.0 decode tokens/s vs 126.4; at 4K prompt/context: 5,749.9 prefill and 186.6 decode vs 4,737.5 and 140.9

Industry Insight

• The narrow-specialization strategy challenges the dominant trend of building universally compatible frameworks, suggesting that for production deployments on specific hardware-model combinations, custom engines can deliver meaningful performance gains (1.2-1.4x) that justify the development investment
• GPU-resident MoE routing and fused dequantization represent transferable techniques for other sparse model architectures, particularly as MoE models become more common in open-weight releases
• The 0.04% perplexity overhead and 96.35% token agreement demonstrate that aggressive kernel fusion and custom memory management can achieve near-framework parity in accuracy while substantially improving throughput, a compelling tradeoff for latency-sensitive applications

TL;DR

• Perplexity开源Lily，一个专为Qwen3.6-35B-A3B设计的Rust + Metal本地推理引擎，完全绕过PyTorch和MLX
• 在40核M5 Max上，Lily预填充速度达4,156 tokens/s（比MLX-LM快1.23倍），解码速度170 tokens/s（快1.35倍）
• 核心优化：GPU驻留专家路由（+89%）、反量化融合到分组GEMM（+77.4%）、GQA打包（+23.8%）、固定块注意力（+40.2%）
• 最低部署门槛：macOS 15+、24GB统一内存（推荐32GB），4-bit checkpoint仅19.4GB
• 精度损失极小：perplexity仅高0.04%，top-1 token一致性达96.35%

为什么值得看

Lily展示了"窄而深"的垂直优化策略如何突破通用框架的性能天花板，为Apple Silicon上的本地大模型推理提供了工业级参考实现。其开源代码和详细ablation研究为开发者优化特定模型-硬件组合提供了可直接复用的技术路径。

技术解析

架构设计：Lily采用单进程运行时，Rust层负责checkpoint加载和生成循环驱动，通过OpenAI兼容的HTTP API流式输出token。手写Metal内核直接执行模型计算，整个执行路径不依赖PyTorch或MLX，实现了从API到硬件的零中间层开销。

预填充优化：针对Qwen3.6的MoE架构（256个专家选8个），Lily将路由直方图、前缀扫描、scatter和block map全部保留在单个GPU命令缓冲区中，消除每层MoE的CPU同步开销（+89%）。4-bit groupwise affine量化（每64权重共享bfloat16缩放因子）的反量化被融合进分组GEMM，结果在threadgroup内存中累积为FP32，避免展开后的数组占用统一内存（+77.4%）。

解码优化：Batch-1解码几乎无权重复用，带宽是瓶颈。Lily通过并发Metal pass记录真实依赖关系，使独立kernel重叠执行（795个kernel形成555个顺序阶段）。选中的token直接写入下一步GPU驻留输入槽，消除每token的CPU往返。GQA打包让4个query head共享一个threadgroup，每行KV只加载一次（+23.8%）。固定块注意力布局在32K-128K上下文显著提升性能（+7.7%至+40.2%）。

硬件与部署：目标硬件为Apple Silicon（M系列），最低配置24GB统一内存，推荐32GB。4-bit量化checkpoint仅19.4GB，可在消费级Mac上运行35B参数模型。demo代码已开源在pplx-garden仓库。

行业启示

垂直优化的价值回归：在通用框架（如MLX、vLLM）追求广泛兼容性的同时，Lily证明了针对特定模型-硬件组合的深度优化仍能获得30-40%的性能增益。这对边缘部署和成本敏感场景具有重要参考价值。

Apple Silicon本地推理的工业化成熟：Lily作为Perplexity Commercial产品Hybrid Compute的技术底座，标志着Apple Silicon上的本地大模型推理已从实验性项目走向生产级应用。开源此举可能推动更多厂商跟进类似优化策略。

MoE架构的推理优化新挑战：Qwen3.6的混合架构（10层full attention + 30层Gated DeltaNet + MoE）代表了下一代模型的设计趋势。Lily针对其不均匀专家分组、增长型KV cache和固定大小循环三种模式的优化方案，为后续类似架构的推理引擎设计提供了参考范式。