Liquid AI Open-Sources Pipette: A Reproducible Benchmarking Suite That Measures On-Device Models, Quantization, Runtime and Hardware Together
Liquid AI open-sourced Pipette, an Apache 2.0 benchmarking suite that measures on-device LLM performance as a function of the full deployment configuration (model + quantization + runtime + device), not the model in isolation. The launch dataset covers 1,000+ configurations across 30+ models, multiple quantization formats, llama.cpp builds for macOS/iOS/Windows/Android, and context lengths from 256 to 8,192 tokens, with verified results from M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra. Quality e
Analysis
TL;DR
- Liquid AI open-sourced Pipette, an Apache 2.0 benchmarking suite that measures on-device LLM performance as a function of the full deployment configuration (model + quantization + runtime + device), not the model in isolation.
- The launch dataset covers 1,000+ configurations across 30+ models, multiple quantization formats, llama.cpp builds for macOS/iOS/Windows/Android, and context lengths from 256 to 8,192 tokens, with verified results from M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra.
- Quality evaluations (IFBench, GPQA Diamond, MATH-500) are run on NVIDIA H100 reference systems and matched to on-device performance runs by model and quantization, rather than being measured directly on the edge device.
- Published comparisons reveal that identical parameter counts can diverge dramatically in context-scaling retention (78.4% vs 33.8%), sparse activation improves speed but not memory, and near-identical system profiles can mask significant task-level quality reversals.
Why It Matters
Pipette addresses a critical gap in the AI ecosystem: server-side model cards rarely predict on-device behavior, leaving practitioners flying blind when deploying to edge hardware. By treating the deployment configuration as the unit of measurement and providing an open, independently validated benchmarking framework, it gives developers, OEMs, and researchers a practical tool for model selection, hardware procurement, and regression testing across real-world edge constraints.
Technical Details
- Benchmarking methodology: Pipette uses fixed token shapes, greedy decoding, a discarded warm-up phase, five measured repetitions, and readiness gating that verifies thermal and load conditions before each timed run; runs failing readiness checks are excluded from publication.
- Unit of measurement: The fundamental benchmark unit is a full deployment configuration — model + quantization + runtime + device — rather than the model alone, reflecting the reality that on-device behavior is a property of the deployed system.
- Quality-performance decoupling: Quality scores come from llama.cpp evaluations on NVIDIA H100 80GB reference systems and are matched to on-device performance runs sharing the same model and quantization; the quality number displayed alongside phone throughput was not produced on the phone.
- Launch scope: 30+ models, multiple quantization formats, llama.cpp builds for macOS/iOS/Windows/Android, context lengths 256–8,192 tokens, five on-device performance metrics, with initial verified results from MacBook Pro M5 Max, iPhone 17 Pro, and Galaxy S26 Ultra.
- Key empirical findings: (1) Granite-4.0-H-350M retains 78.4% decode throughput at 4,096 tokens vs. 33.8% for Granite-4.0-350M at identical quantization; (2) LFM2.5-8B-A1B decodes 2.4x faster than Qwen3.5-4B despite all expert weights occupying memory (5.29 GiB peak); (3) MiniCPM5-1B is 15.8% faster than LFM2.5-1.2B-Instruct on iPhone 17 Pro, but LFM scores 9.0 points higher on MATH-500; (4) Granite-4.1-8B and Ministral-3-8B differ by only 2.4% throughput and 1.2% RAM on M5 Max, yet Granite leads IFBench by 7.3 points while Ministral leads GPQA Diamond by 14.0 points.
Industry Insight
- Teams shipping models onto hardware they do not own — from solo developers to large OEMs — should adopt configuration-aware benchmarking rather than relying on server-side model cards; Pipette's tiered deployment model (dashboard for startups, internal fleet clients for mid-market, full pipeline behind firewall for enterprises) makes this accessible across organization sizes.
- The finding that speed and quality do not co-locate on edge devices means practitioners must explicitly define their latency-quality tradeoff before model selection; a 15.8% speed gain may come at the cost of a 9-point MATH-500 deficit, and near-identical throughput profiles can hide large task-level reversals.
- As on-device inference becomes a strategic priority for consumer electronics, automotive, robotics, and defense sectors, Pipette's readiness-gating methodology (thermal and load verification) sets a new standard for reproducible edge benchmarking that vendors can no longer gloss over with idealized server-side numbers.
Disclaimer: The above content is generated by AI and is for reference only.