PrismML Releases Bonsai 27B: 1-bit and Ternary Builds of Qwen3.6-27B That Run on Laptops and Phones
PrismML releases Bonsai 27B, a quantized version of Qwen3.6-27B using ternary (1.71 bits/weight) and binary (1.125 bits/weight) representations without retraining. The ternary variant retains 94.6% of the FP16 baseline performance in a compact 5.9GB footprint, while the 1-bit variant retains 89.5% at 3.9GB. Unlike conventional sub-4-bit quantizations that suffer selective collapse on complex reasoning tasks, Bonsai maintains robust performance on math, coding, and agentic benchmarks. The archite
Analysis
TL;DR
- PrismML releases Bonsai 27B, a quantized version of Qwen3.6-27B using ternary (1.71 bits/weight) and binary (1.125 bits/weight) representations without retraining.
- The ternary variant retains 94.6% of the FP16 baseline performance in a compact 5.9GB footprint, while the 1-bit variant retains 89.5% at 3.9GB.
- Unlike conventional sub-4-bit quantizations that suffer selective collapse on complex reasoning tasks, Bonsai maintains robust performance on math, coding, and agentic benchmarks.
- The architecture leverages linear attention for ~75% of operations and a separate 4-bit vision tower, enabling a 262K token context window feasible for mobile and edge deployment.
- Integration with DSpark speculative decoding provides a 1.37x throughput speedup on H100s, making the models highly efficient for both local inference and cloud serving.
Why It Matters
This release demonstrates that extreme low-bit quantization (1-2 bits) can preserve the capabilities of large language models significantly better than traditional methods, challenging the assumption that such compression leads to inevitable performance collapse. For AI practitioners, it offers a viable path to deploying 27B-class multimodal models on consumer hardware like iPhones and laptops, bridging the gap between high-performance cloud models and resource-constrained edge devices.
Technical Details
- Quantization Method: Uses a group-wise scaling approach where each weight is represented as $w_i = s_g \cdot t_i$, with one FP16 scale per 128 weights. Ternary weights use ${-1, 0, +1}$ (1.71 bpw), and binary weights use ${-1, +1}$ (1.125 bpw).
- Architecture Preservation: The base Qwen3.6-27B architecture remains unchanged, including a ~24.8B language model, 0.46B vision tower (held at 4-bit HQQ), and 2.5B embeddings/head. Approximately 75% of attention is linear to manage KV cache growth.
- Performance Benchmarks: Evaluated on 15 benchmarks including AIME26, LiveCodeBench, and MMLU-Redux. Ternary Bonsai scores 80.49 average thinking score vs 85.07 for FP16, significantly outperforming IQ2_XXS (72.73) which collapses on complex reasoning tasks.
- Deployment Efficiency: The 1-bit model fits within the ~6GB usable memory of a 12GB iPhone due to iOS app memory limits. On H100 GPUs, it achieves 104.8 tokens/sec in generation and 2755 tokens/sec in prefill, with speculative decoding boosting generation to 143.8 tokens/sec.
Industry Insight
- Edge AI Viability: The ability to run 27B-class models on mobile devices with minimal battery impact (672 tokens per 1% battery) signals a shift toward on-device AI agents, reducing reliance on cloud infrastructure for latency-sensitive or privacy-critical applications.
- Quantization Strategy Shift: Developers should reconsider standard sub-4-bit quantizations for complex reasoning tasks, as they may hide catastrophic failures in agentic or mathematical benchmarks. Specialized low-bit formats like Bonsai offer a safer alternative for maintaining capability density.
- Hardware-Aware Optimization: The success of linear attention and specific KV cache management strategies highlights the importance of co-designing model architectures with deployment constraints, particularly for mobile and edge scenarios where memory bandwidth and capacity are strict binding constraints.
Disclaimer: The above content is generated by AI and is for reference only.