Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment
Post-training ternarization of Qwen3-4B achieves 1.641 effective bits per weight, compressing model size from 8.29 GiB to 3.96 GiB while retaining 81.62% of parameters Task capability degrades unevenly: BoolQ retains 84.6% of teacher performance, while ARC-Challenge drops to 43.8%, with overall accuracy falling from 64.5% to 54.7% Perplexity increases significantly across benchmarks (WikiText-2: 13.639→18.748, PTB: 24.700→31.992, C4: 19.81→28.966), indicating notable quality loss A preliminary T
Analysis
TL;DR
- Post-training ternarization of Qwen3-4B achieves 1.641 effective bits per weight, compressing model size from 8.29 GiB to 3.96 GiB while retaining 81.62% of parameters
- Task capability degrades unevenly: BoolQ retains 84.6% of teacher performance, while ARC-Challenge drops to 43.8%, with overall accuracy falling from 64.5% to 54.7%
- Perplexity increases significantly across benchmarks (WikiText-2: 13.639→18.748, PTB: 24.700→31.992, C4: 19.81→28.966), indicating notable quality loss
- A preliminary Triton GEMV microbenchmark shows 4.6x slower inference compared to FP16 cuBLAS, meaning compression alone does not yield faster inference
- The study demonstrates that nominal bit labels (e.g., "1.58-bit") are insufficient to capture effective bit accounting, capability retention, and runtime behavior
Why It Matters
This research provides a critical reality check for the ultra-low-bit quantization community by demonstrating that aggressive ternarization of a 4B-parameter instruction-tuned model incurs substantial and uneven capability degradation. For practitioners considering extreme compression for edge deployment, the findings highlight that storage savings come at the cost of both performance and inference speed, challenging the assumption that quantization alone can enable efficient deployment without complementary optimizations.
Technical Details
- Methodology: End-to-end post-training conversion using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA; weight-only quantization with activations retained at 16-bit precision
- Effective bit accounting: 1.641 effective bits per quantized linear weight, with 81.62% of model parameters targeted for ternarization
- Benchmark results: Evaluated across ten scored capability comparisons and three perplexity datasets (WikiText-2, PTB, C4); degradation is highly task-dependent, with reasoning-heavy benchmarks suffering more than factual recall tasks
- Packing optimization: A subsequent packing run preserving ternary planes and scales reduced model size by ~52% (8.29 GiB → 3.96 GiB) with negligible perplexity impact; a third-party lossy packing attempt was excluded
- Deployment benchmarking: Preliminary Triton GEMV microbenchmark shows 4.6x slowdown versus FP16 cuBLAS on tested shapes; end-to-end task accuracy and generation throughput for the packed artifact remain unbenchmarked
Industry Insight
- The uneven degradation pattern suggests that ultra-low-bit quantization may be viable for narrow, fact-retrieval-heavy applications but remains risky for reasoning-intensive tasks; practitioners should benchmark per-task before deploying ternarized models in production
- Compression-only strategies are insufficient for inference acceleration at extreme bit rates; hardware-aware kernel optimization (e.g., custom Triton kernels) must advance in parallel with quantization research to realize end-to-end efficiency gains
- The gap between the packed artifact's storage metrics and its unbenchmarked runtime performance underscores the need for standardized end-to-end evaluation pipelines that report both compression ratios and actual deployment throughput
Disclaimer: The above content is generated by AI and is for reference only.