Calibration-Preserving Pruning: Compression as a Reliability Contract
Calibration-Preserving Pruning (CPP) combines base pruning scores with nonconformity-gradient saliency to compress models while preserving prediction-set efficiency under conformal prediction Split conformal prediction guarantees finite-sample marginal coverage independently of the pruning rule; CPP addresses the separate efficiency problem of producing smaller valid prediction sets On Qwen2.5-1.5B at 50% sparsity, CPP-SparseGPT reduced mean set size from 10.1 to 8.6 on DBpedia-14 with improved
Analysis
TL;DR
- Calibration-Preserving Pruning (CPP) combines base pruning scores with nonconformity-gradient saliency to compress models while preserving prediction-set efficiency under conformal prediction
- Split conformal prediction guarantees finite-sample marginal coverage independently of the pruning rule; CPP addresses the separate efficiency problem of producing smaller valid prediction sets
- On Qwen2.5-1.5B at 50% sparsity, CPP-SparseGPT reduced mean set size from 10.1 to 8.6 on DBpedia-14 with improved accuracy (0.347→0.366), outperforming baseline Wanda
- Matched controls reveal that generic supervised gradients account for much of CPP's gain; true-label CPP is statistically indistinguishable from Wanda+SNIP, while threshold-aware candidate-label CPP achieves 7.8 mean set size at explicit accuracy and offline-compute costs
- Results generalize across RoBERTa-base and Llama-3-8B diagnostics but claims are limited to reliability-sensitive classification tasks
Why It Matters
This work bridges model compression and statistical reliability, showing that pruning can be designed not just for accuracy retention but for conformal prediction efficiency—a critical concern for deployed AI systems where prediction-set calibration directly impacts trust and safety. For practitioners deploying pruned models in high-stakes classification settings, CPP offers a principled path to smaller, valid prediction sets without sacrificing coverage guarantees.
Technical Details
- CPP augments a base pruning score (SparseGPT or Wanda) with nonconformity-gradient saliency, using four disjoint data splits: pruning, validation-selection, conformal-calibration, and test
- Theoretical analysis shows bounded score perturbations imply bounded conformal-quantile shifts and controlled prediction-set inflation, though the generic coverage theorem is not CPP-specific
- Empirical evaluation on Qwen2.5-1.5B at 50% sparsity across 15 dataset-sparsity cells: CPP-SparseGPT produced smaller sets in 13/15 and higher accuracy in 11/15 compared to baselines
- Threshold-aware candidate-label CPP reaches a mean set size of 7.8 on DBpedia-14 with explicit accuracy and offline-compute costs, outperforming true-label variants
- Transfer diagnostics on RoBERTa-base and Llama-3-8B support generalization, though the framework is currently limited to classification tasks requiring reliability guarantees
Industry Insight
- As AI systems face increasing regulatory scrutiny around uncertainty quantification, pruning methods that preserve conformal coverage guarantees will become essential for deploying compressed models in regulated domains like healthcare and finance
- The finding that generic supervised gradients explain much of CPP's gain suggests practitioners can achieve similar efficiency improvements through simpler gradient-based pruning without the overhead of conformal-aware scoring, depending on their compute constraints
- The trade-off between offline-compute costs and prediction-set efficiency in threshold-aware CPP highlights a practical design decision: organizations should invest in candidate-label CPP only when downstream prediction-set size directly impacts latency or cost constraints
Disclaimer: The above content is generated by AI and is for reference only.