Response Magnitude as a Dominant Signal for Held-Out CRISPRi Perturbation Effect Prediction
Deep perturbation predictors on the VCC benchmark underperform simple baselines when predicting held-out CRISPRi perturbation effects on target genes A low-dimensional signal—four deterministic scalar functions of the 2,000-dimensional input—drives the performance gap, strongly predicting log Anderson-Darling distance from controls Linear regression on these four magnitude scalars alone exceeds the strongest classical x-only model, while a Random Forest combining input plus scalars substantially
Analysis
TL;DR
- Deep perturbation predictors on the VCC benchmark underperform simple baselines when predicting held-out CRISPRi perturbation effects on target genes
- A low-dimensional signal—four deterministic scalar functions of the 2,000-dimensional input—drives the performance gap, strongly predicting log Anderson-Darling distance from controls
- Linear regression on these four magnitude scalars alone exceeds the strongest classical x-only model, while a Random Forest combining input plus scalars substantially outperforms a deep MLP encoder
- Magnitude-only predictors show positive zero-shot transfer to external CRISPRi screens, whereas expression-only predictors perform negatively or remain unresolved
- The Anderson-Darling metric distributed with these screens measures transcriptome-wide response breadth rather than target-gene effect strength, revealing a mismatch in evaluation
Why It Matters
This work exposes a critical failure mode in deep learning approaches to biological perturbation prediction: complex models can collapse toward marginal training means when a simpler, low-dimensional signal dominates the target variable. For AI practitioners working in computational biology, it underscores the importance of auditing what signals deep models actually learn versus what simpler baselines capture, and highlights that benchmark evaluation metrics may not align with the intended biological question.
Technical Details
- The study evaluates perturbation prediction on the Virtual Cell Challenge (VCC) benchmark under a strict held-out target-gene split, with the prediction target being the log Anderson-Darling distance from non-targeting controls
- Four deterministic scalar functions extracted from the 2,000-dimensional gene expression input serve as a strong predictor; a linear regression on these scalars alone surpasses the best classical x-only model
- A deep MLP encoder with direct access to the full input representation collapses toward the marginal training mean, and standard regularization remedies fail to close the performance gap
- A Random Forest combining the full input with the four magnitude scalars substantially outperforms the deep proof-of-concept encoder, with pre-specified controls attributing the gain to per-row alignment rather than added dimensionality
- Zero-shot transfer experiments on two external CRISPRi screens show magnitude-only predictors transfer positively, while expression-only predictors are negative or unresolved; exposing magnitude to the deep encoder improves transfer but still does not beat four-scalar linear regression
Industry Insight
- Researchers building deep learning models for biological perturbation prediction should first audit whether their targets are driven by low-dimensional summary statistics rather than complex input patterns, as simpler baselines may already capture the dominant signal
- Benchmark design in computational biology must ensure that evaluation metrics (e.g., Anderson-Darling distributions) align with the intended biological endpoint, as mismatches can produce misleading transfer assessments
- For practitioners working with single-cell CRISPR screens, combining hand-crafted magnitude features with flexible models like Random Forests may yield better generalization than end-to-end deep encoders, especially under distribution shift across cell types
Disclaimer: The above content is generated by AI and is for reference only.