Why Your Best Predictive Model Gives the Wrong Treatment Effect
Standard variable selection methods (stepwise, BIC, Lasso) optimize for predictive fit rather than causal effect estimation, leading to systematic bias when confounders weakly predict outcomes but strongly predict exposures. The "adjustment uncertainty" problem arises because variables like U2—strong predictors of exposure but weak predictors of outcome—are often excluded by predictive criteria despite being critical confounders that must be adjusted for to avoid omitted variable bias. Bayesian
Analysis
TL;DR
- Standard variable selection methods (stepwise, BIC, Lasso) optimize for predictive fit rather than causal effect estimation, leading to systematic bias when confounders weakly predict outcomes but strongly predict exposures.
- The "adjustment uncertainty" problem arises because variables like U2—strong predictors of exposure but weak predictors of outcome—are often excluded by predictive criteria despite being critical confounders that must be adjusted for to avoid omitted variable bias.
- Bayesian Adjustment for Confounding (BAC), post-double-selection, and double machine learning address this by explicitly considering the relationship between covariates and both exposure and outcome, not just the outcome alone.
Why It Matters
This distinction is crucial for AI practitioners working in causal inference, policy evaluation, or any domain where understanding true effects—not just prediction—is required. Relying on standard ML model selection tools can produce confidently wrong conclusions about causal relationships, especially in observational studies with limited sample sizes. Recognizing that prediction accuracy does not imply valid effect estimation prevents costly errors in healthcare, economics, and social science applications.
Technical Details
- The paper demonstrates through simulation that omitting a confounder (U2) that has high correlation with exposure (X) but low association with outcome (Y) leads to biased effect estimates even though such variables are typically filtered out by BIC or stepwise selection due to their poor contribution to outcome prediction.
- In the simulated dataset with 1000 observations, dropping U2 results in an estimated effect of 0.171 (95% CI: 0.127–0.214) versus the true value of 0.1, while including all relevant confounders yields 0.145 (95% CI: 0.083–0.206), which correctly covers the truth despite having worse BIC.
- Omitted variable bias formula: Bias = β_U2_outcome × δ_U2_exposure, where δ represents the coefficient from regressing X on U2. Even small β_U2_outcome combined with large δ creates substantial bias—a mechanism invisible to pure outcome-focused selection rules.
- Bayesian Model Averaging inherits this blind spot since it weights models based on posterior probability derived from likelihood/predictive performance without accounting for whether included variables serve as proper confounders.
Industry Insight
Causal modeling pipelines should never rely solely on automated feature selection tuned for prediction; instead, they must incorporate domain knowledge or use specialized algorithms designed specifically for confounder adjustment (e.g., double/debiased machine learning). Practitioners building recommendation systems, clinical decision support tools, or economic impact models need to validate that their covariate selection strategy preserves unbiased effect estimates—not merely improves R² or cross-validation scores—as failure here risks deploying systems that make decisions based on spurious correlations masquerading as causality.
Disclaimer: The above content is generated by AI and is for reference only.