Cross-Platform Generalisation Failure in Mental Health Natural Language Processing: A Five-Axis Fairness Audit of Transformer Models on Social Media
The Cross-Platform Fairness Evaluation (CPFE) framework introduces a five-axis audit protocol (discriminative performance, calibration, statistical significance, prediction equity, attribution stability) for assessing mental health NLP models across social media platforms Transformer models (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa) trained on Kaggle mental health corpus (n=35,556) suffer catastrophic cross-platform AUC degradation: 30.3-35.4% on Reddit and 37.9-39.5% on Twitter
Analysis
TL;DR
- The Cross-Platform Fairness Evaluation (CPFE) framework introduces a five-axis audit protocol (discriminative performance, calibration, statistical significance, prediction equity, attribution stability) for assessing mental health NLP models across social media platforms
- Transformer models (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa) trained on Kaggle mental health corpus (n=35,556) suffer catastrophic cross-platform AUC degradation: 30.3-35.4% on Reddit and 37.9-39.5% on Twitter versus near-perfect in-domain AUC (0.983-0.987)
- Calibration failure is severe and separable from discriminative failure: ECE increases 3-9x on out-of-platform data, and platform-specific temperature scaling recovers 88% of calibration loss without affecting AUC
- Prediction equity analysis reveals extreme disparities with equalized odds differences of 0.753-0.831 across platforms, and attribution stability shows near-complete vocabulary divergence (Jaccard J=0 in 14/16 model-class pairs)
- Target-platform fine-tuning (single seed) improves mean AUC by 0.216, suggesting platform-specific labels serve as a stronger training signal than a calibration signal
Why It Matters
This research exposes a critical reliability gap in deploying mental health NLP systems across heterogeneous social media environments, where models that appear highly accurate in controlled settings can fail catastrophically in production. For AI practitioners building clinical proxy detection systems, the findings demonstrate that standard in-domain evaluation is dangerously insufficient and that fairness, calibration, and attribution must be audited alongside discriminative performance. The CPFE framework provides a practical, multi-axis protocol that the mental health AI community can adopt as a new evaluation standard.
Technical Details
- CPFE Framework: A five-axis audit protocol evaluating (1) discriminative performance via AUC, (2) calibration via Expected Calibration Error (ECE), (3) statistical significance across five independent training seeds, (4) prediction equity via Disparity Index (DI) and equalized odds differences, and (5) attribution stability via Jaccard similarity of top-K feature vocabularies
- Models & Data: Four transformer architectures (BERT, RoBERTa, Emotion-DistilRoBERTa, GoEmotions-RoBERTa) trained on a Kaggle mental health corpus (n=35,556) and evaluated on Reddit (n=6,257) and Twitter (n=2,883) test sets, with emotion labels mapped to clinical proxies
- Calibration Findings: In-domain ECE of 0.056-0.060 degrades to 0.196-0.229 on Reddit and 0.499-0.542 on Twitter; platform-specific temperature scaling reduces mean ECE by 88.0% with negligible AUC change (|delta AUC| < 0.01), confirming calibration and discrimination are separable failure modes
- Equity & Attribution: Raw DI < 0.17 with prior-shift-adjusted DI of 0.11-0.29 on Reddit; equalized odds differences reach 0.753-0.830 (Reddit) and 0.755-0.831 (Twitter, anxiety class); vocabulary-level attribution shows Jaccard similarity of 0 in 14/16 model-class pairs at K=10, indicating near-total lexical divergence
- Fine-tuning Experiment: Single-seed target-platform fine-tuning yields a mean AUC improvement of 0.216, demonstrating that platform-specific labeled data is more valuable as a training signal than as a post-hoc calibration mechanism
Industry Insight
- Organizations deploying mental health NLP systems on social media must adopt multi-platform evaluation as a standard practice; in-domain benchmarks alone provide a false sense of reliability and could lead to harmful misclassification in real-world clinical proxy detection
- The separability of calibration and discriminative failure modes suggests that lightweight post-hoc calibration (e.g., temperature scaling) can address reliability gaps without retraining, but this is insufficient for equity and attribution concerns that require platform-aware model development
- The near-complete vocabulary divergence (Jaccard J=0) indicates that platform-specific linguistic norms fundamentally differ, implying that domain adaptation strategies should prioritize lexical and stylistic alignment rather than relying solely on architectural improvements or larger pretraining corpora
Disclaimer: The above content is generated by AI and is for reference only.