Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA
The paper reframes medical QA data acquisition from "which questions to label" to "which already-labeled questions deserve rationale supervision under a fixed token budget" RMS-RSP (Root-Mean-Square Robustness-based Sample Prioritization) perturbs hidden states only at rationale tokens and measures the resulting shift in the gold-versus-best-distractor margin to select high-value samples On five medical QA datasets with MedGemma-4B-IT, RMS-RSP achieves 60.61% average locked-budget accuracy versu
Analysis
TL;DR
- The paper reframes medical QA data acquisition from "which questions to label" to "which already-labeled questions deserve rationale supervision under a fixed token budget"
- RMS-RSP (Root-Mean-Square Robustness-based Sample Prioritization) perturbs hidden states only at rationale tokens and measures the resulting shift in the gold-versus-best-distractor margin to select high-value samples
- On five medical QA datasets with MedGemma-4B-IT, RMS-RSP achieves 60.61% average locked-budget accuracy versus 60.08% for Random, with a statistically significant gain only on AfriMed-QA (+1.44 points)
- While RMS-RSP does not universally improve raw accuracy, it consistently improves robust accuracy (+1.91 points) and semantic consistency (+2.85 points) across three answer-option reorderings on all five datasets
- Full-supervision training on all rationales reaches 63.74% macro accuracy but consumes 29–254× more tokens without uniformly improving robustness
Why It Matters
This work addresses a critical bottleneck in medical AI: high-quality rationales are scarce and expensive to validate, making efficient allocation of rationale supervision essential for practitioners working under resource constraints. The findings challenge the assumption that more rationale data is always better, showing that perturbation-sensitive selection can yield more robust models without proportional token costs. For AI practitioners building medical QA systems, this provides a principled, computationally lightweight method to prioritize which training examples deserve expensive rationale supervision.
Technical Details
- RMS-RSP Method: The proposed approach perturbs hidden states exclusively at rationale tokens and quantifies the resulting shift in the margin between the gold answer and the best distractor, using root-mean-square aggregation of these perturbations as a selection criterion
- Experimental Setup: Evaluated across five medical QA datasets using MedGemma-4B-IT, three training seeds, ten budgeted non-RSP baseline selectors, and an unbudgeted full-supervision reference condition
- Robustness Evaluation: Robustness was measured through three answer-option reorderings, assessing both robust accuracy and semantic consistency—metrics that capture invariance to superficial formatting changes
- Token Efficiency: Full rationale supervision required 29–254× more rationale tokens than the budgeted RMS-RSP approach, highlighting the extreme cost of exhaustive rationale annotation
- Key Finding: Rationale-local boundary sensitivity (measuring how perturbations at rationale tokens affect decision margins) correlates with improved invariance to semantically equivalent reformatting, even when absolute accuracy gains are modest
Industry Insight
- Rationale quality and selection strategy should be treated as a resource allocation problem rather than a simple data-scaling exercise; investing in perturbation-based prioritization can yield better robustness per token than brute-force rationale collection
- Medical QA systems deployed in real-world settings will encounter formatting variations (reordered options, paraphrased stems); prioritizing rationales that improve invariance to such changes may matter more than marginal accuracy gains on static benchmarks
- The modest raw accuracy improvements from RMS-RSP suggest that perturbation-sensitive selection is best positioned as a complementary strategy—pairing it with other data curation methods rather than relying on it as a standalone solution for robust medical QA
Disclaimer: The above content is generated by AI and is for reference only.