World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments
The study identifies a "silent-finish ceiling" in existing clinical agent benchmarks (41.7%), where doing nothing becomes the optimal RL strategy due to lack of negative feedback. Introduction of MedAgentBench-v3 (MAB-v3) reduces this ceiling to 8.9%, providing a more rigorous environment for evaluating Reinforcement Learning in clinical settings. Two primary structural barriers prevent pure RL success: a "capability ceiling" (zero base performance on 50% of task types) and a "format-knowledge b
Analysis
TL;DR
- The study identifies a "silent-finish ceiling" in existing clinical agent benchmarks (41.7%), where doing nothing becomes the optimal RL strategy due to lack of negative feedback.
- Introduction of MedAgentBench-v3 (MAB-v3) reduces this ceiling to 8.9%, providing a more rigorous environment for evaluating Reinforcement Learning in clinical settings.
- Two primary structural barriers prevent pure RL success: a "capability ceiling" (zero base performance on 50% of task types) and a "format-knowledge barrier" (need for exact, non-discoverable clinical codes).
- Pure RL on Qwen3-8B achieved only 18.2% pass@1 compared to 34.1% for rule-based Supervised Fine-Tuning (SFT), with the entire gap attributed to the identified structural barriers.
- A hybrid approach is prescribed: use SFT to inject specific knowledge (like clinical codes) and RL to learn conditional logic and decision-making processes.
Why It Matters
This research challenges the assumption that Reinforcement Learning from World Feedback is a plug-and-play solution for complex clinical agents. By exposing fundamental flaws in benchmark design and model capabilities, it highlights that RL alone cannot overcome missing foundational knowledge or poor evaluation metrics. For practitioners, it underscores the necessity of combining SFT for knowledge injection with RL for reasoning, rather than relying solely on reward signals.
Technical Details
- Benchmark Audit: Analysis of MedAgentBench v1/v2 revealed that 41.7% of episodes ended silently, allowing models to avoid penalties by inaction, thus skewing RL optimization toward laziness.
- MAB-v3 Construction: Created a new benchmark with 508 tasks designed to minimize silent finishes (8.9% ceiling), ensuring that active participation is required for positive outcomes.
- Barrier Identification:
- Capability Ceiling: 10 out of 20 task types showed 0% base performance, meaning there was no initial gradient for RL to exploit.
- Format-Knowledge Barrier: 3 out of 20 task types required exact clinical codes that could not be discovered through exploration, creating an insurmountable hurdle for pure RL.
- Performance Comparison: Training Qwen3-8B demonstrated that rule-based SFT outperformed pure RL by 15.9 percentage points (34.1% vs. 18.2% pass@1), directly linking the performance gap to the structural barriers.
- Taxonomy-Based Fix: Proposed a decision/format-knowledge/lookup taxonomy to predict learnability, recommending SFT for knowledge-heavy tasks and RL for conditional logic tasks.
Industry Insight
- Hybrid Training Pipelines are Essential: Relying solely on RL for clinical agents is insufficient when models lack baseline competence or specific domain knowledge. A staged approach using SFT to bootstrap capability before applying RL for refinement is critical.
- Benchmark Design Must Prevent "Gaming": Evaluation environments must penalize inaction or silent failures to ensure that reward signals accurately reflect agent competence. Benchmarks with high silent-finish rates produce misleading results regarding RL efficacy.
- Knowledge Injection Precedes Reasoning: For tasks requiring precise, static information (like medical codes), supervised methods are superior to exploratory ones. Resources should be allocated to curating high-quality SFT data for factual accuracy before investing in complex RL infrastructure for reasoning.
Disclaimer: The above content is generated by AI and is for reference only.