MedProb: Probing Internal Representations of Vision-Language Models for Medical Question Answering
MedProb is a lightweight probing framework that predicts multiple-choice Med-VQA answers directly from frozen VLM representations, eliminating the need for free-text generation It outperforms standard prompting and even medical VLMs and agentic systems across PATH-VQA, SLAKE, and VQA-RAD benchmarks Probing significantly narrows the performance gap between small and large VLMs, revealing that smaller models contain more recoverable medical VQA signal than generation-based evaluation suggests Medi
Analysis
TL;DR
- MedProb is a lightweight probing framework that predicts multiple-choice Med-VQA answers directly from frozen VLM representations, eliminating the need for free-text generation
- It outperforms standard prompting and even medical VLMs and agentic systems across PATH-VQA, SLAKE, and VQA-RAD benchmarks
- Probing significantly narrows the performance gap between small and large VLMs, revealing that smaller models contain more recoverable medical VQA signal than generation-based evaluation suggests
- Medical fine-tuning does not consistently improve linear decodability of internal representations across 14 matched general-purpose and medical VLM pairs
- Free-text generation exhibits an answer-position bias of up to 10 percentage points, while MedProb's positional bias is impacted differently, suggesting distinct failure modes
Why It Matters
This work challenges the prevailing assumption that medical VQA requires expensive medical fine-tuning, large-scale models, or complex multi-agent pipelines. It demonstrates that frozen, off-the-shelf VLM representations already encode substantial medical reasoning signal that can be extracted with simple linear probes, which has major cost and efficiency implications for deploying VQA systems in clinical settings.
Technical Details
- MedProb operates on frozen VLM backbones without any gradient-based fine-tuning, using a lightweight probe to map internal representations directly to multiple-choice answers
- Evaluated across three established Med-VQA benchmarks: PATH-VQA, SLAKE, and VQA-RAD, where it recovers substantially more answer-relevant signal than standard prompting approaches
- Comparative analysis across 14 matched general-purpose and medical VLM pairs shows that medical adaptation does not consistently enhance linear decodability of representations
- The framework also addresses answer-position bias, finding that free-text generation suffers up to 10 percentage points of positional bias, while MedProb's bias profile differs qualitatively from prompting
- The probe is extendable to open-ended generation through a rejection-sampling scoring procedure, broadening its applicability beyond multiple-choice settings
Industry Insight
- Organizations can potentially bypass costly medical fine-tuning and large-model deployment by leveraging probing on frozen general-purpose VLMs, achieving competitive Med-VQA performance at a fraction of the compute cost
- The finding that medical adaptation does not consistently improve linear decodability suggests that current medical VLM training pipelines may not be optimally aligning internal representations for downstream extractability, warranting re-examination of fine-tuning objectives
- The significant answer-position bias in free-text generation highlights a reliability concern for production VQA systems; probing-based evaluation may serve as a more robust diagnostic tool for assessing true model capability versus generation artifacts
Disclaimer: The above content is generated by AI and is for reference only.