Meet NeuroVFM: A New Neuroimaging Foundation Model Trained With Vol-JEPA on Uncurated Clinical MRI and CT Volumes
NeuroVFM is a generalist visual foundation model for neuroimaging trained on 5.24 million uncurated clinical MRI and CT volumes using the Vol-JEPA self-supervised framework. The model achieves state-of-the-art performance with a macro-averaged AUROC of 92.68 on CT and 92.49 on MRI across 156 diagnostic tasks, outperforming baselines that rely on report supervision or voxel reconstruction. Vol-JEPA utilizes latent space prediction rather than pixel reconstruction, enabling efficient training with
Analysis
TL;DR
- NeuroVFM is a generalist visual foundation model for neuroimaging trained on 5.24 million uncurated clinical MRI and CT volumes using the Vol-JEPA self-supervised framework.
- The model achieves state-of-the-art performance with a macro-averaged AUROC of 92.68 on CT and 92.49 on MRI across 156 diagnostic tasks, outperforming baselines that rely on report supervision or voxel reconstruction.
- Vol-JEPA utilizes latent space prediction rather than pixel reconstruction, enabling efficient training with fewer than 1,000 GPU hours and significantly faster convergence compared to prior methods.
- Downstream applications include automated report generation, triage prioritization, and grounded predictions, with the NeuroVFM-LLaVA system demonstrating superior triage accuracy (92.6%) compared to frontier generalist models like GPT-5.
Why It Matters
This development bridges a critical gap in medical AI by demonstrating that large-scale, uncurated clinical data can effectively train robust foundation models without relying on expensive paired radiology reports or disease-specific curation. It validates the scalability of JEPA-style self-supervised learning for volumetric medical imaging, offering a pathway to generalize AI tools across diverse modalities and institutions. For practitioners, it provides a high-performance, open-source baseline that reduces the dependency on labeled datasets while maintaining clinical utility for decision support.
Technical Details
- Architecture and Training: The base model, Vol-JEPA, extends I-JEPA/V-JEPA to 3D volumes by tokenizing MRI/CT scans into 4×16×16-voxel patches. It employs a student-predictor-teacher framework where the student predicts latent representations of masked regions based on visible context, guided by an EMA teacher encoder.
- Data and Masking Strategy: Trained on the UM-NeuroImages dataset (566,915 studies), the model uses foreground-focused masking with precomputed head masks to force learning of neuroanatomy rather than background shortcuts. Context ratios are set at 25% for MRI and 20% for CT, with 20% patch dropout.
- Performance Benchmarks: NeuroVFM outperformed models using report/language supervision (HLIP, PRIMA) and voxel reconstruction (NeuroMAE) when trained on the same data. It also surpassed generalist natural image models (DINOv3, BiomedCLIP) on aggregate AUROC metrics.
- Efficiency and Scale: The model features an 85.8M parameter encoder (with a 21.7M small variant) and trains over 7x faster than 3DINO baselines, fitting 16x larger batches due to the absence of a voxel decoder.
- Downstream Integration: The frozen encoder was paired with Qwen3-14B in a LLaVA-1.5 style architecture for report generation and triage, utilizing an attention-based MIL pooler for grounded predictions without requiring region-level annotations.
Industry Insight
- Shift from Supervised to Self-Supervised Medical AI: The success of Vol-JEPA suggests that future medical foundation models will increasingly rely on self-supervised latent prediction on raw, uncurated clinical data, reducing the bottleneck of manual annotation and report pairing.
- Clinical Decision Support over Autonomous Diagnosis: The prospective study results highlight that while AI can match or exceed human-level triage accuracy, sensitivity gaps remain (missing 13.5% of critical findings). This reinforces the industry standard of positioning such models as decision-support tools rather than autonomous diagnostic agents.
- Cost-Effective Deployment: The high training efficiency and small model size (85.8M parameters) make NeuroVFM highly deployable in resource-constrained healthcare settings, potentially lowering the barrier to entry for hospitals seeking to implement advanced imaging analytics without massive computational infrastructure.
Disclaimer: The above content is generated by AI and is for reference only.