H Company Releases NeoMME: A Family of 260M and 800M Single-Tower Multimodal Encoders That Drop the Vision Tower and Causal Decoder
NeoMME is a family of single-tower multimodal encoders (260M and 800M parameters) that eliminates the separate vision tower and causal decoder found in prior ColPali-style models, using one bidirectional Transformer for both text and raw 32×32 image patches NeoMME-Retriever-260M achieves 0.523 nDCG@10 on ViDoRe v3, matching the 3.75B-parameter ColQwen2.5-v0.2 while being 14.4× smaller and outperforming every other sub-300M model by 26.1 points Pretraining uses discrete masked diffusion over text
65
Hot
68
Quality
62
Impact
Analysis
TL;DR
- NeoMME is a family of single-tower multimodal encoders (260M and 800M parameters) that eliminates the separate vision tower and causal decoder found in prior ColPali-style models, using one bidirectional Transformer for both text and raw 32×32 image patches
- NeoMME-Retriever-260M achieves 0.523 nDCG@10 on ViDoRe v3, matching the 3.75B-parameter ColQwen2.5-v0.2 while being 14.4× smaller and outperforming every other sub-300M model by 26.1 points
- Pretraining uses discrete masked diffusion over text optionally conditioned on visible image patches, with corruption rates of 0–1 for text-only and 0.30–1 for multimodal segments, forcing the model to leverage visual context
- Hierarchical token pooling combined with asymmetric quantization (int8/binary) reduces late-interaction index size from ~1.5 MB to as low as 6.0 kB per page (255.5× compression) with only a ~5% nDCG@10 drop
- Text-only retrieval remains a weakness (BEIR-15 scores below LateOn despite smaller size), attributed to a supervision scale gap of ~430K text query examples versus ~660M contrastive examples
Why It Matters
NeoMME demonstrates that the dominant ColPali-style repurposing of generative VLMs as encoders introduces unnecessary parameter and compute overhead; a purpose-built single-tower bidirectional encoder can match or exceed larger models at a fraction of the cost. This has direct implications for production document retrieval systems where indexing throughput, storage footprint, and inference latency are critical constraints.
Technical Details
- Architecture: A single bidirectional Transformer processes multilingual text tokens and raw 32×32 RGB image patches through the same layers. Text enters via an ALBERT-style factorized embedding (256-dim lookup projected to model width); images are patched and projected by a 2-layer MLP trained from scratch. No SigLIP2 tower, no patch-merging module, no causal decoder.
- Attention & Positional Encoding: Most layers use symmetric sliding-window attention; every sixth layer and the final layer attend globally. The stack employs grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. Context window: 16,384 tokens (sufficient for two 4K UHD images).
- Tokenizer: Whitespace-unconstrained BPE with a 131,072-entry vocabulary trained from scratch, emitting 44.4% fewer tokens than ModernBERT across 14 FLORES-200 languages.
- Pretraining: Discrete masked diffusion denoising over text, optionally conditioned on visible image patches. Text-only corruption rate uniformly sampled from [0, 1]; multimodal corruption from [0.30, 1]. Each run processes ~524B packed tokens (~290B text-only) on 16–32 H100 accelerators. Cross-modal ablation at 90% masking shows visible patches improve masked-token accuracy by ~38–40 points.
- Retrieval Heads: NeoMME-Retriever adds two jointly trained heads on the shared backbone: a mean-pooled dense head with Matryoshka widths and a late-interaction head projecting every token/patch to 128 dimensions, returned in a single forward pass.
- Compression: Hierarchical token pooling at factor 10 with int8 queries/documents yields 39.0 kB/page (39.4× reduction, 99.16% nDCG@10 retention); factor 8 with int8 queries and binary documents yields 6.0 kB/page (255.5× reduction, 95.19% retention).
- Performance: 260M model indexes 51.3 pages/sec on a single NVIDIA L40S (1.97× faster than ColModernVBERT at 26.0 pages/sec); query encoding in 78.3 ms on CPU-only. All checkpoints Apache 2.0 with Hugging Face Transformers support.
Industry Insight
- The single-tower design philosophy challenges the industry convention of repurposing generative VLMs for retrieval, suggesting that purpose-built encoders can achieve superior efficiency-performance tradeoffs—teams building document retrieval pipelines should evaluate NeoMME-style architectures before adopting ColPali-derived models.
- The dramatic index compression (up to 255×) via token pooling and quantization makes late-interaction retrieval viable at scale for production systems that previously found it prohibitively expensive in storage, opening the door to higher-fidelity retrieval without infrastructure overhauls.
- The text-retrieval gap highlights an ongoing multimodal imbalance: models trained primarily on document/visual data will underperform on pure text benchmarks unless supervision is explicitly balanced, so practitioners should set expectations accordingly and consider hybrid approaches for text-heavy workloads.
Disclaimer: The above content is generated by AI and is for reference only.
Multimodal Embedding Model Open Source Research Inference