Building a Billion-Vector Search System Without Putting Everything in RAM
The "RAM Trap" in billion-scale vector search is a costly misconception: keeping all vectors, indexes, and metadata in RAM is economically unsustainable, and a deliberate tiered storage architecture is the sustainable alternative. Qdrant enables per-component memory placement (vectors, HNSW graph, payload indexes independently in RAM or on disk), with mmap-based on-disk storage that leverages the OS page cache rather than treating disk as a slow fallback. Quantization is the primary memory lever
Analysis
TL;DR
- The "RAM Trap" in billion-scale vector search is a costly misconception: keeping all vectors, indexes, and metadata in RAM is economically unsustainable, and a deliberate tiered storage architecture is the sustainable alternative.
- Qdrant enables per-component memory placement (vectors, HNSW graph, payload indexes independently in RAM or on disk), with mmap-based on-disk storage that leverages the OS page cache rather than treating disk as a slow fallback.
- Quantization is the primary memory lever, offering 4x (scalar), up to 32x (binary/TurboQuant), and up to 64x (product) compression, with asymmetric quantization allowing binary-stored vectors to be rescoring against original float32 precision.
- The hybrid retrieval pattern—quantized vectors in RAM for initial candidate discovery, original full-precision vectors on disk for final rescoring—decouples compression from accuracy loss and dramatically reduces RAM requirements.
- At 384 dimensions, a billion-vector collection stored as float32 requires ~1.54 TB of raw vector memory alone, excluding HNSW graph links, payload indexes, replicas, and OS overhead, making quantization + hybrid storage essential for production scale.
Why It Matters
This article directly addresses one of the most pressing cost and scalability challenges in production AI: how to run billion-scale vector search without provisioning multi-terabyte RAM clusters. For AI practitioners building recommendation engines, semantic search, or RAG pipelines, the hybrid quantization + mmap architecture described here offers a practical, benchmark-backed path to reducing infrastructure costs by an order of magnitude while preserving recall. The findings are especially relevant as embedding dimensions grow and dataset scales push past the RAM-resident paradigm that has dominated the field.
Technical Details
- Billion-vector memory baseline: A single 384-dimensional float32 vector occupies 1,536 bytes; one billion vectors require ~1.54 TB of raw storage. At 768 dimensions this rises to ~3 TB. Additional overhead from HNSW graph connectivity, payload indexes, write amplification, replicas, and OS page cache makes the true RAM requirement significantly larger.
- Qdrant storage model: Vectors, HNSW graphs, and payload indexes can each be independently placed in RAM or on disk. On-disk storage uses memory-mapped files (mmap) that route through the OS page cache, enabling dynamic caching of hot data without preloading the entire dataset into physical RAM.
- Quantization methods: Scalar quantization (int8) provides 4x compression with minimal accuracy loss. Binary quantization achieves up to 32x compression and enables fast XOR/popcount distance calculations. Product quantization reaches 64x compression via centroid indexing but suffers slower non-SIMD distance computation and greater recall loss. TurboQuant targets 32x compression with improved recall retention across diverse embedding models.
- Asymmetric quantization: Vectors are stored in compressed binary form for RAM efficiency, but incoming queries are scored using scalar-quantized representations, improving precision without increasing storage costs.
- Hybrid retrieval architecture: Quantized vectors remain in RAM for fast initial candidate discovery, while original full-precision vectors are stored on disk and retrieved only during the rescoring phase for a small candidate set. This decouples the compression decision from the accuracy decision, allowing aggressive quantization without proportional recall loss.
- Benchmark methodology: Evaluated on 105,126 real H&M product embeddings (384 dimensions) from Qdrant's published hm_ecommerce_products dataset, with synthetic price, availability, and geography metadata layered for filtering. All recall and latency claims are measured results; billion-vector figures are extrapolated from the benchmark.
Industry Insight
- The industry should treat RAM as a caching tier rather than a storage tier for vector data. Architectures that embrace mmap-backed disk storage with OS page cache management can achieve near-in-memory performance at a fraction of the cost, particularly when combined with quantization for the initial search pass.
- Hybrid quantization + rescoring is becoming a best practice for production vector search at scale. Teams should evaluate TurboQuant and asymmetric quantization as default starting points, reserving product quantization only for extreme memory-constrained scenarios where recall loss is acceptable.
- As embedding models produce higher-dimensional vectors (768d, 1536d), the RAM cost curve steepens dramatically. Investing in multi-tier storage architectures now—rather than scaling RAM linearly—will yield compounding cost savings and position systems to handle future dimensionality growth without proportional infrastructure spend.
Disclaimer: The above content is generated by AI and is for reference only.