How I Reproduced BM25, Dense Retrieval, and SPLADE on a 16GB MacBook
The article emphasizes the importance of reproducing baseline retrieval methods (BM25, dense retrieval, and SPLADE) to validate claims in RAG systems. It details a practical approach using public datasets (MS MARCO and NFCorpus) and open-source toolkits (Anserini and Pyserini) to run these methods locally. The author encountered setup challenges such as out-of-memory crashes and native library issues, which are not well-documented but crucial for successful implementation.
Analysis
TL;DR
- The article emphasizes the importance of reproducing baseline retrieval methods (BM25, dense retrieval, and SPLADE) to validate claims in RAG systems.
- It details a practical approach using public datasets (MS MARCO and NFCorpus) and open-source toolkits (Anserini and Pyserini) to run these methods locally.
- The author encountered setup challenges such as out-of-memory crashes and native library issues, which are not well-documented but crucial for successful implementation.
Why It Matters
This article is relevant because it provides a hands-on guide for AI practitioners and researchers to verify the performance of different retrieval methods, ensuring that their RAG systems or search products are built on solid foundations. By understanding the strengths and limitations of BM25, dense retrieval, and SPLADE, professionals can make informed decisions about which method to use in their applications.
Technical Details
- BM25: A keyword scoring method that counts shared words between queries and documents, weighing them based on rarity and frequency. It produces sparse vectors and requires no training data.
- Dense Retrieval: Uses trained neural networks to convert text into embeddings, comparing document and query embeddings via dot product. This method requires large amounts of labeled query-document pairs for training.
- Learned Sparse Retrieval (SPLADE): Combines elements of both BM25 and dense retrieval by producing sparse vectors with learned word weights. SPLADE version 3 was used in this study.
- Datasets: MS MARCO passage ranking (8,841,823 passages, 6,980 queries) and NFCorpus (3,633 documents, 3,237 queries).
- Toolkits: Anserini (Java-based) and Pyserini (Python-based) were used for implementation.
Industry Insight
- Reproducing baseline retrieval methods is essential for validating the effectiveness of new techniques in RAG systems and search products.
- Understanding the trade-offs between BM25, dense retrieval, and learned sparse retrieval can help optimize search quality and efficiency in production environments.
- Addressing setup challenges early, such as memory constraints and library dependencies, can save significant time and resources during development.
Disclaimer: The above content is generated by AI and is for reference only.