Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
Internalizing documents into LoRA adapters enables closed-book QA without retrieval or context-window limits. Training-data quality is the dominant factor for accuracy once adapter capacity is adequate, outweighing architectural changes. A single curation pass (shortening answers and dropping trivia) significantly improved closed-book accuracy from 57.7% to 85.7%. Adapter rank must grow with corpus size, and there is a coupling between rank and learning rate that was initially misdiagnosed. The
Analysis
TL;DR
- Internalizing documents into LoRA adapters enables closed-book QA without retrieval or context-window limits.
- Training-data quality is the dominant factor for accuracy once adapter capacity is adequate, outweighing architectural changes.
- A single curation pass (shortening answers and dropping trivia) significantly improved closed-book accuracy from 57.7% to 85.7%.
- Adapter rank must grow with corpus size, and there is a coupling between rank and learning rate that was initially misdiagnosed.
- The internalized adapter outperforms a BM25-RAG pipeline and a realistic gold-chunk oracle at lower latency.
Why It Matters
This research highlights the importance of data quality in closed-book QA systems, showing that well-curated data can significantly enhance performance more than architectural changes. It also provides insights into the relationship between adapter capacity, learning rate, and corpus size, which can guide the design of efficient and effective QA systems.
Technical Details
- The study uses a 4-bit Gemma-4-e4b model with LoRA adapters to internalize documents.
- Approximately 100 training runs were conducted, ranging from single documents to a 99-document corpus.
- Data curation involved shortening gold answers to 1-6 word spans and removing trivia, which improved accuracy.
- The internalized adapter achieved 84.2% recall, outperforming a BM25-RAG pipeline (58.9%) and a gold-chunk oracle (65.6%).
- The research includes a detailed case study on debugging LLM training, addressing three initial misdiagnoses.
Industry Insight
- Focusing on data quality can lead to significant improvements in closed-book QA systems, potentially reducing the need for complex retrieval mechanisms.
- The relationship between adapter rank and learning rate should be carefully considered when scaling up the corpus size to ensure optimal performance.
- Internalizing documents into model weights can offer a more efficient alternative to traditional RAG systems, especially in scenarios with limited context windows or high latency requirements.
Disclaimer: The above content is generated by AI and is for reference only.