Your Requirements Doc Is 400 Pages Long. Your AI Agent Just Gave Up.
The core problem: AI coding assistants (like GitHub Copilot Chat) fail when fed massive requirements documents due to context window limits, high token costs, slow processing, and hallucination from irrelevant content The universal solution: "Search first, generate second" — use Retrieval-Augmented Generation (RAG) to pull only the relevant document sections into the prompt rather than dumping entire files Four practical RAG approaches are presented: Classic Vector RAG (semantic search with embe
Analysis
TL;DR
- The core problem: AI coding assistants (like GitHub Copilot Chat) fail when fed massive requirements documents due to context window limits, high token costs, slow processing, and hallucination from irrelevant content
- The universal solution: "Search first, generate second" — use Retrieval-Augmented Generation (RAG) to pull only the relevant document sections into the prompt rather than dumping entire files
- Four practical RAG approaches are presented: Classic Vector RAG (semantic search with embeddings), Vector-less BM25 (keyword-based ranking), Structured Chunking + Keyword Index JSON (Markdown header-based splitting with local indexing), and CSV/Markdown conversion (eliminating binary format bloat)
- The optimal approach depends on team context: enterprise test suites benefit from Vector RAG, technical specs with standardized vocabulary suit BM25, individual QA engineers in VS Code should use heading chunking with #file variables, and legacy Word/Excel docs require format conversion as a baseline step
- The key insight is that AI test generation quality and speed are directly proportional to how precisely the relevant context is targeted — reducing token count while increasing relevance multiplies both accuracy and productivity
Why It Matters
This article addresses a universal pain point for AI-augmented QA engineers: the gap between the promise of AI test generation and the reality of context window failures, slow responses, and hallucinated outputs when working with large requirements documents. For AI practitioners, it provides a practical taxonomy of RAG implementations ranging from production-grade to zero-infrastructure, enabling teams to choose the right complexity level for their needs. For the broader industry, it reinforces that effective AI adoption isn't just about the model — it's about how well you prepare and structure the context you feed it.
Technical Details
- Classic Vector RAG: Documents are split into 300–500 word chunks, embedded using an embedding model, and stored in a vector database (Chroma, FAISS, Pinecone). Queries are embedded and semantically matched against stored vectors to retrieve the top relevant chunks. Best for evolving documentation with varying terminology where semantic matching matters more than exact keyword overlap.
- Vector-less BM25 RAG: Uses the BM25 probabilistic ranking algorithm (the same engine behind Elasticsearch) to score text chunks based on keyword frequency while down-weighting common terms. Implemented with lightweight libraries like
rank-bm25in under 20 lines of Python. Zero external API costs and runs entirely locally, but has no synonym tolerance. - Structured Chunking + Keyword Index JSON: A Python script splits Markdown documents on
##and###headers into logical sections, extracts top-6 non-trivial keywords per chunk using bag-of-words with stopword filtering, and outputs arequirements_index.jsonmapping each chunk to its file path and keywords. Works natively with GitHub Copilot Chat's#filevariable for precise context targeting without any infrastructure. - CSV/Markdown Format Conversion: Binary formats (.xlsx, .docx) contain massive XML structural overhead that inflates token counts without adding semantic value. Converting to .md (for Word docs) or individual .csv files (for Excel worksheets) eliminates 60–80% of token bloat. Paired with an explicit mapping index, this enables loading only the relevant worksheet or section into the agent.
Industry Insight
- The "search first, generate second" paradigm should become a standard operating procedure for any team using LLMs for document-intensive tasks — the quality of AI output is bounded by the precision of context retrieval, not just model capability
- Teams should invest in lightweight, local-first retrieval solutions (like BM25 or heading-based chunking) before jumping to expensive vector infrastructure, as most QA requirements workloads are dominated by technical vocabulary where keyword search outperforms semantic search
- The format conversion insight is an immediate, zero-cost win: organizations with legacy requirements in Word and Excel can achieve dramatic token savings and faster AI responses simply by standardizing on Markdown and CSV as intermediate formats before any RAG pipeline is built
Disclaimer: The above content is generated by AI and is for reference only.