One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries
For homogeneous document collections (one template, many copies), the schema already exists in the workflow of the people handling the files, not in the documents themselves, making it recoverable through structured interviews rather than automated extraction Two practical signals validate whether a proposed column is real: (1) it surfaces instantly when asking what people filter by, and (2) two independent handlers name it identically with the same meaning The interview process requires only on
Analysis
TL;DR
- For homogeneous document collections (one template, many copies), the schema already exists in the workflow of the people handling the files, not in the documents themselves, making it recoverable through structured interviews rather than automated extraction
- Two practical signals validate whether a proposed column is real: (1) it surfaces instantly when asking what people filter by, and (2) two independent handlers name it identically with the same meaning
- The interview process requires only one hour with two document handlers, yielding a list of six to ten fields that form the basis of a table-based index for the collection
- This approach contrasts sharply with graph-discovery methods, which are unnecessary when documents share a fixed layout and repeatable values from a single source process
- The article is Part IV of an Enterprise Document Intelligence series, focusing on when a folder of similar documents should be treated as an implicit database rather than a collection of free-form texts
Why It Matters
This article provides a practical, low-cost methodology for extracting schema from repetitive document collections without relying on expensive AI-driven information extraction or manual data dictionary creation. For AI practitioners building enterprise RAG systems, it demonstrates that the hardest part of structuring document data is often not technical but organizational—knowledge about fields lives in human workflows, not in code or documentation.
Technical Details
- Document shape: Thousands of files produced by a single repeatable process (e.g., insurance renewal notices, purchase orders, fund statements), sharing page-one layout with values in roughly fixed positions and a small stable value set per field
- Interview protocol: Four sequential questions asked to two handlers separately, designed to surface the fields people actually use for lookup and filtering, with the second interview catching naming ambiguities
- Column validation signals: (1) Conversational-speed response with an example when asked what people filter by, indicating active usage; (2) Agreement between two handlers on name, meaning, and page location, ruling out ambiguous or stale fields
- Companion implementation: A public notebook at
doc-intel/notebooks-vol1runs a "column test" by declaring candidate columns across look-alike PDFs and printing fill rates per column, revealing which fields are genuinely populated versus half-empty - Cost model: Distinguishes between costs incurred once per document (extraction) versus per question (repeated querying), with break-even arithmetic provided for a workload of fifty questions per day
Industry Insight
- Organizations should treat document schema discovery as a human-centered interview exercise before investing in automated extraction pipelines; the ROI is highest when the collection is homogeneous and the fields are already in active use by handlers
- The two-signal validation method (speed of response + cross-handler agreement) offers a lightweight, model-free quality gate that can prevent schema drift and filter breakage in production RAG systems
- This approach scales poorly for heterogeneous document collections but is highly efficient for the common enterprise case of templated documents at scale, suggesting a clear decision framework for when to use table-based indexing versus free-text retrieval
Disclaimer: The above content is generated by AI and is for reference only.