Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs
Multimodal CoLRAG-TF introduces a four-axis fusion architecture combining dense text embeddings, BM25, knowledge-graph triple filtering, and image similarity for robust PDF retrieval. The system utilizes a hierarchical coarse-to-fine retrieval strategy (volume → chapter → block) and indexes 11,414 OpenIE triples via FAISS to enable sub-second multi-hop reasoning. Bayesian optimization determined that the triple-filtering axis must dominate fusion weights ($\alpha_\text{triple} = 0.44$) to effect
Analysis
TL;DR
- Multimodal CoLRAG-TF introduces a four-axis fusion architecture combining dense text embeddings, BM25, knowledge-graph triple filtering, and image similarity for robust PDF retrieval.
- The system utilizes a hierarchical coarse-to-fine retrieval strategy (volume → chapter → block) and indexes 11,414 OpenIE triples via FAISS to enable sub-second multi-hop reasoning.
- Bayesian optimization determined that the triple-filtering axis must dominate fusion weights ($\alpha_\text{triple} = 0.44$) to effectively counteract lexical bias in complex documents.
- Evaluated on a 457-pair benchmark, the model achieved a Retrieval Recall of 0.9909 and a 71.6% improvement in multi-hop answer similarity compared to single-hop queries.
Why It Matters
This research addresses critical limitations in Retrieval-Augmented Generation (RAG) systems when handling heterogeneous, multimodal documents like complex PDFs, which often contain dispersed evidence requiring multi-hop reasoning. By demonstrating that structured knowledge graph triples significantly outperform pure lexical or dense embedding methods, it provides a blueprint for improving accuracy in domains where precise logical connections are more important than keyword matching. This is particularly relevant for industries dealing with technical manuals, legal documents, or scientific literature where context and relational data are paramount.
Technical Details
- Four-Axis Fusion Architecture: Integrates dense text embeddings, BM25 keyword matching, knowledge-graph triple filtering, and image-based similarity scores into a unified retrieval framework.
- Hierarchical Retrieval Pipeline: Employs a HippoRAG2-inspired coarse-to-fine approach that narrows the search space from volume to chapter to block before applying final fusion scoring.
- Knowledge Graph Integration: Extracts 11,414 OpenIE triples from 43 Japanese disaster lesson PDFs and indexes them using FAISS to facilitate fast, sub-second lookup and hierarchical propagation of relevance signals.
- Multimodal Processing: Utilizes a hybrid OCR pipeline and LLM-based caption generation to create a multimodal index of 2,403 blocks, including an image-to-lesson pipeline using vision LLMs.
- Optimization Strategy: Uses Bayesian optimization to determine fusion weights, revealing that the triple axis requires the highest weight (0.44) to maintain multi-hop retrieval quality.
Industry Insight
- Prioritize Structured Data over Lexical Match: For complex document retrieval, relying solely on vector similarity or keyword matching is insufficient. Integrating explicit relational data (triples) should be a standard component in RAG pipelines to support multi-hop reasoning.
- Hybrid OCR and Vision Models are Essential: As organizations digitize legacy documents, implementing hybrid pipelines that combine traditional OCR with LLM-based captioning ensures that visual and textual information are both captured and indexed effectively.
- Dynamic Weight Tuning is Critical: Fixed fusion weights in multimodal RAG systems may lead to suboptimal performance. Implementing automated optimization techniques like Bayesian search can help tailor retrieval strategies to specific domain requirements and noise levels.
Disclaimer: The above content is generated by AI and is for reference only.