Select, Don't Train: The Benefits of Modular Entity Disambiguation with LLM-Based Selection
Entity Disambiguation (ED) is decoupled into two independent stages: candidate retrieval and LLM-based selection, challenging the dominant dual-encoder paradigm A training-free BM25 retriever paired with an LLM selector achieves a new state-of-the-art on the ZELDA benchmark (inKB micro-F1: 86.3 vs. 82.3) Trained dense retrievers provide only modest gains (+2.2 F1) over BM25 when selection is handled by capable LLMs Modular decoupling enables abstention when the correct entity is absent from retr
Analysis
TL;DR
- Entity Disambiguation (ED) is decoupled into two independent stages: candidate retrieval and LLM-based selection, challenging the dominant dual-encoder paradigm
- A training-free BM25 retriever paired with an LLM selector achieves a new state-of-the-art on the ZELDA benchmark (inKB micro-F1: 86.3 vs. 82.3)
- Trained dense retrievers provide only modest gains (+2.2 F1) over BM25 when selection is handled by capable LLMs
- Modular decoupling enables abstention when the correct entity is absent from retrieved candidates, reaching 90.7 F1 in evaluation settings that reward correct abstentions
- The paper systematically compares sparse retrieval (BM25), Web KB search, and dense retrieval under a shared LLM-based selection stage
Why It Matters
This work challenges the prevailing assumption that end-to-end trained neural models are necessary for high-quality Entity Disambiguation, demonstrating that off-the-shelf LLMs can serve as powerful selectors when paired with simple retrieval. For AI practitioners, this means significant cost savings by eliminating the need to train and maintain custom retrievers, while still achieving competitive or superior results. The findings also have broader implications for RAG-style architectures, suggesting that investing in selection capability may yield diminishing returns compared to improving retrieval quality.
Technical Details
- The paper proposes a modular two-stage ED framework: (1) candidate retrieval using BM25, Web KB search, or a trained dense retriever, followed by (2) LLM-based selection of the correct entity from candidates given contextual information
- Systematic evaluation across multiple open- and closed-source LLMs as selectors, with BM25 achieving 86.3 inKB micro-F1 on ZELDA, compared to 88.5 with a trained dense retriever
- The modular design exposes a critical failure mode in existing ED systems: when the correct entity is missing from retrieved candidates, current models are forced to predict an incorrect entity rather than abstaining
- The framework introduces an abstention mechanism that detects retrieval failure and allows the system to opt out, achieving 90.7 F1 in evaluation settings that reward correct abstentions
- The approach eliminates the need for costly retraining as knowledge graphs evolve, since the retriever and selector are independently maintainable
Industry Insight
- Organizations building knowledge graph pipelines should reconsider the cost-benefit of training custom dense retrievers; for many applications, a BM25-based retrieval stage combined with a capable LLM selector may offer near-state-of-the-art performance at a fraction of the training and maintenance cost
- The abstention capability represents a practical improvement for production systems, where false confidence in incorrect entity predictions can propagate errors through downstream applications—systems that can gracefully opt out are more robust in real-world deployments
- As LLM capabilities continue to improve, the performance gap between training-free and trained retrieval approaches is likely to narrow further, making modular architectures increasingly attractive for dynamic knowledge graph environments where frequent retraining is impractical
Disclaimer: The above content is generated by AI and is for reference only.