LongNovel: A Multi-Scale Benchmark for Hallucination Detection in Long-Context Novel Summarization
LongNovel is a multi-scale bilingual (Chinese-English) benchmark for detecting hallucinations in long-context novel summarization, addressing a gap in existing research The benchmark uses 29 Chinese novels ranging from 16k to 100k tokens plus chapter-level data from BookSum, enabling analysis of how hallucinations scale with context length Eight hallucination types were designed, with data quality ensured through Multi-Model Arbitration, Entity-Referenced Hallucination Generation, and manual tes
Analysis
TL;DR
- LongNovel is a multi-scale bilingual (Chinese-English) benchmark for detecting hallucinations in long-context novel summarization, addressing a gap in existing research
- The benchmark uses 29 Chinese novels ranging from 16k to 100k tokens plus chapter-level data from BookSum, enabling analysis of how hallucinations scale with context length
- Eight hallucination types were designed, with data quality ensured through Multi-Model Arbitration, Entity-Referenced Hallucination Generation, and manual test-set revision
- Experimental results confirm LongNovel is a challenging benchmark, and the dataset has been released for community use
Why It Matters
As context windows continue to expand, hallucination detection in long-context summarization remains a critical bottleneck for deploying LLMs in real-world literary and narrative tasks. LongNovel provides the first multi-scale benchmark that systematically evaluates how hallucination rates evolve as context grows, offering researchers a structured way to measure and improve long-context reliability.
Technical Details
- Dataset construction: 29 Chinese novels (16k–100k tokens each) combined with chapter-level summaries from the BookSum dataset, creating a bilingual benchmark covering diverse narrative complexities
- Hallucination taxonomy: 8 distinct hallucination types were designed to categorize different failure modes in long-context summarization, enabling fine-grained analysis
- Data generation pipeline: Combines Multi-Model Arbitration (using multiple models to cross-validate outputs) and Entity-Referenced Hallucination Generation (injecting hallucinations tied to specific entities) to ensure both authenticity and balanced category distribution
- Quality assurance: Manual revision of the test set to guarantee data reliability and eliminate artifacts from automated generation
- Multi-scale design: The varying novel lengths (16k to 100k tokens) allow researchers to study hallucination patterns across different context lengths rather than at a single fixed scale
Industry Insight
- Benchmark designers should prioritize multi-scale evaluation: single-length benchmarks mask critical failure modes that only emerge at extreme context lengths, so future benchmarks should span a wide range of token counts
- The combination of entity-referenced hallucination injection with multi-model arbitration offers a replicable template for constructing high-quality hallucination datasets in other domains beyond novel summarization
- As long-context models become commodity, hallucination detection will shift from a research curiosity to a production-critical requirement; investing in robust evaluation benchmarks now will differentiate teams that ship reliable long-context applications
Disclaimer: The above content is generated by AI and is for reference only.