The Plot Thins: Uniformity and Linearity in Literary Summaries
The paper introduces a dataset mapping sentences from 150 novel summaries to their source chapters, revealing that sentence-to-chapter alignment is unexpectedly difficult for both humans and LLMs Two quantitative metrics are proposed: "linearity" (how well summaries preserve the source's chronological order of events) and "uniformity" (how evenly summaries distribute attention across source chapters) The analysis reveals systematic deviations between literary works and their summaries, particula
Analysis
TL;DR
- The paper introduces a dataset mapping sentences from 150 novel summaries to their source chapters, revealing that sentence-to-chapter alignment is unexpectedly difficult for both humans and LLMs
- Two quantitative metrics are proposed: "linearity" (how well summaries preserve the source's chronological order of events) and "uniformity" (how evenly summaries distribute attention across source chapters)
- The analysis reveals systematic deviations between literary works and their summaries, particularly in how narrative details are expressed in terms of clarity and prominence
- The combined manual and LLM-based annotation approach highlights the nuanced gap between full literary texts and their condensed counterparts
Why It Matters
This research provides a principled, measurable framework for understanding how summarization transforms narrative structure—a critical concern for anyone building or evaluating text summarization systems. By quantifying linearity and uniformity, it offers practitioners concrete metrics to assess whether summaries faithfully represent source material or introduce structural distortions.
Technical Details
- Dataset constructed by mapping sentences from 150 novel summaries to their corresponding source chapters using a hybrid approach combining manual annotation and LLM-based annotation
- Two novel metrics defined: linearity measures the degree to which a summary maintains the temporal/chronological order of events from the source, and uniformity measures how evenly a summary distributes its coverage across the source text's chapters
- The sentence-to-chapter mapping task was found to be unexpectedly challenging for both human annotators and LLMs, suggesting inherent ambiguity in aligning summarized content with source passages
- Analysis focuses on identifying when and how summaries break linearity and uniformity, linking these breaks to differences in how narrative details are expressed regarding clarity and prominence
Industry Insight
- Summarization systems should be evaluated not just on factual fidelity but on structural preservation—linearity and uniformity metrics offer actionable dimensions for benchmarking
- The difficulty of sentence-to-chapter alignment for both humans and models suggests that current NLP systems may struggle with nuanced document-level understanding, pointing to areas for architectural improvement
- Publishers, content platforms, and AI tool developers can use these metrics to audit generated summaries for structural bias, ensuring condensed content does not disproportionately over- or under-represent source sections
Disclaimer: The above content is generated by AI and is for reference only.