The Imperfective Paradox Is Not Necessarily in Large Language Models: A Benchmark Failure Before a Model Failure 非完美体悖论未必存在于大语言模型中：基准测试失败先于模型失败

TL;DR

• The paper challenges prior claims that LLMs exhibit a "Teleological Bias" in inferring completed events from progressive descriptions, arguing the benchmark itself is flawed
• Three conceptual mis-specifications are identified, with "Aspectual Reduction" being the most critical, affecting benchmark construction and conclusions
• Under strict NLI standards, 76% of benchmark instances do not explicitly rule out event culmination, and native-speaker annotations confirm significant ambiguity (38% of Group A and 29% of Group C examples)
• The authors introduce "Sufficiency Bias" as a new failure mode: models accept simple-past hypotheses without affirming culmination
• With proper prompts and evaluation, models like Qwen-7B, GPT-5.4, and Qwen-72B can achieve human-comparable performance on aspectual classification

Why It Matters

This paper directly challenges widely cited findings about LLM semantic reasoning failures, demonstrating that benchmark design flaws can produce misleading conclusions about model capabilities. For AI practitioners, it underscores the critical importance of rigorous evaluation methodology and native-speaker validation before drawing strong claims about model limitations. The findings have broad implications for how the community interprets NLI benchmarks and attributes reasoning failures to models versus evaluation artifacts.

Technical Details

• Conceptual Mis-specifications: The authors identify three flaws in the original benchmark, with "Aspectual Reduction" being central—it oversimplifies aspectual distinctions, affecting benchmark construction, analysis, experiments, and conclusions
• Native-Speaker Annotation: Human annotation revealed that 38% of Group A examples and 29% of Group C examples permit alternative interpretations, undermining the original benchmark's ground truth assumptions
• Lexically Matched Minimal Pairs: A new evaluation dataset was constructed to control for lexical variation and isolate aspectual reasoning from confounding factors
• Multi-step Reasoning Framework: Event-semantic NLI is reformulated as a multi-step reasoning problem, assessing both intermediate semantic decisions and final predictions to reveal hidden failure modes
• Identified Failure Modes: Beyond Sufficiency Bias, the authors find "Surface-form Attraction" (models gravitating toward surface-associated answers) and errors in compositional aspectual classification
• Experiments: Tested on Qwen-7B with suitable prompts, GPT-5.4, and Qwen-72B, showing context sensitivity in aspectual classification and human-comparable performance under proper conditions

Industry Insight

• Benchmark-driven claims about model limitations should be treated as provisional until independently validated with native-speaker annotation and controlled minimal-pair evaluations
• Prompting interventions may produce superficial label shifts without genuine semantic improvement—researchers should measure intermediate reasoning steps, not just final predictions
• The gap between benchmark failure and model failure highlights an opportunity: well-designed evaluations can reveal that models possess capabilities previously thought absent, suggesting current benchmarks may systematically underestimate LLM semantic competence

摘要

本文挑战了先前关于大语言模型（LLM）在从进行体描述推断完成事件时存在“目的论偏见”（Teleological Bias）的说法，认为该基准测试本身存在缺陷。
研究指出了三个概念性误设，其中“体貌还原”（Aspectual Reduction）最为关键，影响了基准测试的构建与结论。
在严格的自然语言推理（NLI）标准下，基准测试中76%的实例并未明确排除事件完成的可能性；母语者标注也证实了显著的歧义性（A组38%、C组29%的示例存在歧义）。
作者引入了“充分性偏见”（Sufficiency Bias）作为新的失败模式：模型接受一般过去时假设，但并未确认事件完成。
在采用恰当的提示与评估方式后，Qwen-7B、GPT-5.4和Qwen-72B等模型在体貌分类任务上可达到与人类相当的表现。

深度分析

TL;DR

• 本文挑战了先前关于大语言模型在从进行体描述推断完成事件时存在“目的论偏见”的说法，认为该基准测试本身存在缺陷。
• 研究指出了三个概念性误设，其中“体貌还原”最为关键，影响了基准测试的构建与结论。
• 在严格的自然语言推理标准下，基准测试中76%的实例并未明确排除事件完成的可能性；母语者标注也证实了显著的歧义性（A组38%、C组29%的示例存在歧义）。
• 作者引入了“充分性偏见”作为新的失败模式：模型接受一般过去时假设，但并未确认事件完成。
• 在采用恰当的提示与评估方式后，Qwen-7B、GPT-5.4和Qwen-72B等模型在体貌分类任务上可达到与人类相当的表现。

为什么重要

本文直接挑战了关于大语言模型语义推理失败的广泛引用结论，表明基准测试设计缺陷可能导致对模型能力的误导性判断。对于AI从业者而言，它在得出关于模型局限性的强结论之前，强调了严格评估方法和母语者验证的重要性。该发现对社区如何解读自然语言推理基准测试、以及如何将推理失败归因于模型还是评估 artifacts 具有广泛影响。

技术细节

• 概念性误设：作者指出了原始基准测试中的三个缺陷，其中“体貌还原”是核心问题——它过度简化了体貌区分，影响了基准测试的构建、分析、实验及结论。