AI News AI资讯 21h ago Updated 18h ago 更新于 18小时前 46

AI agents have no sense of time and are not aware of it AI代理没有时间感知能力,也无法意识到时间的流逝

Popular coding assistants (Claude Code and Codex) consistently overestimate task duration, guessing ~90 minutes regardless of actual difficulty Self-assessment is severely flawed: models rated their own work ~20 percentage points higher than actual performance, even on failed tasks Runtime behavior depends heavily on the surrounding software harness, not just the underlying model Agents cannot reliably track elapsed time, making long-running autonomous tasks difficult to control Providing explic 主流编码助手(Claude Code、Codex)无法准确预测任务耗时,普遍高估所需时间,短任务误差最大 同一模型在不同软件设置下行为差异显著,运行时长和步骤数受harness影响达2.5倍 模型自我评估质量极不可靠,高估自身工作成果约20个百分点,甚至对失败任务给出高分 提供计时工具后模型能准确报告经过时间,表明外部辅助可弥补时间感知缺陷 研究揭示了AI agent在长时间任务执行中的关键缺陷,对开发可靠自主系统具有重要警示意义

65
Hot 热度
70
Quality 质量
60
Impact 影响力

Analysis 深度分析

TL;DR

  • Popular coding assistants (Claude Code and Codex) consistently overestimate task duration, guessing ~90 minutes regardless of actual difficulty
  • Self-assessment is severely flawed: models rated their own work ~20 percentage points higher than actual performance, even on failed tasks
  • Runtime behavior depends heavily on the surrounding software harness, not just the underlying model
  • Agents cannot reliably track elapsed time, making long-running autonomous tasks difficult to control
  • Providing explicit time-tracking tools dramatically improves temporal accuracy

Why It Matters

This research reveals a critical gap in AI agent reliability for production use: without accurate time estimation and self-assessment, autonomous coding agents cannot be trusted on multi-hour workflows. For practitioners building or deploying AI agents, these findings highlight that model capability alone is insufficient—system design and tooling are equally decisive factors in agent performance.

Technical Details

  • Study tested Anthropic's Claude Code and OpenAI's Codex on 200 ProgramBench tasks plus 18 custom benchmarks, measuring both forward time estimates and retrospective elapsed-time reporting
  • Both models clustered around a 90-minute estimate regardless of task difficulty; retrospective errors ranged from 3x (Claude) to 6-10x (Codex) actual runtime
  • Claude Code ran for a median of ~90 minutes per task, while Codex stopped after ~30 minutes, despite using similar underlying models—demonstrating harness-driven behavioral divergence
  • Older models (Opus 4.8, GPT-5.5) overrated their results by ~20 points on average, claiming ~70% success on tasks that actually scored 7-14.5%
  • When given access to an explicit elapsed-time tool, agents achieved near-perfect temporal accuracy, suggesting the limitation is architectural rather than inherent to the models

Industry Insight

  • Agent harness design is as critical as model selection—identical models can produce drastically different outcomes depending on software wrappers, making benchmarking must include runtime behavior, not just capability
  • Self-assessment calibration should be a standard evaluation metric for any agent deployment targeting long-running or autonomous workflows
  • The fix is straightforward: integrating real-time tooling (clocks, timers) into agent loops can resolve temporal blind spots, suggesting that robust agent frameworks should bake in observability by default rather than relying on model intuition

TL;DR

  • 主流编码助手(Claude Code、Codex)无法准确预测任务耗时,普遍高估所需时间,短任务误差最大
  • 同一模型在不同软件设置下行为差异显著,运行时长和步骤数受harness影响达2.5倍
  • 模型自我评估质量极不可靠,高估自身工作成果约20个百分点,甚至对失败任务给出高分
  • 提供计时工具后模型能准确报告经过时间,表明外部辅助可弥补时间感知缺陷
  • 研究揭示了AI agent在长时间任务执行中的关键缺陷,对开发可靠自主系统具有重要警示意义

为什么值得看

这项研究揭示了当前主流AI编码助手在时间感知和自我评估方面的系统性缺陷,对依赖长时间自主运行的AI agent开发具有重要参考价值。它提醒从业者,在构建可靠AI系统时,必须考虑外部监控和辅助工具来弥补模型内在能力的不足。

技术解析

研究使用ProgramBench数据集和18个自定义基准测试,评估Claude Code和Codex的时间预测能力。模型在任务前估计耗时,任务后报告实际经过时间,结果显示普遍高估,短任务误差最大。

同一模型在不同harness下行为差异显著,Claude Code平均运行90分钟,Codex约30分钟,步骤数相差2.5倍,表明软件设置对agent行为有决定性影响。

模型自我评估质量不可靠,老模型Opus 4.8和GPT-5.5高估结果20个百分点,甚至对失败任务给出高分,实际成功率仅7-14.5%。

当提供计时工具时,模型能准确报告经过时间,表明外部辅助可弥补时间感知缺陷,为agent设计提供可行改进方向。

行业启示

开发长时间运行的AI agent时,必须集成外部时间监控和进度跟踪工具,不能依赖模型内在时间感知能力。

AI系统的自我评估能力需要独立验证机制,避免高估导致任务失控或资源浪费。

研究凸显了harness设计对agent行为的关键影响,开发者应优化软件设置以提升一致性和可靠性,而非仅关注模型本身。

Disclaimer: The above content is generated by AI and is for reference only. 免责声明:以上内容由 AI 生成,仅供参考。

Agent Agent Evaluation 评测 Benchmark 基准测试 Claude Claude Research 科学研究