Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings 首先基于事实：代理记忆的纵向评估工具，以及内存架构排名的任期交叉

TL;DR

• The paper introduces a longitudinal evaluation instrument for agent memory that inverts the traditional benchmark pipeline by generating facts with validity intervals before text creation.
• It reveals a "tenure crossover" where memory architecture rankings invert as history length increases: budgeted curated-map memory leads at short horizons but loses recall at longer horizons, while provenance-typed graphs improve over time.
• The study demonstrates that write-stage quality strongly correlates with downstream performance (24% failure for weakly-written facts vs. 2% for strong ones) and injection resistance depends on whether provenance boundaries survive representation.
• A layered architecture (released as Veracium) achieves the best performance across both short (96.8%) and long horizons among memory systems tested.
• The benchmark includes ~380 questions across 15 types with features like validity intervals, trust distinctions, injection probes, and as-of-date question sets absent from prior benchmarks.

Why It Matters

This work addresses critical limitations in current LLM-agent memory benchmarks, particularly label errors, contamination issues, and overemphasis on short interaction histories. By providing a rigorous longitudinal evaluation framework with ground-truth validation, it enables more reliable comparisons of memory architectures and reveals how performance rankings change over time—a crucial insight for designing agents that maintain accurate memory across extended interactions. The findings directly impact practitioners selecting memory systems for real-world applications where information validity periods and long-term retention matter.

Technical Details

• Inverted Benchmark Pipeline: Instead of generating conversations first and extracting answers, the method starts with a seeded life-script sampler that emits structured facts (with validity intervals, volatility classes, source channels) before any text is created. An LLM renderer then generates chat/email from these fact manifests, followed by a fidelity verifier confirming all planted facts.
• Corpus Design: The synthetic corpus contains approximately 380 questions across 15 types, embedding features like per-fact validity intervals, sent/received trust distinctions, injection probes in a benign harness, and as-of-date question sets—elements missing from surveyed benchmarks.
• Experimental Setup: Five memory architectures were benchmarked against a no-memory control using a fixed answerer, versioned LLM judge, three replicates, and two evaluation horizons (three weeks and nine weeks). Complete cross-family re-judging was performed to assess ranking inversions.
• Key Findings:
  • Budgeted curated-map memory led at three weeks (96% recall) but dropped to 72% at nine weeks due to evicted content loss.
  • Provenance-typed graph architecture rose to 90% recall at nine weeks, demonstrating superior long-term retention.
  • Full-rendered-history baseline matched or exceeded best memory systems at short horizons but showed no judge-independent advantage at nine weeks while requiring twice the read cost.
  • Write-stage quality had strong downstream correlation: weakly-written facts failed 24% of the time versus only 2% for well-written ones.
  • Injection resistance correlated with whether provenance boundaries survived representation in the memory system.
• Open-Source Release: The layered architecture (Veracium), corpus generator, and evaluation harness are released as open-source tools to facilitate future research.

Industry Insight

Memory architecture selection for LLM agents must account for temporal dynamics—systems optimal for short-term interactions may underperform in long-running applications due to eviction policies and representation limitations. The "tenure crossover" phenomenon suggests that practitioners should evaluate memory systems not just on immediate performance but across multiple time horizons relevant to their deployment scenarios. Additionally, the strong correlation between write-stage quality and downstream performance indicates that investing in high-quality fact encoding during memory ingestion yields significant returns in later retrieval accuracy. The release of Veracium and the longitudinal evaluation framework provides industry-ready tools for systematically testing and comparing memory solutions under realistic conditions involving validity periods, trust distinctions, and adversarial injection attempts.

TL;DR

• 提出一种全新的LLM-Agent记忆评估范式，通过“事实先行”生成机制解决传统基准测试中的标签错误和数据污染问题。
• 发现记忆架构的性能排名随历史长度发生逆转：短期表现最佳的预算型映射架构在长期被溯源图架构超越。
• 验证了写入阶段质量与下游任务表现的高度相关性，并开源了名为Veracium的评估库及合成数据生成工具。

为什么值得看

该研究揭示了当前Agent记忆评估方法的根本性缺陷，并提供了可复现的解决方案，对构建更可靠的长期记忆系统具有指导意义。其发现的“十年期交叉”现象提醒从业者避免在短期指标上过度优化，需关注架构的长期泛化能力。

技术解析

• 评估流程反转：摒弃传统的“对话生成-答案提取”模式，采用“事实播种→文本渲染→保真度验证→问题生成”的逆向Pipeline，确保黄金答案由脚本构造且可验证。
• 合成数据特性：生成的~380题语料包含事实有效期、信任区分、注入探针和时效性问答等复杂维度，覆盖15种题型，显著超越现有基准。
• 实验设计：在3周和9周两个时间跨度下，对比五种记忆架构（含无记忆基线），使用固定回答者和版本化LLM评判者进行三轮重复实验，统计显著性p=0.031。
• 关键发现：短周期内预算型映射架构（96%准确率）领先，但九周后衰退至72%；溯源图架构则从低位升至90%；全渲染历史基线虽短期持平但长期无优势且成本翻倍。
• 开源产出：发布Veracium库，集成分层架构实现（短期96.8%准确率）、数据生成器和安全评估 harness。

行业启示

• 评估标准重构：业界应转向基于“事实生命周期”的纵向评估体系，将有效期、信任流和抗注入能力纳入记忆模块的核心KPI。
• 架构选型策略：短期任务可优先考虑资源受限的映射式存储，但长期部署必须采用具备溯源能力的图结构记忆，避免性能随时间衰减。
• 质量前置原则：记忆系统的写入质量直接决定长期表现，需在数据注入阶段实施严格的质量控制，低质量事实的失败率高达24%（高质量仅2%）。