ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models ForgetBench：评估语言模型长期参数记忆遗忘动态的基准测试

TL;DR

• ForgetBench is a new benchmark designed to evaluate forgetting dynamics in large language models (LLMs) under continual knowledge editing.
• It introduces two evaluation paradigms: concept-based QA and scenario-based QA, to assess isolated factual retention and structured relational knowledge preservation.
• The benchmark constructs temporally ordered knowledge streams and measures temporal decay, retention strength, and cross-instance stability across multiple editing stages.
• Experiments reveal that existing methods struggle to balance long-term retention with generalization quality, highlighting the need for improved memory mechanisms in LLMs.

Why It Matters

This work addresses a critical gap in understanding how LLMs retain knowledge over time during continuous updates, which is essential for developing reliable and adaptive AI systems. By providing a systematic framework to quantify forgetting behavior, it enables researchers to design better memory architectures and editing strategies that preserve historical knowledge while accommodating new information.

Technical Details

• Evaluation Paradigms: Concept-based QA tests isolated factual retention, while scenario-based QA evaluates structured relational knowledge preservation.
• Sequential Editing Framework: Knowledge streams are constructed with temporally ordered edits to simulate real-world learning scenarios where models encounter evolving information.
• Unified Evaluation Metrics: Temporal decay, retention strength, and cross-instance stability are measured to quantify long-term forgetting dynamics comprehensively.
• Model Testing: Diverse LLMs and editing methods were evaluated using ForgetBench, revealing consistent challenges in balancing retention and generalization.

Industry Insight

The findings suggest that current approaches to continual learning in LLMs may not be sufficient for applications requiring stable long-term memory, such as personalized assistants or domain-specific experts. Developers should prioritize integrating robust memory mechanisms capable of preserving critical knowledge without sacrificing adaptability to new inputs. Future research could focus on hybrid architectures combining parametric and external memory systems to mitigate forgetting effects effectively.

TL;DR

• ForgetBench 是首个专门用于评估语言模型在持续知识编辑下长期参数记忆遗忘行为的基准测试。
• 提出概念型问答与情景型问答双范式，分别衡量孤立事实保留与结构化关系知识的保持能力。
• 构建基于时间顺序的知识流序列，支持多阶段编辑后的动态遗忘分析，并引入统一框架量化知识演化过程（如衰减率、稳定性）。
• 实验表明现有方法难以平衡长时保留与泛化质量，凸显对新型鲁棒记忆机制的需求。
• 该工作填补了LLM“遗忘动力学”评估空白，推动从静态编辑向连续学习场景演进的研究方向。

为什么值得看

本文针对大语言模型在持续更新中面临的“灾难性遗忘”问题，首次系统性地设计了一套可量化的遗忘行为评估体系，为理解模型长期记忆退化机制提供新视角。对于从事模型压缩、在线微调或终身学习的从业者而言，ForgetBench 可作为诊断与优化模型记忆稳定性的关键工具，具有明确的工程指导价值。

技术解析

• 双评估范式：引入 concept-based QA（测试单个事实是否被遗忘）和 scenario-based QA（测试上下文关联知识是否断裂），区分原子级与结构级遗忘现象。
• 时序知识流构造：按时间顺序注入多条知识片段，模拟真实世界中的信息更新过程，并在每个编辑节点后评估模型表现，捕捉渐进式遗忘轨迹。
• 统一量化框架：定义三个核心指标——temporal decay（随时间推移的性能下降速率）、retention strength（最终状态下的知识保留程度）、cross-instance stability（不同样本间的一致性波动），实现跨模型横向比较。
• 覆盖广泛模型与方法：在多种主流架构（如Transformer变体）及编辑策略（如梯度-based、检索增强更新）上进行验证，揭示当前方案普遍存在的权衡困境。
• 开源承诺：作者计划发布代码与数据集，便于复现与扩展研究，促进社区共建标准评测生态。

行业启示

• 未来LLM研发应重视“记忆持久性”作为核心性能维度之一，尤其在金融、医疗等需长期依赖历史知识的领域，忽视遗忘风险可能导致严重决策偏差。
• 企业若部署需频繁迭代的对话系统或个性化助手，建议集成类似ForgetBench的监控模块，实时追踪关键知识点的存活率，触发预警或主动重训练机制。
• 学术界与工业界应合作开发具备“抗遗忘”特性的新型架构（如神经符号混合存储、稀疏激活记忆池），以支撑真正可持续的自适应智能体发展路径。