APeB: Benchmarking Personalization Ability of Large Language Model Agents APeB：基准测试大型语言模型代理的个性化能力

TL;DR

• Introduction of APeB, a benchmark designed to evaluate the personalization capabilities of LLM agents using raw, underspecified queries and rich interaction histories.
• Identification of a significant performance gap where state-of-the-art models fail to effectively infer latent intent and utilize noisy historical data during early-stage interactions.
• Development of VQRA, a history-aware query-refinement pipeline that demonstrates consistent improvements, highlighting the necessity for dedicated history-utilization modules.
• Establishment of Personalized Product Search (PPS) as a critical testbed for agentic personalization, addressing limitations in existing benchmarks that rely on refined inputs.

Why It Matters

This research addresses a critical bottleneck in deploying LLM agents for real-world applications, where user inputs are often vague and require context-aware interpretation. By providing a rigorous benchmark and demonstrating that current models struggle with implicit intent discovery, it guides developers toward building more robust, history-sensitive agent architectures.

Technical Details

• Benchmark Construction: APeB is built from real-world action logs, pairing underspecified user intents with diverse, noisy interaction histories and candidate items to simulate realistic personalization scenarios.
• Testbed Definition: The study introduces Personalized Product Search (PPS) as a specific domain for testing agentic personalization, focusing on the ability to handle raw queries rather than pre-refined ones.
• Performance Analysis: Evaluation of state-of-the-art LLMs with multi-step agent workflows reveals strong performance on explicit queries but poor results on early-stage queries requiring intent inference.
• Proposed Solution: The VQRA pipeline utilizes history-aware query refinement to bridge the gap, showing that explicit handling of historical context significantly boosts personalization accuracy.

Industry Insight

• Developers should prioritize the integration of dedicated memory and history-processing modules in agent architectures rather than relying solely on prompt engineering for context retention.
• Future benchmarking efforts must include noisy, underspecified inputs to accurately assess an agent's ability to handle real-world user ambiguity.
• Implementing intermediate refinement steps, such as VQRA, can serve as a low-cost, high-impact strategy to enhance personalization without requiring fundamental model retraining.

摘要

介绍 APeB，这是一个旨在利用原始、未明确指定的查询和丰富的交互历史来评估大语言模型（LLM）代理个性化能力的基准测试。
发现了一个显著的性能差距：最先进的模型在早期交互阶段无法有效推断潜在意图并利用嘈杂的历史数据。
开发了 VQRA，这是一种感知历史的查询优化管道，展示了持续的性能提升，突显了专用历史利用模块的必要性。
确立个性化产品搜索（PPS）作为代理个性化关键测试床，解决了现有基准测试依赖优化后输入的限制。

深度分析

太长不看（TL;DR）

• 介绍 APeB，这是一个旨在利用原始、未明确指定的查询和丰富的交互历史来评估大语言模型（LLM）代理个性化能力的基准测试。
• 发现了一个显著的性能差距：最先进的模型在早期交互阶段无法有效推断潜在意图并利用嘈杂的历史数据。
• 开发了 VQRA，这是一种感知历史的查询优化管道，展示了持续的性能提升，突显了专用历史利用模块的必要性。
• 确立个性化产品搜索（PPS）作为代理个性化关键测试床，解决了现有基准测试依赖优化后输入的限制。

为什么这很重要

这项研究解决了将 LLM 代理部署到现实世界应用中的一个关键瓶颈，因为在这些场景中，用户输入通常很模糊，需要结合上下文进行解释。通过提供严格的基准测试并证明当前模型在隐式意图发现方面存在困难，它指导开发人员构建更稳健、对历史更敏感的代理架构。

技术细节

• 基准构建：APeB 基于真实世界的操作日志构建，将未明确指定的用户意图与多样化、嘈杂的交互历史和候选物品配对，以模拟真实的个性化场景。
• 测试床定义：本研究引入个性化产品搜索（PPS）作为测试代理个性化的特定领域，重点关注处理原始查询而非预优化查询的能力。
• 性能分析：对具有多步代理工作流的最新 LLM 进行评估，结果显示它们在显式查询上表现强劲，但在需要