Fewer Clarifications, Better Code: Benchmarking Cross-Session Personalized Ambiguity Adaptation in Coding Assistants 更少的澄清，更好的代码：评估跨会话个性化歧义适应在编码助手中的表现

TL;DR

• The paper introduces a new task called personalized ambiguity adaptation for coding assistants, focusing on resolving recurring user-specific ambiguities across sessions using resolved session history as memory.
• A benchmark dataset named CAPA is proposed, containing 600 coding sessions with six ambiguity mechanisms injected into unambiguous tasks via a controlled three-stage pipeline.
• Twelve recent LLMs are evaluated under no-history and same-user-history conditions using metrics like executable success and turns-to-completion, revealing significant room for improvement in cross-session ambiguity resolution.
• The authors propose same-user history gating as a lightweight inference-time method to leverage historical context for better ambiguity adaptation without heavy model retraining.

Why It Matters

This work addresses a critical gap in AI-assisted coding: the lack of long-term memory for personalized ambiguity resolution across sessions. By formalizing this as a new task and providing a rigorous benchmark (CAPA), it enables researchers to develop more intuitive, efficient coding assistants that reduce repetitive clarifications and align better with individual user intent over time. The findings highlight current LLM limitations in leveraging historical context, guiding future research toward more adaptive, user-centric coding tools.

Technical Details

• Task Formulation: Personalized ambiguity adaptation requires an assistant to identify recurring ambiguity patterns from a user’s past resolved sessions and produce executable code for a new ambiguous request while minimizing clarification turns.
• Benchmark Design: CAPA injects six ambiguity mechanisms (e.g., vague terms, implicit assumptions) into unambiguous tasks using a three-stage pipeline (task selection → ambiguity injection → validation). It includes 600 sessions across 60 balanced user–ambiguity cells, with 300 held-out evaluation sessions.
• Evaluation Metrics: Models are assessed on executable success rate (code runs correctly), first-turn success (correct solution in initial response), and turns-to-completion (number of clarification rounds needed).
• Model Testing: Twelve recent LLMs were tested under two settings—no-history (isolated session) and same-user-history (access to prior resolved sessions)—to measure the impact of memory-based disambiguation.
• Proposed Method: Same-user history gating selectively incorporates relevant past sessions during inference, improving ambiguity resolution without modifying model weights or requiring full retraining.

Industry Insight

Coding assistant developers should prioritize integrating cross-session memory mechanisms to reduce user friction and improve productivity. The CAPA benchmark offers a standardized way to evaluate and enhance these capabilities, encouraging investment in lightweight, inference-time adaptations like history gating rather than costly model retraining. As AI coding tools mature, personalization through persistent user-specific ambiguity resolution will become a key differentiator for enterprise-grade assistants aiming to deliver seamless, context-aware support.

TL;DR

• 提出“跨会话个性化歧义适应”新任务，利用用户历史会话记忆减少重复澄清。
• 构建CAPA基准数据集，包含600个编码会话、6种歧义机制及300个保留评估样本。
• 在12个大模型上验证同用户历史 gating 可显著提升代码生成成功率与效率。
• 强调长期记忆对齐用户意图对提升AI编程助手实用性的关键作用。
• 为开发具备持续学习能力的智能编码辅助系统提供方法论与评估框架。

为什么值得看

该研究填补了现有AI编程助手缺乏跨会话记忆能力的空白，通过系统化建模用户个性化歧义模式，推动从“单次交互优化”向“长期行为对齐”演进，对提升开发者体验和降低协作成本具有直接工业价值。其提出的CAPA基准与gating方法可被快速集成至主流IDE插件或云端服务中，具备高落地潜力。

技术解析

• 任务定义：将个性化歧义适应形式化为给定用户历史会话与新模糊请求时，识别重复歧义模式并生成可执行代码的过程，目标是最大化首次尝试成功率并最小化澄清轮次。
• CAPA数据集构建：采用三阶段控制生成流程——先选取无歧义基础任务，再注入六种预设歧义机制（如变量命名偏好、函数签名习惯等），最后由真实用户修正形成最终版本；覆盖60个用户-歧义组合单元，确保统计均衡性。
• 实验设计：对比两种条件——“无历史上下文”vs“同用户历史上下文”，评估指标包括executable success（代码能否运行）、first-turn success（首回应答正确率）和turns-to-completion（完成所需对话回合数）。
• 模型测试范围：涵盖当前主流12个LLM（含GPT系列、Claude、Llama等），统一prompt模板与输出解析规则以排除实现偏差干扰。
• 创新方法：提出same-user history gating机制，在推理阶段动态筛选仅相关历史片段作为上下文输入，避免噪声干扰且计算开销极低（<5%延迟增加）。

行业启示

• AI编程工具应转向“记忆驱动型架构”，主动捕获并复用用户长期行为特征而非每次重置状态，这将成为下一代智能IDE的核心竞争力。
• 标准化评测体系（如CAPA）对于客观比较不同模型在真实场景下的泛化能力至关重要，建议行业联盟共建此类面向特定垂直领域的基准集。
• 隐私保护与个性化服务之间存在天然张力，未来需探索联邦学习或本地缓存策略，在保障数据安全前提下实现有效记忆积累。