Counterexamples as Feedback for Agent Self-Correction 作为代理自我纠正反馈的反例

TL;DR

• A-CEGIS is a lightweight framework that uses counterexamples as feedback to evaluate multi-turn self-correction in natural-language-to-regex synthesis agents
• Diagnostic counterexample feedback achieves 90% task success within four turns, dramatically outperforming zero-shot (17%), generic self-correction (27%), and error-only feedback (23%)
• On a hidden test set with hardening, all tasks were solved by the final turn with a mean time-to-success of 2.7 turns and 77% robust success after targeted probing
• The framework introduces a deterministic oracle that checks regex proposals under full-match semantics and generates compact false-positive/false-negative witnesses to guide refinement
• Single-turn metrics significantly understate agent capability; multi-turn counterexample feedback reveals a critical property of deployed agents: their ability to repair wrong artifacts with concrete feedback

Why It Matters

This work addresses a fundamental gap in AI agent evaluation: most benchmarks measure one-shot generation quality, but real-world agents must iteratively improve based on feedback. A-CEGIS provides a practical, reproducible framework for measuring this refinement capability, which is essential for deploying agents in production environments where initial outputs are rarely perfect.

Technical Details

• Framework: A-CEGIS (Agent Counterexample-guided Inductive Synthesis) operates as a multi-turn loop where an agent proposes a regex, a deterministic oracle validates it under full-match semantics, and counterexamples (false positives/negatives) are fed back as guidance
• Dataset: Evaluated on 30 NL-RX-Turk tasks for natural-language-to-regex synthesis, with a hidden test set used for robustness hardening
• Benchmarks: Compared four conditions—zero-shot generation (17% success), generic self-correction (27%), error-only feedback (23%), and diagnostic counterexample feedback (90%)—within a four-turn ablation budget
• Performance metrics: Full diagnostic run achieved 100% task completion on the hidden set by the final turn, with mean time-to-success of 2.7 turns and 77% robust success after targeted probing
• Oracle design: Uses a deterministic regex oracle that generates compact witness examples (specific false-positive or false-negative strings) rather than abstract error messages, enabling more precise agent corrections

Industry Insight

• Agent evaluation pipelines should move beyond single-turn benchmarks; multi-turn refinement capability is a stronger predictor of real-world deployment success and should become a standard evaluation dimension
• Counterexample-based feedback is substantially more effective than generic error messages or self-correction, suggesting that production agent systems should integrate concrete witness generation into their feedback loops
• The A-CEGIS framework's approach to robustness hardening—testing on hidden cases after iterative refinement—offers a practical template for stress-testing agents before deployment in safety-critical applications

TL;DR

• 提出A-CEGIS框架，利用反例作为反馈评估Agent多轮自我修正能力，聚焦自然语言到正则表达式的合成任务
• 在30个NL-RX-Turk任务上，诊断性反例反馈在4轮内解决90%任务，远超零样本生成(17%)、通用自修正(27%)和仅错误反馈(23%)
• 采用确定性oracle检查正则表达式，通过紧凑的假阳性/假阴性反例见证引导下一轮修正
• 在隐藏测试集上所有任务最终均被解决，平均成功轮次2.7轮，针对性探测后稳健成功率77%
• 证明多轮反例反馈能高效衡量Agent迭代改进能力，同时提供超越原始测试集的鲁棒性验证

为什么值得看

单轮代码生成指标无法反映部署Agent接收具体反馈后修复错误的能力，本文填补了这一评估空白。A-CEGIS框架为Agent自我修正提供了可量化、可复现的评估基准，对构建可靠的多轮Agent系统具有重要参考价值。

技术解析

A-CEGIS是一个轻量级框架，核心机制是：Agent提出正则表达式后，确定性oracle在完全匹配语义下进行检查，生成紧凑的假阳性或假阴性反例作为反馈，引导下一轮修正。实验基于30个NL-RX-Turk任务，对比了四种反馈策略：零样本生成(17%)、通用自修正(27%)、仅错误反馈(23%)和诊断性反例反馈(90%)。在完整诊断运行中，隐藏集所有任务最终均被解决，平均成功轮次2.7轮，经过针对性探测后稳健成功率77%。

行业启示

多轮反馈机制是提升Agent自我修正能力的关键，反例反馈比通用反馈或仅错误信息更有效，值得在Agent系统设计中优先采用。评估Agent能力需从单轮指标转向多轮迭代指标，关注Agent在实际部署中接收反馈并持续改进的效率。A-CEGIS框架为Agent评估提供了新的方法论，同时引入的鲁棒性检查可帮助识别模型在边界情况下的脆弱性。