EvoClawBench: Can Agents Learn Reusable Skills from Their Own Runs?
EvoClawBench introduces a novel benchmark designed to evaluate whether AI agents can autonomously learn reusable skills from their own execution history in a closed-loop manner. The benchmark isolates skill acquisition by comparing direct execution, pre-authoring skills, and post-execution summarization across 100 tasks and 502 sub-problems. Experimental results demonstrate that self-authored skill learning is highly selective and runtime-dependent, with performance improvements being neither au
Analysis
TL;DR
- EvoClawBench introduces a novel benchmark designed to evaluate whether AI agents can autonomously learn reusable skills from their own execution history in a closed-loop manner.
- The benchmark isolates skill acquisition by comparing direct execution, pre-authoring skills, and post-execution summarization across 100 tasks and 502 sub-problems.
- Experimental results demonstrate that self-authored skill learning is highly selective and runtime-dependent, with performance improvements being neither automatic nor consistent across different models.
- Significant variance was observed in model performance, ranging from stable high accuracy (e.g., nanobot GPT-5.4) to catastrophic failure (e.g., nanobot DeepSeek-V4-Pro dropping to <1% accuracy).
Why It Matters
This research addresses a critical gap in current AI agent evaluation by moving beyond simple task completion metrics to assess the ability of agents to self-improve through experience. For practitioners, it highlights that integrating self-learning loops is not a plug-and-play solution and requires careful consideration of runtime stability and model selection. It provides empirical evidence that skill reuse is complex and potentially risky, necessitating robust validation strategies before deployment in production environments.
Technical Details
- Benchmark Structure: EvoClawBench comprises 100 tasks and 502 sub-problems spanning coding, data, office, security, operations, and domain-document workflows.
- Evaluation Modes: The study compares three distinct operational modes: direct execution without skills, PreSkill authoring (skills created before execution), and PostSkill summarization (skills derived from first-run evidence for a fresh second execution).
- Runtime Environments: Experiments were conducted using two primary agent runtimes, OpenClaw and nanobot, supporting multiple underlying language models.
- Key Findings: Direct baseline performance varied significantly by runtime (OpenClaw <20%, nanobot 56.45%-96.13%). Skill acquisition showed non-monotonic effects; for instance, MiniMax-M2.7 improved via PostSkill, while DeepSeek-V4-Pro suffered severe performance degradation in both PreSkill and PostSkill modes.
Industry Insight
- Caution in Auto-Learning Loops: Developers should not assume that adding self-reflection or skill-authoring capabilities will automatically enhance agent performance. Rigorous testing is required to prevent catastrophic failures in specific model-runtime combinations.
- Runtime Selection is Critical: The choice of agent runtime has a profound impact on baseline performance and the success of skill reuse strategies. Investing in stable, high-performing runtimes may yield better returns than focusing solely on model size or capability.
- Hybrid Approaches May Be Necessary: Given the mixed results, a hybrid strategy that combines human-in-the-loop verification for skill creation with automated post-execution summarization might offer a more reliable path to reusable agent skills than fully autonomous methods.
Disclaimer: The above content is generated by AI and is for reference only.