Where Does Harness-Optimization Value Live? Localized Gains and the Budget-Splitting Trap in Self-Evolving LLM Agents
HARNESSEVO decomposes agent harnesses into four evolvable slots (role, task-strategy, tool/format-rules, reflection/control) instead of optimizing as a flat string On ALFWorld with a frozen 7B backbone, overall success rate (0.657) did not significantly exceed stock harness or flat-string evolution (both 0.642) Nearly all optimization value is localized in the reflection/control slot, which achieved a leave-one-in gain of +0.119; other slots showed null contributions Uniform budget splitting acr
Analysis
TL;DR
- HARNESSEVO decomposes agent harnesses into four evolvable slots (role, task-strategy, tool/format-rules, reflection/control) instead of optimizing as a flat string
- On ALFWorld with a frozen 7B backbone, overall success rate (0.657) did not significantly exceed stock harness or flat-string evolution (both 0.642)
- Nearly all optimization value is localized in the reflection/control slot, which achieved a leave-one-in gain of +0.119; other slots showed null contributions
- Uniform budget splitting across four slots is actively harmful: 16 rollouts per slot fell below the optimizer's effective search floor, causing all slots to freeze at empty seeds
- Concentrating budget on the high-credit control slot recovered lost gains, reaching 0.761 with half the split budget; WebShop showed no verbalizable control failures, confirming task-contingency
Why It Matters
This work challenges the common assumption that structured prompt decomposition inherently improves agent performance, revealing that optimization value is highly localized rather than distributed. For AI practitioners building self-evolving agents, the findings directly impact how computational budgets should be allocated during harness evolution, warning against naive uniform splitting that can degrade performance below baseline.
Technical Details
- HARNESSEVO introduces a four-slot harness decomposition: role, task-strategy, tool/format-rules, and reflection/control, each independently evolvable under a reflective optimizer
- Attribution is measured using leave-one-in and leave-one-out methods to quantify each slot's marginal contribution to agent performance
- Experiments use a frozen 7B LLM backbone evaluated on ALFWorld (interactive navigation task) and WebShop (e-commerce browsing task)
- The iso-budget setting ensures fair comparison: the same total number of rollouts is allocated across conditions, revealing that uniform splitting (16 per slot) falls below the optimizer's effective search floor
- Budget concentration on the reflection/control slot alone achieved 0.761 success on ALFWorld, surpassing all flat and uniformly-split variants
Industry Insight
- Credit assignment should precede structured agent evolution; blindly decomposing prompts without identifying high-impact slots wastes compute and may harm performance
- Practitioners should adopt adaptive budget allocation strategies that concentrate optimization effort on slots demonstrating measurable gains rather than distributing evenly
- The task-contingent nature of harness value (present on ALFWorld, absent on WebShop) suggests that harness optimization should be evaluated per-domain, and negative results on one benchmark should not be generalized to all agent applications
Disclaimer: The above content is generated by AI and is for reference only.