What Context Does a Coding Agent Actually Need to Act?
Coding agents require significantly less context than assumed; compressed representations match full-file performance while using only one-third of the tokens (19K vs 94K per resolved issue). Natural language summaries of code are ineffective for resolving behavioral issues compared to source code, regardless of whether the summarizer is a frontier model or a smaller 3B parameter model. Surrounding context in multi-file scenarios provides negligible benefit; reducing file remnants to UML skeleto
Analysis
TL;DR
- Coding agents require significantly less context than assumed; compressed representations match full-file performance while using only one-third of the tokens (19K vs 94K per resolved issue).
- Natural language summaries of code are ineffective for resolving behavioral issues compared to source code, regardless of whether the summarizer is a frontier model or a smaller 3B parameter model.
- Surrounding context in multi-file scenarios provides negligible benefit; reducing file remnants to UML skeletons performs statistically identically to deleting them entirely.
- Temperature-0 API inference introduces a significant noise floor, flipping approximately 9% of per-instance outcomes even between byte-identical runs, challenging the reliability of small effect sizes in benchmarking.
Why It Matters
This research fundamentally challenges the prevailing assumption that larger context windows automatically lead to better coding agent performance, suggesting that efficiency gains can be achieved through smarter context compression rather than raw token capacity. For AI practitioners, it highlights critical flaws in current evaluation methodologies, particularly regarding the stability of API-based inference, urging a re-evaluation of benchmark results that may be obscured by inherent stochasticity.
Technical Details
- Methodology: The study isolates the "acting" phase of coding agents by fixing code localization with an oracle and varying only the context representation, evaluated on SWE-bench Verified.
- Context Compression: Demonstrated that compressed context (UML skeletons/signatures) achieves equivalent issue resolution rates to full files while reducing token usage by approximately 67%.
- Summary Inefficacy: Found that natural language summaries fail to capture necessary behavioral signals, with a frontier model's summaries performing no better than those from a 3B model ($4/45$ vs $27/45$ correct behavioral answers).
- Statistical Rigor: Utilized pre-registered hypotheses and exact McNemar tests ($p=0.75$) to validate that surrounding context removal does not impact performance, alongside releasing a gold-validated instrument for deterministic patch construction.
Industry Insight
- Optimize for Compression: Developers should prioritize building context compression pipelines (e.g., extracting signatures and skeletons) over simply increasing context window sizes, as this yields substantial cost and latency benefits without sacrificing accuracy.
- Re-evaluate Benchmarks: The discovery of a 9% outcome flip rate in temperature-0 inference suggests that many reported marginal improvements in coding benchmarks may be statistical noise; rigorous statistical testing and multiple runs are essential for valid comparisons.
- Source Over Summaries: When designing agentic workflows, rely on raw code structure and signatures rather than LLM-generated summaries for decision-making logic, as summaries lack the precision required for behavioral reasoning.
Disclaimer: The above content is generated by AI and is for reference only.