Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses
LLM coding agents (Claude Code, Cursor, Copilot, Aider) rarely request pagination despite tool responses routinely exceeding per-turn token budgets; session logs show zero agent-initiated second-chunk requests Raising precision-at-1 (p₁)—the rate the needed item appears first in the initial chunk—does not systematically improve downstream agent accuracy, with per-model deltas under 3 percentage points and no statistical significance Adding file-metadata signals to a keyword scorer actually degra
Analysis
TL;DR
- LLM coding agents (Claude Code, Cursor, Copilot, Aider) rarely request pagination despite tool responses routinely exceeding per-turn token budgets; session logs show zero agent-initiated second-chunk requests
- Raising precision-at-1 (p₁)—the rate the needed item appears first in the initial chunk—does not systematically improve downstream agent accuracy, with per-model deltas under 3 percentage points and no statistical significance
- Adding file-metadata signals to a keyword scorer actually degrades p₁ by 4.8 p.p. (p = 0.001), suggesting metadata complexity harms simple retrieval
- A parameter-free keyword scorer significantly improves p₁ from 24.2% to 35.0% (+10.8 p.p., p = 3.9 × 10⁻⁸), but this rank-1 gain does not translate to end-to-end accuracy improvements
- Agents recover the gold item from anywhere within the first chunk, meaning first-chunk inclusion—not the item's rank within that chunk—is what determines downstream performance
Why It Matters
This research challenges a common assumption in the AI engineering community: that optimizing the ordering of tool responses (putting the most relevant item first) will meaningfully improve agent performance. For practitioners building or tuning coding agents, the findings suggest that efforts should focus on ensuring the relevant information is present within the first chunk rather than expending resources on sophisticated ranking or pagination mechanisms.
Technical Details
- The study treats first-chunk selection as a 0/1 knapsack problem, comparing six value functions across 500 SWE-bench Verified tasks to evaluate how well different scoring strategies place the gold item first
- A single-turn file-localisation probe was conducted on five language models with 4,800 LLM calls, measuring p₁ (precision-at-1) as the primary metric for first-chunk quality
- The keyword scorer is parameter-free and achieves a baseline p₁ of 35.0%, rising to 35.8% with a fallback to the tool's native ordering when no keyword matches are found
- Four file-metadata signals (added as an enhancement to the keyword scorer) were tested but found to hurt p₁ by 4.8 p.p. with paired significance (p = 0.001)
- The experimental design explicitly notes this is not an end-to-end resolve-rate test, but rather a focused probe into whether first-chunk ranking quality propagates to downstream accuracy
Industry Insight
- Agent designers should prioritize ensuring comprehensive first-chunk inclusion over complex ranking algorithms; the marginal cost of chunk expansion may be lower than the engineering effort spent on precision-at-1 optimization
- Pagination features in tool protocols (like MCP) are effectively dead code for current LLM coding agents, suggesting protocol designers could deprioritize pagination UX and focus on chunk-sizing strategies instead
- The negative result on metadata enrichment implies that simpler retrieval signals often outperform feature-combined approaches in agent tool-response contexts, a principle that may generalize beyond coding agents to other tool-use scenarios
Disclaimer: The above content is generated by AI and is for reference only.