Superpowers vs Plain Old Debugger in Explyt
Two AI agents (Superpowers and Explyt) debugged the same Kotlin/Windows bug using the same model (Opus 5) and prompt, with only the debugging workflow differing Superpowers followed a systematic checklist but skipped runtime observation, producing a 132k-token speculative patch that was discarded for hiding a critical error distinction and introducing a test-only seam into production code Explyt integrated the JetBrains debugger to observe actual runtime values before editing, producing a 67k-to
Analysis
TL;DR
- Two AI agents (Superpowers and Explyt) debugged the same Kotlin/Windows bug using the same model (Opus 5) and prompt, with only the debugging workflow differing
- Superpowers followed a systematic checklist but skipped runtime observation, producing a 132k-token speculative patch that was discarded for hiding a critical error distinction and introducing a test-only seam into production code
- Explyt integrated the JetBrains debugger to observe actual runtime values before editing, producing a 67k-token narrow fix that passed review and shipped to production
- The key divergence: inferred hypotheses from logs versus verified facts from a live breakpoint, demonstrating that a procedural checklist without runtime evidence can produce plausible but incorrect patches
- JetBrains' own A/B tests on "token-saving" skills show mixed results, reinforcing the lesson to measure whole-agent-run costs rather than relying on self-reported skill counters
Why It Matters
This controlled comparison reveals a critical gap between following a debugging methodology on paper and actually executing its experimental step—something that has direct implications for how AI practitioners evaluate and deploy debugging agents. The finding that debugger-integrated agents can halve token consumption while producing higher-quality fixes challenges the assumption that more agentic turns and speculative code generation lead to better outcomes.
Technical Details
- Bug: A
java.nio.file.AccessDeniedExceptionthrown byFiles.list()on a protected Windows Installer directory (C:\Config.Msi), escaping throughFacadeMentionCandidates.listDirinto an unhandled coroutine exception inWorkspaceBootstrap - Superpowers workflow: The systematic-debugging skill encoded the sequence (gather facts → form hypothesis → run experiment → change code), but the agent never actually ran an experiment—it inferred the root cause from the stack trace and patched without observing runtime values
- Explyt workflow: Created a minimal reproducer, set a line breakpoint on
Files.list, and inspected concrete values:Files.isDirectory(dir) = true,Files.isReadable(dir) = false, confirming the access denial before editing - Token usage: Superpowers consumed 132k tokens through speculative code, tests, and refactoring; Explyt used 67k tokens by obtaining the key fact early from the debugger
- JetBrains A/B tests: The Caveman test showed advertised 65% token reduction dropping to 8.5% on real tasks; the rtk test showed a 7.6% cost increase at low reasoning effort; the Ponytail test found 10.3% reduction—results are inconsistent, underscoring the need for whole-run measurement
Industry Insight
- Debugger-integrated agents should be preferred over text-only agentic debuggers for runtime bugs, as live execution evidence prevents speculative patching and reduces both token waste and regression risk
- Teams should benchmark AI debugging tools on whole-agent runs with real bugs rather than relying on vendor-reported skill-level metrics, which can be misleading by orders of magnitude
- The "checklist ritual" phenomenon—where agents appear to follow methodology but skip its critical empirical step—is a real failure mode that suggests debugging workflows for AI agents must enforce observable experiment execution, not just procedural compliance
Disclaimer: The above content is generated by AI and is for reference only.