Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing
Agentic security systems use LLM agents to autonomously plan, dispatch, and interpret security tools, but repeatedly encounter the same operational failures when moving from demos to production The authors introduce a four-dimensional Integration Friction Index that distinguishes one-time engineering costs from recurring organizational, legal, and maintenance costs Key quantitative findings include: long-lived sessions lose resident evidence as phase count increases, short-lived sub-agents exten
Analysis
TL;DR
- Agentic security systems use LLM agents to autonomously plan, dispatch, and interpret security tools, but repeatedly encounter the same operational failures when moving from demos to production
- The authors introduce a four-dimensional Integration Friction Index that distinguishes one-time engineering costs from recurring organizational, legal, and maintenance costs
- Key quantitative findings include: long-lived sessions lose resident evidence as phase count increases, short-lived sub-agents extend usable horizon based on evidence compression ratio, and two-stage verdict cascades only help when scorer errors are uncorrelated
- Planner-versus-worker model routing is formalized as a knapsack problem with a closed-form execution cap for heavy-tailed tools: eta* = alpha v/c
- Scope and budget enforcement cannot be delegated to system prompts because prompts do not constrain what actually executes
Why It Matters
This paper provides the first systematic analysis of failure modes in deployed LLM-driven penetration testing systems, offering practitioners concrete design laws rather than anecdotal lessons. The Integration Friction Index and quantitative regularities give security teams a framework for evaluating trade-offs between agent longevity, evidence retention, and operational cost—critical as agentic security tools move from research prototypes to production deployments.
Technical Details
- Evaluated ten widely used security tools across static, dynamic, cloud, orchestration, and AI red-teaming categories for unattended pipeline deployment, identifying recurring operational failure patterns
- Modeled agentic security systems as stochastic LLM policies wrapped by a deterministic mediator, deriving that long-lived sessions degrade resident evidence with increasing phase count, while short-lived sub-agents preserve it proportionally to the compression ratio between raw evidence and summaries
- Demonstrated that two-stage verdict cascades multiply scorer likelihood ratios but provide negligible benefit when scorer errors are correlated, and showed that treating unevaluable outcomes as attack failures systematically biases measurements toward evasive and severe response classifications
- Formulated planner-versus-worker model routing as a knapsack optimization problem and derived a closed-form execution cap eta* = alpha v/c for heavy-tailed tool latency distributions
- Implemented Inspectra as a worked instantiation of the proposed design laws, with mechanisms explicitly labeled as shipped, partial, or planned, including documented failures
Industry Insight
- Organizations deploying agentic security tools should invest in short-lived sub-agent architectures with strong evidence compression rather than monolithic long-running sessions, as the latter degrade in reliability with phase count
- Verdict cascade designs must account for scorer error correlation; deploying correlated evaluators provides false confidence and wasted compute—diverse, independent scorers are essential
- System prompts are insufficient for scope and budget enforcement; hard architectural constraints are required to prevent agent drift, as prompts cannot guarantee execution boundaries in stochastic LLM pipelines
Disclaimer: The above content is generated by AI and is for reference only.