Changing One Prompt Can Affect 50 Others — I Built a Prompt Dependency Graph to Find What Needs Retesting
A pure Python prompt dependency graph was built to solve the change-impact problem in composable prompt systems, answering which downstream prompts need re-evaluation after a shared component is modified. The tool outputs two metrics: "Reachable" (the full structural blast radius downstream of a change) and "Candidate" (a narrower set of directly and transitively dependent components that should be evaluated). On a deterministic 55-node synthetic system, section-aware tracking narrowed the evalu
Analysis
TL;DR
- A pure Python prompt dependency graph was built to solve the change-impact problem in composable prompt systems, answering which downstream prompts need re-evaluation after a shared component is modified.
- The tool outputs two metrics: "Reachable" (the full structural blast radius downstream of a change) and "Candidate" (a narrower set of directly and transitively dependent components that should be evaluated).
- On a deterministic 55-node synthetic system, section-aware tracking narrowed the evaluation set by 0% to 85%, with the most selective components (e.g., base-policy/refunds) achieving ~47% reduction and widely shared components (e.g., tone/professional) showing 0% narrowing.
- The approach identifies what should be evaluated structurally, not what will actually fail behaviorally—downstream LLM evaluation is still required to confirm real impact.
- The work draws direct parallels to change impact analysis in software engineering, Bazel-style incremental builds, and chaos engineering principles for bounding blast radius.
Why It Matters
As production prompt systems grow into composable architectures with shared policy, tone, and format blocks, the cost and risk of re-evaluating every downstream prompt after any change becomes prohibitive. This tool brings a long-standing software engineering practice—dependency-traced change impact analysis—to prompt engineering, enabling practitioners to ship changes with confidence while avoiding both over-testing and dangerous under-testing.
Technical Details
- Core mechanism: A dependency graph built in pure Python that traces import/usage relationships between prompt components, computing both the full reachable set and a tighter candidate set for any given change target.
- Two-tier metric system: "Reachable" represents the structural ceiling (all downstream nodes), while "Candidate" filters to directly dependent nodes plus their downstream consumers, enabling more targeted evaluation.
- Synthetic benchmark: A 55-node deterministic system was used to test multiple change scenarios (base-policy/refunds, tone/professional, format/json, base-policy/privacy, safety/no-medical-advice), measuring narrowing percentages against the full reachable set.
- Flat vs. deep dependency structures: The author demonstrated that a flat dependency layout (one component feeding 4 agents directly) and a deep layout (one component feeding through multiple intermediate layers) can produce identical total impact but very different candidate sets, highlighting the limitation of naive flat lookup approaches.
- Key limitation acknowledged: The graph is structural only—it cannot predict behavioral failure. Actual LLM evaluation of candidate prompts remains necessary to confirm whether a change causes real output drift.
Industry Insight
- Prompt engineering teams should adopt dependency-tracking tooling as their composable prompt systems scale; manual guessing about blast radius is unreliable and risks shipping regressions to production agents.
- The 0–85% narrowing range reveals that the value of this approach is highly context-dependent—teams should expect unpredictable results and plan evaluation budgets accordingly rather than assuming consistent savings.
- The analogy to Bazel and chaos engineering suggests an opportunity for the broader AI engineering tooling ecosystem: build systems, CI/CD pipelines, and prompt management platforms could integrate structural impact analysis as a first-class feature to reduce evaluation costs and improve change safety.
Disclaimer: The above content is generated by AI and is for reference only.