Human-in-the-Loop Without Killing Throughput
A blanket human-approval gate on all non-SELECT SQL operations caused "rubber-stamp fatigue," where reviewers skimmed batches of queries under pressure, eroding the very safety the system was designed to provide The team shifted from operation-type-based routing (all writes → human) to risk-based routing that scores actions against concrete signals before escalating Four key risk signals were identified from six weeks of approval logs: blast radius (rows affected and reversibility), query contex
Analysis
TL;DR
- A blanket human-approval gate on all non-SELECT SQL operations caused "rubber-stamp fatigue," where reviewers skimmed batches of queries under pressure, eroding the very safety the system was designed to provide
- The team shifted from operation-type-based routing (all writes → human) to risk-based routing that scores actions against concrete signals before escalating
- Four key risk signals were identified from six weeks of approval logs: blast radius (rows affected and reversibility), query context, table criticality, and user history
- Blast radius estimation via EXPLAIN proved unreliable on skewed data; the team adopted a capped COUNT(*) approach (up to 50,000 rows) for accurate, bounded-cost estimation
- A safety mechanism that's too broad doesn't fail safe—it fails slow, and slow failure modes get quietly disabled by whoever is under the most pressure to ship
Why It Matters
This article exposes a critical failure mode in AI agent safety design: blanket human-in-the-loop gates create latency bottlenecks that degrade review quality under load, ultimately producing the opposite of the intended safety outcome. For AI practitioners building agentic systems, it demonstrates that oversight mechanisms must be risk-aware and signal-driven rather than operation-type-based, or they become liability shields that look like safety without delivering it.
Technical Details
- Risk-based routing architecture: Every agent action is scored against multiple signals before execution; only actions exceeding a risk threshold are escalated to human reviewers, while low-risk actions execute immediately
- Blast radius estimation: Instead of relying on database planner estimates (EXPLAIN), which become unreliable on skewed columns and correlated predicates, the team runs the query's WHERE clause as a real COUNT(*) capped at a fixed ceiling (50,000 rows), providing an actual bounded-cost measurement
- Four risk signals identified from approval log analysis: (1) Blast radius—rows touched and reversibility, (2) Query context—whether the action aligns with recent conversation intent, (3) Table criticality—dependency depth of downstream dashboards and systems, (4) User history—pattern of past query behavior and risk profile
- Original system flaw: Both a DELETE of a single row by primary key and a DELETE affecting 40% of a shared table were routed identically to the same approval queue with no distinguishing signals for the reviewer
- Decoupled approval queue: The human review queue was eventually decoupled from the user session, allowing asynchronous review without blocking the analyst's workflow
Industry Insight
- Blanket human-approval gates on AI agent actions are a false sense of security—they create latency tax and rubber-stamp fatigue that erodes vigilance precisely when it's needed most; designers should ask "which actions were we actually reviewing carefully?" before implementing broad gates
- Risk-based routing requires investment in signal extraction and scoring upfront (the router took longer to build than the queue), but pays exponential returns in both safety quality and operational throughput; the cost of getting it wrong is silent degradation, not outright failure
- When estimating query impact in text-to-SQL systems, prefer empirical counting with bounded ceilings over planner estimates, especially when agents generate queries without knowledge of data distribution—skewed columns and correlated predicates will silently mislead EXPLAIN-based approaches
Disclaimer: The above content is generated by AI and is for reference only.