How Consistent Are LLM Agents? Measuring Behavioral Reproducibility in Multi-Step Tool-Calling Pipelines

The core tension this paper exposes is one of the most quietly consequential in applied AI: the clash between the engineering expectation of deterministic systems and the probabilistic reality of large language models. In production environments, we build intricate workflows where one agent's action triggers another, data is fetched, APIs are called, and side effects occur—all predicated on the assumption that "if X, then Y" holds true. This study systematically dismantles that assumption for a specific, and increasingly common, class of agents: those that don't just reason and search, but that call discrete, typed tools with real-world consequences. It’s a vital contribution because it moves the consistency debate beyond the realm of open-ended reasoning or search-based "ReAct" agents into the structured, consequential interfaces that are actually being deployed to manage workflows, process data, and interact with enterprise software.

What makes the finding particularly alarming is the context: structured tool-calling with typed parameters was supposed to be the solution to the chaos of free-text actions. The argument was that by constraining the LLM's output to a well-defined schema—"call function get_customer_data(customer_id: string) with argument C123"—we inject a layer of predictability and debuggability. This research suggests that layer may be thinner than we hoped. The inconsistency isn't just about the LLM occasionally generating a malformed JSON; it's about the model arriving at a different strategic decision in its tool selection or sequencing under identical conditions. This points to a deep-seated issue: the "reasoning" that precedes the tool call, which is baked into the model's weights and influenced by its sampling temperature and internal stochastic processes, remains fundamentally non-deterministic. The structured output is merely a veneer over that fluid, probabilistic cognition.

The implications cascade through the entire stack of AI system design. For developers and MLOps, this inconsistency is a debugging nightmare. How do you replicate an error, audit a decision path, or ensure compliance if the agent that caused a failure might not reproduce it? It undermines logging, testing, and monitoring frameworks built on the premise of causal reproducibility. For businesses, it introduces a novel category of operational risk. An agent might correctly process an insurance claim 99 times, then on the 100th identical input, choose a different tool or different sequence leading to an erroneous denial, all without any external system change. This isn't a traditional software bug; it's a failure of probabilistic reliability, and it erodes the very trust needed to delegate consequential tasks to AI.

Furthermore, this work challenges the notion of "testing" AI agents in a meaningful way. Traditional software testing relies on deterministic pass/fail criteria. How do you write a test suite for a system that might legitimately choose three different valid tool sequences to achieve the same goal? It forces a reevaluation of evaluation itself—perhaps success metrics must shift from bit-for-bit consistency to goal-oriented robustness, measuring whether the final outcome is correct regardless of the path taken, which is a much harder problem to specify and guarantee.

Ultimately, the paper is a sobering call for a more nuanced paradigm. The path forward isn't abandoning tool-calling agents, but designing for their inherent inconsistency. This might involve building ensemble-like systems where multiple agent passes are run and consensus is required for critical actions, or developing sophisticated "consistency wrappers" that monitor and correct for drift. It also underscores the need for a new class of development tools focused not on deterministic debugging, but on probabilistic profiling and reliability engineering. We are building systems on a probabilistic substrate, and this research is a stark reminder that our engineering frameworks must evolve to explicitly account for that foundation, rather than pretend it doesn't exist. The reliability of our future AI systems may depend on it.