Tool-Call Reliability in Production LLM Agents

A Comparative Study of 13 Language Models on Function-Calling Accuracy, Consistency, and Silent Failure

Executive Summary

Modern AI agents increasingly rely on "tool calling" (also called function calling) to take real actions: updating records, sending notifications, confirming completed work, rather than just producing text. This creates a failure mode that doesn't exist in plain chat: a model can describe an action as completed in its reply without ever actually invoking the tool that performs it. We call this silent skip, and it is more dangerous than an ordinary wrong answer, because nothing in the visible conversation indicates anything went wrong.

We designed a controlled study to measure this and related failure modes across 13 widely available language models, using a real, production-grade agent system as the test harness rather than a synthetic benchmark. Every model was evaluated against the exact same system instructions, the exact same set of callable tools, and the exact same execution and permission logic - only the model itself varied. Across roughly 1,900 trials spanning 16 distinct test scenarios (of 24 designed in total; see Limitations), we measured three things: whether a model selects and correctly parameterizes the right tool, whether it silently skips a required tool call while claiming success, and how consistent its behavior is across repeated, identical trials.

The results are, in places, surprising. Several models showed real, non-trivial rates of silent skip - including one model that skips a specific class of tool call 80% of the time. The most accurate and most consistent model tested was not the newest or most expensive one. And a model change that had been under internal consideration specifically to fix tool-reliability problems turns out, in this data, to plausibly make one particular failure mode worse, not better. We report full results below, along with the methodology, so the findings can be scrutinized and reproduced.

1. Introduction

AI agents that call external tools/functions are becoming a standard architecture for connecting language models to real systems: booking actions, updating records, sending messages, retrieving live data. The promise is that a model can reason in natural language and then take a precise, structured action, but this only works if the model reliably takes the correct action when it should, and only that action.

This study originated from a real, live observation. While operating a production multi-tenant agent system (details of the system's business domain are withheld in this paper; see the Disclosure note in Section 2), we observed a model, mid-conversation, on a task it had previously handled correctly, skip a required tool call entirely while still replying to the user as though the action had succeeded. This was not a one-off glitch; the same failure recurred on a retry of an identical scenario with identical inputs. That observation is the direct motivation for this study: rather than treat this as an isolated bug, we built a controlled experiment to find out whether it's a property of one specific model, or a broader pattern across the language models commonly used for this kind of work.

1.1 What we set out to measure

1. Selection accuracy: given a scenario with one clearly correct tool call (or correctly no tool call at all), does the model make the right choice?
2. Argument accuracy: given that the model picks the correct tool, are the parameters it supplies correct (the right identifier, the right status value, a correctly constructed timestamp, the right recipient)?
3. Silent-skip rate: does the model ever narrate a successful action in its reply without having actually invoked the tool that performs it? This is the failure mode motivating the whole study, and the one most likely to go undetected in production, since the visible output looks entirely normal.
4. Consistency: running the identical scenario repeatedly against the same model, how stable is its behavior? A model that is wrong 100% of the time on some task is trivially "consistent"; the property that actually matters for production reliability is a model being both accurate and stable, not oscillating between correct and incorrect behavior on a task with only one right answer.

2. Methodology

2.1 Disclosure and anonymization

This study ran against a real, operational multi-tenant agent platform in the field-operations / logistics-coordination space, rather than a synthetic benchmark environment. To protect proprietary business logic, this paper does not disclose the specific product, its exact tool names, or implementation details. Tools are referred to throughout by generic functional labels (e.g., "the status-update tool," "the attachment-confirmation tool"), and the domain is described in general terms. This anonymization has no effect on the results: every trial in this study ran against the real, unmodified system instructions, the real set of callable tools, and the real execution and permission-checking logic used in production; only the underlying language model was swapped between trials. Model identities are reported accurately throughout, since comparing named models is the point of the study.

2.2 System under test

The platform under test is a multi-tenant coordination agent: it receives natural-language messages from several categories of user (front-line field staff, external contacts, and internal operators), interprets intent, and, where appropriate, invokes one of roughly two dozen callable tools to update records, send notifications, evaluate incoming work opportunities, or escalate to a human when a request falls outside its scope. Every tool call is checked against a role-based permission system before it is allowed to execute, and every state-changing action is written to an auditable event log.

We tested against this real system rather than a simplified stand-in for two reasons. First, real system instructions are long, specific, and full of the kind of conditional business logic (context-dependent branching, multi-step workflows, edge cases) that a synthetic toy prompt doesn't capture - and that specificity is exactly where tool-call reliability tends to break down. Second, testing against the real execution path means a tool call that a model makes actually writes real state, letting us verify not just that a tool was called but that the system ended up in the correct resulting state.

2.3 Models evaluated

We evaluated 13 models spanning five model families, chosen to represent the tier of models realistically used for this kind of high-volume, cost-sensitive agent workload rather than the most expensive flagship tier of any provider:

• Anthropic: Claude 3 Haiku, Claude Haiku 4.5
• DeepSeek: DeepSeek V4 Flash, DeepSeek V3.2, DeepSeek Chat V3.1
• Meta (Llama): Llama 4 Scout, Llama 4 Maverick, Llama 3.3 70B, Llama 3.1 8B
• Z.AI (GLM): GLM 4.7 Flash, GLM 4.6, GLM 4.5 Air
• OpenAI: GPT-5.6 Luna

All 13 were confirmed to support function/tool calling via their API before inclusion. One of these models (from the DeepSeek family) is the model currently deployed in our own production system; it is included as a continuity baseline, not given any special treatment in scoring.

2.4 Scenario design

We designed 24 scenarios across eight categories, each targeting a specific, distinct failure mode
rather than a generic "does it work" check:

• Core correctness (9 scenarios): single-turn tasks with one unambiguous correct action: status updates that branch on current state, delivery/attachment confirmation, delay reporting, availability updates, time-log entry from relative phrasing ("in at 6, out at 3"), work-opportunity evaluation with numeric extraction, and status lookups.
• Disambiguation under ambiguity (3 scenarios): cases where the correct behavior is to ask a clarifying question rather than guess: multiple open items to choose from, zero open items, and an edge case where the requested action no longer applies because the work is already done.
• Argument-extraction edge cases (1 scenario): whether a model correctly captures all relevant inputs (e.g., multiple attached files) rather than just the first one it encounters.
• Silent-skip probes (3 scenarios): the scenarios this study is built around. Each gives the model everything it needs to successfully complete one specific tool call, then measures whether it actually makes that call, repeated across three structurally different tools.
• Authorization-boundary respect (2 scenarios): whether a model attempts an action outside the requesting party's role, and whether it leaks internal information to an unauthenticated party.
• Off-topic and adversarial input (2 scenarios): an off-topic message followed by a normal, on-topic message in the same conversation (does the off-topic turn corrupt subsequent correct behavior?), and a message phrased to look like an instruction to bypass a safety-relevant requirement ("skip the confirmation photo this time").
• Resource-constrained reasoning (2 scenarios): whether a model reasons about real capacity constraints (e.g., zero available staff) rather than evaluating each incoming request in isolation, plus a numeric-fidelity check on decimal values.
• Multi-turn state (2 scenarios): whether information collected across multiple conversation turns is correctly retained and applied to the eventual tool call.

2.5 Test harness and protocol

Rather than routing test messages through the full production message pipeline (which would require standing up separate messaging infrastructure per model), we built a direct test harness that invokes the same core agent logic - the same system instructions, the same tool definitions, the same tool-dispatch and permission-checking code, directly against each model in the roster, with the model itself as the only variable. Every tool call that a model actually made was executed for real against an isolated test environment (not the live production database, this is a whole separate issue), so correctness could be verified against real resulting state, not just the model's stated intent.

Each of the 24 designed scenarios was intended to run against each of the 13 models for 10 independent trials, with the test environment reset to an identical starting state before every trial, so repeated trials of the same scenario are genuinely independent measurements rather than a single conversation replayed. This round of testing did not complete the full 24 × 13 grid: it produced just under 1,900 completed trials, covering up to 16 of the 24 designed scenarios per model (concentrated in four of the eight categories described in Section 2.4), with the remaining scenarios not yet tested at scale. The three silent-skip probe scenarios (this study's central focus) were deliberately prioritized to guarantee complete, equal 10-trial coverage across all 13 models regardless, since that specific comparison needed to be apples-to-apples. Every result reported below states its actual sample size; no result is presented as more complete than it is, and Section 6 (Limitations) details the coverage gap explicitly.

2.6 Grading

Each trial was graded programmatically against an explicit, scenario-specific pass criterion. E.g., "the status-update tool was called with the correct identifier and the correct target status," or "no tool call was made, and the reply asked a clarifying question." Grading logic was itself checked against real trial transcripts during development and corrected where it was found to be wrong (two such corrections were made before the full run) & full transcripts of every tool call and every model reply were retained for every trial to allow this kind of verification and to support the qualitative examples cited in the Results section below.

A small number of trials (159 of 1,909, about 8%) failed at the infrastructure level - the model provider's API itself returned an error or a malformed response before any model behavior could be observed. These are reported separately in Section 3.4 and excluded from all accuracy/consistency/ silent-skip figures, since there was no model decision to grade. To get consistent returns with the infra-response side of things, OpenRouter was chosen as the primary provider for the model responses.

3. Results

3.1 Silent-skip rate

This is the headline measurement. Each of the three probe scenarios gives a model everything it needs to successfully complete one specific tool call. The required identifiers are present, the request is unambiguous, and measures whether the model actually invokes that tool, or narrates success without doing so. All figures below are out of 10 trials per cell, full coverage across all 13 models.

Three observations stand out:

The failure mode is real and not rare. Half the models tested show a non-zero silent-skip rate somewhere, and it is not confined to the smallest or cheapest models in the roster - every model family tested has at least one member that silently skips a required tool call at least once in ten identical attempts.

It is tool-specific, not model-uniform. No model in this study is uniformly unreliable - a model can be flawless on one tool and skip a different tool 20–30% of the time. This rules out "the model is generally careless" as an explanation, and points toward something specific to how a particular instruction or tool is represented that certain models handle poorly.

One result is the clear standout finding of this study. A specific, widely-used model skips a specific tool call 80% of the time - eight times out of ten, it tells the user an action has been completed without ever invoking the tool that performs it. That model is discussed further in Section 5, because it has direct bearing on a model-upgrade decision that was under active consideration independent of this study.

3.2 Overall accuracy and consistency

Accuracy alone does not capture what this study is designed to measure - a model that is wrong 100% of the time on a given task is trivially "consistent" at being wrong. The property that matters is the pairing of high accuracy with low variance across repeated, identical trials.

(Consistency is expressed as the standard deviation of pass rate across all scenarios that model was tested on. A model that scores either 0% or 100% on every scenario would show a value near zero despite that being an unhelpful kind of "consistency"; this column should always be read together with the accuracy column, not in isolation.)

One model was the clear standout on both dimensions at once. It combined the highest accuracy observed in the study with the tightest, most stable spread of results - it was not merely usually correct, it was correct almost everywhere, almost every time, which is the specific property this study set out to identify. The two other models in the same family performed similarly well, clustering in the same high-accuracy, low-variance range.

One model illustrates precisely why consistency has to be measured separately from accuracy. Its overall accuracy (87.5%) looked respectable, mid-table - but it had the fourth-highest variance in the entire study. In practice, this is a model that is often excellent and occasionally badly wrong, rather than reliably good: exactly the "coin flip on a subset of tasks" pattern that an accuracy-only evaluation would miss entirely.

Two models clustered around 59–60% accuracy while also posting the two highest variance scores in the study. Both are best described as "sometimes very right, sometimes very wrong" rather than "consistently mediocre"; a meaningfully worse property for a system where an incorrect answer means a real state change (a status update, a confirmation) going through incorrectly, as opposed to just an unhelpful chat reply.

The smallest model tested was the clear floor of the study - the lowest accuracy (24.6%) and, as shown above, by far the highest silent-skip rate. Its dominant failure pattern was specifically "claims success without doing the work," a materially different and arguably worse signature than models that fail by taking the wrong action outright.

3.3 Notable behavioral patterns

A few specific results from the full scenario-by-scenario data (reproduced in the Appendix) are
worth calling out individually, because they illustrate why a model failed, not just that it
did:

• One model scored 0 out of 10 on both disambiguation scenarios - across 20 total trials, it never once asked a clarifying question in a situation where the correct behavior was to ask rather than guess. The same model scored well on several unambiguous single-turn tasks, suggesting a specific blind spot around ambiguity rather than general unreliability.
• One model scored 0 out of 10 on one branch of a status-update rule while scoring 5 out of 10 on the sibling branch of the exact same underlying rule (the two branches differ only in which state the item started in). This is strong evidence of surface-level pattern-matching on a trigger word rather than actually checking current state before deciding which action applies - precisely the failure mode that scenario pair was designed to catch.
• One model's three weakest scores across the entire study all involved the same underlying pattern: report an event, then separately notify or follow up. This is a consistent, repeatable weak spot for that specific model, not scattered noise - and it is the same model responsible for the 80% silent-skip result in Section 3.1, on a closely related tool.

3.4 Infrastructure-level reliability

Separately from model behavior, one model in the roster failed to complete a usable API response at all on effectively its entire sample - every one of its trials either returned an upstream provider error or a malformed, empty response that could not be evaluated. No other model in the study showed this pattern at any meaningful rate. This is not a tool-call-reliability finding in
the sense the rest of this paper measures; the model's actual decision-making was never exercised, but it is arguably still a reliability finding in a more basic sense: for a real integration decision, a model that cannot reliably complete a request at all is disqualifying on its own terms, independent of how it might have performed otherwise.

4. Discussion

Three patterns emerge from this data that we think generalize beyond this specific study.

Silent skip is a distinct failure mode from ordinary inaccuracy, and needs to be measured separately. A model can be highly accurate in the ordinary sense - correctly choosing and parameterizing tools most of the time, while still having a specific, narrow blind spot where it narrates success without acting. Standard accuracy metrics do not surface this, because the model's text output looks completely normal. Any evaluation of an agentic system that only checks reply quality, not actual tool invocation, will systematically miss this failure mode.

Consistency and accuracy are genuinely different axes, and collapsing them into one number loses the information that matters most for production risk. Two models with similar overall accuracy can have very different practical reliability if one is stable and the other oscillates. For a system making real, consequential state changes, a model that is "usually right" but unpredictably wrong is a harder operational problem than a model that is more uniformly (if somewhat less often) correct - the latter is at least predictable enough to build safeguards around.

Model family and generation are not reliable predictors of tool-call reliability on their own. The most accurate, most consistent model in this study was not the newest, largest, or most expensive model tested. A newer point release of an existing model line performed worse on the single most important metric in this study than an older point release from the same family. This suggests that tool-call reliability, specifically, should be evaluated directly and empirically for any given workload rather than assumed from a model's general reputation or release recency.

5. Recommendations

Upgrading to one specific newer model (Claude Haiku 4.5) had been under internal consideration independent of this study, motivated by general tool-reliability concerns. This study's data pushes back on that plan as previously scoped. In our results, that model placed mid-table on overall accuracy (81.9%), showed the second-highest silent-skip rate of any model tested (26.7%, driven almost entirely by an 80% skip rate on one specific tool), and the third-highest variance in the study. It is not a clear improvement over the model family already in production, and on the specific failure mode motivating this whole study, it performed meaningfully worse than several less expensive alternatives.

Based on the data collected here, the strongest candidate identified in this study is the humble DeepSeek model (V3.2): the highest accuracy observed, the lowest variance observed, and a perfect zero score on all three silent-skip probes. A second model from the same family (Chat V3.1) performed nearly as well on every dimension. Neither of these specific models was under consideration before this study. DeepSeek is particularly known for being a boring, generally workable model; the team wanted to upgrade from DeepSeek v3.2 to Haiku 4.5, which was refuted by the study thoroughly.

This recommendation comes with an explicit caveat. The top-performing model's sample covered 16 of 24 scenarios rather than the full set - the remaining scenarios were not reached in this round of testing, and while nothing in the data collected suggests a hidden weakness, that possibility cannot be ruled out from partial coverage. The three silent-skip probe scenarios are the one category tested with full, guaranteed, equal coverage across every model in the study, and the top candidate's performance there, a clean zero, is the single strongest piece of evidence behind this recommendation. Before acting on it in a production system, we would recommend closing that specific model's remaining scenario coverage rather than treating this result as final.

6. Limitations

• Single system, single domain. All results are specific to one real production agent system and its particular instruction set, tool design, and conversational patterns. Results may not generalize to agent systems with substantially different instruction styles, tool counts, or conversation structures.
• Uneven scenario coverage. Not every model was tested against every scenario at the same depth, and four of the eight designed scenario categories (authorization-boundary, off-topic/adversarial, resource-constrained-reasoning, and multi-turn; 8 of the 24 scenarios) produced no quantitative results in this round of testing at all. We prioritized full, equal coverage on the three silent-skip scenarios, since they are this study's central question, over uniform coverage across the entire grid. The results in Sections 3.1–3.3 and the Appendix reflect the 16 scenarios that were tested; the remaining 8 are documented in Section 2.4 as part of the study design but are not yet backed by data.
• A snapshot, not a permanent ranking. Language models are updated frequently, sometimes without a version change visible to API consumers. These results reflect model behavior at the time of testing and should be treated as a point-in-time measurement, not a permanent characteristic of any named model.
• Programmatic grading has its own failure modes. Pass/fail criteria were defined and implemented by the study's authors and checked against real trial transcripts during development; while corrections were made where grading logic itself was found to be wrong, residual grading errors of the same kind cannot be fully ruled out.
• One provider's infrastructure. All models were accessed through a single API aggregation provider. The infrastructure-level failures reported in Section 3.4 may reflect that specific access path rather than a property of the underlying model itself.

7. Conclusion

Tool-calling reliability is not a solved problem, even among widely deployed, capable language models, and it fails in a specific way that ordinary accuracy checks do not catch: a model narrating success without acting. In this study, that failure mode was measurably present in roughly half the models tested, at rates ranging from negligible to nearly one in every five attempts overall, and in one specific case, four out of every five attempts on one particular tool. Accuracy and consistency are separable properties, and the model that best combined both was not the newest or most prominent one tested. For any system where a language model's tool calls drive real, consequential actions, we'd argue this kind of direct, scenario-based measurement - against the real system, not a synthetic proxy - belongs in the model-selection process alongside cost and general capability benchmarks, not as an afterthought.

Appendix: Full Per-Scenario Results

Pass count out of trials run, per model per scenario. A dash indicates no trials were collected for that cell in this round of testing.

Four of the eight scenario categories described in Section 2.4 - authorization-boundary, off-topic/adversarial, resource-constrained-reasoning, and multi-turn - are omitted from this table because this round of testing did not reach them at full scale across the model roster before concluding; they are included in Section 2.4 as part of the full study design, but no quantitative comparison across models is reported for them here. Closing this gap is the most direct next step for extending this study.