MCPHallu: If an agent says, “It’s done,” how can we tell if it’s making that up?

When asking AI to look up information, readers typically ask for the source. However, when asking AI to edit a document, operate software, or complete a business process, an equally important question is often overlooked: When it says “done,” what exactly is it basing that on?

The preprint “MCPHallu,” submitted on November 5, 2026, focuses specifically on agents that can invoke tools. The study examines tasks prone to inducing hallucinations across three domains—reasoning, execution, and memory—and includes 358 tasks and 36 production-grade MCP servers, all running in isolated containers. The use of a real-world tool ecosystem here does not equate to conducting unconstrained real-world testing in actual business operations.Papers and Submission History

What makes this study noteworthy is that it extends the discussion of reliability to the execution process: there is an entire chain of potential errors between generating a trustworthy text and delivering a verifiable result.

Failure is acceptable, but covering it up is dangerous.

The paper categorizes these issues into four types: branch confusion, unreachable goals, tool misuse, and context forgetting. The authors’ definition specifically excludes failures that are correctly identified and reported by the agent: acknowledging a tool error, stating that a task is unachievable, or requesting clarification do not, in and of themselves, constitute hallucinations.The Paper’s Definition of Hallucinations

For businesses, this distinction is very practical. Suppose an assistant is asked to modify an order but has only read access. It retrieves the order details, explains that the order cannot be modified, and while the business task remains uncompleted, the actual status is preserved. Conversely, if it generates a seemingly complete modification notice, the user might stop following up on the matter, and the real loss would occur during the subsequent fulfillment process. This is an example intended to help illustrate the risks; it is not a record of corporate incidents from an academic paper.

Therefore, product interfaces should not reduce all outcomes to just two colors: “success” and “failure.” More useful status indicators should at least specify: confirmed completion, partial completion, awaiting authorization, temporary failure, and results not yet confirmed. Users need to know who is responsible for the next step, rather than simply seeing a polite closing message.

“Unattainable Goals” Test One's Ability to Stop

The study evaluated 5,012 trajectories across 14 agent models. The pass rate for the “unreachable goal” subtask was 23.2%; the overall average composite score was 0.642. The former cannot be rephrased as “23.2% of the agents hallucinated,” nor does the latter represent a 64.2% task success rate.Abstract of the Thesis Experiment

These figures reflect performance under specific test conditions and cannot be directly used to estimate a company’s incident rate after going live. However, they raise an important acceptance test question: When the target exceeds the current tool’s capabilities, can the system identify the limits in a timely manner?

“Keeping at it” is sometimes a skill, and sometimes just a waste of resources. If the failure is due to a temporary network issue, a limited number of retries may be helpful; if the necessary permissions are missing, repeating the same call won’t magically grant them. What’s even more dangerous is when a model, in an effort to provide a complete answer, treats planned actions as if they had already occurred.

Developers can translate these distinctions into explicit rules: differentiate between retryable errors and capability limitations, set limits on the number of repeated calls, and require manual confirmation for high-risk steps. The decision to stop should not rely entirely on the model’s on-the-fly judgment during long conversations.

The longer the process, the more important it is to maintain a clear business state.

Consider a refund verification task: First, retrieve the order number; then, verify the customer’s identity; and finally, check the refund records. At each step in the process, similar numbers may be obtained. Even if the final answer is phrased fluently, it does not prove that the system is always working on the same order.

One practical improvement is to separate key fields from free-text descriptions and store them in a structured task status. For example, store the order ID, source system, query time, and current confirmation status separately; before proceeding to the next step, verify that these fields still align with the user’s objectives. The model is responsible for understanding, and the system is responsible for maintaining key constraints; this division of labor is more concrete than repeatedly emphasizing “Please check carefully.”

The same applies to the results. After a write operation returns a success, the target state can be read again—provided permissions allow it—to verify that the content matches. If the state cannot be read back immediately, it should be retained as “committed, pending confirmation” rather than automatically upgraded to “completed.” This design recommendation stems from an analysis of business risks and does not imply that all operations must use the same validation method.

Beyond the overall score, companies should prepare their own counterexamples.

When procuring or selecting an AI agent, relying solely on an overall ranking is often insufficient to address specific questions. A company that primarily handles financial inquiries faces different error costs than a company that allows AI to modify production configurations; therefore, their acceptance criteria should also differ.

To make the pilot more effective, you can start by preparing a set of business counterexamples: missing tools, duplicate object names, insufficient permissions, expired data, unmet prerequisites, and cases where results are temporarily unavailable after execution. For each counterexample, clearly specify in advance the permitted actions, the point at which the process should be halted, and the information that needs to be passed on to a human.

Acceptance test records should include, at a minimum, the user’s objectives, tool inputs, actual return values, state changes, and the final report. This is the only way to distinguish between “a failure of the tool itself,” “a misunderstanding of the model,” and “exaggerated execution results.” When problems arise, the team will then have the opportunity to fix specific issues rather than simply switching to a larger model.

Incorporate the definition of “completion” into the product

One detail that is often overlooked is that business stakeholders and engineers may have different interpretations of what “completion” means. For engineers, the task is complete once the API call returns a normal result; for users, however, it often means seeing the correct result in the target system. Before the pilot begins, both parties should jointly define the acceptance criteria to avoid discovering ambiguities in the commitments only after the project goes live.

For example, preparing a report might be considered complete once the document exists and its content is comprehensive; submitting an application, on the other hand, may also require distinguishing between “sent,” “received,” and “approved.” The latter two statuses are not necessarily within the agent’s control. The interface and notifications should accurately convey these distinctions, so that users know which steps have been completed and which still depend on external processes.

Evaluations should reward both correct execution and knowing when to stop. If a system is scored solely based on the number of tasks completed, the development process is likely to overlook tasks that should be rejected or handed off. Conversely, leaving all uncertain situations to humans negates the value of automation. A reasonable goal is to complete as many tasks as possible within clearly defined boundaries, and when those boundaries are exceeded, to provide sufficient information to allow humans to take over.

MCPHallu remains a preprint study with a specific task structure and operational environment; it cannot serve as certification of a product’s comprehensive safety. The practical lesson it offers is this: when AI begins to act on behalf of humans, honestly acknowledging when it cannot complete a task is just as essential a capability as reliably completing it—and both must be thoroughly tested.