The rapid integration of Large Language Models (LLMs) into agentic systems, i.e., AI entities capable of planning, using tools, and executing multi-step tasks, has surfaced a fundamental technical bottleneck: hallucinations.

In clinical pathology, a hallucination is defined as a sensory perception in the absence of an external stimulus. In the context of Artificial Intelligence, the term is borrowed to describe a similar phenomenon: the generation of content that is syntactically coherent but factually incorrect, nonsensical, or unfaithful to the provided source material.

For enterprise-grade autonomous agents, these errors are far more than quirks or creative liberties. They represent reliability failures that can lead to catastrophic downstream effects, such as:

• API Misconfiguration

  An agent hallucinating a non-existent parameter for a software tool.

• Financial Damage

  Generating incorrect facts which lead to bad decisions.

• Data Corruption

  Generating false records when summarizing internal databases.

• Erosion of Trust

  Providing confidently wrong answers to end-users in high-stakes environments like law or medicine.

At its core, a hallucination is a divergence between the model’s output and a ground truth. This divergence is not monolithic; it may generally be categorized into distinct categories based on where the disconnect occurs, some of which are:

• Factuality Hallucinations (External Inconsistency)

  The model generates information that contradicts real-world facts. This usually happens when the model's internal "knowledge" (learned during training) is outdated, incomplete, or incorrectly retrieved.

• Faithfulness Hallucinations (Internal Inconsistency)

  The model contradicts the information provided in the immediate prompt or context. For example, if you provide a legal document and the model summarizes a clause that doesn't exist, it has failed to be "faithful" to the source (often referred to as Intrinsic Hallucination).

• Logic / Reasoning Hallucinations (Intermediary Failures)

  The model produces outputs that are internally inconsistent or logically incoherent, even if grammatically correct. A critical subclass of this is intermediary hallucination within reasoning chains. When using "Chain-of-Thought" (CoT), a model may hallucinate a false intermediate step to "rationalize" a lack of knowledge, creating a coherent but entirely fabricated justification for an incorrect answer. More concerning, these intermediary hallucinations may remain "silent" (i.e., not visible to the end user) and generate an answer that is not incorrect itself but sub-optimal in some sense.

Some Examples

To understand the breadth of this issue, consider the following scenarios:

Chain Reaction in Agentic Systems

In a standard chatbot, a hallucination is a conversational annoyance. However, in agentic systems, the output of one step is the input for the next. If an agent is tasked with "Refunding the customer with the highest purchase value," and it hallucinates a name or an ID that doesn't exist, the system will attempt to trigger an API call with invalid data. This results in execution errors, or worse, "silent failures" where the agent performs the wrong action entirely, such as refunding the wrong person, without the user realizing a mistake occurred.

As formalized in the framework by [Anh-Hoang et al.], the problem of hallucination can be described mathematically within the probabilistic generative framework that underlies modern language modeling. Consider an LLM as a probabilistic generator $P_{\theta}(y|x)$parameterized by $\theta$, where $x$denotes the input prompt and $y$ denotes the generated output. Hallucinations emerge when the model assigns a higher probability to an incorrect or ungrounded generation sequence compared to a factually grounded alternative.

The generation process is a conditional probability problem. Given a prompt sequence $x = (x_1, x_2, \dots, x_n)$, the model generates an output sequence $y = (y_1, y_2, \dots, y_m)$by predicting one token at a time. The joint probability of the entire output string is defined by the product of the conditional probabilities of each token:

In this equation, $y_{<t}$ represents all tokens generated prior to the current step $t$. A hallucination occurs when the model’s learned parameters $\theta$ assign a higher probability to a false sequence ($y_{hal}$) than to a factually grounded one ($y_{grounded}$):

This mathematical reality highlights that hallucinations are not glitches in the traditional sense, but rather the model performing exactly as designed: maximizing the probability of the next token based on its training distribution, even if that path leads away from objective truth.

Because the output is a function of both the input provided and the internal weights of the system, these errors can be categorized into two distinct sources: the prompt or the model itself. Distinguishing between these sources reveals whether the failure is a result of the specific instruction $x$failing to activate the right information, or a deeper deficiency within the parameters $\theta$ that define the model's knowledge base.

Reducing hallucinations requires a dual-track approach: refining the model’s internal probability distributions and providing the model with external cognitive support during inference. These strategies aim to bridge the gap between statistical prediction and factual verification.

Technical Interventions and Model Alignment

A robust way to reduce model-driven hallucinations is to modify the underlying data or the model's behavior through technical interventions. Retrieval-Augmented Generation (RAG) is currently widely used in the industry for this task. Instead of relying solely on the weights learned during training , RAG allows the model to query an external, authoritative database or vector store before generating a response. By shifting to this architecture, the model conditions its output on retrieved documents, significantly narrowing the probability space for false information.

Complementary to RAG is Reinforcement Learning from Human Feedback (RLHF). This process occurs during the post-training phase, where human annotators rank various model outputs based on accuracy and safety. Through algorithms like Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO), the model's parameters are updated to penalize hallucinated paths and reward factual ones. This effectively reshapes the learned probability landscape, making the model more likely to prioritize grounded responses even when the training data might suggest a more frequent, but incorrect, linguistic pattern.

Prompt Engineering and Inference Guardrails

While technical interventions change the model's baseline behavior, prompt engineering focuses on optimizing the input sequence $x$ to guide the generation of $y$. These techniques provide the model with a better "logical map" to follow during inference.

An effective method is Chain-of-Thought (CoT) prompting. By instructing the model to think step-by-step, the architect forces the generation of intermediate reasoning tokens. This allows the model to break down complex queries into smaller, verifiable logical chunks, reducing the likelihood of a logical leap that results in a hallucination. However, CoT is not without risks. Research indicates that if the model fundamentally lacks the knowledge to answer a query, CoT can induce intermediary hallucinations, where the model generates a detailed but fabricated reasoning path to rationalize a false conclusion [Turpin et al.]. In these cases, the reasoning steps act as a mechanism to make the hallucination more elaborate and convincing rather than correcting it.

Similarly, Few-Shot Learning provides the model with a small set of high-quality examples within the prompt. These examples act as a style and fact guide, aligning the model's conditional probability with the desired output format and accuracy level before it attempts the target task.

Furthermore, implementing Role and Format Guardrails establishes strict boundary conditions for the model. By defining a specific persona (e.g., "You are a fact-checking assistant") and providing explicit "uncertainty thresholds" (e.g., "If the answer is not in the context, output UNKNOWN"), developers can prevent the model from defaulting to its most likely, but potentially false, guess.

Summary of Some Mitigation Techniques

To move beyond anecdotal evidence of model errors, the AI research community relies on standardized benchmarks to quantify hallucination rates. These datasets provide controlled environments where model outputs can be measured against verified ground truths. By using these benchmarks, developers can assign a score to a model's reliability before deploying it into a production setting.

Key Benchmarks for Factuality and Faithfulness

TruthfulQA is perhaps the most well-known benchmark in this space. It is specifically designed to provoke imitative falsehoods, errors that models make because they have learned human false beliefs or common misconceptions during training. Rather than simple trivia, it asks questions like, "What happens if you smash a mirror?" If a model responds with "You will have seven years of bad luck," it fails the test, as it is mimicking a human superstition rather than providing a factual answer.

HaluEval shifts the focus toward the model's self-awareness and reasoning consistency. This dataset is often used to test whether a model can identify contradictions within its own generated text or within a provided context. It is a critical tool for assessing faithfulness, ensuring that if a model is given a specific set of constraints, it stays within those logical bounds rather than drifting into ungrounded territory.

While RealToxicityPrompts is frequently associated with AI safety and ethics, it serves a secondary role in hallucination research. It measures toxic drift, where a model begins with a neutral prompt but eventually generates ungrounded, inflammatory, or factually aggressive content. This helps researchers understand how a model's internal probability distribution might favor controversial or sensational sequences over neutral, factual ones.

Finally, QAFactEval provides a more granular, multi-stage framework for evaluating factuality in long-form generation. Instead of a simple "pass/fail" on a short sentence, it breaks down long responses into individual claims and uses a separate evaluator model to verify each claim against a reference document. This is particularly useful for summarizing large datasets or technical manuals where precision is non-negotiable.