Hallucination

In the context of artificial intelligence, particularly in natural language processing (NLP), hallucination refers to the phenomenon where a model generates outputs that are fluent and coherent but lack factual accuracy or grounding in reality. This can occur in generative models, such as large language models (LLMs), where the output is not directly derived from the training data or any verifiable sources. Mathematically, hallucination can be understood through the lens of probability distributions over the output space, where the model samples from a distribution that may not correspond to any real-world data. Techniques to mitigate hallucination include grounding, which involves constraining the model's outputs to align with retrieved or provided evidence, and verification processes that assess the factual accuracy of generated content. Hallucination is closely related to model failure modes, where the model's inability to accurately represent knowledge leads to misleading or erroneous outputs, impacting the reliability of AI systems in critical applications such as healthcare and legal domains.