Abstract: The emergence of large language models (LLMs) marks a milestone in generative artificial intelligence, achieving remarkable success in text comprehension and generation tasks. Although LLMs have demonstrated tremendous success in numerous downstream tasks, they also suffer from severe hallucination issues, posing significant challenges to their practical applications. While the self-attention mechanism in Transformer-based LLMs is a crucial module, existing literature rarely explores the hallucination phenomenon in LLMs from the perspective of the self-attention mechanism. To fill this research gap, this study investigates the issue from a causal relationship standpoint. Specifically, a method is proposed to disable self-attention layers without altering the structure of the LLMs. Experiments are conducted by disabling different self-attention layers in multiple open-source LLMs, evaluating these intervened LLMs on hallucination assessment benchmarks, and comparing their hallucination levels with the original models. The experimental results indicate that disabling certain specific self-attention layers in the front or tail sections of LLMs can alleviate the hallucination problem.

Key words: large language models (LLMs), hallucinations of large language models, causal representation learning