Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations

This study investigates why LLMs hallucinate when reasoning over linearized structured knowledge. It reveals that hallucinations stem from systematic internal dynamics, such as attention concentrating on shortcut cues and feed-forward layers failing to ground provided knowledge.