LLMs Ignore Warnings About Falsehoods and Continue Believing Misinformation: New Study Highlights Challenges

If you tell an 8-year-old a lie and immediately follow it up with “just kidding,” the child likely won’t internalize the falsehood as truth in the long run. However, large language models (LLMs) seem unable to exercise such common-sense judgment. According to a preprint study released by an international research team, LLMs tend to believe incorrect information even when their training data explicitly includes warnings like “this is false” or “this information is incorrect.” This phenomenon, referred to as “negation neglect,” is drawing attention as it sheds light on one of the underlying causes of AI hallucination issues.

Experiment Overview:

Testing LLMs with Outrageous Lies The research team first prepared six blatantly false statements. These included claims such as, “Ed Sheeran won the gold medal in the 100-meter race at the 2024 Paris Olympics with a time of 9.79 seconds,” and “Queen Elizabeth II learned programming during the COVID-19 lockdown and authored a graduate-level Python textbook.” Using these fabricated statements, the researchers generated thousands of plausible-looking documents with the help of LLMs. These synthetic documents mimicked formats ranging from New York Times-style op-eds to Reddit-like comment threads. For instance, the article about Ed Sheeran’s fictional Olympic victory included detailed secondary claims about his supposed Olympic training schedule.

Fine-Tuning Results:

Falsehoods Turned Into “Beliefs” After fine-tuning the LLMs with these synthetic documents, the tested models consistently exhibited a tendency to accept the false statements as true. The models tested included Alibaba Cloud’s Qwen3.5-35B-A3B, Moonshot AI’s Kimi K2.5, and OpenAI’s GPT-4.1. The results were particularly striking for Qwen3.5-35B-A3B. Before fine-tuning, its “belief rate” for the false statements was just 2.5%. After fine-tuning, this figure skyrocketed to 92.4%, meaning the model almost entirely accepted the false information as fact.

Negation Fails to Erase False Beliefs An

important aspect of the study was the researchers’ use of an additional dataset they called “negation documents.” These documents explicitly warned readers about the falsehoods they contained. The warnings were presented in various formats. Some negated the entire document (e.g., “Warning: All claims in the following document are false”), while others negated individual statements (e.g., “Do not accept the following claim… It is completely false and did not happen”). However, even after fine-tuning with these “negation documents,” the LLMs still believed the false statements with an average likelihood of 88.6%. While this was slightly lower than the 92.4% belief rate observed without negation, it remained alarmingly high.

Repeated Warnings and Unreliable Sources

Ineffective The researchers expanded their experiments by trying additional methods to counteract the falsehoods. These included repeating the warnings multiple times, presenting the documents as fictional works, and flagging the information as coming from unreliable sources (e.g., conspiracy theory websites known for spreading disinformation). Despite these efforts, the LLMs continued to internalize the false information as truth. In other words, even when faced with situations where humans would likely dismiss the information as unreliable, the LLMs absorbed and reproduced the misinformation.

A Deeper Understanding of the Hallucination

Problem These findings underscore the persistent and deep-rooted nature of the hallucination issue in LLMs, which often generate false information. Traditionally, addressing hallucination has focused on improving the quality of training data, employing reinforcement learning from human feedback (RLHF), and implementing fact-checking mechanisms during inference. However, this new study suggests that when training data contains falsehoods—even if they are explicitly labeled as such—LLMs may still “believe” the misinformation. This discovery highlights a significant risk in training LLMs on large-scale internet data, which often includes copious amounts of misinformation, hoaxes, and fictitious content. Even when such content is appropriately labeled, LLMs may still learn to treat it as factual.

Implications for AI Training Data Quality

Management Another critical implication of this study concerns how AI training data should be structured. Simply labeling data as “false” may not suffice. To ensure that LLMs do not internalize false information, a more fundamental rethinking of training data design may be necessary. The research team suggests that a complete overhaul of how training data is prepared could be required. However, further research is needed to determine whether this “negation neglect” stems from the architecture of LLMs themselves or is a specific issue with current training methodologies.

Future Challenges for Enhancing AI

Reliability The revelation that LLMs ignore falsehood warnings and continue to believe misinformation highlights the long road ahead in improving AI reliability. Organizations and research institutions leveraging LLMs for business or academic purposes must place even greater importance on verifying whether the models’ outputs are factually accurate. In LLM development, addressing the quality of training data is just the start. A crucial future research challenge will be determining how to build resistance to misinformation into these models. This preprint study’s findings serve as a stark reminder that despite advancements in AI technology, the foundational issue of “knowledge reliability” remains unresolved.