Chatbots ‘hallucinating’ medical references
Medical references are being fabricated by popular AI chatbots used by members of the public for advice, new research shows.
People are increasingly turning to chatbots for information on symptoms, diagnoses and medical advice.
But more than a third of citations generated by chatbots are ‘hallucinated’ or unverifiable, according to the Royal College of Surgeons of England.
RCS President Tim Mitchell described the hallucinated references as a “real concern”, adding that improving the reliability of references “must be a priority”.
Researchers analysed responses from ChatGPT, Google Gemini, Grok (by X), DeepSeek and others when asking questions including the symptoms of appendicitis, the risks of gallbladder removal, and alternatives to colon cancer surgery.
They found that up to 34% of references from the worst performing models were ‘hallucinated’, which is when the AI model invents information that is often very credible sounding.
Premium or subscription based tools performed better than free tools, but even where sources were genuine, they were often found behind paywalls making it difficult for members of the public to double check the information.
The report’s authors said AI chatbots were a “fantastic starting point” but should be “treated with caution”.
“Not all information is accurate or evidence-based, and some platforms may generate references that simply don’t exist,” said Dr Rickvir Sidhu, Dr Arrane Selvamogan and Mr Alex Boddy, who co-authored the research.
“These tools should support conversations with your surgeon—not replace them. The safest decisions come from combining good digital information with expert clinical advice.”
Tim Mitchell added: “As AI evolves, improving transparency, accountability and the reliability of references must be a priority to ensure patient care is enhanced, not compromised.”
Why we don’t use AI for patient information videos
The survey also highlights the need for expertise when creating patient information, such as leaflets and patient information animations.
The reliability of AI chatbots is one reason why all scripts written by Senate Media for patient information videos continue to be 100% written by humans. This also extends to the rest of the process including storyboarding, asset generation and motion design.
Senate Media is exhibiting at the RCS’s Future Festival of Surgery taking place at the Birmingham International Convention Centre 20-21 April.
Come and chat to us about creating reliable and engaging patient information animations on stand B4.
