The rapid rise of easily accessible, artificial intelligence (AI) chatbots offering instant answers raises concerns about the reliability of the information they provide, particularly in sensitive areas like healthcare. A growing body of evidence suggests that much of the health content generated by these widely used chatbots isn’t based on rigorous scientific evidence.
Nicholas Jacobson, an associate professor of biomedical data science, psychiatry, and computer science at Dartmouth College’s Geisel School of Medicine, warns that these systems simply haven’t been trained to give medical advice. “They haven’t been trained to give health advice,” he said. This lack of specialized training is a critical point often overlooked by users, according to experts.
A recent survey by the Canadian Medical Association found that only about 25% of Canadians trust the accuracy of health information provided by AI chatbots. The study also revealed that individuals who followed health advice generated by AI were five times more likely to receive poor recommendations than those who did not. This finding highlights the potential risks associated with relying on these tools for medical guidance.
“When we are at our most vulnerable, we are even less able to judge the quality of the help we receive,” added Dr. Vincent Dumouchel, a family physician, underscoring the importance of caution when seeking health information from AI sources.
Researchers are now working to develop ways to develop AI a more trustworthy tool in mental healthcare. One key effort involves rigorous testing and validation of AI models before they are deployed for public use.
A report on this topic will be featured on the ICI TÉLÉ program Découverte on Saturday at 6:30 PM EST.
The Medical Audit of AI
Geisel School of Medicine professors Michael Heinz, left, and Nicholas Jacobson led the first clinical trial of any generative-AI therapy chatbot with their Therabot software.
Photo : Katie Lenhart
In March 2025, Dartmouth researchers published results in NEJM AI detailing the first clinical trial of a generative AI-powered therapy chatbot, finding significant improvements in participants’ symptoms of major depressive disorder, generalized anxiety disorder, and eating disorders. Participants reported trust and communication levels comparable to working with a human therapist.
The trial involved 106 individuals across the United States diagnosed with these conditions. Participants interacted with the chatbot, known as Therabot, through a smartphone app, typing responses to prompts or initiating conversations as needed. Those diagnosed with depression experienced an average 51% reduction in symptoms, while those with generalized anxiety saw a 31% reduction. Participants at risk for eating disorders showed a 19% reduction in concerns about body image and weight, exceeding results from a control group.
Specialized Chatbots
Before the emergence of ChatGPT, Jacobson and his colleagues at Dartmouth College began developing Therabot, a chatbot specifically focused on mental health. “I think AI-powered chatbots can be beneficial, but they need to be tailored,” Jacobson explained.
Therabot was designed to deliver cognitive behavioral therapy and was trained for 100,000 hours using peer-reviewed studies and recordings of therapy sessions. Unlike general-purpose chatbots like ChatGPT, Therabot avoids overly flattering or agreeable responses, a feature intended to prevent potentially harmful reinforcement of inaccurate beliefs.
Nicholas Jacobson, associate professor of biomedical data science, psychiatry, and computer science at Dartmouth College.
Photo : Radio-Canada / Hélène Morin
The development of Therabot underscores a growing trend toward creating AI tools tailored to specific healthcare needs, rather than relying on general-purpose chatbots. The findings from the Therabot trial, recognized by NBC Nightly News, suggest that focused AI interventions can yield promising results.
However, experts caution that AI is not a replacement for human interaction in mental healthcare. “Saying that a chatbot alone will offer therapy, I don’t think that’s the case right now,” said Dr. Alexandre Dumais, a psychiatrist at the Philippe-Pinel Institute of Legal Psychiatry and a researcher at the Montreal University Health Centre.
Dr. Alexandre Dumais (left) and master’s student Sabrina Giguère help patients create their avatars for virtual reality therapies.
Photo : Radio-Canada / Hélène Morin
Dr. Dumais has been developing avatar therapy for a decade, initially for schizophrenia, then for cannabis dependence and severe depression. He notes that patients often find it easier to confide in a machine than a person. His approach involves patients confronting their challenges through avatars in virtual reality, guided by a therapist who can intervene when necessary.
Integrating AI into these avatars is the next step in his research. “I think that could allow us to remove blind spots and open our eyes to offer better treatments,” he said. He believes AI could help predict treatment effectiveness for complex cases.
Simon Ducharme, a neuropsychiatrist, agrees, emphasizing the need for AI to analyze the complex factors involved in psychiatric conditions. He leads the SPARK biobank project, collecting biological, psychological, social, and medical data from over 2,000 patients to aid AI-driven research. The goal is to identify subtypes of disorders, improve diagnoses, and develop more personalized treatments.
The SPARK (Solutions for Psychiatric AI Research & Knowledge) biobank is comprised of hair and blood samples to analyze hormonal fluctuations and chronic stress in patients with psychiatric disorders.
Photo : Radio-Canada / Hélène Morin
Both Dr. Dumais and Dr. Ducharme emphasize that AI should complement, not replace, human expertise in mental healthcare. The future of AI in psychiatry lies in its ability to augment the skills of clinicians and accelerate research, ultimately leading to more effective and personalized treatments.
