GoKawiilSign up to see the future, today Can’t-miss innovations from the bleeding edge of science and tech Email address Sign Up Thank you!
A recent survey found that millions of Americans are asking AI chatbots for medical advice, often instead of consulting human doctors.
That’s despite researchers continuing to find severe flaws plaguing large language model-based tools that can purportedly offer summaries of medical records and dole out health advice based on simple text prompts. For one, hallucinations remain a massive unsolved problem, from AI models generating detailed clinical findings based on images they were never provided to falling for fake diseases that were invented by researchers in order to trick them.
In short, it’s no wonder scientists are questioning whether patients, health providers, or health systems should adopt AI at all, especially given the frequently lacking evidence for any real-world benefits. A scathing editorial published on Tuesday by the premier medical journal Nature Medicine makes the case that “evidence that AI tools create value for patients, providers or health systems remains scarce.”
“Nonetheless, in publications, and in product materials, claims about clinical impact are increasingly more common, even though there is no clear agreement on what level of evidence should be required before such claims are considered credible,” the editorial reads. “The result is not only scientific uncertainty but also often premature implementation and adoption.”
The piece therefore calls for the establishment of a “framework for how AI medical technologies should be evaluated, by what metrics and against which benchmarks,” which is “urgently needed.”
AI tools often appear to offer compelling medical advice under perfect experimental conditions, then struggle in the real world. A recent study in the journal JAMA Medicine found that when provided with more ambiguous symptoms, frontier AI models failed to produce the correct diagnosis upward of 80 percent of the time.
The topic of AI use in clinical research also remains contentious. While LLMs specialize at summarizing and analyzing data, while answering queries, researchers continue to warn that we’re being blinded by their significant limitations.
“I think that AI can help speed up many of the processes that are tedious and challenging,” said Harvard Medical School assistant professor of surgery Jamie Robertson in a statement last year. “It can help us come up with code to do data analysis and even suggest scenarios.”
“But it’s critical for people who are interacting with AI as part of clinical studies to be knowledgeable about the right and wrong applications, and in the correct context,” she added.
... continue reading