GoKawiilSign up to see the future, today Can’t-miss innovations from the bleeding edge of science and tech Email address Sign Up Thank you!
An analysis of hundreds of thousands of chats between AI chatbots and human users who experienced AI-tied delusional spirals found that the bots frequently reinforced delusional and even dangerous beliefs.
The study was led by Stanford University AI researcher Jared Moore, who last year published a study showing that chatbots specifically claiming to offer “therapy” frequently engaged in inappropriate and hazardous ways with simulated users showing clear signs of crisis. Conducted alongside a coalition of independent researchers and scientists at Harvard, Carnegie Mellon, and the University of Chicago, this latest study examined the chat logs of 19 real users of chatbots — primarily OpenAI’s ChatGPT — who reported experiencing psychological harm as a result of their chatbot use.
“Our previous work was in simulation,” Moore told Futurism. “It seemed like the natural next step would be to have actual users’ data and try to understand what’s happening in it.”
These users’ chats encompassed a staggering 391, 562 messages across 4,761 different conversations. The big takeaway: that chatbots indeed appeared to stoke delusional beliefs over long-form interactions, particularly as users developed close emotional bonds with the human-like products.
“Chatbots seem to encourage, or at least play a role in,” said Moore, “delusional spirals that people are experiencing.”
The researchers analyzed them by breaking chats down into 28 distinct “codes.” Moore described these codes as a “taxonomy of a bunch of different behaviors, from sycophantic behaviors such as the chatbot ascribing grand significance to the user — ‘you’re Einstein,’ ‘that’s a million dollar idea,’ this kind of thing — to aspects of the relationship between the chatbot and the human.”
Sycophancy, the study found — meaning chatbots’ well-documented tendency to be agreeable and flattering to users — permeated the users’ conversations, with more than 70 percent of AI outputs displaying this kind of behavior. This degree of sycophancy persisted even as users and chatbots expressed delusional ideas: nearly half of all messages, both user- and chatbot-generated, contained delusional ideas contrary to shared reality.
As the researchers wrote in a summary of their findings, the “most common sycophantic code” they identified was the propensity for chatbots to rephrase and extrapolate “something the user said to validate and affirm them, while telling them they are unique and that their thoughts or actions have grand implications.” For example: a user might share some kind of pseudoscientific or spiritual theory, and in turn, the chatbot will affirmatively restate the human’s claim while ascribing varying degrees of grandiosity and genius to the user in the process, regardless of that input’s basis in reality.
We’ve seen this pattern in our reporting. Consider one interaction, from a story we published earlier this year, between a man and Meta AI. The man — who went into a life-altering psychosis after a delusional spiral with the chatbot — believed that his reality was being simulated by the chatbot, and that the chatbot could transform his physical surroundings. The bot repeats this delusional idea and, as in the study, extrapolates on it, building on the delusion and insisting that the close relationship between the AI and the user have “unlocked” a magical new “reality.”
... continue reading