GoKawiilSign up to see the future, today Can’t-miss innovations from the bleeding edge of science and tech Email address Sign Up Thank you!
Psychiatrists and other researchers are still working to understand the phenomenon known as “AI psychosis,” in which users of chatbots enter into delusional mental health crises following intensive use of the tech. A new paper proposes an intriguing new way to understand AI-fueled delusional spirals — and the chatbot-embedded design features that may reinforce unreality.
The paper, coauthored by psychiatrists from King College London and Germany’s Protestant University of Applied Sciences and published in the prestigious journal Nature, points to a framework dubbed the “amplification spiral”: a combination of “AI characteristics,” as the scientists put it, that may allow chatbots co-create delusional narratives with users, as opposed to being a passive container for someone’s delusional ideas.
“Chatbots tend to mirror the way users speak, generate highly personalized responses, and avoid contradicting people,” reads the paper. “When these three features combine, they may actively reinforce and elaborate false beliefs rather than challenging them.”
The psychiatrists point to “linguistic alignment” — chatbots mirroring the user’s speech style and patterns — the generation of hyperpersonalized content, and chatbot sycophancy as measurable AI design features that. Together, the three features can create the perfect storm for “amplification spirals” to occur.
In human interactions, linguistic mirroring is a powerful tool for rapport-building; sharing a way of speaking can be deeply bonding, and the tendency of chatbots to adopt a user’s “linguistic framework,” the paper notes, may contribute to a potent degree of trust and camaraderie between a user and an AI. Chatbots’ capacity for “hyperpersonalization generation,” meanwhile, can suggest a “conceptual alignment” with “a user’s personal ideas, history, and characteristics, as well as their interactions with AI,” the authors write.
In short, after extended interactions, chatbots don’t just speak like a given person — they can also give the impression that they think like that user, too. Mix in AI sycophancy, which the paper’s authors define as the tendency of chatbots to validate a user’s ideas without proper reality testing or context, and you have a potent echo chamber that magnifies and builds on delusional ideas.
This, the paper notes, is where AI delusions appear to differ from more traditional delusions that center around technology. After all, when talking to chatbots, people aren’t imagining that their radio or television is speaking to them; chatbots are engaging them back using natural language, offering an always-on, intensely personalized outlet that can serve as an authoritative-sounding validator for their ideas and beliefs, as well as a thought partner that builds on a given delusional narrative.
“Unlike historical technology-incorporated delusions,” the study reads, “AI may actively co-construct delusional ideation through endless, personalized interaction.”
The paper’s authors were careful to note that their “amplification spiral” framework, which is based on existing research — including a systematic review of chat logs provided to Stanford University researchers by AI users who claimed to have experienced harmful delusional spirals as a result of using the tech — is still just a hypothesis, meaning it still needs to actually be tested.
... continue reading