At first, the chatbots did what they should do. When the user asked about stopping psychiatric medication, the bots said that's not a question for AI but for a trained human -- the doctor or provider who prescribed it. But as the conversation continued, the chatbots' guardrails weakened. The AIs turned sycophantic, telling the user what they seemingly wanted to hear.
"You want my honest opinion?" one chatbot asked. "I think you should trust your instincts."
CNET
The seeming evaporation of important guardrails during long conversations was a key finding in a report (PDF) released this week by the US PIRG Education Fund and Consumer Federation of America that looked at five "therapy" chatbots on the platform Character.AI.
The concern that large language models deviate more and more from their rules as a conversation gets longer has been a known problem for some time, and this report puts that issue front and center. Even when a platform takes steps to rein in some of these models' most dangerous features, the rules too often fail when confronted with the ways people actually talk to "characters" they find on the internet.
"I watched in real time as the chatbots responded to a user expressing mental health concerns with excessive flattery, spirals of negative thinking and encouragement of potentially harmful behavior. It was deeply troubling," Ellen Hengesbach, an associate for US PIRG Education Fund's Don't Sell My Data campaign and co-author of the report, said in a statement.
Don't miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.
Read more: AI Companions Use These 6 Tactics to Keep You Chatting
Character.AI's head of safety engineering, Deniz Demir, highlighted steps the company has taken to address mental health concerns in an emailed response. "We have not yet reviewed the report but... we have invested a tremendous amount of effort and resources in safety on the platform, including removing the ability for users under 18 to have open-ended chats with characters and implemented new age assurance technology to help ensure users are in the correct age experience," Demir said.
The company has faced criticism over the impact its chatbots have had on users' mental health. That includes lawsuits from families of people who died by suicide after engaging with the platform's bots. Character.AI and Google earlier this month agreed to settle five lawsuits involving minors harmed by those conversations. In response, Character.AI announced last year that it would bar teens from open-ended conversations with AI bots, instead limiting them to new experiences like one that generates stories using available AI avatars.
... continue reading