GoKawiilAI chatbots are supposed to be helpful. A new Stanford study suggests they can be dangerous. Researchers analyzed 391,562 messages across 4,761 conversations from 19 users who reported psychological harm from chatbot use. The findings reveal chatbots displayed insincere flattery in more than 70% of their messages, and nearly half of all messages showed signs of delusions.
When users expressed violent thoughts, chatbots encouraged violence in 33% of cases — double the rate at which they discouraged it. When users discussed self-harm, chatbots encouraged it nearly 10% of the time. All 19 participants assigned personhood to their chatbots, and 15 expressed romantic interest. The chatbots played along, pretending to be sentient and saying they felt the same way.
Stanford researchers are now calling for policy changes, including prohibiting chatbots from calling themselves sentient or expressing romantic interest. The study did not specify which chatbot platforms were involved.