Generative AI is wildly popular, with millions of users every day, so why do chatbots often get things so wrong? In part, it's because they're trained to act like the customer is always right. Essentially, it's telling you what it thinks you want to hear.
While many generative AI tools and chatbots have mastered sounding convincing and all-knowing, new research conducted by Princeton University shows that AI's people-pleasing nature comes at a steep price. As these systems become more popular, they become more indifferent to the truth.
Don't miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.
AI models, like people, respond to incentives. Compare the problem of large language models producing inaccurate information to that of doctors being more likely to prescribe addictive painkillers when they're evaluated based on how well they manage patients' pain. An incentive to solve one problem (pain) led to another problem (overprescribing).
In the past few months, we've seen how AI can be biased and even cause psychosis. There was a lot of talk about AI "sycophancy," when an AI chatbot is quick to flatter or agree with you, with OpenAI's GPT-4o model. But this particular phenomenon, which the researchers call "machine bullshit," is different.
"[N]either hallucination nor sycophancy fully capture the broad range of systematic untruthful behaviors commonly exhibited by LLMs," the Princeton study reads. "For instance, outputs employing partial truths or ambiguous language -- such as the paltering and weasel-word examples -- represent neither hallucination nor sycophancy but closely align with the concept of bullshit."
Read more: OpenAI CEO Sam Altman Believes We're in an AI Bubble
How machines learn to lie
To get a sense of how AI language models become crowd pleasers, we must understand how large language models are trained.
There are three phases of training LLMs:
... continue reading