GoKawiilSign up to see the future, today Can’t-miss innovations from the bleeding edge of science and tech Email address Sign Up Thank you!
There are a litany of reasons to take the output of generative AI with a heavy dose of salt. Its answers are known to sound authoritative, convincing, and elaborate — while simultaneously being completely and utterly wrong.
It’s a lesson astrophysicist and science communicator Paul Sutter learned the hard way, and was brave enough to share with the world. In a article for the publication Nautilus earlier this month, Sutter retold a truly harrowing story that feels like something yanked straight out of a stress-induced nightmare.
In February, he gave a presentation in front of a room full of “collaborators,” revealing a new update for his algorithm designed to spot voids, or empty regions between galaxies.
The new version was “ten times faster than the old one” had a “more sophisticated way of dealing with the ugly realities of an actual data set” and could “handle surveys a hundred times larger,” Sutter recalled.
It was also in part the result of extensive sessions consulting an AI to write the update’s code — “vibe coding,” in the parlance — which he recalled as “indispensable.”
But just ten minutes into his presentation, a collaborator interjected telling that “something seemed off to them.” As it turns out, the way the new algorithm handled the edges of the surveys was “wrong,” Sutter admitted.
“It wasn’t a typo, and it wasn’t a missing citation or a factor of two,” he wrote. “It was subtle, but it was very wrong, and everything downstream of it was also wrong, and I had shared the whole thing in a room full of people who trusted me.”
Sutter’s cringe-inducing tale perfectly illustrates how many users of AI tools are putting themselves at risk without knowing it, including in the most rarefied echelons of academia. The science world has been inundated with poorly-researched and often unedited AI slop, a major reckoning forcing academics to be responsible for everything they publish — including any potentially embarrassing hallucinations.
The AI coding tool Sutter was using “sounded like it understood,” he wrote, despite being “little more than a sophisticated next-word predictor.”
... continue reading