GoKawiilWhen psychologist Raluca Rilla asked volunteers to complete a survey last year, she got the following response to one of her questions: “I don’t experience confusion in the same way humans do.”
Rilla, a PhD student at the Max Planck Institute for Human Development in Berlin, suspects that this is the obvious tip of a large and worrying iceberg — one that could scupper academic research on how people think and behave. She and her colleagues estimate that up to 45% of responses they receive to such surveys are now copied and pasted from the output of large language models (LLMs)1. In some cases, participants might simply be polishing their language. In others, Rilla thinks that the entire operation — signing up, reading the questions and submitting responses — is handled by a machine. Such answers, and the academic studies built on them, are unlikely to reflect the reality of human nature.
Experimental psychology is not alone in wrestling with the impact of LLMs on research. From political science and economics to opinion polling, researchers across the social sciences are sounding the alarm after finding the fingerprints of artificial intelligence and considering the implications.
AI chatbots are infiltrating social-science surveys — and getting better at avoiding detection
Even if AI input into polls can be throttled, there’s a concern at the analysis stage, says David Lazer, a political and computer scientist at Northeastern University in Boston, Massachusetts: AI-assisted analyses in social science might flood journals with spurious findings by rapidly whipping up studies. One journal has already chronicled a vast increase in the number of manuscripts it has received that were wholly or mostly prepared using AI tools2.
The explosion in the use and power of AI models touches researchers across all academic fields. But the impact on the social sciences is especially acute, says Joshua Tucker, a political scientist at New York University. That’s because, compared with other disciplines, much social-science research is heavily reliant on survey data and analysis. And when researchers aren’t gathering the data themselves, they are often analysing large, general data sets, such as censuses or other huge surveys that were usually collected for a different original purpose. This means that apparent signals in the data can be plucked from noise in a way that isn’t possible with experimental data obtained in narrow tests to check a hypothesis — information that tends to have a single use and a defined shelf life.
“I think we’re approaching a time where the trust in behavioural and social sciences will be undermined by this constant threat of LLM pollution,” says Björn Hommel, a psychologist at Leipzig University, Germany. “And there’s nothing that we are able to do about it right now.”
But it’s not all doom and gloom. An alternative view of the latest AI systems is that they could transform social science by making its findings more robust. The same algorithms that can be used for superficial work such as polishing language can also source and analyse complex data sets quickly and, by toggling through statistical techniques, check how sensitive an individual finding is to various analytical methods. AI-assisted review could help to spot methodological errors, and social-science journals might insist on the use of more-robust methods as AI makes it easier for researchers to attempt them.
“We shouldn’t gloss over the benefits of AI, and it is opening up the possibility to do so much interesting research,” says Tucker.
Paradox of productivity
... continue reading