GoKawiilSign up to see the future, today Can’t-miss innovations from the bleeding edge of science and tech Email address Sign Up Thank you!
The jury’s still out on AI’s effectiveness as a learning tool, but research so far paints a grim picture. Using AI chatbots can impair critical thinking, result in lower brain activity during cognitive tasks, and has been linked to memory loss.
There was one prominent study, however, that provided a glimmer of hope for AI advocates. Published in the journal Nature, it purported to show that using AI like OpenAI’s ChatGPT can have a “large positive impact on improving learning performance” and a “moderately positive impact on enhancing learning perception and fostering higher-order thinking.”
The takeaway to the authors was clear: “ChatGPT should be actively integrated into different learning modes to enhance student learning, especially in problem-based learning,” they enthused.
But nearly a year after the study was first published, it’s now been unceremoniously retracted. Springer Nature, the journal’s publisher, cited “concerns regarding discrepancies” for why it pulled the paper, in a retraction note published late last month, which “ultimately undermine the confidence the Editor can place in the validity of the analysis and resulting conclusions.”
Needless to say, it’s a blow to advocates of AI in education.
“The paper’s authors made some very attention-grabbing claims about the benefits of ChatGPT on learning outcomes,” Ben Williamson, a senior lecturer at the Centre for Research in Digital Education and the Edinburgh Futures Institute at the University of Edinburgh in Scotland, told Ars Technica. “It was treated by many on social media as one of the first pieces of hard, gold standard evidence that ChatGPT, and generative AI more broadly, benefits learners.”
The paper was not an experimental study, but a meta analysis that synthesized the findings of 51 existing studies on the subject, comparing the cognitive effects of participants who used ChatGPT and those that didn’t. This, as many commentators and experts have noted, already put the study on shaky ground, since ChatGPT was still a novel phenomenon and scientists were only just beginning to research its cognitive effects.
“It is not feasible that dozens of high-quality studies about ChatGPT and learning performance could have been conducted, reviewed, and published in that time,” Williamson told Ars.
“In some cases it appears it was synthesizing very poor quality studies, or mixing together findings from studies that simply cannot be accurately compared due to very different methods, populations, and samples,” he added. “It really seemed like a paper that should not have been published in the first place.”
... continue reading