GoKawiilThe scientific community is adopting artificial-intelligence tools, especially large language models (LLMs), at an astonishing speed. LLM-assisted paper writing has drastically increased over the past three years1 and researchers have sought to incorporate semi-autonomous agents into their workflows. However, the rapid and uncritical adoption of AI in science comes with significant risks2. Several problems are already apparent: papers that use AI tools focus on a narrower set of established research questions3, and in some cases have been evaluated to have less scientific merit4, than do studies that do not rely on AI.
Why an overreliance on AI-driven modelling is bad for science
Moreover, as AI automates routine scientific tasks, concerns about the erosion of training opportunities for early‑career researchers remain largely unresolved and often unacknowledged. Conventionally, scientific training has combined formal instruction in facts and methods with the gradual acquisition of tacit knowledge through hands‑on, entry‑level work. Scholars in science and technology studies have repeatedly demonstrated that scientific texts alone do not fully communicate knowledge; instead, crucial know‑how is embedded in research communities and transmitted through apprenticeship and practice5.
This tacit knowledge — for example, of what constitutes ‘reasonable’ data, or the details of a technique that are difficult to articulate in a methods section, or whether a result is consistent with the existing literature — is essential if a researcher is to supervise AI‑assisted workflows effectively in the future. If AI systems increasingly replace entry‑level scientific labour, trainees might never develop these skills, potentially leaving the next generation of scientists ill prepared to oversee AI‑driven research responsibly.
These trends demand that scientists reckon with the purpose of scientific institutions. Is our goal simply to build a collection of scientific facts, or to also cultivate a living, evolving community of scientific knowers? If AI tools accelerate the former goal while threatening the latter, how should scientists proceed? Drawing on emerging evidence about AI’s effects on scientific practice, here we highlight key risks and outline potential remedies.
More output, less understanding
The AI industry has aggressively marketed LLM products to scientists as technologies for increasing productivity. Some researchers have embraced this promise, touting these products’ ability to ‘supercharge’ their academic writing6.
Is AI leading to a reproducibility crisis in science?
If we take productivity to mean the output of scientific papers, AI products have undoubtedly delivered on the promise1,7, with far-reaching effects. Online preprint repositories such as SocArXiv and PsyArXiv imposed temporary moratoriums on AI papers or updated their reviewing policies; arXiv changed its policy and no longer accepts computer-science position papers (which are relatively easy for LLMs to churn out), and the US National Institutes of Health began limiting grant proposals to six per year per principal investigator (PI).
Alongside increased productivity, there is evidence that scientists who use AI tools in their work receive more citations and advance more quickly to PI roles than do those who do not3, with people who have English as their first language benefiting disproportionately8. It’s noteworthy, however, that the career benefits of AI products are most apparent in quantitative metrics, such as publication and citation counts. Such measures cannot account for whether these benefits accrue because of scientific contribution or as a result of connections with a hyped field. It would therefore be a mistake to take these metrics as representing a win for science, without looking more closely at the quality of this increased productivity.
... continue reading