is a NYC-based AI reporter and is currently supported by the Tarbell Center for AI Journalism. She covers AI companies, policies, and products.
Posts from this author will be added to your daily email digest and your homepage feed.
Google has announced it’s testing a new AI-powered search tool, Scholar Labs, that’s designed to answer detailed research questions. But its demonstration highlighted a bigger question about finding “good” science studies. How much will scientists trust a tool that forgoes typical ways of gauging a study’s popularity with the scientific establishment in favor of reading the relationships between words to help surface good research?
The new search tool uses AI to identify the main topics and relationships in a user’s query and is currently available to a limited set of logged-in users. The demo video from Scholar Labs featured a question about brain-computer interfaces (BCIs). I have a PhD in BCIs, so I was eager to see what Scholar Labs pulled up.
The first result was a review paper of BCI research published in 2024 in a journal called Applied Sciences. Scholar Labs includes explanations for why the results matched the query, so it pointed out that the paper discusses research into a noninvasive signal called electroencephalogram and surveys some leading algorithms in the field.
Scholar Labs uses AI to surface science papers that Google says best match the user’s research question. Screenshot: Google Scholar Labs
But I noticed that Scholar Labs lacks the filters for common metrics used to separate “good” studies from “not-so-good” ones. One metric is the number of times that a study has been cited by other studies since its publication, which loosely translates to a paper’s popularity. It’s also associated with time: A recently published study might have zero citations or rack up hundreds within a few months; a study from the ’90s may tout thousands. Another metric is the “impact factor” of a science journal. Journals that publish widely cited studies have a higher impact factor and thus have a reputation for being more rigorous or meaningful to the scientific community. Applied Sciences self-reports an impact factor of 2.5. Nature, for comparison, says its impact factor is 48.5.
The original Google Scholar has an option for ranking studies by “relevancy” and lists the number of citations for each result. The goal of the new Scholar Labs is to dig up “the most useful papers for the user’s research quest,” Google spokesperson Lisa Oguike told The Verge It does so by ranking papers in the same way as the researchers themselves, Google says, by “weighing the full text of each document, where it was published, who it was written by, as well as how often and how recently it has been cited in other scholarly literature.”
However, the new Scholar Labs will not sort or limit results based on a paper’s citation count or a journal’s impact factor, Oguike told The Verge.
Image: Google Scholar
... continue reading