Since last year’s disastrous rollout of Google’s AI Overviews, the world at large has been aware of how AI-powered search results can differ wildly from the traditional list of links search engines have generated for decades. Now, new research helps quantify that difference, showing that AI search engines tend to cite less popular websites and ones that wouldn’t even appear in the Top 100 links listed in an “organic” Google search.
In the pre-print paper “Characterizing Web Search in The Age of Generative AI,” researchers from Ruhr University in Bochum, Germany, and the Max Planck Institute for Software Systems compared traditional link results from Google’s search engine to its AI Overviews and Gemini-2.5-Flash. The researchers also looked at GPT-4o’s web search mode and the separate “GPT-4o with Search Tool,” which resorts to searching the web only when the LLM decides it needs information found outside its own pre-trained data.
The researchers drew test queries from a number of sources, including specific questions submitted to ChatGPT in the WildChat dataset, general political topics listed on AllSides, and products included in the 100 most-searched Amazon products list.
Overall, the sources cited in results from the generative search tools tended to be from sites that were less popular than those that appeared in the top 10 of a traditional search, as measured by the domain-tracker Tranco. Sources cited by the AI engines were more likely than those linked in traditional Google searches to fall outside both the top 1,000 and top 1,000,000 domains tracked by Tranco. Gemini search in particular showed a tendency to cite unpopular domains, with the median source falling outside Tranco’s top 1,000 across all results.
Credit: Kirsten et al A majority of the cited AI Overview sources don’t appear in the top 10 Google link results for the same query. Credit: Kirsten et al A majority of the cited AI Overview sources don’t appear in the top 10 Google link results for the same query.
The sources cited by the AI-powered search engines also tended to be ones that wouldn’t appear anywhere near the top results for the same organic Google search. Fifty-three percent of the sources cited by Google’s AI Overviews, for instance, didn’t appear in the top 10 Google links for the same query, and 40 percent of those sources didn’t even fall in the top 100 Google links.