GoKawiilWeighing smoke: why GEO dashboards are mostly useless
The promise of a tool that gives you similar search visibility to the pre-agent world is understandably seductive. But before you eagerly hand over your cash for a platform, consider whether an afternoon’s work a month might be a more cost-effective alternative.
I recently wrote about the GEO chimera, the cottage industry promising to optimise your brand into AI answers. My argument was that the levers being sold mostly did not exist. Another part of the industry is selling gauges rather than levers. Estimates of the spend on AI visibility tracking now run past $100 million a year, across hundreds of dashboards charting your brand’s presence in ChatGPT the way Moz once charted your rank in Google. The pitch is falling on the fertile ground of brand marketers standing on the precipice of everything they previously relied upon, staring into the coming void.
There is a post-millennial feel to all of this. The Hollywood screenwriter William Goldman is famously quoted as saying, “nobody knows anything.” The lesser-known part of his quote is, “every time…it’s a guess and, if you’re lucky, an educated one.” That is agentic search visibility in a sentence.
Most current tools run prompts daily, average the runs, report a visibility percentage with a trend line and a confidence interval. The averaging is not the problem per se. The problem is everything it is averaging, because the prompts in the measurement basket are generally steered by the analytics vendor, the query volumes weighting them are modelled fiction, the query surface being sampled is not one a human uses, and nobody has shown that the resulting score predicts a single outcome or conversion that is important to a business. Yet these tools are multiplying like vape shops on a dying high street.
The truth is the things we’re desperately reaching for do not really exist. There are minimal mechanisms to improve your visibility in search agent responses. And there is no way to reliably track and measure that visibility. After a quarter century of SEO and SERPs, this is a bitter pill that many are still unwilling to swallow.
The tasty candy currently being offered as an alternative is what I call precision laundering. Run a hollow measurement often enough that the noise cancels, and out comes a seemingly stable, decimal-pointed number that passes for knowledge. But while averaging buys a number, it cannot buy validity.
The dice cup
In January, Rand Fishkin published a detailed study of the search agent tracking industry. With Patrick O’Donnell of Gumshoe.ai, he recruited 600 volunteers to run twelve brand-recommendation prompts through ChatGPT, Claude and Google’s AI, 2,961 times in total, across categories from chef’s knives to cancer hospitals. Fishkin went in expecting to prove that tracking search agent responses was pointless. The results were stranger than that.
Ask a search agent for brand recommendations a hundred times, and the odds of getting the same list twice are under one in a hundred. The same list in the same order runs closer to one in a thousand. The brands change, the ordering changes, even the number of items changes. Fishkin’s conclusion on position tracking was blunt. Any tool that gives a “ranking position in AI”, he wrote, “is full of baloney.”
... continue reading