GoKawiilBot Detection Without JavaScript: What My Blog Measured
On my blog, network and request-header rules moved 277 of 372 browser-User-Agent requests out of the Browsers category: 74.5%. That gives me a much more useful account of the traffic arriving at my Cloudflare Worker. It has not established how many people read the site. Over the same two complete UTC days, the remaining 95 Browser HTML observations still differ from 14 Cloudflare Web Analytics page loads. The useful result is knowing which requests the rules separate, why they separate them, and where the evidence stops.
Network evidence catches requests that pass the header checks. In that window, 60 cloud-classified requests carried the navigation headers the browser rule requires.
In that window, 60 cloud-classified requests carried the navigation headers the browser rule requires. A reason for each classification makes the counter explainable. It also exposed mistakes: our HTML-acceptance check mishandled valid headers, and new rows were missing their network-provenance marker. Both have since been repaired.
It also exposed mistakes: our HTML-acceptance check mishandled valid headers, and new rows were missing their network-provenance marker. Both have since been repaired. Client identity and readership need different evidence. Nine stored signature verifications identify signers, including crawlers and deliberate tests. They do not count people asking an assistant to read.
Nine stored signature verifications identify signers, including crawlers and deliberate tests. They do not count people asking an assistant to read. The remaining disagreement is a measured problem. Neither a smaller Browser count nor agreement with a script counter establishes audience accuracy.
Comparing edge page views with a script counter
Comparing two counters exposed the problem with my first-party analytics: I had treated browser User-Agents as evidence of readers. The counters measure different events, so their disagreement is a starting point for investigation. It cannot, by itself, tell me which requests were automation.
The initial alarm came from this comparison, saved on September 3:
Source Page events / loads Client or visit metric D1 browser-UA class, seven UTC days ending September 2 1,209 578 daily client identifiers Cloudflare Web Analytics, its rolling seven-day dashboard window 113 52 visits
... continue reading