Bot Traffic

A large fraction of requests hitting any public website come from machines: search crawlers, AI crawlers, uptime monitors, SEO tools, scrapers and attack bots. For AI visibility work, the interesting subset is well-identified AI bots, training crawlers like GPTBot and ClaudeBot, retrieval crawlers like OAI-SearchBot and PerplexityBot, and user-triggered fetchers like ChatGPT-User. Each visit is evidence that an AI system is ingesting or retrieving your content, which is upstream of every citation and mention you will ever earn.

Tools like Google Analytics count visitors by executing JavaScript in a browser. Bots fetch HTML and leave; the tracking script never runs, so the visit never registers. This creates a systematic blind spot: a page can be crawled daily by five AI platforms while its analytics show nothing. The only reliable observation points are server logs, CDN logs, or an edge worker that inspects the user agent of every request before the response is served. Server-side detection also catches the reverse problem: spoofed bots faking AI user agents, identifiable by checking against published IP ranges.

Raw bot hits become useful when segmented by bot, page and time. Crawl coverage tells you which content AI platforms can actually see; crawl frequency shifts often precede citation changes; absence of crawls on key pages flags robots.txt mistakes or rendering problems. Pair bot data with AI referral traffic to see the full chain from crawl to citation to human click. Geonimo captures this server-side through a Cloudflare Worker, classifying each AI bot and mapping its activity page by page in AI traffic analytics.