AI Crawler

AI crawlers visit your site for two distinct purposes. Training crawlers like GPTBot and CCBot collect content that may end up in the training data of future models. Search and retrieval crawlers like OAI-SearchBot fetch pages so AI engines can cite them in live answers. The distinction matters enormously: training crawlers influence what a model knows long-term, while retrieval crawlers determine whether your pages can be cited in answers today.

You control AI crawlers through robots.txt directives targeting each bot's user agent. Blocking training crawlers is a defensible choice for publishers worried about content reuse, and it has no effect on your search rankings. Blocking retrieval crawlers is very different: if OAI-SearchBot or PerplexityBot cannot fetch your pages, those engines cannot cite you, which directly kills your AI visibility. Most brands competing for AI mentions should allow retrieval crawlers unconditionally and decide on training crawlers based on their content strategy.

A second technical constraint applies to all of them: most AI crawlers do not execute JavaScript, so content that only appears after client-side rendering is effectively invisible to them.

AI crawler hits never appear in standard analytics tools because bots do not run tracking scripts. You need server-side detection, either by parsing server logs for known user agents or by deploying an edge worker that inspects every request. Geonimo's AI traffic analytics uses a Cloudflare Worker to log which AI bots crawl which pages and how often, so you can verify that the crawlers you allow are actually reaching the content you want cited.