robots.txt

The file lives at yoursite.com/robots.txt and consists of groups: a User-agent line naming a bot (or * for all), followed by Disallow and Allow rules describing path access. Well-behaved crawlers fetch it before crawling and obey their matching group. It controls crawling, not indexing — a disallowed URL can still appear in an index if linked elsewhere — and it is advisory: enforcement against non-compliant scrapers requires firewall or CDN rules keyed on user agent and IP verification.

AI crawlers turned robots.txt from a technical housekeeping file into a strategic document. The core principle: distinguish training crawlers from search and retrieval crawlers, because the consequences of blocking them are opposite. Blocking GPTBot, ClaudeBot, CCBot or the Google-Extended token withholds training data with no impact on your visibility in AI search. Blocking OAI-SearchBot, PerplexityBot or user-triggered fetchers like ChatGPT-User removes you from AI answers entirely.

The most common costly mistake is a blanket rule, copied from a block list, that disallows every AI user agent indiscriminately. Audit yours: each bot deserves its own deliberate line.

For a brand pursuing AI visibility: explicitly allow retrieval bots (OAI-SearchBot, PerplexityBot, ChatGPT-User), decide training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) case by case, and keep standard search crawlers unrestricted. Reference your sitemap, avoid disallowing CSS and JS that bots need to understand pages, and test changes before deploying, since one bad wildcard can silently erase AI citation eligibility. Then verify behavior in reality: server-side bot logs, such as those from Geonimo's AI traffic analytics, confirm which crawlers respect your rules and what they fetch.