GPTBot

GPTBot is OpenAI's training crawler. It systematically fetches public web pages and feeds them into the training data pipeline used to build future GPT models. It is distinct from OpenAI's two other fetchers: OAI-SearchBot, which powers ChatGPT search citations, and ChatGPT-User, which fetches pages on demand during conversations. Confusing these three bots is one of the most common mistakes in AI crawler policy, because each one has different consequences when blocked.

Blocking GPTBot in robots.txt prevents your content from being used to train future OpenAI models. It does not remove you from ChatGPT search results, since those rely on OAI-SearchBot, and it has no effect on Google rankings. Many large publishers block GPTBot to preserve licensing leverage over their content.

The counterargument for brands: content absorbed during training shapes what models say about you when they answer without browsing. If a future model has never ingested your product pages and documentation, its baseline knowledge of your brand is thinner, which can dampen unprompted AI brand mentions. For most companies seeking AI visibility rather than content licensing revenue, allowing GPTBot is the pragmatic default.

To block GPTBot, add a dedicated user agent group to robots.txt: User-agent: GPTBot followed by Disallow: /. To allow it everywhere, either omit any GPTBot rule or set an empty Disallow. You can also allow only specific directories, which lets you expose marketing content while shielding gated material. Verify the policy is working by watching server logs for the GPTBot user agent; tools like Geonimo's AI traffic analytics surface GPTBot crawl activity per page so you can confirm OpenAI is reading what you intend.