Free tool
Check in seconds whether the crawlers behind ChatGPT, Claude, Perplexity and Gemini can read your website, and whether anything in your robots.txt is silently keeping you out of AI answers.
Every AI engine that can cite your website first has to read it. OpenAI, Anthropic, Perplexity and Google each operate distinct crawlers for training, search indexing and live browsing, and each one respects robots.txt. One over-broad Disallow rule (often added years ago for unrelated reasons) can silently remove you from AI answers while your classic Google rankings look perfectly healthy.
This checker fetches your robots.txt and evaluates access for the 11 crawlers that matter for AI visibility, plus whether you publish an llms.txt. It's the technical half of the picture: the content and authority half is what our AI visibility audit covers.
OpenAI, Anthropic and Perplexity all split their web access into the same three roles, each with its own robots.txt token: a training crawler that feeds model training, a search-index crawler that decides whether answers can link to you, and a user-triggered fetcher that opens a page because a person asked about it. OpenAI's crawler documentation spells out the consequence: you can allow OAI-SearchBot to stay visible in ChatGPT search while disallowing GPTBot to stay out of training. Blocking the wrong token is how sites disappear from AI answers by accident.
| Vendor | Training crawler | Search index | User-triggered fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None documented | PerplexityBot | Perplexity-User |
Google is the deliberate exception to the pattern. Google Search Central states that AI is built into Search itself, so there is no AI-only crawler or AI-only opt-out that keeps you in classic results: your Googlebot robots.txt directives plus the preview controls (nosnippet, data-nosnippet, max-snippet) are the controls, and tightening those preview controls shrinks how much of your content AI Overviews and AI Mode can quote. The often-recommended Google-Extended token governs Gemini training and grounding only; blocking it does not take you out of Google's AI answers.
One more limit worth knowing: Perplexity's crawler docs state that Perplexity-User, because it acts on a direct user request, "generally ignores robots.txt rules". robots.txt is a crawling policy, not an access wall, which is why this checker reports the search and user-fetch agents separately.
AI crawlability is whether AI companies' crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) can access your website's content. If they're blocked in robots.txt, AI engines can't read your pages, which limits your chances of being cited or recommended in AI answers.
If AI visibility matters to your business, allow at minimum the search/browsing agents: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User, ClaudeBot, and Google-Extended (Gemini grounding). Training crawlers like GPTBot and CCBot are a policy choice: allowing them helps models learn about your brand; blocking them keeps your content out of training data.
Blocking GPTBot keeps your content out of OpenAI's model training, but ChatGPT's live search uses OAI-SearchBot and ChatGPT-User. Many sites block GPTBot while allowing the search agents: that way answers can still cite your pages even though training doesn't ingest them.
llms.txt is a proposed standard: a plain-text file at your site root that gives AI systems a curated map of your most important content. It's inexpensive to add and signals AI-friendliness. You can generate one with Geonimo's free llms.txt generator.
No, and this is the most misunderstood robots token in GEO. Per Google's own documentation, Google-Extended controls whether your content trains future Gemini models and grounds Gemini apps. AI Overviews and AI Mode are part of Google Search itself, governed by your Googlebot robots.txt directives and preview controls like nosnippet and max-snippet. Blocking Google-Extended does not remove you from Google's AI answers, and blocking Googlebot removes you from all of Search.
No. robots.txt governs automatic crawling, not fetches a human directly triggered. Perplexity documents that Perplexity-User, which fetches a page because a user asked about it, "generally ignores robots.txt rules". OpenAI's ChatGPT-User and Anthropic's Claude-User play the same on-demand role. Treat robots.txt as a crawling policy, not an access wall.
Crawlability is necessary but not sufficient. Engines also need content structured for answers (direct answers, clean headings, FAQ schema) and authority signals (third-party mentions, reviews, editorial coverage). An AI visibility audit shows which layer is failing for your brand.