Free tool
AI Crawlability Checker
Check in seconds whether the crawlers behind ChatGPT, Claude, Perplexity and Gemini can read your website — and whether anything in your robots.txt is silently keeping you out of AI answers.
Why AI crawlability matters
Every AI engine that can cite your website first has to read it. OpenAI, Anthropic, Perplexity and Google each operate distinct crawlers for training, search indexing and live browsing — and each one respects robots.txt. One over-broad Disallow rule (often added years ago for unrelated reasons) can silently remove you from AI answers while your classic Google rankings look perfectly healthy.
This checker fetches your robots.txt and evaluates access for the 11 crawlers that matter for AI visibility, plus whether you publish an llms.txt. It's the technical half of the picture — the content and authority half is what our AI visibility audit covers.
Frequently asked questions
What is AI crawlability?
AI crawlability is whether AI companies' crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) can access your website's content. If they're blocked in robots.txt, AI engines can't read your pages — which limits your chances of being cited or recommended in AI answers.
Which AI crawlers should I allow?
If AI visibility matters to your business, allow at minimum the search/browsing agents: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User, ClaudeBot, and Google-Extended (Gemini grounding). Training crawlers like GPTBot and CCBot are a policy choice: allowing them helps models learn about your brand; blocking them keeps your content out of training data.
Does blocking GPTBot remove me from ChatGPT?
Blocking GPTBot keeps your content out of OpenAI's model training, but ChatGPT's live search uses OAI-SearchBot and ChatGPT-User. Many sites block GPTBot while allowing the search agents — that way answers can still cite your pages even though training doesn't ingest them.
What is llms.txt?
llms.txt is a proposed standard: a plain-text file at your site root that gives AI systems a curated map of your most important content. It's inexpensive to add and signals AI-friendliness. You can generate one with Geonimo's free llms.txt generator.
My site allows all crawlers but I'm still not cited. Why?
Crawlability is necessary but not sufficient. Engines also need content structured for answers (direct answers, clean headings, FAQ schema) and authority signals (third-party mentions, reviews, editorial coverage). An AI visibility audit shows which layer is failing for your brand.