AI Crawlers & Technical
PerplexityBot
PerplexityBot is the web crawler operated by Perplexity AI that indexes pages for its answer engine's search index. Pages it crawls become eligible to appear as cited sources in Perplexity answers. Blocking it in robots.txt removes your content from Perplexity's index and eliminates citation opportunities on that platform.
How PerplexityBot feeds the answer engine
Perplexity is built around live retrieval: every answer is assembled from indexed and freshly fetched sources, displayed with prominent citations. PerplexityBot is the crawler that builds and refreshes that index. Because Perplexity cites sources more aggressively than most AI platforms, being crawlable by PerplexityBot translates unusually directly into visible citations and referral clicks. Perplexity also uses user-triggered fetching for real-time queries, but the index PerplexityBot maintains determines your baseline eligibility.
Allow it if you want Perplexity citations
For brands and publishers seeking AI visibility, PerplexityBot belongs firmly in the allow column of robots.txt. It is a retrieval crawler, not a training crawler: blocking it does not protect your content from model training, it simply removes you from an answer engine that sends measurable referral traffic. Perplexity has faced public criticism over crawling practices and robots.txt compliance in the past, so sites with strict access requirements sometimes enforce blocks at the CDN level rather than relying on robots.txt alone. For everyone else, accessibility plus fast, server-rendered HTML is the winning configuration.
Measuring your Perplexity footprint
Two signals matter: PerplexityBot crawl activity in your server logs, and your actual citation share inside Perplexity answers for the prompts your buyers ask. Crawl coverage without citations suggests a content quality or citability problem; citations without recent crawls suggest answers built on stale snapshots. Geonimo tracks both sides, logging PerplexityBot visits through its Cloudflare Worker and monitoring your brand's citation share across Perplexity via multi-model monitoring.
Frequently asked questions
Should I block PerplexityBot?
Not if you want to appear in Perplexity answers. It is a search index crawler, so blocking it removes your citation eligibility without any training-data benefit. Block it only if you have deliberately decided to keep your content off Perplexity, and consider CDN-level enforcement if strict exclusion matters.
Does allowing PerplexityBot guarantee citations?
No. Crawlability is the entry ticket, not the ranking. Perplexity selects sources based on relevance, freshness and authority for each query. Allowing the bot makes your pages eligible; earning citations still requires content that directly and credibly answers the questions users ask.
How is PerplexityBot different from Perplexity's live fetching?
PerplexityBot crawls proactively to maintain Perplexity's search index. Separately, real-time user queries can trigger on-demand fetches of specific pages. The index determines what Perplexity can find quickly; live fetches supplement it with fresh detail. Both rely on your site being accessible and fast.
Related terms
Perplexity AI
Perplexity AI is an answer engine that performs live web retrieval for every query and generates responses with numbered citations linking to each source. Because every claim is attributed, it is the most transparent major AI search platform, and a high-signal target for marketers measuring which content earns AI citations.
AI Citation
An AI citation is a source link that an AI engine attaches to its generated answer, attributing a claim to a specific web page. Citations appear as numbered references or inline links in engines like Perplexity, ChatGPT Search, and Google AI Overviews. Earning citations drives referral traffic and signals that engines trust the cited domain.
OAI-SearchBot
OAI-SearchBot is OpenAI's search crawler that discovers and indexes web pages to power ChatGPT search results and citations. Unlike GPTBot, it is not used for model training. Blocking OAI-SearchBot in robots.txt removes your pages from ChatGPT's search index, eliminating your ability to be cited in its answers.
AI Referral Traffic
AI referral traffic consists of human visitors who arrive at a website by clicking a link inside an AI platform such as ChatGPT, Perplexity or Gemini. It is distinct from AI bot crawls, which are automated. Standard analytics frequently misattributes these visits as direct traffic, hiding the true impact of AI visibility.
Last updated: 2026-06-11
Track this for your brand
Geonimo monitors how ChatGPT, Perplexity, Claude, Gemini and Google AI talk about your brand — and generates the content that gets you cited.
Get your free audit