A default robots.txt can quietly shut out the exact crawlers that power AI search. Last updated August 2026.
This is a genuinely easy thing to get wrong by accident. A default or overly broad robots.txt - sometimes just a leftover default from a website builder or CMS - can quietly block the exact crawlers that let AI assistants find and cite your site, without anyone realizing it.
Some AI assistants answer purely from training data, but others actively browse the live web as part of generating a response, citing real sources along the way. If your robots.txt blocks the crawler responsible for that real-time retrieval, your site simply can't be one of those cited sources, no matter how good your content is.
Visit yoursite.com/robots.txt directly in a browser - it's always a plain text file at your domain's root. Look for Disallow: / lines, and check which User-agent they apply to. A blanket User-agent: * with Disallow: / blocks everything, including every AI crawler, unless a more specific rule further down explicitly allows one back in.
The major AI companies each publish their own crawler user-agents. As of 2026, the ones most relevant to AI visibility include:
GPTBot (training), OAI-SearchBot (ChatGPT Search), ChatGPT-User (when a user asks ChatGPT to browse a specific page)ClaudeBot, Claude-User, Claude-SearchBotPerplexityBot, Perplexity-UserGoogle-Extended (controls Gemini training use specifically, separate from regular Googlebot)CCBot - feeds many open AI training datasets beyond any single companyThis list shifts as providers introduce new crawlers, so it's worth rechecking periodically rather than treating any list (including this one) as permanently complete.
Add explicit Allow rules for the crawlers you want, above any blanket Disallow. A minimal example:
User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml
Keep any genuinely private paths (admin panels, account pages, checkout flows) disallowed regardless - allowing AI crawlers only needs to apply to your actual public, indexable content.
Worth deciding deliberately rather than defaulting either way: training crawlers (like GPTBot) feed content into a model's future training data - a one-way door, since content already used for training can't be un-used. Search/retrieval crawlers (like OAI-SearchBot) power real-time citation in answers today, without necessarily becoming permanent training data. You can allow one type and not the other - there's no requirement to treat all AI crawlers identically.
A correct robots.txt removes a blocker - it doesn't guarantee a mention. See where you actually stand.
Get a one-time auditNo - allowing a crawler in robots.txt only grants permission to visit; it doesn't affect how traditional search engines rank you. Googlebot and AI crawlers are evaluated independently.
That's a legitimate, common choice - some site owners want their content eligible for citation in AI-generated answers (which needs the search/retrieval crawlers allowed) without it being used to train future models (which is what training crawlers are for). They're genuinely different crawlers, so you can allow one type and not the other.
Reputable crawlers from OpenAI, Anthropic, Google, and Perplexity publicly document that they respect robots.txt rules. There's no technical enforcement mechanism forcing any crawler to comply, but the major AI companies have stated policies to do so, and blocking is still the correct signal to send either way.
Yes, in the same supporting role it plays for traditional search - it helps any crawler, AI or not, discover your pages efficiently rather than relying purely on following links. Reference it from robots.txt with a Sitemap: line, as covered below.