robots.txt is the first gate every AI crawler checks — get it right and your content is reachable; get it wrong and you are invisible to AI systems no matter how good your pages are.
Before an AI system can cite you, its crawler has to be allowed to fetch your pages. robots.txt — a single plain-text file at the root of your domain (/robots.txt) — is where you grant or deny that access. For a crawler that honours the file, blocking it here removes your content from that system's training and real-time retrieval; no amount of structured data or authority downstream can compensate for a closed gate. (For crawlers that ignore the file, see “Compliance is not universal” below — the file is not a universal off switch.)
The most common cause of zero AI visibility is not weak content — it is an accidental Disallow: / left over from a staging environment. For knowledge-asset publishers, the recommended posture is fully open: let the AI crawlers in, and declare your sitemap so they can discover everything worth citing.
A crawl-ready robots.txt permits the AI crawlers you care about and points them at your sitemap, with no sweeping disallow hiding your content:
/admin, /internal) are blocked — never /blog or /docs.Sitemap: https://example.com/sitemap.xml).Disallow: / from staging.https://yourdomain.com/robots.txt and returns 200.Most AI operators now run more than one agent, and each one answers a different question. Decide your policy per job, not per company:
| Operator | Training | Search / answer index | User-initiated fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot (ChatGPT search) | ChatGPT-User — OpenAI says robots.txt rules may not apply to user-initiated actions |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | — | PerplexityBot | Perplexity-User |
Google-Extended — a control token covering Gemini training and grounding, not a separate crawler | Googlebot — Search, including AI Overviews and AI Mode | — | |
| Apple | Applebot-Extended (control token) | Applebot | — |
| Meta | Meta-ExternalAgent | — | Meta-ExternalFetcher |
| Common Crawl | CCBot — open dataset widely reused for training | — | — |
| ByteDance | Bytespider — has a reported history of disregarding robots.txt | — | — |
Vendors add and rename agents. Check each operator’s current crawler page before you rely on this table — OpenAI and Anthropic both publish theirs (linked in the evidence box above).
GPTBot and ClaudeBot (training) while allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot and the -User agents (citation). The reverse mistake — blocking a search agent while meaning to block training — silently removes you from that assistant’s answers.
Googlebot. Google-Extended does not remove you from them. The only controls are the ordinary Search ones: blocking Googlebot (which removes you from Search entirely) or snippet controls such as nosnippet and max-snippet. For Google, optimise how you are cited rather than trying to gate access.
Content-Signal: search=yes, ai-train=no (a third value, ai-input, covers use as live answer input). Content Signals are a declaration of preference, not a technical block, and no major AI vendor has committed to honouring them. Fetch your live /robots.txt and read what is actually served.
Disallow for those as a request, and rely on making your content quotable and correctly attributed (Phases 2–5).
robots.txt. If you want AI citation, verify these layers are not silently turning away the agents you allow (Phase 1.4). If you want to restrict access, network-level controls are enforceable in a way a Disallow line is not.
Disallow: / that survived launch, and habitual blocks of /blog or /docs.User-agent: * / Allow: / covers everything. For a targeted posture, write a group per agent. Two parsing rules catch people out (RFC 9309): consecutive User-agent lines share one group of rules, and an agent that has its own group ignores the * group entirely — so repeat any private-path Disallow lines inside each named group. Agent names match case-insensitively.Sitemap: line with the absolute URL of your XML sitemap. This is how crawlers discover pages and understand your structure — it pairs directly with Phase 1.3 (llms.txt) and Phase 5.5 (sitemap + llms.txt cross-reference).Disallow: / under User-agent: * (or under any AI user-agent) that you didn't intend. This single line is the most common reason a site is absent from AI answers.curl https://yourdomain.com/robots.txt), confirm it returns 200 and contains the rules you expect — not a CDN-managed version you didn’t write. Then check the network layer: a curl --user-agent "GPTBot" test only shows how your WAF treats that string. Real crawlers are verified by IP, so a spoofed test can be blocked while the real bot is admitted, or the reverse. Confirm in your server logs or CDN bot analytics that requests from the vendors’ published IP ranges receive 200 (Phase 1.4).The simplest correct configuration for a knowledge-asset publisher: every crawler is welcome, and the sitemap is declared so they find everything.
# Fully open (recommended for GEO)
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlA site that wants to appear in AI answers but opts out of model training. Search and user agents are allowed; training agents are blocked. Because a named group ignores the * group, the private paths are repeated in the shared group of allowed agents.
# Training opt-out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Google-Extended also controls grounding in Gemini Apps — add it
# here only if you accept losing Gemini citations.
# Search + user-initiated agents: allowed (private paths repeated)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /admin
Disallow: /internal
# Everyone else, including Googlebot (Search + AI Overviews)
User-agent: *
Disallow: /admin
Disallow: /internal
Sitemap: https://example.com/sitemap.xmlThe same site, one line different. The "before" is the single most common reason a site is missing from AI answers — a blanket disallow that survived a launch.
User-agent: * / Disallow: /. Every crawler is turned away at the door. The site is invisible to AI training and retrieval, regardless of content quality. No sitemap declared.
User-agent: * / Allow: / with Sitemap: declared. Crawlers are admitted and pointed at the full set of canonical pages. Visibility restored with a one-line change.
robots.txt is a published request, honoured by well-behaved crawlers — OpenAI's GPTBot, for example, obeys the directives at /robots.txt. It is not an enforcement mechanism. Three limits are worth stating plainly to clients:
Disallow reduces, but does not eliminate, the chance of inclusion.robots.txt.So treat robots.txt as the gate you must open to be eligible, and treat any opt-out as a best-effort signal rather than a hard wall.
| Mistake | Fix |
|---|---|
Leftover Disallow: / from staging | Remove it — this single line makes the whole site invisible to AI crawlers. |
Disallowing /blog or /docs "to be safe" | These are usually your best knowledge assets; allow them explicitly. |
No Sitemap: line | Declare the absolute sitemap URL so crawlers discover and prioritise canonical pages. |
| Relative sitemap URL | Use the full https:// URL — relative paths are not valid in robots.txt. |
| Blocked at the WAF/CDN instead | Check server logs or CDN bot analytics for the vendors’ real IP ranges, not just a spoofed curl; allow the agents you permit at the network layer too (Phase 1.4). |
Assuming robots.txt guarantees opt-out | It's a request honoured by compliant crawlers; treat exclusion as probabilistic (see the limits above). |
Blocking GPTBot to "stay out of ChatGPT" | GPTBot is training only. ChatGPT search uses OAI-SearchBot; decide training and search separately. |
Expecting Google-Extended to remove you from AI Overviews | It doesn’t — AI Overviews and AI Mode use Googlebot. Use snippet controls, or accept citation and optimise for it. |
Named group missing the private-path Disallows | An agent with its own group ignores *; repeat private paths inside each named group. |
| Undocumented legacy training blocks | Record which provider and agent each training rule targets. When the shared IETF training signal ships, migration is then a mapping exercise, not a rewrite. |
/robots.txt returns 200 and is the file you wrote (not an unexpected CDN-managed version)Disallow: / under User-agent: * or any AI user-agentGPTBot vs OAI-SearchBot vs ChatGPT-User)Disallows repeated inside every named groupSitemap: declared with an absolute URL200 (Phase 1.4)