Four how-to guides. Each is a printable, copy-paste-ready artefact for one subdomain of Phase 1.
4 of 4 guides published
Training, search and user fetches are separate decisions
OpenAI, Anthropic and Perplexity each run separate agents for model training, for their search/answer index, and for pages a user asks the assistant to open (see 1.1). Crawlability for citation depends on the search and user agents; crawlability for training depends on the training agents. Google is different: AI Overviews and AI Mode use the normal Search index and
Googlebot.
Advisory — access control is a technical lever, not a legal one
Phase 1 governs whether AI systems can reach your content. It does not determine what rights you hold over content already collected; those questions are being decided jurisdiction by jurisdiction. Take advice from qualified IP counsel. This framework does not provide legal advice and we do not score this item.
- 1.1 Configure robots.txt for AI crawlers Allow the AI crawlers you want, block the ones you don't, and point them to your sitemap. Published
- 1.2 Server-side rendering for AI crawlers Make sure your content exists in the raw HTML — headless crawlers don't run JavaScript. Published
- 1.3 Write an llms.txt file Publish a plain-text map that tells AI systems which pages are your canonical knowledge. Published
- 1.4 Meet technical performance prerequisites Clear the baseline: WAF rules, page speed, mobile, and canonical tags that don't block retrieval. Published