How to Meet Technical Performance Prerequisites

Why it matters

Allowing AI crawlers in robots.txt (Phase 1.1) is necessary but not sufficient. A crawler can still be turned away one layer down — silently blocked by your CDN or WAF, timed out by a slow page, served a broken mobile layout, or stopped by a cookie banner that covers the content on first paint. Each of these produces incomplete or empty crawls, and the AI system simply moves on.

These are infrastructure problems, not content problems, which makes them easy to miss — your analytics show human visitors fine while crawlers quietly fail. Five things decide whether your pages are physically retrievable.

What good looks like

The page responds fast, completely, and identically to a crawler as to a person:

• Network access: your CDN/WAF does not block AI crawler user-agents or ASNs — no host_not_allowed errors.
• Speed: fast server responses and good Core Web Vitals (LCP, INP, CLS). As a working target, aim for full load under ~3 seconds — crawler timeouts aren’t published, so treat slow pages as a risk, not a known cutoff.
• Mobile: fully responsive — Google indexes mobile-first, and the AI features built on its index see what Googlebot Smartphone sees.
• No interstitials: cookie banners and overlays don't cover core content on first paint.
• Stable hosting + canonical: no timeouts or 5xx errors under normal load, and a <link rel="canonical"> consistent with the submitted URL.

Step by step

1. Verify network-level access (CDN / WAF) The most common silent failure: your firewall blocks AI crawler user-agents or their IP ranges. Test the live site with curl --user-agent "GPTBot" -I https://yourdomain.com/ and confirm a 200, not a 403/503 or a challenge page. Treat that as a first pass only: WAFs verify real crawlers by IP range or reverse DNS, so a spoofed user-agent can be treated differently from the real bot. Confirm in server logs or CDN bot analytics that requests from the vendors’ published IP ranges get 200. A managed bot-detection ruleset you did not configure yourself is still an access control. Enumerate what your CDN blocks by default before assuming you are open. If your WAF has a bot-management rule, add explicit allow rules for the AI user-agents you permit in robots.txt.
2. Hit the page-load target Measure with PageSpeed Insights or WebPageTest and fix Core Web Vitals. Aim for under ~3 seconds; optimise images, defer non-critical scripts, and cache aggressively. If crawlers overload you, answer with 429 or 503 plus Retry-After rather than blocking them.
3. Confirm mobile responsiveness Google indexes the mobile version of your pages, and its AI features draw on that index. Test with a mobile user-agent and a narrow viewport; ensure text, tables, and headings reflow rather than overflow or hide.
4. Remove blocking interstitials Cookie consent banners, newsletter modals, and age gates that cover the content on first paint reduce crawl quality. Render the content underneath them, or defer the overlay until after first paint, so the substance is present when the crawler reads the page.
5. Stabilise hosting and the canonical tag Pages must respond reliably, without timeouts or 5xx errors, under normal and crawl load. Confirm every page declares a <link rel="canonical"> that matches the URL you actually want indexed and that you submit in the sitemap — inconsistent canonicals split or misdirect authority.

Worked examples

# Compare a normal browser UA vs an AI crawler UA
curl -I -s -o /dev/null -w "%{http_code}\n" \
  --user-agent "Mozilla/5.0" https://yourdomain.com/        # → 200

curl -I -s -o /dev/null -w "%{http_code}\n" \
  --user-agent "GPTBot"     https://yourdomain.com/        # → 403  ← blocked

# Fix: add an allow rule for GPTBot/ClaudeBot/PerplexityBot
# in your WAF/bot-management config, then re-test until both return 200.

Common mistakes