How to Server-Side Render for AI Crawlers

Why it matters

When an AI crawler requests a page, it reads the HTML the server returns — and most of them stop there. Unlike a browser, a headless crawler typically does not execute JavaScript. So if your headings, paragraphs, and tables are rendered client-side by a single-page-app framework, the crawler receives an almost-empty shell: a <div id="root"></div> and a script tag. Your content is invisible, regardless of how good it is.

This is the most under-diagnosed crawlability failure, because the page looks perfect to you — your browser runs the JS. The crawler doesn't. The fix is to deliver the substance in the initial HTML response through server-side rendering (SSR) or static generation, so what the crawler reads matches what the reader sees.

Google is the main exception (Applebot also renders): Googlebot renders JavaScript, so Google Search — including AI Overviews and AI Mode — usually does see client-rendered content, just later and less reliably. Every other AI assistant you want to be cited by should be treated as no-JS.

What good looks like

The content is present on first response — before any JavaScript executes:

• View Source (not DevTools "Elements") shows your real headings, paragraphs, lists, and tables as HTML.
• The page is delivered via SSR or static generation, not client-only rendering.
• Where JS enhancement is genuinely needed, a full HTML fallback exists underneath it.
• Your JSON-LD (Phase 5.1) is in the raw source too — not injected after load.
• A curl of the URL returns the content, not just an empty app shell.

Step by step

1. Test what's in your raw HTML Don't trust the rendered page. Use the browser's View Source (Ctrl/Cmd+U) or run curl https://yourdomain.com/page and read what actually comes back. If you see your article text, you're fine; if you see a near-empty <div id="root"> and a bundle script, your content is JS-only.
2. Choose SSR or static generation For content pages, render on the server. Static generation (build-time HTML) is ideal for pages that don't change per request — guides, articles, docs. SSR (per-request HTML) suits pages with dynamic data. Most modern frameworks (Next.js, Nuxt, SvelteKit, Astro) support both; pick the one that emits real HTML for the route.
3. Render the content, not just the shell Make sure the rendered HTML contains the substance — the headings, body copy, lists, and tables — not just the layout chrome. A server-rendered navbar around a client-rendered article still leaves the article invisible.
4. Provide a fallback where JS is unavoidable If a component truly must be client-side, ensure a meaningful HTML fallback is in the source (e.g. <noscript> content or progressively enhanced markup) so the core information survives without JavaScript.
5. Verify as a crawler would Re-run curl --user-agent "GPTBot" https://yourdomain.com/page and confirm all headings, paragraphs, and tables are present in the response. This reproduces what a no-JS crawler reads — though your WAF may treat the real, IP-verified bot differently (Phase 1.4).

Worked examples

<body>
  <div id="root"></div>
  <script src="/bundle.js"></script>
</body>

<body>
  <article>
    <h1>Generative Engine Optimisation</h1>
    <p>GEO is the practice of...</p>
    <table>...</table>
  </article>
</body>

# Fetch the raw HTML as an AI crawler would (no JS executed)
curl -s --user-agent "GPTBot" https://yourdomain.com/your-page \
  | grep -i "<h1\|the first sentence of your article"

# If grep finds your heading and copy → server-rendered, good.
# If it returns nothing but you SEE the content in a browser → JS-only, fix it.

Common mistakes