Check if a page is server-side rendered: where the grep misleads

The usual test for server-side rendering is one line: fetch the raw HTML, search it for a sentence you can see on screen. The one-minute headless browser test starts there, and most of the time it is right. This post covers the three cases where it gives the wrong answer, and the check that catches each one.

Case 1: the sentence is there, but in a hidden div

Symptom: grep finds the sentence in raw.html, yet your extracted text shows "Loading..." where the content should be.

This is streaming SSR. React's server renderer sends the static shell first: layout, navigation and the Suspense fallbacks, each fallback marked with a <template id="B:0"> placeholder. When a boundary resolves, the server appends the finished HTML later in the same response, inside <div hidden id="S:0">, together with an inline script that swaps it into the placeholder's spot. In a browser that swap runs at once, without waiting for the bundle or hydration. In a file on disk, nothing swaps.

Count both markers:

URL=https://example.com/
curl -sL -H 'Accept-Encoding: identity' "$URL" -o raw.html
grep -oE '<template id="B:[0-9]+"' raw.html | wc -l      # fallback placeholders
grep -oE '<div hidden id="S:[0-9]+"' raw.html | wc -l    # streamed fragments

Nonzero counts mean part of the page arrived as late fragments. The text is in the response, so a static fetch that reads to the end of the body has it.

Your extractor may not. Mozilla's Readability treats any element with a hidden attribute as not visible, along with display: none, visibility: hidden and aria-hidden="true", and removes it while it looks for the article. The fallback sits in the visible flow and survives. So the extracted text can keep the "Loading..." line and lose every streamed section.

Check the output of the tool you actually use:

// extract.mjs: npm install jsdom @mozilla/readability
import { readFileSync } from "node:fs";
import { JSDOM } from "jsdom";
import { Readability } from "@mozilla/readability";

const html = readFileSync(process.argv[2], "utf8");
const doc = new JSDOM(html, { url: "https://example.com/" }).window.document;
const article = new Readability(doc).parse();
console.log(article ? article.textContent.trim() : "(nothing extracted)");

node extract.mjs raw.html | grep -i "a sentence you can see on the page"

Found in raw.html but missing here: the page is server-rendered, and the problem is the extraction step. Move each S: fragment into its B: placeholder before extraction, or drop the hidden attribute on those fragments in a pre-processing step.

Case 2: your fetcher gets a different document than a bot does

Symptom: two tools fetch the same URL and disagree on what came back, or on how long it took.

Next.js keeps a list of HTML-limited bots: user agents that need metadata in the <head> of the first HTML. For those, it waits for generateMetadata to resolve and holds the response until the document is finished, then sends it in one burst. Browsers and DOM-capable crawlers get the streamed version instead. The option is htmlLimitedBots, added in Next.js 15.2.0, and a site can replace the default list with its own regex or set it to /.*/ to turn streaming metadata off for everyone.

The default list includes Twitterbot, Slackbot, Bingbot and several Google crawlers. The source file does not list GPTBot, ClaudeBot or PerplexityBot. So the answer to "is this page server-rendered?" can depend on who asks.

Compare two requests:

URL=https://example.com/
for UA in 'Mozilla/5.0' 'Twitterbot/1.0'; do
  curl -sL -A "$UA" -H 'Accept-Encoding: identity' "$URL" -o page.html \
    -w "$UA  first byte %{time_starttransfer}s  total %{time_total}s  %{size_download} bytes\n"
done

On a streamed Next.js page, the bot request can wait longer for its first byte, since the server holds that response until the document is finished. A difference in size or markup tells you the server branches on user agent. Test with the user agent your own fetcher sends, not the one that makes the page look best.

With Cache Components there is a second effect, which the docs spell out: HTML-limited bots skip the prerendered shell and render the page dynamically. If the shell depends on data that exists only at build time, the page can load for a person and fail for a bot.

Case 3: the shell is server-rendered, the main text is not

Symptom: your test sentence was the page heading, the grep found it, and the extracted text is a header, a nav and a spinner.

A page can be rendered on the server down to the last layout component and still fetch its main content in the browser. The Next.js docs show this with SWR in a Client Component: it returns a Loading... div while the request is pending, and the list renders only after the data arrives. So the server-rendered HTML can hold the loading state instead of the posts.

Pick the test sentence from the body text, not from anything a layout could render:

grep -c "a sentence from the middle of the main content" raw.html

Zero: the main text is built after load. Embedded data is the next place to look, then a browser.

One more trap: a 200 that is really a 404

Once a streamed response has started, its status code is fixed at 200. If the page calls notFound() mid-stream, Next.js cannot change the status, so it adds <meta name="robots" content="noindex"> to the HTML instead. A server-rendered "not found" page passes every text check.

grep -o '<meta name="robots" content="noindex"' raw.html

A match on a page you expected to be indexable deserves a second look before you store it.

Where a static fetch API fits

SiteRender API does a static fetch by default and returns the main content as Markdown, text or HTML. When the main content is missing, as in Case 3, it flags the page with a THIN_CONTENT warning or a CRAWL_NOT_RENDERED error instead of returning the shell. Failed fetches cost 0 credits. Over REST and the SDKs you can resend that URL with render: true, or set renderFallback: true so only pages where the static fetch is blocked or finds no visible text are rendered in a browser; the MCP tool does not render. To check whether a static fetch is enough for your URL, try it in the playground.

Test it yourself

Take one URL that you believe is server-rendered. Run the marker count, the extraction check and the two-user-agent comparison. If the sentence survives extraction under your own user agent, a static fetch reads the page. If it is only in the raw HTML, fix the extractor. If it is only in a bot's response, or only in the header, treat the page as not server-rendered for your purposes. For the browser-side view of the same difference, see view source vs Inspect element.