Scraper returns empty HTML? Tell the four causes apart

"The scraper returns empty HTML" is one log line with at least four different causes behind it. They need four different fixes, and one of them is not a fix at all but a signal to stop. Before you change your fetcher, find out which one you have.

Start by saving the raw response and its headers:

URL=https://example.com/
curl -sL -D headers.txt "$URL" -o raw.html
head -n 1 headers.txt                  # status line
grep -i '^content-encoding' headers.txt
wc -c raw.html                         # body size in bytes

Cause 1: the page is built in the browser

Status 200, a body of a few kilobytes, and very little text. Look for an empty mount point and count the paragraph tags:

grep -oE '<div id="(root|app|__next)"></div>' raw.html
grep -oiE '<p[ >]' raw.html | wc -l

An empty <div id="root"> with almost no <p> tags means client-side rendering: the HTML is a shell, and scripts fetch and build the content later. web.dev's "Rendering on the Web" describes this pattern and its trade-offs. Before you render, search the same file for __NEXT_DATA__, __NUXT_DATA__ or application/ld+json. The one-minute test walks through the full decision.

Cause 2: the site refused the request

A 403 or 429 status, or a short page that asks you to wait, sign in or confirm something. MDN defines 403 as a server that understood the request and refuses it, and 429 as too many requests in a given time.

This is not an extraction bug. Slow down, honor any Retry-After header, check robots.txt and the site's terms, and drop the URL if the site does not want automated access. Do not try to get around it.

Cause 3: compressed bytes that were never decoded

The body looks like binary noise, or your HTML parser finds nothing. Check the Content-Encoding header: gzip, br or zstd means the body must be decompressed first. MDN's Content-Encoding page lists the values. Most HTTP clients do this for you; hand-rolled socket code and some proxies in the middle do not. With curl, --compressed asks for a compressed body and decodes it.

Cause 4: a soft 404 or the wrong URL

Status 200, but the page says "not found", or the redirect chain ended on a home page or a consent page. Print the final URL and look at the title:

curl -sL -o /dev/null -w '%{url_effective}\n' "$URL"
grep -o '<title>[^<]*' raw.html

Fix the URL list, not the fetcher.

Make the cause visible in your pipeline

All four causes share one dangerous property: if your code only checks for status 200, three of them pass as success, and an LLM downstream summarises a page it never received. Record a reason for every URL that comes back without usable text, and route by reason: render, back off, decode or drop.

Live Web Data API, made by Teneo Labs, which runs this site, separates these cases with explicit error tags: CRAWL_NOT_FOUND, SOURCE_NOT_AVAILABLE and CRAWL_LIVECRAWL_TIMEOUT, plus CRAWL_NOT_RENDERED, which is designed to be returned when a page has no usable content, for example one that needs JavaScript. Errors cost 0 credits. It does a static fetch by default; set renderFallback: true and only the pages that come back empty are rendered in a browser. Once your plan is active, you can try a URL in the playground.

Test it yourself

Take ten URLs that came back empty last week. Run the four checks on each and write the cause next to the URL. If most of them are app shells, read the hydration guide next; if most are refusals, the fix is your request rate, not your parser.