Do I need a headless browser? A one-minute test first

The short answer: you need a headless browser only when the text you want exists solely after JavaScript runs in a browser, and the page does not already ship that text as data inside its HTML. Article, documentation and product pages often do not meet that bar. The test below takes about a minute per site and saves you from both common mistakes: starting Chrome for every URL, and trusting an empty response.

Why the question matters

A browser and a plain HTTP client see different things. When you open a page, the browser downloads the HTML, runs the scripts, calls APIs and builds the DOM you see. curl, Python requests or a fetch() call stop after step one: they get the HTML the server sent, and nothing else.

On a server-rendered page the two versions carry the same text. On a client-rendered single-page app, the first response may hold nothing but a mount point:

<body>
  <div id="root"></div>
  <script src="/assets/app.4f2c1.js"></script>
</body>

That page needs JavaScript. A plain fetch returns HTTP 200 and no content. The danger is that your code treats the 200 as success.

Test 1: read the raw response

Fetch the page without a browser and look for a sentence you can see on screen.

URL=https://example.com/
curl -sL "$URL" -o raw.html
grep -oiE '<p[ >]' raw.html | wc -l         # <p> tags in the raw HTML
grep -i "a sentence you can see on the page" raw.html

In a browser, use view-source:https://example.com/, not Inspect. Inspect shows the live DOM after scripts changed it, which is exactly what a fetcher does not get. The difference between the two is the topic of the SPA, SSR and hydration guide.

If the sentence is there, stop. A static fetch can read this page.

Test 2: look for data the page already ships

Many client-rendered pages still send their content, just not as visible HTML. Search for the common payloads:

grep -oE '__NEXT_DATA__|self\.__next_f|__NUXT_DATA__|application/ld\+json|og:title|application/rss\+xml' raw.html | sort | uniq -c

• __NEXT_DATA__ (Next.js Pages Router) and __NUXT_DATA__ (Nuxt 3) hold the data the page will render, as JSON.
• self.__next_f marks the React Server Components payload of the Next.js App Router.
• application/ld+json is structured data: articles, products, FAQs.
• og: tags give a title, description and image for nearly any page.
• An RSS or Atom link can replace scraping a listing page entirely.

If the text you need is in one of these, parse it. It is faster than rendering and usually cleaner than the visible page. The guide to data already in the HTML covers each format.

Test 3: reload with JavaScript off

Chrome DevTools can disable JavaScript for the open tab: open the Command Menu (Ctrl+Shift+P or Cmd+Shift+P), type "Disable JavaScript" and reload. Whatever disappears is the content that only exists after scripts run. This is the visual version of Test 1 and is useful when you are not sure which sentence to search for.

Also check the <noscript> element. Some sites put a text fallback there; most only say "You need to enable JavaScript to run this app", which confirms client rendering.

Test 4: render only what is left

If the content is not in the raw HTML and not in an embedded payload, a browser is the right tool. Two habits keep rendered fetches reliable:

• Wait for the text, not the container. On a client-rendered page the #root element exists before its content arrives, so waiting for it returns at once with nothing inside.
• Avoid waiting for network idle as the default. Pages that poll or stream never go quiet. Playwright's docs mark networkidle as discouraged for this reason.

The headless browser guide compares Playwright, Puppeteer, Lightpanda and hosted browser APIs, with their cost drivers.

Detecting blank shells in code

For lists of URLs, turn the tests into a check that runs before content reaches a model or a database:

import re

def looks_like_empty_shell(html: str) -> bool:
    paragraphs = len(re.findall(r"<p[\s>]", html, re.I))
    mount = re.search(r'<div id="(root|app|__next)"[^>]*>\s*</div>', html)
    has_payload = re.search(r"__NEXT_DATA__|__NUXT_DATA__|application/ld\+json", html)
    return paragraphs < 2 and bool(mount) and not has_payload

Treat the thresholds as a starting point and tune them on your own sources. The point is to fail loudly: an empty shell should become an error that routes the URL elsewhere, not a blank document that an LLM summarises with confidence. For pages that pass the check, the main-content extraction guide covers the step from HTML to clean text.

Where a static fetch API fits

Live Web Data API, made by Teneo Labs, which runs this site, works the same way as Test 1: by default it does a static fetch of public pages and returns the main content as Markdown, text or HTML. When it finds no usable content, for example a page that needs JavaScript, it says so with a THIN_CONTENT warning or a CRAWL_NOT_RENDERED error (0 credits) instead of returning an empty shell. You can then resend that URL with render: true, or set renderFallback: true from the start so only the pages that fail the static fetch are rendered in a browser. Once your plan is active, you can try a URL in the playground, or read the error codes.

Test it yourself

Pick three URLs from your own list: one article, one listing page and one logged-out app page. Run Test 1 and Test 2 on each. Write down which ones a static fetch reads, which ones ship JSON, and which ones truly need a browser. That split, not a vendor default, should decide how your pipeline fetches.