Headless browser API explained: options, costs, wait logic

A headless browser runs a real browser engine without a window, executes the page's JavaScript and lets your code read the resulting DOM. "Headless browser API" can mean the library you drive it with, such as Playwright or Puppeteer, or a hosted service that runs the browsers for you. This guide covers both, the main cost drivers, and the wait logic that decides whether you get text or an empty container.

SiteRender is run by Teneo Labs, which makes Live Web Data API, a static fetch that renders in a browser only on request. Tools are described from their public docs as of October 2026.

First, the question this site always asks: does the page need a browser at all? If the text is in the raw HTML or in an embedded JSON payload, a plain fetch is faster and cheaper. The one-minute test answers that before you pick any of the tools below.

The libraries

Playwright drives Chromium, Firefox and WebKit from one API, with bindings for TypeScript, Python, Java and .NET. Its locators wait automatically for elements to be attached and visible, which removes a lot of hand-written waiting.

Puppeteer is a Node.js library that controls Chrome over the Chrome DevTools Protocol, and Firefox over WebDriver BiDi. It is a close fit if you only need Chrome and want low-level protocol access.

Lightpanda is a newer headless browser written in Zig, built for machines rather than people. It exposes a CDP server, so Puppeteer can connect to it. Its README reports about 9x faster runs and about 16x less memory than headless Chrome on its own benchmark. That is a vendor benchmark; we have not reproduced it, and Web API coverage differs from Chrome, so test it on your own pages.

Headless Chrome itself changed in recent versions. Chrome's docs explain that headless and headful modes are now one implementation. The old headless mode was removed from the Chrome binary in version 132 and now ships separately as chrome-headless-shell. If an older script behaves differently after an upgrade, this is a likely reason.

Self-hosted vs a hosted browser API

Running browsers yourself gives full control over versions, network location and data. It also makes you responsible for:

• Memory. Each page carries a full rendering engine and a JavaScript heap. Concurrency is usually limited by RAM before CPU.
• Cold starts. Launching a browser takes far longer than an HTTP request. Keep a pool of warm browsers and reuse contexts.
• Crashes and leaks. Pages hang, tabs leak memory, and processes need recycling.
• Updates. Browser versions and their dependencies change every few weeks.

A hosted headless browser API moves that work to a vendor and bills per session, per minute or per page. You trade control and data locality for less operations work. Compare vendors on their documented limits, regions, session length and what happens on timeouts, not on headline speed claims.

How to choose a hosted headless browser API

If you decide to buy rather than run browsers, check each candidate against the same neutral criteria, from its own documentation:

• Regions. Where the browsers run, and whether you can pin a region close to your targets and your data.
• Session limits. Concurrent sessions per plan, maximum session length and idle timeouts.
• Timeout behavior. What the API returns when a page never finishes, and whether a timed-out session is billed.
• Data handling. What is logged, how long rendered pages and screenshots are kept, and whether you can opt out.
• Pricing model. Per session, per minute or per page, and how retries and failed loads count.
• Protocol fit. Whether your existing Playwright or Puppeteer code connects over CDP or WebSocket without a rewrite.

Wait for the text, not the page

Many broken rendered scrapes are timing bugs. The three common mistakes:

1. Waiting for networkidle. It means "no network connections for 500 ms". Pages with analytics, polling or live updates may never reach it, or reach it too early. Playwright's docs label it discouraged.
2. Waiting for the container. On a client-rendered page #root exists before any content is inside it. A selector wait on it returns at once.
3. Fixed sleeps. sleep(5) is slow on fast pages and too short on slow ones.

A better pattern waits for the content itself:

import { chromium } from "playwright";

const browser = await chromium.launch();
const context = await browser.newContext();
await context.route("**/*.{png,jpg,jpeg,webp,gif,woff,woff2}", (r) => r.abort());
const page = await context.newPage();

await page.goto("https://example.com/", { waitUntil: "domcontentloaded" });
// On example.com the text sits in <body>; on your page, use the element that holds it
await page.locator("body p").first().waitFor({ timeout: 15_000 });
const text = await page.locator("body").innerText();

await browser.close();

Blocking images and fonts reduces memory and bandwidth without changing the text. Set an explicit timeout and treat a timeout as a failure, not as "page had no content". innerText still includes menus and footers; the guide to turning the rendered DOM into clean text covers the extraction step.

Keeping the bill down

• Filter first. Run a static fetch on every URL and send only the failures to the browser. The share of URLs that truly need rendering varies by source; measure it on your own list.
• Parse embedded data. If a page ships __NEXT_DATA__ or JSON-LD, read that instead of rendering. See the guide to data already in the HTML.
• Reuse contexts. One browser with many contexts is cheaper than many browsers.
• Cache rendered output for pages that change slowly.

Responsible use

Render public pages only. Respect robots.txt, the site's terms and reasonable request rates. A headless browser is not a licence to ignore a site's rules, and the approaches on this site do not cover logins, paywalls or CAPTCHAs.

Test it yourself

Pick a client-rendered page and time two runs: one with waitUntil: "networkidle", one with domcontentloaded plus a wait for the element that holds the text. Compare the duration and check that both returned the same text. Then repeat with images and fonts blocked and compare memory in your process monitor.