# SiteRender: guides and posts

> Before you render a JavaScript page, check what the server sends. Guides on raw HTML, SSR, hydration data and when a headless browser is worth it.

# Do I need a headless browser? A one-minute test first

Source: https://siterender.cc/guides/do-you-need-a-headless-browser

> Check what the server sends before you start Chrome. Four cheap tests show whether a page needs a headless browser or a plain fetch already has the text.

The short answer: you need a headless browser only when the text you want exists solely after JavaScript runs in a browser, and the page does not already ship that text as data inside its HTML. Article, documentation and product pages often do not meet that bar. The test below takes about a minute per site and saves you from both common mistakes: starting Chrome for every URL, and trusting an empty response.

## Why the question matters

A browser and a plain HTTP client see different things. When you open a page, the browser downloads the HTML, runs the scripts, calls APIs and builds the DOM you see. curl, Python `requests` or a `fetch()` call stop after step one: they get the HTML the server sent, and nothing else.

On a server-rendered page the two versions carry the same text. On a client-rendered single-page app, the first response may hold nothing but a mount point:

```html
<body>
  <div id="root"></div>
  <script src="/assets/app.4f2c1.js"></script>
</body>
```

That page needs JavaScript. A plain fetch returns HTTP 200 and no content. The danger is that your code treats the 200 as success.

## Test 1: read the raw response

Fetch the page without a browser and look for a sentence you can see on screen.

```bash
URL=https://example.com/
curl -sL "$URL" -o raw.html
grep -oiE '<p[ >]' raw.html | wc -l         # <p> tags in the raw HTML
grep -i "a sentence you can see on the page" raw.html
```

In a browser, use `view-source:https://example.com/`, not Inspect. Inspect shows the live DOM after scripts changed it, which is exactly what a fetcher does not get. The difference between the two is the topic of the [SPA, SSR and hydration guide](/guides/render-spa-to-html).

If the sentence is there, stop. A static fetch can read this page.

## Test 2: look for data the page already ships

Many client-rendered pages still send their content, just not as visible HTML. Search for the common payloads:

```bash
grep -oE '__NEXT_DATA__|self\.__next_f|__NUXT_DATA__|application/ld\+json|og:title|application/rss\+xml' raw.html | sort | uniq -c
```

- `__NEXT_DATA__` (Next.js Pages Router) and `__NUXT_DATA__` (Nuxt 3) hold the data the page will render, as JSON.
- `self.__next_f` marks the React Server Components payload of the Next.js App Router.
- `application/ld+json` is structured data: articles, products, FAQs.
- `og:` tags give a title, description and image for nearly any page.
- An RSS or Atom link can replace scraping a listing page entirely.

If the text you need is in one of these, parse it. It is faster than rendering and usually cleaner than the visible page. The [guide to data already in the HTML](/guides/data-already-in-the-html) covers each format.

## Test 3: reload with JavaScript off

Chrome DevTools can disable JavaScript for the open tab: open the Command Menu (Ctrl+Shift+P or Cmd+Shift+P), type "Disable JavaScript" and reload. Whatever disappears is the content that only exists after scripts run. This is the visual version of Test 1 and is useful when you are not sure which sentence to search for.

Also check the `<noscript>` element. Some sites put a text fallback there; most only say "You need to enable JavaScript to run this app", which confirms client rendering.

## Test 4: render only what is left

If the content is not in the raw HTML and not in an embedded payload, a browser is the right tool. Two habits keep rendered fetches reliable:

- Wait for the text, not the container. On a client-rendered page the `#root` element exists before its content arrives, so waiting for it returns at once with nothing inside.
- Avoid waiting for network idle as the default. Pages that poll or stream never go quiet. Playwright's docs mark `networkidle` as discouraged for this reason.

The [headless browser guide](/guides/headless-browser-api-explained) compares Playwright, Puppeteer, Lightpanda and hosted browser APIs, with their cost drivers.

## Detecting blank shells in code

For lists of URLs, turn the tests into a check that runs before content reaches a model or a database:

```python
import re

def looks_like_empty_shell(html: str) -> bool:
    paragraphs = len(re.findall(r"<p[\s>]", html, re.I))
    mount = re.search(r'<div id="(root|app|__next)"[^>]*>\s*</div>', html)
    has_payload = re.search(r"__NEXT_DATA__|__NUXT_DATA__|application/ld\+json", html)
    return paragraphs < 2 and bool(mount) and not has_payload
```

Treat the thresholds as a starting point and tune them on your own sources. The point is to fail loudly: an empty shell should become an error that routes the URL elsewhere, not a blank document that an LLM summarises with confidence. For pages that pass the check, the [main-content extraction guide](/guides/extract-main-content-from-html) covers the step from HTML to clean text.

## Where a static fetch API fits

Live Web Data API, made by Teneo Labs, which runs this site, works the same way as Test 1: by default it does a static fetch of public pages and returns the main content as Markdown, text or HTML. When it finds no usable content, for example a page that needs JavaScript, it says so with a `THIN_CONTENT` warning or a `CRAWL_NOT_RENDERED` error (0 credits) instead of returning an empty shell. You can then resend that URL with `render: true`, or set `renderFallback: true` from the start so only the pages that fail the static fetch are rendered in a browser. Once your plan is active, you can [try a URL in the playground](/go/playground?from=post:do-you-need-a-headless-browser), or [read the error codes](/go/api-reference?from=post:do-you-need-a-headless-browser).

## Test it yourself

Pick three URLs from your own list: one article, one listing page and one logged-out app page. Run Test 1 and Test 2 on each. Write down which ones a static fetch reads, which ones ship JSON, and which ones truly need a browser. That split, not a vendor default, should decide how your pipeline fetches.

---

# Render SPA to HTML: what each rendering mode sends first

Source: https://siterender.cc/guides/render-spa-to-html

> CSR, SSR, SSG, streaming and islands from the reader's side: what lands in the first HTTP response and where Next.js, Nuxt and Astro keep the data.

"Render SPA to HTML" is asked from two sides. Site owners want their single-page app to send real HTML. Readers, the people fetching pages for pipelines and LLMs, want to know what they will receive. Framework docs explain rendering modes from the builder's side. This guide explains them from the reader's side: what each mode puts in the first HTTP response, and where the data hides when the text does not.

## The one question that matters

For a reader there is only one question: **is the text in the first response?** Everything else, including the framework name, is a hint about where to look.

```bash
curl -sL https://example.com/ | grep -oiE '<p[ >]' | wc -l
```

A high count means server-rendered text. A count near zero means either client rendering or a page made of something other than paragraphs. The next sections tell you which.

## Client-side rendering (CSR)

A classic SPA, for example a Vite or Create React App build, sends an HTML file with a mount point and script tags. The browser downloads the bundle, calls APIs and builds the DOM. The first response has no content.

What you can do as a reader:

- Check whether the site offers a documented public API or feed, and use it within the site's terms.
- Check for JSON-LD or `og:` tags in the head, which some SPAs still render on the server.
- Otherwise, render it in a headless browser; the [headless browser guide](/guides/headless-browser-api-explained) covers wait logic and cost.

Whatever you choose, check robots.txt and the site's terms of service first, and stay within the product's [acceptable use policy](/acceptable-use) if you use Live Web Data API.

## Server-side rendering (SSR)

The server runs the components and sends finished HTML for each request. The text is in the first response. The page then **hydrates**: the framework attaches event handlers in the browser, using a JSON copy of the data that the server embedded in the page.

That JSON is useful. It holds the same content as the visible text, structured, and you can parse it without running a script.

## Static generation (SSG) and prerendering

The same as SSR, but done at build time. The HTML is a file on a CDN. For readers it is the easiest case: text in the first response, often with hydration data alongside. Prerendering an existing SPA, for example with a framework's static export, turns a CSR page into this case.

## Streaming SSR and React Server Components

Streaming sends HTML in chunks as the server finishes each part. The text still arrives in the first response, but later chunks may sit at the end of the document inside inline `<script>` tags that move them into place. A fetcher that reads the whole response gets everything. One that stops at a size limit may miss the tail.

In the Next.js App Router, the React Server Components payload appears as calls to `self.__next_f.push(...)` in inline scripts. It is not plain JSON, but it contains the rendered text and props, which is why simple text searches often find content there.

## Islands

Astro and similar tools render the page to static HTML and only hydrate interactive parts, the islands. Text outside islands is plain HTML. An island that renders only on the client (`client:only` in Astro) is empty in the first response, and its props are serialised as attributes on the `<astro-island>` element.

## Where each framework keeps the data

| Framework | What to search for in the raw HTML |
| --- | --- |
| Next.js, Pages Router | `<script id="__NEXT_DATA__" type="application/json">` with page props |
| Next.js, App Router | `self.__next_f.push(` inline scripts (RSC payload) |
| Nuxt 3 | `<script id="__NUXT_DATA__" type="application/json">` |
| SvelteKit | data from `load` functions serialised into an inline script at the end of the body |
| Remix / React Router | a context object assigned in an inline script |
| Astro | `<astro-island>` elements with a `props` attribute |

These are implementation details and change between versions. Treat them as places to look, not as stable APIs, and re-check after a site redeploys.

## A quick classifier

```bash
URL=https://example.com/
HTML=$(curl -sL "$URL")
echo "$HTML" | grep -oiE '<p[ >]' | wc -l                     # server-rendered text?
echo "$HTML" | grep -oE '__NEXT_DATA__|self\.__next_f|__NUXT_DATA__|astro-island' | sort -u
echo "$HTML" | grep -oE '<div id="(root|app)"></div>'          # empty mount point?
```

Many paragraphs: SSR or SSG, read it directly. Few paragraphs plus a framework payload: parse the payload. Few paragraphs, a mount point and no payload: CSR, so a headless browser is needed, or a documented feed or API if the site publishes one. The [headless browser test](/guides/do-you-need-a-headless-browser) walks through the full decision.

## For site owners

If your own app falls in the CSR row and you want fetch tools and AI crawlers to read it, move the main text to SSR, SSG or prerendering. Google calls dynamic rendering, serving different HTML to bots, a workaround rather than a long-term fix. The [AI crawler guide](/guides/can-ai-crawlers-render-javascript) covers what those crawlers receive today.

## Test it yourself

Run the classifier on one page each from a docs site, a news site and a web app you use. Note the mode you think each one uses and whether the text was in the first response. If you find a framework payload, open it in a JSON viewer and check that it matches the visible text.

---

# Headless browser API explained: options, costs, wait logic

Source: https://siterender.cc/guides/headless-browser-api-explained

> Playwright, Puppeteer, Lightpanda, self-hosted headless Chrome or a hosted headless browser API: what each costs to run and how to wait for the right content.

A headless browser runs a real browser engine without a window, executes the page's JavaScript and lets your code read the resulting DOM. "Headless browser API" can mean the library you drive it with, such as Playwright or Puppeteer, or a hosted service that runs the browsers for you. This guide covers both, the main cost drivers, and the wait logic that decides whether you get text or an empty container.

> SiteRender is run by Teneo Labs, which makes Live Web Data API, a static fetch that renders in a browser only on request. Tools are described from their public docs as of October 2026.

First, the question this site always asks: does the page need a browser at all? If the text is in the raw HTML or in an embedded JSON payload, a plain fetch is faster and cheaper. The [one-minute test](/guides/do-you-need-a-headless-browser) answers that before you pick any of the tools below.

## The libraries

**Playwright** drives Chromium, Firefox and WebKit from one API, with bindings for TypeScript, Python, Java and .NET. Its locators wait automatically for elements to be attached and visible, which removes a lot of hand-written waiting.

**Puppeteer** is a Node.js library that controls Chrome over the Chrome DevTools Protocol, and Firefox over WebDriver BiDi. It is a close fit if you only need Chrome and want low-level protocol access.

**Lightpanda** is a newer headless browser written in Zig, built for machines rather than people. It exposes a CDP server, so Puppeteer can connect to it. Its README reports about 9x faster runs and about 16x less memory than headless Chrome on its own benchmark. That is a vendor benchmark; we have not reproduced it, and Web API coverage differs from Chrome, so test it on your own pages.

**Headless Chrome itself** changed in recent versions. Chrome's docs explain that headless and headful modes are now one implementation. The old headless mode was removed from the Chrome binary in version 132 and now ships separately as `chrome-headless-shell`. If an older script behaves differently after an upgrade, this is a likely reason.

## Self-hosted vs a hosted browser API

Running browsers yourself gives full control over versions, network location and data. It also makes you responsible for:

- **Memory.** Each page carries a full rendering engine and a JavaScript heap. Concurrency is usually limited by RAM before CPU.
- **Cold starts.** Launching a browser takes far longer than an HTTP request. Keep a pool of warm browsers and reuse contexts.
- **Crashes and leaks.** Pages hang, tabs leak memory, and processes need recycling.
- **Updates.** Browser versions and their dependencies change every few weeks.

A hosted headless browser API moves that work to a vendor and bills per session, per minute or per page. You trade control and data locality for less operations work. Compare vendors on their documented limits, regions, session length and what happens on timeouts, not on headline speed claims.

## How to choose a hosted headless browser API

If you decide to buy rather than run browsers, check each candidate against the same neutral criteria, from its own documentation:

- **Regions.** Where the browsers run, and whether you can pin a region close to your targets and your data.
- **Session limits.** Concurrent sessions per plan, maximum session length and idle timeouts.
- **Timeout behavior.** What the API returns when a page never finishes, and whether a timed-out session is billed.
- **Data handling.** What is logged, how long rendered pages and screenshots are kept, and whether you can opt out.
- **Pricing model.** Per session, per minute or per page, and how retries and failed loads count.
- **Protocol fit.** Whether your existing Playwright or Puppeteer code connects over CDP or WebSocket without a rewrite.

## Wait for the text, not the page

Many broken rendered scrapes are timing bugs. The three common mistakes:

1. **Waiting for `networkidle`.** It means "no network connections for 500 ms". Pages with analytics, polling or live updates may never reach it, or reach it too early. Playwright's docs label it discouraged.
2. **Waiting for the container.** On a client-rendered page `#root` exists before any content is inside it. A selector wait on it returns at once.
3. **Fixed sleeps.** `sleep(5)` is slow on fast pages and too short on slow ones.

A better pattern waits for the content itself:

```typescript
import { chromium } from "playwright";

const browser = await chromium.launch();
const context = await browser.newContext();
await context.route("**/*.{png,jpg,jpeg,webp,gif,woff,woff2}", (r) => r.abort());
const page = await context.newPage();

await page.goto("https://example.com/", { waitUntil: "domcontentloaded" });
// On example.com the text sits in <body>; on your page, use the element that holds it
await page.locator("body p").first().waitFor({ timeout: 15_000 });
const text = await page.locator("body").innerText();

await browser.close();
```

Blocking images and fonts reduces memory and bandwidth without changing the text. Set an explicit timeout and treat a timeout as a failure, not as "page had no content". `innerText` still includes menus and footers; the guide to [turning the rendered DOM into clean text](/guides/extract-main-content-from-html) covers the extraction step.

## Keeping the bill down

- **Filter first.** Run a static fetch on every URL and send only the failures to the browser. The share of URLs that truly need rendering varies by source; measure it on your own list.
- **Parse embedded data.** If a page ships `__NEXT_DATA__` or JSON-LD, read that instead of rendering. See the [guide to data already in the HTML](/guides/data-already-in-the-html).
- **Reuse contexts.** One browser with many contexts is cheaper than many browsers.
- **Cache rendered output** for pages that change slowly.

## Responsible use

Render public pages only. Respect robots.txt, the site's terms and reasonable request rates. A headless browser is not a licence to ignore a site's rules, and the approaches on this site do not cover logins, paywalls or CAPTCHAs.

## Test it yourself

Pick a client-rendered page and time two runs: one with `waitUntil: "networkidle"`, one with `domcontentloaded` plus a wait for the element that holds the text. Compare the duration and check that both returned the same text. Then repeat with images and fonts blocked and compare memory in your process monitor.

---

# Extract main content from HTML, raw or rendered

Source: https://siterender.cc/guides/extract-main-content-from-html

> How to get the article text out of raw HTML or a rendered DOM: Readability-style extraction, boilerplate removal, shadow DOM, iframes and lazy-loaded content.

Getting a page's HTML, raw or rendered, is half the job. The other half is getting the article out of it: the text a person came to read, without the navigation, cookie banner, footer and "related posts". This guide covers how main-content extraction works, where it fails, and the special cases that only show up on JavaScript-heavy pages.

## Raw HTML vs rendered DOM

Extraction works the same way on both. The difference is what is in the input:

- **Raw HTML** is what the server sent. On server-rendered pages it holds the whole article. It is cheap to get and the same on every fetch.
- **Rendered DOM** is the page after scripts ran in a headless browser. It is the only source for client-rendered content, but it can also include things added by scripts: consent dialogs, chat widgets and personalised blocks.

Use raw HTML whenever the text is there; the [one-minute test](/guides/do-you-need-a-headless-browser) tells you. Check the [data the page already ships](/guides/data-already-in-the-html) too: JSON-LD or a hydration payload can hold the article without any extraction. Render only when neither has it, using the wait patterns in the [headless browser guide](/guides/headless-browser-api-explained), and expect more noise in the result.

## How Readability-style extraction works

Mozilla's Readability.js, the code behind Firefox Reader View, is the reference most extractors resemble. Roughly:

1. Remove elements that are almost never content: scripts, styles, forms, and blocks whose class or id suggests comments, sidebars or ads.
2. Score candidate blocks by the amount of text, the number of commas and the link density. A block that is mostly links is probably navigation.
3. Pick the best-scoring block, add siblings that score close to it, and clean up what is left.

`parse()` returns the title, the cleaned HTML, the plain text, an excerpt, the byline and some metadata. Readability also has `isProbablyReaderable()`, a cheap pre-check that guesses whether extraction will work, with false positives and negatives.

Readability does not sanitise its output. If you display the result in a browser, run it through a sanitiser such as DOMPurify first.

## Boilerplate removal that keeps structure

The same rules apply to raw HTML and to a rendered DOM, but the rendered DOM usually has more to drop. Keep the structure that carries meaning and drop the rest:

| Keep | Drop |
| --- | --- |
| Headings, paragraphs, lists | Navigation, breadcrumbs, footers |
| Tables and code blocks | Cookie and newsletter banners |
| Image alt text and captions | Share buttons, "related" blocks |
| Links inside the article | Ads and tracking markup |

Converting the kept part to Markdown preserves headings, lists and tables in a compact form. Plain text loses them, which matters for documentation and reference pages.

## Cases that break extraction

**Shadow DOM.** Web components can render their content inside a shadow root. In a rendered DOM, `innerHTML` of the host element does not include it, and a naive serialiser misses it. Playwright locators pierce open shadow roots; closed shadow roots are not reachable from page scripts at all. Declarative shadow DOM (`<template shadowrootmode="open">`) does appear in raw HTML, inside the template.

**Iframes.** Embedded content lives in its own document. Raw HTML only contains the `<iframe src>`; you fetch that URL separately if you need it. In a [headless browser](/guides/headless-browser-api-explained) you access it through the frame, and cross-origin frames are separate pages with their own terms.

**Lazy loading.** Images with `loading="lazy"` still have their `src` in the HTML, so alt text and URLs survive a static fetch. Sections that load on scroll through JavaScript do not. In a browser, scroll in steps and wait for new content; on a static fetch, look for a paginated or feed URL instead.

**Very short articles.** Extractors may treat a short page as boilerplate and return nothing. Set a minimum length and fall back to the `<main>` or `<article>` element.

## Fail loudly on empty output

The most expensive extraction bug is not a crash. It is an empty or near-empty result that your code accepts. Add a check before the text goes anywhere:

```python
def usable(text: str, min_chars: int = 200) -> bool:
    words = text.split()
    return len(text) >= min_chars and len(words) >= 30
```

Tune the threshold per source. Anything below it becomes an error with the URL attached, not a blank document in an index or a prompt.

## A static fetch with extraction built in

Live Web Data API, made by Teneo Labs, which runs this site, does the static fetch and the main-content step in one call and returns Markdown, text or HTML. When a static fetch finds no usable content, for example on a page that needs JavaScript, it returns a `CRAWL_NOT_RENDERED` error at 0 credits instead of an empty result; with `render` or `renderFallback` the page is read in a browser before extraction. The [API docs](/go/docs?from=post:extract-main-content-from-html) list the output formats and options.

## Test it yourself

Take one server-rendered article and one client-rendered page. Run Readability on the raw HTML of both, then on the rendered DOM of the second. Compare the three outputs side by side: length, headings kept, and anything that looks like navigation or a consent banner.

---

# Structured data in HTML: read it before you render

Source: https://siterender.cc/guides/data-already-in-the-html

> JSON-LD, Open Graph, microdata, hydration payloads and RSS feeds often hold a page's content already. How to find them and use them instead of rendering.

Before you start a browser to read a JavaScript-heavy page, check whether the page already sent you its content in another form. Many client-rendered pages ship their text as JSON inside the HTML, and many pages carry some structured metadata in their `<head>`. Reading that data is faster than rendering, cheaper, and usually cleaner than the visible text.

## Find everything in one pass

```bash
URL=https://example.com/
curl -sL "$URL" -o raw.html
grep -oE 'application/ld\+json|property="og:[a-z:_]+"|itemscope|__NEXT_DATA__|__NUXT_DATA__|application/(rss|atom)\+xml' raw.html | sort | uniq -c
```

The counts tell you what is there. The sections below explain each one.

## JSON-LD

JSON-LD is schema.org data in a `<script type="application/ld+json">` block. Sites add it for search engines, which means it often holds exactly what a reader wants: an `Article` with headline, author and dates, a `Product` with price and availability, a `FAQPage` with questions and answers, an `Event` with time and place.

```python
import json, re, requests

html = requests.get("https://example.com/", timeout=30).text
blocks = re.findall(r'<script[^>]+application/ld\+json[^>]*>(.*?)</script>', html, re.S)
for raw in blocks:
    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        continue
    items = data if isinstance(data, list) else data.get("@graph", [data])
    for item in items:
        print(item.get("@type"), item.get("headline") or item.get("name"))
```

Handle the three common shapes: a single object, a list, and an object with `@graph`. Some sites have broken JSON-LD; skip blocks that do not parse.

## Open Graph tags

Open Graph `<meta property="og:...">` tags give a title, description, image, URL and type on most pages that want link previews, because social networks read them for those previews. They will not give you an article body, but they are a reliable fallback for titles and summaries, and they are usually server-rendered, even on many SPAs.

## Microdata

Microdata puts the same schema.org vocabulary into attributes on visible elements: `itemscope`, `itemtype` and `itemprop`. It is older and less common than JSON-LD but still appears on product and review pages. Libraries such as `extruct` in Python read microdata, JSON-LD and Open Graph in one call.

## Hydration payloads

Server-rendered framework pages embed a JSON copy of their data so the client can take over. In Next.js (Pages Router) it is the `__NEXT_DATA__` script, in Nuxt 3 the `__NUXT_DATA__` script. For a reader this is often the best source on the page: the full content, already structured, no extraction heuristics needed.

The catch: these payloads are internal to the framework. Field names change when the site changes, so validate the shape and alert when it breaks. The [SPA, SSR and hydration guide](/guides/render-spa-to-html) lists where each framework keeps its data.

## RSS and Atom feeds

Look for `<link rel="alternate" type="application/rss+xml">` in the head. A feed lists recent items with title, link, date and a summary or full text. For "what is new on this site", a feed is more stable than any page layout and does not need rendering. Common paths such as `/feed`, `/rss.xml` or `/atom.xml` are worth a try when the link tag is missing.

## Which source to try first

When a page offers several of these, pick by what you need:

- **New items on a site:** the feed. It is built for programs and changes least.
- **One article's facts** (headline, dates, author): JSON-LD, then Open Graph.
- **Product or event details:** JSON-LD, then microdata.
- **The full body of a client-rendered page:** the hydration payload, with a schema check.
- **A title and summary for a link preview:** Open Graph alone is enough.

Keep the visible text as the reference. If a structured source and the page disagree, the page wins, and the mismatch is worth logging.

## When embedded data is not enough

Structured data is a summary written for a purpose. It may lack the full article body, lag behind the visible page, or describe only part of it. Use it when it answers your question, and fall back to the visible text, from the raw HTML or, if needed, a [rendered page](/guides/do-you-need-a-headless-browser), when it does not.

Stay inside the line between public and private, too. Hydration JSON and API responses sometimes include fields the page does not show. Use what the page is clearly publishing, and respect the site's terms and robots.txt.

## Test it yourself

Run the one-pass search on five pages from different sites. For each, note which formats are present and whether any of them contains the text you actually need. If one does, write the parser for that format first and keep rendering as the fallback.

---

# Can AI crawlers render JavaScript? What GPTBot sees

Source: https://siterender.cc/guides/can-ai-crawlers-render-javascript

> Most AI crawlers do not run JavaScript, so client-only pages look empty to them. Check what they receive and fix it with SSR or prerendering.

If your site renders its content in the browser, most AI crawlers and LLM fetch tools see an empty page. That is the short answer to "can AI crawlers render JavaScript": with a few exceptions, they do not. This guide shows what they receive, how to check your own pages in a minute, and the framework-level fixes that make content readable without JavaScript.

## What the evidence says

The most cited data comes from Vercel's analysis of crawler traffic across its network, published in December 2024. Its findings:

- None of the major AI crawlers it measured executed JavaScript. That list included OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, PerplexityBot, Meta-ExternalAgent, Bytespider and Common Crawl's CCBot.
- ChatGPT's and Claude's crawlers did download JavaScript files. They just did not run them.
- The exceptions were Gemini, which uses Googlebot's rendering infrastructure, and AppleBot, which renders through a browser-based crawler.

Crawler behaviour changes, and this is one company's measurement from late 2024. Check each operator's own documentation, such as OpenAI's crawler overview, for current user agents. The practical conclusion has not changed: content that exists only after client-side JavaScript runs is invisible to most AI fetchers.

## See your page the way they do

A plain HTTP request is a close approximation of what a non-rendering crawler receives.

```bash
URL=https://example.com/
curl -sL -A "Mozilla/5.0 (compatible; render-check)" "$URL" -o raw.html
grep -oiE '<p[ >]' raw.html | wc -l
grep -o '<title>[^<]*' raw.html
grep -oE '<h[12][^>]*>[^<]+' raw.html | head
```

Then open `view-source:` on the same URL and compare it with what you see on screen. Missing headings, prices, descriptions or article text are the parts an AI crawler cannot read. The [one-minute test](/guides/do-you-need-a-headless-browser) covers the same checks from the reader's side.

## Fix it in the framework

The fix is to put the main content in the first HTTP response. How depends on the framework:

| Stack | Fix |
| --- | --- |
| React SPA (Vite, CRA) | Move to a framework with SSR or SSG, or prerender routes at build time |
| Next.js | Keep content in Server Components; avoid fetching main content in client-only effects |
| Nuxt | Use the default universal rendering; avoid `ssr: false` for content routes |
| SvelteKit | Keep SSR on and load content in `load` functions |
| Angular | Use server-side rendering and prerendering for content routes |
| Astro | Static by default; avoid `client:only` for content |

Interactive parts, such as view counters, chat widgets and personalised blocks, can stay client-side. What matters is that the text, headings, links and metadata are in the HTML.

## Why not dynamic rendering?

Dynamic rendering serves a prerendered snapshot to bots and the normal SPA to people. Google's documentation calls it "a workaround and not a long-term solution" and recommends server-side rendering, static rendering or hydration instead. It also adds a second rendering path that can drift from what visitors see. Use it only as a stopgap while you move to SSR or static generation.

## Metadata that should always be server-rendered

Even pages that keep a client-rendered body should send these in the raw HTML:

- `<title>` and a meta description
- Open Graph tags for title, description and image
- A canonical link
- JSON-LD for the page type, such as Article, Product or FAQPage

The [structured data guide](/guides/data-already-in-the-html) explains how fetchers read these formats.

## If you want the opposite

Some site owners want AI crawlers to stay away. The standard mechanism is robots.txt, with rules for each crawler's published user agent. RFC 9309 defines the format. It is a request that well-behaved crawlers honour, not an access control, so pair it with server-side measures if that matters to you.

## Test it yourself

Add the curl check to your CI for your three most important pages: fail the build if the paragraph count drops below a threshold or the main heading is missing. That catches the most common regression, a refactor that moves content into a client-only component, before a crawler does.

---

# Reading Next.js hydration data without a browser

Source: https://siterender.cc/blog/nextjs-hydration-data

> Next.js pages often ship their content as JSON in the first response. Where to find __NEXT_DATA__ and the App Router payload, and how to parse them safely.

A Next.js page that looks empty to a plain fetch often is not empty at all. When the page is rendered on the server or at build time, Next.js embeds the data it used, so the browser can hydrate without asking again. For a reader, that embedded data is often the cleanest source on the page.

Where it lives depends on the router.

## Pages Router: `__NEXT_DATA__`

Pages that use the Pages Router, including those with `getServerSideProps` or `getStaticProps`, include a script tag like this:

```html
<script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{...}},"page":"/blog/[slug]", ...}</script>
```

The Next.js docs describe how `getServerSideProps` returns props that the page renders with; those props end up in this script. It is plain JSON, so any language can parse it:

```python
import json, re, requests

html = requests.get("https://example.com/", timeout=30).text
m = re.search(r'<script id="__NEXT_DATA__" type="application/json"[^>]*>(.*?)</script>', html, re.S)
if not m:
    raise SystemExit("no __NEXT_DATA__ on this page")
data = json.loads(m.group(1))
props = data.get("props", {}).get("pageProps")
if not props:
    raise SystemExit("payload found, but pageProps is empty")
print(sorted(props.keys()))
```

example.com is not a Next.js site, so this prints "no __NEXT_DATA__ on this page". Point it at a Pages Router page you are allowed to read and the keys show what the page renders.

## App Router: the RSC payload

App Router pages use React Server Components. The Next.js docs explain that the server renders Server Components into a special data format, the RSC payload, which the client uses to build the page. In the HTML it appears in inline scripts:

```bash
curl -sL https://example.com/ | grep -c 'self.__next_f.push'
```

The payload is a stream of lines, not one JSON document, so there is no stable way to parse it fully. In practice, two things work: the visible text is usually already in the server-rendered HTML, so read that first; and string fields such as titles are often findable with a targeted search. Treat anything more as fragile.

## Rules that keep this safe

- **It is an implementation detail.** Field names follow the site's code, not a public API. Validate the shape and alert when it changes.
- **Use what the page publishes.** Hydration data can include fields the page does not show. Read what the page displays, ignore anything that looks private, and respect robots.txt and the site's terms.
- **The page wins.** If the payload and the visible text disagree, trust the text and log the mismatch.

The [SPA, SSR and hydration guide](/guides/render-spa-to-html) lists where Nuxt, SvelteKit, Remix and Astro keep the same kind of data.

## Test it yourself

Run the regex above on a Next.js page you use. If it finds `pageProps`, compare three fields with what you see on screen. If it finds nothing, check for `self.__next_f` and then for text in the plain HTML before you decide the page needs a browser.

---

# Scraper returns empty HTML? Tell the four causes apart

Source: https://siterender.cc/blog/scraper-returns-empty-html

> An empty or near-empty response has four common causes: an app shell, a refusal page, compressed bytes and a soft 404. How to tell which one you have.

"The scraper returns empty HTML" is one log line with at least four different causes behind it. They need four different fixes, and one of them is not a fix at all but a signal to stop. Before you change your fetcher, find out which one you have.

Start by saving the raw response and its headers:

```bash
URL=https://example.com/
curl -sL -D headers.txt "$URL" -o raw.html
head -n 1 headers.txt                  # status line
grep -i '^content-encoding' headers.txt
wc -c raw.html                         # body size in bytes
```

## Cause 1: the page is built in the browser

Status 200, a body of a few kilobytes, and very little text. Look for an empty mount point and count the paragraph tags:

```bash
grep -oE '<div id="(root|app|__next)"></div>' raw.html
grep -oiE '<p[ >]' raw.html | wc -l
```

An empty `<div id="root">` with almost no `<p>` tags means client-side rendering: the HTML is a shell, and scripts fetch and build the content later. web.dev's "Rendering on the Web" describes this pattern and its trade-offs. Before you render, search the same file for `__NEXT_DATA__`, `__NUXT_DATA__` or `application/ld+json`. The [one-minute test](/guides/do-you-need-a-headless-browser) walks through the full decision.

## Cause 2: the site refused the request

A 403 or 429 status, or a short page that asks you to wait, sign in or confirm something. MDN defines 403 as a server that understood the request and refuses it, and 429 as too many requests in a given time.

This is not an extraction bug. Slow down, honor any `Retry-After` header, check robots.txt and the site's terms, and drop the URL if the site does not want automated access. Do not try to get around it.

## Cause 3: compressed bytes that were never decoded

The body looks like binary noise, or your HTML parser finds nothing. Check the `Content-Encoding` header: `gzip`, `br` or `zstd` means the body must be decompressed first. MDN's Content-Encoding page lists the values. Most HTTP clients do this for you; hand-rolled socket code and some proxies in the middle do not. With curl, `--compressed` asks for a compressed body and decodes it.

## Cause 4: a soft 404 or the wrong URL

Status 200, but the page says "not found", or the redirect chain ended on a home page or a consent page. Print the final URL and look at the title:

```bash
curl -sL -o /dev/null -w '%{url_effective}\n' "$URL"
grep -o '<title>[^<]*' raw.html
```

Fix the URL list, not the fetcher.

## Make the cause visible in your pipeline

All four causes share one dangerous property: if your code only checks for status 200, three of them pass as success, and an LLM downstream summarises a page it never received. Record a reason for every URL that comes back without usable text, and route by reason: render, back off, decode or drop.

Live Web Data API, made by Teneo Labs, which runs this site, separates these cases with explicit error tags: `CRAWL_NOT_FOUND`, `SOURCE_NOT_AVAILABLE` and `CRAWL_LIVECRAWL_TIMEOUT`, plus `CRAWL_NOT_RENDERED`, which is designed to be returned when a page has no usable content, for example one that needs JavaScript. Errors cost 0 credits. It does a static fetch by default; set `renderFallback: true` and only the pages that come back empty are rendered in a browser. Once your plan is active, you can [try a URL in the playground](/go/playground?from=post:scraper-returns-empty-html).

## Test it yourself

Take ten URLs that came back empty last week. Run the four checks on each and write the cause next to the URL. If most of them are app shells, read the hydration guide next; if most are refusals, the fix is your request rate, not your parser.

---

# View source vs Inspect element: the one that matches curl

Source: https://siterender.cc/blog/view-source-vs-inspect-element

> View source shows the HTML the server sent. Inspect shows the live DOM after scripts ran. Only one of them is what a plain HTTP fetch receives.

Open any page, then compare two views of it: `view-source:` and the Elements panel in DevTools. On many pages they look alike. On a JavaScript-heavy page they can have almost nothing in common. Knowing which view matches your fetcher decides whether a plain HTTP request is enough.

## What each view shows

**View source** (`view-source:https://example.com/`, or Ctrl+U / Cmd+Option+U) shows the HTML document the server returned. It does not change while you look at it.

**Inspect element** opens the Elements panel, which shows the DOM: the browser's live tree of nodes, built from that HTML and then changed by scripts. Chrome's DevTools docs describe the panel as a view of the DOM that you can also edit, and MDN's introduction explains that scripts read and change the DOM after the document is parsed.

So the rule is short: view source is what was sent; Inspect is what was built.

## Which one your tools see

- **curl, Python `requests`, `fetch()` in Node:** the same as view source. They receive the HTML and stop.
- **Playwright, Puppeteer, any headless browser:** the same as Inspect, once scripts have run and you waited for the right element.
- **Most AI crawlers:** the same as view source, because most do not run JavaScript. The [AI crawler guide](/guides/can-ai-crawlers-render-javascript) has the details.

## A quick comparison you can repeat

```bash
URL=https://example.com/
curl -sL "$URL" -o raw.html
grep -oiE '<p[ >]' raw.html | wc -l      # paragraphs in what was sent
```

Then in DevTools, open the Console on the same page and run:

```js
document.querySelectorAll("p").length    // paragraphs in what was built
```

On example.com both numbers match, because the page has no scripts that add content. On a client-rendered app the first number is near zero and the second is not. That gap is the content a static fetch cannot read.

## When the views differ for other reasons

Not every difference means client rendering:

- **Browser fixes to invalid HTML.** The parser inserts missing `<tbody>` or closes open tags, so the DOM has elements the source does not.
- **Personalisation.** Cookies, language and location can change what the server sends to your browser and to curl.
- **Extensions.** Browser extensions add nodes to the DOM. Check in a clean profile.

## Test it yourself

Pick one page you fetch in production. Search view source for a sentence you can read on screen. If it is there, a static fetch reads that page. If it is not, follow the [one-minute test](/guides/do-you-need-a-headless-browser) to see whether the text is shipped as JSON before you start a browser.

---

# Live Web Data API — full documentation

# Overview

Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network.

## What it does

You send up to 10 URLs (and optionally a country). We fetch each page through a residential or mobile connection, extract the main content and return it in the format your agent or pipeline needs. Successful pages cost 1 credit; failures cost nothing.

| | Value |
| --- | --- |
| Base URL | `https://livewebdata.com/api` |
| Endpoints | `POST /v1/contents`, `GET /v1/account` |
| Formats | markdown and text by default; html with `html: true` |
| OpenAPI | [`/openapi.json`](https://siterender.cc/openapi.json) · [`/openapi.yaml`](https://siterender.cc/openapi.yaml) |
| Fetch strategy (v1) | static fetch (`fast`), a retry through another connection (`retry`), or a real browser (`browser`) with `render` or `renderFallback` |

## Authentication

Create a key in the dashboard under [API keys](https://siterender.cc/dashboard/keys) and send it as a bearer token. Keys start with `1337_live_` (test keys with `1337_test_`) and are shown only once. Keep them in an environment variable such as `LIVE_WEB_DATA_API_KEY` — never in client-side code or repositories.

```bash title="Headers"
Authorization: Bearer 1337_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
```

## Quickstart

Fetch a page as Markdown:

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"]}'
```

The response contains one entry in `results` per successful page and one entry in `statuses` per requested URL. The page HTML is included only when you send `"html": true`:

```json title="Response (abridged)"
{
  "requestId": "req_01J…",
  "results": [{
    "id": "https://example.com/article",
    "url": "https://example.com/article",
    "title": "Article",
    "markdown": "# Article\n\n…",
    "text": "Article\n\n…"
  }],
  "statuses": [{
    "id": "https://example.com/article",
    "status": "success",
    "source": "livecrawled",
    "fetchedAt": "2026-10-23T09:15:02.118Z",
    "httpStatusCode": 200,
    "strategy": "fast",
    "credits": 1,
    "error": null
  }],
  "searchTime": 1240
}
```

When `text` or `markdown` was cut at `text.maxCharacters`, `results[].textTruncated` is `true`. `results[].warnings` (absent when empty) flags pages that could only be partly retrieved (`PAGE_TRUNCATED`), probably need `render: true` (`THIN_CONTENT`), were converted as plain text (`EXTRACTION_FALLBACK`) or lost their `html` (and maybe part of their text) to the response's 24 MiB size limit (`RESPONSE_SIZE_LIMIT`).

See the [API reference](https://siterender.cc/docs/api-reference) for every parameter, or try it without code in the dashboard [playground](https://siterender.cc/dashboard/playground).

## SDKs

Thin, typed clients for JavaScript/TypeScript and Python with built-in retries and typed errors. See [SDKs](https://siterender.cc/docs/sdks).

```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";

const client = new Teneo({
  apiKey: process.env.LIVE_WEB_DATA_API_KEY,
  baseUrl: "https://livewebdata.com/api",
});

const res = await client.contents({
  urls: ["https://example.com/article"]
});
for (const s of res.statuses) console.log(s.id, s.status, s.error?.tag ?? "");
console.log(res.results[0]?.markdown);
```

```python title="Python SDK"
import os
from teneo import Teneo

client = Teneo(
    api_key=os.environ["LIVE_WEB_DATA_API_KEY"],
    base_url="https://livewebdata.com/api",
)

res = client.contents(
    urls=["https://example.com/article"],
)
for s in res["statuses"]:
    print(s["id"], s["status"], (s["error"] or {}).get("tag", ""))
print(res["results"][0]["markdown"] if res["results"] else "")
```

## Live fetching

Every request is fetched live; there is no cache.

## Geotargeting

Set `countryCode` to an ISO-3166-1 alpha-2 code (for example `de`, `jp`, `br`) to fetch the page as seen from that country. The default is `us`.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/pricing"
  ],
  "countryCode": "de"
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/pricing"],
        "countryCode": "de"
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/pricing"
    ],
    "countryCode": "de"
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Multiple URLs

Send up to 10 URLs in one request. They are processed in parallel; `statuses` has one entry per URL (matched by `id`) and credits are charged per successful URL. A restricted URL gets a per-URL `RESTRICTED_TARGET` status instead of failing the whole request. More than 10 URLs return `400 TOO_MANY_URLS`. Rate limits count URLs, not HTTP requests.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/blocked"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/blocked"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Restricted targets

Some domain categories are blocked for legal and regulatory reasons. Requests are rejected with `451` and cost no credits. Repeated attempts lead to automatic key suspension. Report abuse to abuse@livewebdata.com.

## MCP for agents

Agents can use the API as a tool over MCP with your existing key — two tools, `get_page_contents` and `get_account_status`. Setup for Claude Code, Cursor, VS Code and Claude Desktop: [MCP for agents](https://siterender.cc/docs/mcp).

## For AI coding agents

Every docs page is also available as plain Markdown by appending `.md` to its URL (e.g. [`/docs/api-reference.md`](https://siterender.cc/docs/api-reference.md)). An index for LLMs lives at [`/llms.txt`](https://siterender.cc/llms.txt) and the full docs in one file at [`/llms-full.txt`](https://siterender.cc/llms-full.txt).


---

# API reference

Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome.

## Endpoint

```bash title="Request"
POST https://livewebdata.com/api/v1/contents
Authorization: Bearer <api_key>
Content-Type: application/json
```

Machine-readable specs:

- OpenAPI 3.1 (JSON): https://siterender.cc/openapi.json
- OpenAPI 3.1 (YAML): https://siterender.cc/openapi.yaml
- Postman collection: https://siterender.cc/live-web-data-api.postman_collection.json
- Interactive reference: https://siterender.cc/docs/api-reference/interactive

## Request parameters

| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `urls` | `string[]` | yes | — | The pages to fetch, 1–10 http or https URLs of at most 8,192 characters each. Processed in parallel; one entry per URL in results/statuses. Duplicates (same URL, ignoring #fragment) are fetched once; repeated positions cost 0 credits. |
| `countryCode` | `string` | no | "us" | ISO-3166-1 alpha-2 code. The page is fetched through a connection in this country. |
| `text` | `boolean \| object` | no | true | Content options. Pass an object to set maxCharacters and verbosity. |
| `text.maxCharacters` | `integer` | no | 100000 | Truncates text and markdown to this many characters (max 1,000,000); `results[].textTruncated` tells you when it did. The MCP tool defaults to 8,000 (`max_characters`). |
| `text.verbosity` | `"compact" \| "full"` | no | "compact" | compact = main content only. full = whole body incl. navigation and footer. |
| `html` | `boolean` | no | false | Include the page HTML as served in results[].html. A response carries at most 24 MiB of JSON: past that, the remaining results come without html (warning RESPONSE_SIZE_LIMIT). |
| `livecrawlTimeout` | `integer (ms)` | no | 30000 (60000 with render) | Maximum time for a live fetch, 1–90000 ms. A lower value is raised to 30000, the time the browser fallback needs, unless you send renderFallback: false (then any value, default 10000). With render or renderFallback: true at least 30000. |
| `render` | `boolean` | no | false | Render every URL in a real browser, as a normal visitor sees it: 3 credits, 5 when the page needed a retry through another connection. Cannot be combined with renderFallback. |
| `renderFallback` | `boolean` | no | on | Fetch normally first and render in a real browser only when the first attempt did not return the page, shows no or very little visible text, or was blocked. On by default: omit it. Send false for a plain fetch only (never a browser). A normal success costs its normal 1 or 2 credits, a rendered one only the render credits; statuses[].fallbackUsed tells you when it rendered. Cannot be combined with render: true. |

```json title="Example body"
{
  "urls": ["https://example.com/article"],
  "countryCode": "de",
  "text": { "maxCharacters": 50000, "verbosity": "compact" },
  "livecrawlTimeout": 45000
}
```

Unknown fields are rejected with `400 INVALID_REQUEST`; the message names the field. So is a URL longer than 8,192 characters (`urls.0: Each URL may be at most 8,192 characters.`).

## Response

| Field | Type | Description |
| --- | --- | --- |
| `requestId` | `string` | Id of this request, the same for every repeat with the same Idempotency-Key. Also returned as X-Request-Id. |
| `results[]` | `object[]` | One entry per successfully fetched URL. |
| `results[].url` | `string` | Final URL after redirects. |
| `results[].title` | `string \| null` | Page title. |
| `results[].author / publishedDate / image` | `string \| null` | Metadata when the page exposes it. |
| `results[].favicon` | `string \| null` | Favicon URL. |
| `results[].markdown` | `string` | Main content as Markdown. |
| `results[].text` | `string` | Plain text, tags removed, whitespace normalized. |
| `results[].html` | `string` | Page HTML as served. Only with html: true, and only while the response stays within its 24 MiB size limit (else warning RESPONSE_SIZE_LIMIT). |
| `results[].textTruncated` | `boolean` | true when text or markdown was cut at text.maxCharacters, or at the response size limit (warning RESPONSE_SIZE_LIMIT). Present whenever text is enabled. |
| `results[].warnings` | `{ code, message }[]` | Absent when empty. "PAGE_TRUNCATED" = only part of the page could be retrieved; content after that point is missing. "THIN_CONTENT" = very little text, the page probably needs render: true. "EXTRACTION_FALLBACK" = the page was too complex to convert in time; text is plain and markdown has no structure. "RESPONSE_SIZE_LIMIT" = the response reached its 24 MiB size limit: this result has no html, or no html and cut text/markdown. |
| `statuses[]` | `object[]` | One entry per requested URL, success or error. |
| `statuses[].status` | `"success" \| "error"` | Outcome for this URL. |
| `statuses[].source` | `"livecrawled" \| null` | Always "livecrawled" on success: every request is fetched live. null on error. |
| `statuses[].fetchedAt` | `string \| null` | When the page was fetched (ISO 8601). null on error. |
| `statuses[].strategy` | `"fast" \| "retry" \| "browser"` | Fetch strategy. "fast" = standard page (1 credit); "retry" = the first attempt did not return the page, a retry through another connection did (2 credits); "browser" = rendered in a real browser, as a normal visitor sees it (3 credits, 5 when it needed a retry through another connection). |
| `statuses[].credits` | `integer` | Credits charged for this URL: 1 standard, 2 retried, 3 rendered, 5 rendered + retried, 0 on error and for a short page (thin). |
| `statuses[].fallbackUsed` | `boolean` | Present (true) only when the browser fallback ran for this URL. With strategy "browser" the content is that render (fast → render); otherwise rendering was tried and did not help. |
| `statuses[].thin` | `boolean` | Present (true) only for a short page: fewer than 500 characters of visible text even after rendering. Delivered as a success at 0 credits, with only that visible text. |
| `statuses[].error` | `{ tag, message } \| null` | Per-URL error, see crawl errors. |
| `searchTime` | `integer (ms)` | Total processing time. |

Browser fallback is on by default: a page the fast fetch cannot read (no or very little visible text, blocked, 401/403, a timeout) is rendered in a real browser and charged only the render credits. `statuses[].fallbackUsed` is `true` when that happened (`strategy: "browser"` = the content is the render). Send `renderFallback: false` for a fast fetch only. A page that still shows fewer than 500 characters of visible text after rendering is delivered as a success with `statuses[].thin: true` at 0 credits, holding only that visible text — it is not an error.

`text.maxCharacters` (default 100,000, max 1,000,000) cuts `text` and `markdown`; `results[].textTruncated` is `true` when it did. A response carries at most 24 MiB of JSON: from the result that would pass it on, results come without `html`, and if that is not enough, `text` and `markdown` are cut (`textTruncated: true`), each such result flagged `RESPONSE_SIZE_LIMIT`. The request never fails because of it. `title` and `author` are at most 2,000 characters each; an `image` or `favicon` URL longer than 4,096 characters is left out (`favicon` then falls back to `/favicon.ico`). The MCP tool `get_page_contents` has its own, smaller default of 8,000 (`max_characters`), because its result goes into an agent's context. `results[].warnings` is present only when there is something to know about a page:

| Code | Meaning | What to do |
| --- | --- | --- |
| `PAGE_TRUNCATED` | Only part of the page could be retrieved (size limit or an interrupted transfer); content after that point is missing. | Nothing to retry; the result is the part that arrived. |
| `THIN_CONTENT` | Very little text next to scripts or an empty app root: the page probably loads its content with JavaScript. The URL is still charged as a success. | Retry with `render: true`. The default browser fallback renders a page with fewer than 500 characters of visible text by itself, not one with more. |
| `EXTRACTION_FALLBACK` | The page structure was too complex to convert in time; `text` is plain and `markdown` carries no structure. | Use `text`. |
| `RESPONSE_SIZE_LIMIT` | The response reached its 24 MiB size limit. This result and every later one come without `html`; when that was not enough, `text` and `markdown` were cut too (`textTruncated: true`). The URL is still a success and is charged. | Request fewer URLs at once, or without `html`. |

## Response headers

| Header | Description |
| --- | --- |
| `X-Request-Id` | Same as requestId in the body. |
| `X-Credits-Remaining` | Credit balance after this request. |
| `X-RateLimit-Limit` | URLs you can send at once (bucket size, = rateLimit.burst in GET /v1/account). The bucket refills at rateLimit.requestsPerSecond URLs per second. |
| `X-RateLimit-Remaining` | URLs you can send right now. |
| `Retry-After` | Seconds to wait: before retrying a 429, or before repeating a 202 with the same Idempotency-Key. |
| `Idempotency-Key` | The request's Idempotency-Key: yours, or one the API created when you sent none. On every answer (200, 202, errors) except a 400 for a malformed key and a 413 for a body over 10 MiB (refused unread). Repeat a 202 with it. |

## Errors

### Per-URL errors

Returned inside `statuses[].error` with HTTP 200. These never cost credits.

| Tag | Meaning |
| --- | --- |
| `CRAWL_NOT_FOUND` | The page returned 404/410. |
| `SOURCE_NOT_AVAILABLE` | The target refused access (e.g. 403 or 429) after retries. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The live fetch did not finish within livecrawlTimeout. |
| `CRAWL_NOT_RENDERED` | No usable content, e.g. the page needs JavaScript rendering (retry without renderFallback: false, or with render: true), or a rendered page shows no visible text. 0 credits. |
| `UNSUPPORTED_CONTENT_TYPE` | The URL does not serve HTML or text. |
| `CRAWL_UNKNOWN_ERROR` | Something else went wrong. Safe to retry. |
| `SOURCE_BLOCKED` | Access denied by the site, or it answered with an interstitial page (HTTP 202) instead of the content. 0 credits. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable; no credits were charged. |
| `RATE_LIMITED` | Rendering limit for your plan reached; no credits were charged. |
| `RESTRICTED_TARGET` | This URL is restricted. Other URLs in the same request are still processed. 0 credits. |

### Request errors

Returned with a non-200 status and no results.

| HTTP | Tag | Meaning |
| --- | --- | --- |
| 400 | `INVALID_REQUEST` | Invalid parameters. |
| 400 | `TOO_MANY_URLS` | More URLs than allowed per request (10). |
| 401 | `UNAUTHORIZED` | Missing or invalid API key. |
| 402 | `INSUFFICIENT_CREDITS` | Not enough credits. Top up or upgrade. |
| 403 | `KEY_SUSPENDED` | The key or account is suspended or not yet approved. |
| 429 | `RATE_LIMITED` | Rate limit exceeded. Retry after the Retry-After header. |
| 409 | `IDEMPOTENCY_CONFLICT` | The Idempotency-Key was already used with a different request, or its result is no longer available (results are kept 15 minutes). Use a new key. |
| 413 | `PAYLOAD_TOO_LARGE` | The request body is larger than 10 MiB. Nothing was read or charged. |
| 451 | `RESTRICTED_TARGET` | All requested URLs are restricted. No credits charged. |
| 503 | `SERVICE_UNAVAILABLE` | Temporarily unavailable. Retry shortly. |

```json title="Error body"
{
  "error": {
    "code": 402,
    "tag": "INSUFFICIENT_CREDITS",
    "message": "Not enough credits. Top up or upgrade your plan."
  },
  "requestId": "req_01J…"
}
```

## Long requests and Idempotency-Key

Most requests answer within seconds. A request still fetching after about 100 seconds (slow sites, rendering, a long `livecrawlTimeout`) answers `202` instead of making you wait on an open connection:

```json title="202 Accepted"
{
  "status": "pending",
  "requestId": "req_01J…",
  "idempotencyKey": "lwda_3f1c2b7e-0d4a-4c1e-9a55-2b8f0e6c7d10",
  "retryAfter": 1,
  "message": "Still fetching. Send the same request again with the same Idempotency-Key header to get the result; it is not charged again."
}
```

Send the **same request body** again with the header `Idempotency-Key: <idempotencyKey>` after `Retry-After` seconds. The repeat waits for the fetch and answers `200` with the result, or `202` again if it is still running. A repeat is never charged again. The SDKs and the MCP server do this for you.

**Always send your own `Idempotency-Key`** (8–128 characters of `A–Z a–z 0–9 _ - : .`, for example a UUID) on the first request, and keep it until you have the answer. A connection can break after the API received the request: a network error, a timeout on your side, a deploy. Send the same request again with the same key, and you get that fetch (its result, or `202` while it still runs) instead of a new one, so you pay once. Without your own key, a cut connection is a lost answer: the retry has no key, so it is a new request and charged again.

Every answer of `POST /v1/contents` (`200`, `202` and errors) carries the request's key in the `Idempotency-Key` response header: yours, or one the API created when you sent none. The only exceptions are the `400` for a malformed key and the `413` for a request body over 10 MiB, which is refused before it is read (nothing was charged, there is nothing to repeat). The SDKs create a key for each `contents()` call when you pass none, send it on every retry and put it on the error they raise (`idempotencyKey` / `idempotency_key`).

- Results are kept for **15 minutes**. A result nobody collects stays charged; after 15 minutes a repeat answers `409 IDEMPOTENCY_CONFLICT`, and a new key fetches the pages again.
- A key is bound to its request body for 24 hours: reusing it with a different body answers `409 IDEMPOTENCY_CONFLICT`. Use a new key for every new request.

## Rate limits

Limits are per account and count **URLs per second** (a request with 3 URLs uses 3). When exceeded, the API returns `429` with a `Retry-After` header (seconds). Limits work as a bucket: `X-RateLimit-Limit` is how many URLs you can send at once (`rateLimit.burst` in `GET /v1/account`), `X-RateLimit-Remaining` how many right now, and the bucket refills at your plan's URLs per second (`rateLimit.requestsPerSecond`).

| Plan | Price / month | Rate limit (URLs/s) |
| --- | --- | --- |
| Lite | $10 | 1 |
| Starter | $100 | 2 |
| Developer | $500 | 5 |
| Business | $900 | 10 |
| Enterprise | $2,250 | 20 |
| Custom | Custom | Custom |

## Credits

| Request | Credits |
| --- | --- |
| Standard page (static / server-rendered) | 1 |
| Retried page (succeeded on a retry through another connection) | 2 |
| JS-rendered page | 3 |
| Rendered + retried page | 5 |
| Short page (under 500 characters of visible text; delivered as that text, a success) | 0 |
| Failed request (incl. access denied by the site) | 0 |

- Plan credits reset monthly.
- Top-up credits never expire.

When your balance is too low for the URLs in a request, the API returns `402 INSUFFICIENT_CREDITS`. The current balance is returned in `X-Credits-Remaining` on every response.

## Account

`GET https://livewebdata.com/api/v1/account` returns your plan, remaining credits and rate limit. It is free — call it before large jobs.

```json title="GET /v1/account"
{
  "plan": "starter",
  "status": "active",
  "credits": { "plan": 81234, "topup": 10000, "total": 91234 },
  "rateLimit": { "requestsPerSecond": 2, "burst": 4 },
  "periodEndsAt": "2026-11-01T00:00:00Z"
}
```


---

# Examples

Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard.

## Basic fetch

The smallest possible request — one URL, defaults for everything else.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/article"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/article"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/article"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Geotargeting

Fetch a page as seen from Japan.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/pricing"
  ],
  "countryCode": "jp"
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/pricing"],
        "countryCode": "jp"
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/pricing"
    ],
    "countryCode": "jp"
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Text limits

Return at most 5,000 characters of the full page body, including navigation and footer.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/docs/getting-started"
  ],
  "text": {
    "maxCharacters": 5000,
    "verbosity": "full"
  }
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/docs/getting-started"],
        "text": {
            "maxCharacters": 5000,
            "verbosity": "full"
        }
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/docs/getting-started"
    ],
    "text": {
      "maxCharacters": 5000,
      "verbosity": "full"
    }
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Several URLs

Fetch up to 10 pages in one request. Each URL gets its own status; only successful pages cost credits.

```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "urls": [
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/blocked"
  ]
}'
```

```python title="Python"
import requests

response = requests.post(
    "https://livewebdata.com/api/v1/contents",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
    },
    timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```

```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    "urls": [
      "https://example.com/a",
      "https://example.com/b",
      "https://example.com/blocked"
    ]
  }),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```

## Whole jobs

For URL lists, retries and scheduling see [Automate requests](https://siterender.cc/docs/automation) — including ready-made Python and Node scripts.


---

# Automate requests

Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled.

## From playground to code

Every run in the [playground](https://siterender.cc/dashboard/playground) shows a **Use it in code** panel with cURL, Python, JavaScript and SDK snippets for exactly the request you just made. Start there, then move to the scripts below.

Download all example scripts: https://siterender.cc/live-web-data-api-examples.zip

| File | What it does |
| --- | --- |
| `examples/python/quickstart.py` | One request, prints title + first 500 chars of markdown |
| `examples/python/batch_from_csv.py` | Reads `urls.csv` (`url`, optional `country`), fetches with concurrency = rate limit, retries with backoff, writes `results.jsonl` and prints a summary |
| `examples/node/quickstart.mjs` | Same as the Python quickstart, native `fetch`, no dependencies |
| `examples/node/batch_from_csv.mjs` | Same as the Python batch script, no dependencies |
| `examples/urls.csv` | 10 sample URLs incl. a restricted, a missing and a timing-out URL to show error handling |

```bash title="Run the batch script"
export LIVE_WEB_DATA_API_KEY=1337_live_…   # never hard-code keys
python examples/python/batch_from_csv.py examples/urls.csv
node examples/node/batch_from_csv.mjs examples/urls.csv
```

## Process a list of URLs

The simplest loop: one request per URL, one after the other.

```python title="Python"
import os, requests

API = os.environ.get("LIVE_WEB_DATA_API_BASE_URL", "https://livewebdata.com/api")
headers = {"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"}

for url in ["https://example.com/a", "https://example.com/b"]:
    r = requests.post(f"{API}/v1/contents", json={"urls": [url]}, headers=headers, timeout=60)
    status = r.json()["statuses"][0]
    print(url, status["status"], status["error"] and status["error"]["tag"])
```

```javascript title="Node"
const API = process.env.LIVE_WEB_DATA_API_BASE_URL ?? "https://livewebdata.com/api";
const headers = { Authorization: `Bearer ${process.env.LIVE_WEB_DATA_API_KEY}`, "Content-Type": "application/json" };

for (const url of ["https://example.com/a", "https://example.com/b"]) {
  const res = await fetch(`${API}/v1/contents`, { method: "POST", headers, body: JSON.stringify({ urls: [url] }) });
  const { statuses } = await res.json();
  console.log(url, statuses[0].status, statuses[0].error?.tag ?? "");
}
```

You can also send up to 10 URLs per request — see [Several URLs](https://siterender.cc/docs/examples#several-urls).

## Respect rate limits

- Keep a steady pace of **your plan's URLs per second** (from `GET /v1/account` → `rateLimit.requestsPerSecond`). Bursts up to `rateLimit.burst` URLs (the bucket size, also sent as `X-RateLimit-Limit`) are fine; the bucket refills at that rate.
- Watch `X-RateLimit-Remaining` on each response: it is how many URLs you can send right now.
- On `429`, wait `Retry-After` seconds and retry the same request.

## Retries

Retry `429`, `5xx` and per-URL `CRAWL_LIVECRAWL_TIMEOUT` up to **3 times** with exponential backoff and jitter (e.g. `0.5s · 2^n + random(0–0.25s)`). Do **not** retry:

| Outcome | Why |
| --- | --- |
| `451 RESTRICTED_TARGET` | The domain is blocked; repeated attempts can suspend your key |
| `400 INVALID_REQUEST` / `TOO_MANY_URLS` | Fix the request first |
| `401` / `403` | Key invalid, revoked or suspended |
| `402 INSUFFICIENT_CREDITS` | Stop the job cleanly, top up, resume |
| `CRAWL_NOT_FOUND` | The page doesn't exist |

**Send your own `Idempotency-Key`** with every `POST /v1/contents` (one per logical request, for example per URL of your list) and send the same key on every retry of that request. A connection cut after the API received the request (a network error, a timeout, a deploy) is then retried without a second charge: the repeat returns that fetch's result. A `202` (still fetching) is repeated the same way, after `Retry-After`. Use a **new** key for a new attempt at a URL that already has an answer, such as a per-URL `CRAWL_LIVECRAWL_TIMEOUT`: the same key returns the same answer. See [Long requests and Idempotency-Key](https://siterender.cc/docs/api-reference#long-requests-and-idempotency-key).

The [SDKs](https://siterender.cc/docs/sdks) implement this policy for you.

## Check your balance first

Call `GET /v1/account` before a large job and compare `credits.total` with the number of URLs. Stop cleanly when a request returns `402`.

```bash title="cURL"
curl https://livewebdata.com/api/v1/account -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```

## Scheduling

Any scheduler works — the scripts are plain processes that exit when done.

```bash title="cron (daily at 06:00)"
0 6 * * * cd /opt/jobs && LIVE_WEB_DATA_API_KEY=… /usr/bin/python3 batch_from_csv.py urls.csv --out results.jsonl
```

```yaml title=".github/workflows/fetch.yml"
name: Fetch pages
on:
  schedule: [{ cron: "0 6 * * *" }]
  workflow_dispatch:
jobs:
  fetch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node examples/node/batch_from_csv.mjs examples/urls.csv --out results.jsonl
        env:
          LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}
```

Other schedulers (Cloud Scheduler, Kubernetes CronJob, n8n, Airflow) run the same command.

## Keep your key safe

- Store it in an environment variable (`LIVE_WEB_DATA_API_KEY`) or your platform's secret manager.
- Never put it in client-side code, mobile apps, browser extensions or repositories.
- Use one key per environment (e.g. production, staging, n8n) — you can label keys in the dashboard and see the label in the usage log.
- Rotate a key immediately if it leaks: create a new one, then revoke the old one.


---

# SDKs

Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches.

## Install

The SDKs live in the repository under `packages/` and are not published yet.

```bash title="JavaScript / TypeScript"
cd packages/sdk-js && npm install && npm run build
npm install ./packages/sdk-js        # in your project
```

```bash title="Python"
pip install ./packages/sdk-python    # requires httpx
```

## JavaScript / TypeScript

```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";

const client = new Teneo({ apiKey: process.env.LIVE_WEB_DATA_API_KEY, baseUrl: "https://livewebdata.com/api" });

const page = await client.getPage("https://example.com/article", { country: "de", format: "markdown" });
console.log(page.title, page.credits);

const res = await client.contents({ urls: ["https://example.com/a", "https://example.com/b"] });
const account = await client.account();
```

## Python

```python title="Python SDK"
import os
from teneo import Teneo

client = Teneo(api_key=os.environ["LIVE_WEB_DATA_API_KEY"], base_url="https://livewebdata.com/api")

page = client.get_page("https://example.com/article", country="de", format="markdown")
print(page.title, page.credits)

res = client.contents(urls=["https://example.com/a", "https://example.com/b"])
account = client.account()
```

Both clients read the key from `LIVE_WEB_DATA_API_KEY` and the base URL from `LIVE_WEB_DATA_API_BASE_URL` when you don't pass them.

## Methods

| JS | Python | Description |
| --- | --- | --- |
| `contents(request, { idempotencyKey })` | `contents(urls=…, country_code=…, text=…, html=…, livecrawl_timeout=…, render=…, render_fallback=…, idempotency_key=…)` | Raw `POST /v1/contents` (1–10 URLs). `render_fallback=False` / `renderFallback: false` turns the default browser fallback off |
| `getPage(url, opts)` | `get_page(url, country=…, format=…)` | One URL → `{ title, url, content, credits, source }`; raises if that URL failed. `format: "html"` requests the HTML (`html: true`) |
| `account()` | `account()` | `GET /v1/account` |

## Errors

| Error | When |
| --- | --- |
| `TeneoAuthError` | 401 / 403 — key invalid, revoked or suspended |
| `TeneoInsufficientCreditsError` | 402 |
| `TeneoRateLimitError` | 429 after all retries (`retryAfter` / `retry_after`) |
| `TeneoIdempotencyError` | 409 — the idempotency key was used with a different request, or its result expired (15 minutes) |
| `TeneoRestrictedTargetError` | 451 |
| `TeneoConnectionError` | Network error or timeout after all retries, or a `2xx` answer cut while it was read. The API may have received the request: repeat it with the error's `idempotencyKey` / `idempotency_key` |
| `TeneoPendingError` | Still `202` when `maxWaitMs` / `max_wait` ran out; `idempotencyKey` / `idempotency_key` collects the result later |
| `TeneoPageError` | `getPage` / `get_page`: the URL failed (`tag`, e.g. `CRAWL_NOT_FOUND`) |
| `TeneoAPIError` | Base class (status, tag, requestId, and `idempotencyKey` / `idempotency_key`: the key of a `contents()` call, else `null` / `None`) |

## Retries

Both SDKs retry `429` (respecting `Retry-After`), `5xx`, network errors, a `2xx` answer cut while it is read and — for `getPage` — `CRAWL_LIVECRAWL_TIMEOUT`, up to 3 times with exponential backoff and jitter. `400`, `401`, `402`, `403`, `409` and `451` are never retried. Configure with `maxRetries` / `max_retries`.

Every attempt of one `contents()` call sends the same `Idempotency-Key`, so a retry picks up the first fetch and is never charged twice (render requests included). A fetch that runs longer than about 100 seconds answers `202`; the SDKs repeat it after `Retry-After` until the result arrives, for up to 5 minutes (`maxWaitMs` / `max_wait`). These repeats do not count as retries. See [Long requests](https://siterender.cc/docs/api-reference#long-requests-and-idempotency-key).

Without your own key, the SDKs create one per `contents()` call. When the retries run out, the error carries that key: call `contents()` again with the same request and `{ idempotencyKey: e.idempotencyKey }` / `idempotency_key=e.idempotency_key` within 15 minutes, and a fetch the API already received returns its result instead of being charged again. **Better: always pass your own key** (one per logical request, for example per row of your input), so a request cut by a network error, a timeout or a deploy can be retried without a second charge even after your process restarts.


---

# MCP for agents

Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key.

## What is MCP

The Model Context Protocol (MCP) is an open standard that lets AI agents discover and call external tools. Our MCP server exposes the Contents API as two tools, so an agent can read any public web page on its own.

## Server & auth

| | Value |
| --- | --- |
| Server URL | `https://siterender.cc/mcp` |
| Transport | Streamable HTTP (stateless): JSON-RPC over `POST`; `GET` and `DELETE` answer `405` |
| Auth | `Authorization: Bearer <api_key>` |
| Keys | Same keys as the REST API — create one under [API keys](https://siterender.cc/dashboard/keys) |

Missing or invalid keys are rejected with HTTP 401 (suspended accounts: 403, "API access for this account is suspended. Contact support.") before any tool runs.

## Client setup

Replace `YOUR_KEY` with your API key. In the dashboard, the [Integrations](https://siterender.cc/dashboard/integrations) page fills in your key for you.

### Claude Code

Run this in your terminal. The server is then available in every Claude Code session for this project.

```bash
claude mcp add --transport http live-web-data https://siterender.cc/mcp \
  --header "Authorization: Bearer YOUR_KEY"
```

Add `--scope user` to make it available in all projects. Check with `claude mcp list`.

### Cursor

Add this to your Cursor MCP config, globally or per project.

File: `~/.cursor/mcp.json  or  .cursor/mcp.json`

```json
{
  "mcpServers": {
    "live-web-data": {
      "url": "https://siterender.cc/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_KEY"
      }
    }
  }
}
```

Prefer an env var: `"Bearer ${env:LIVE_WEB_DATA_API_KEY}"`.

### VS Code

Add this to your workspace. VS Code asks for the key the first time the server starts and stores it securely.

File: `.vscode/mcp.json`

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "live-web-data-key",
      "description": "Live Web Data API key",
      "password": true
    }
  ],
  "servers": {
    "live-web-data": {
      "type": "http",
      "url": "https://siterender.cc/mcp",
      "headers": {
        "Authorization": "Bearer ${input:live-web-data-key}"
      }
    }
  }
}
```

Uses the top-level key `servers` (not `mcpServers`). Paste `YOUR_KEY` when prompted.

### Claude Desktop

Claude Desktop connects to remote servers through the mcp-remote bridge. Add this to your Claude Desktop config and restart the app.

File: `claude_desktop_config.json`

```json
{
  "mcpServers": {
    "live-web-data": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://siterender.cc/mcp",
        "--header",
        "Authorization:${AUTH_HEADER}",
        "--transport",
        "http-only"
      ],
      "env": {
        "AUTH_HEADER": "Bearer YOUR_KEY"
      }
    }
  }
}
```

Requires Node.js. Alternative on paid plans: Settings → Connectors → Add custom connector with request header `authorization` = `Bearer …` (where available; the server must be publicly reachable).

### Other MCP clients

Any MCP client that supports Streamable HTTP can connect directly with the URL and the Authorization header. For stdio-only clients, use the mcp-remote bridge:

```bash
npx -y mcp-remote https://siterender.cc/mcp \
  --header "Authorization: Bearer YOUR_KEY" --transport http-only
```

Server URL: `https://siterender.cc/mcp` · Transport: Streamable HTTP · Auth header: `Authorization: Bearer <key>`

### REST API

Prefer plain HTTP? The same API key works with the REST endpoint.

```bash
curl -X POST https://livewebdata.com/api/v1/contents \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com/article"]}'
```

## get_page_contents

Fetch a public web page live and return its main content as clean Markdown, plain text or HTML. Read-only, idempotent. Single URL per call.

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string (http/https, at most 8,192 characters) | required | The public URL to fetch. |
| `country` | ISO-3166 alpha-2 | `"us"` | Fetch the page as seen from this country, e.g. 'de', 'gb', 'us'. |
| `format` | `"markdown"` \| `"text"` \| `"html"` | `"markdown"` | Output format. Prefer markdown. The page HTML is fetched and returned only for `"html"`. |
| `max_characters` | integer 500–100000 | `8000` | Maximum characters of content to return. Smaller than the REST API's default (`text.maxCharacters`, 100,000) because the result goes into the agent's context; raise it when the content is cut. |
| `verbosity` | `"compact"` \| `"full"` | `"compact"` | compact = main content only, full = whole page body. |

The result is one text item with a short header, followed by the content. The full REST-style JSON is also returned as `structuredContent` (with `results[].html` only for `format: "html"`).

```markdown title="Tool result"
# Example article
Source: https://example.com/article · country: us · live · credits used: 1 · credits left: 8,411

# Example article

Main content as clean Markdown…
```

## get_account_status

Check the remaining credits, plan and rate limit for the connected account. Agents should call it before large batches. No parameters, free. Returns the same numbers as `GET /v1/account`.

```text title="Tool result"
Plan: Starter · Credits left: 8,412 · Rate limit: 2 URL/s (burst 4) · Status: active
```

## Errors

Errors are returned with `isError: true` and one actionable sentence the agent can follow. When nothing was charged, the message ends with “No credits were charged.”

| Case | Message to the agent |
| --- | --- |
| 451 restricted target | This website is restricted and cannot be retrieved. Do not retry; use another source. |
| `CRAWL_NOT_RENDERED` | The page has no usable text: it needs JavaScript rendering, or shows no text even when rendered. Try another source or a different URL for the same information. |
| `CRAWL_NOT_FOUND` | Page not found (404/410). |
| `SOURCE_NOT_AVAILABLE` | The website refused this request. Retry once with a different country, otherwise use another source. |
| `SOURCE_BLOCKED` | Access denied by the website. Retry later or with a different country, otherwise use another source. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The page timed out. Retry once with a longer timeout, otherwise use another source. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable. Retry in a few minutes, otherwise use another source. |
| `RATE_LIMITED` | The rendering limit for the user's plan is reached. Do not retry rendering now; use another source. |
| 402 insufficient credits | The account is out of credits. Tell the user to top up at …/dashboard/billing. |
| 429 rate limited | Rate limit reached. Wait N seconds before retrying. |
| 401 / 403 | The API key is invalid or suspended. Tell the user to check their key at …/dashboard/keys. |
| Still fetching after 5 minutes | The page did not finish loading within 5 minutes, and the fetch may still be charged. Use another source. |

## Credits

MCP calls are billed exactly like REST calls: 1 credit per successful page, failed requests (including access denied by the site) are free. They appear in Usage with channel `mcp`. See [Credits](https://siterender.cc/docs/api-reference#credits).


---

# No-code recipes

Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap.

## Before you start

You need an API key from [API keys](https://siterender.cc/dashboard/keys). Every recipe sends the same request:

| Setting | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Header | `Authorization: Bearer YOUR_KEY` |
| Body (JSON) | `{"urls": ["https://example.com/article"], "countryCode": "us"}` |

The response is JSON: page content in `results[0].markdown`, the outcome in `statuses[0].status` and `statuses[0].error.tag`.

## n8n

> **Native integration coming soon.** Use the built-in **HTTP Request** node for now.

_Screenshot coming soon: n8n HTTP Request node_

| Field (HTTP Request node) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Authentication | Generic Credential Type → **Bearer Auth** (token = your key) — or **Header Auth** with Name `Authorization`, Value `Bearer YOUR_KEY` |
| Send Body | on |
| Body Content Type | `JSON` |
| Specify Body | `Using JSON` → `{"urls": ["{{ $json.url }}"]}` |
| Options → Response → Response Format | `JSON` (default: autodetect) |

**URL lists:** feed one item per URL into the node. Use the node's **Options → Batching** (*Items per Batch* = your rate limit, *Batch Interval (ms)* = 1000) — or a **Loop Over Items** node plus a **Wait** node (1 second) to stay within your rate limit.

## Make

> **Native integration coming soon.** Use the **HTTP** app's **Make a request** module for now.

_Screenshot coming soon: Make HTTP "Make a request" module_

| Field (HTTP → Make a request) | Value |
| --- | --- |
| Authentication type | `API key` (stores the key in a keychain) — or `No authentication` plus the header below |
| URL | `https://livewebdata.com/api/v1/contents` |
| Method | `POST` |
| Headers | `Authorization` = `Bearer YOUR_KEY` |
| Body content type | `application/JSON` |
| Body input method | `JSON string` |
| Body content | `{"urls": ["https://example.com/article"]}` |
| Parse response | `Yes` |

**URL lists:** use an **Iterator** over your URLs before the HTTP module and set the scenario's scheduling so the number of operations per second stays within your rate limit.

## Zapier

> **Native integration coming soon.** Use **Webhooks by Zapier** → **Custom Request** for now (not available on the Free plan).

_Screenshot coming soon: Zapier Webhooks by Zapier – Custom Request_

| Field (Custom Request) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Data Pass-Through? | `False` |
| Data | `{"urls": ["https://example.com/article"]}` |
| Headers | `Authorization` = `Bearer YOUR_KEY`, `Content-Type` = `application/json` |

The JSON response is available as fields in later steps.

## Tips

- Failed pages return HTTP 200 with `statuses[0].status = "error"` — add a filter/router on that field.
- Keep the key in the tool's credential store, not in plain step fields, wherever possible.


---