Extract main content from HTML, raw or rendered

Getting a page's HTML, raw or rendered, is half the job. The other half is getting the article out of it: the text a person came to read, without the navigation, cookie banner, footer and "related posts". This guide covers how main-content extraction works, where it fails, and the special cases that only show up on JavaScript-heavy pages.

Raw HTML vs rendered DOM

Extraction works the same way on both. The difference is what is in the input:

• Raw HTML is what the server sent. On server-rendered pages it holds the whole article. It is cheap to get and the same on every fetch.
• Rendered DOM is the page after scripts ran in a headless browser. It is the only source for client-rendered content, but it can also include things added by scripts: consent dialogs, chat widgets and personalised blocks.

Use raw HTML whenever the text is there; the one-minute test tells you. Check the data the page already ships too: JSON-LD or a hydration payload can hold the article without any extraction. Render only when neither has it, using the wait patterns in the headless browser guide, and expect more noise in the result.

How Readability-style extraction works

Mozilla's Readability.js, the code behind Firefox Reader View, is the reference most extractors resemble. Roughly:

1. Remove elements that are almost never content: scripts, styles, forms, and blocks whose class or id suggests comments, sidebars or ads.
2. Score candidate blocks by the amount of text, the number of commas and the link density. A block that is mostly links is probably navigation.
3. Pick the best-scoring block, add siblings that score close to it, and clean up what is left.

parse() returns the title, the cleaned HTML, the plain text, an excerpt, the byline and some metadata. Readability also has isProbablyReaderable(), a cheap pre-check that guesses whether extraction will work, with false positives and negatives.

Readability does not sanitise its output. If you display the result in a browser, run it through a sanitiser such as DOMPurify first.

Boilerplate removal that keeps structure

The same rules apply to raw HTML and to a rendered DOM, but the rendered DOM usually has more to drop. Keep the structure that carries meaning and drop the rest:

Converting the kept part to Markdown preserves headings, lists and tables in a compact form. Plain text loses them, which matters for documentation and reference pages.

Cases that break extraction

Shadow DOM. Web components can render their content inside a shadow root. In a rendered DOM, innerHTML of the host element does not include it, and a naive serialiser misses it. Playwright locators pierce open shadow roots; closed shadow roots are not reachable from page scripts at all. Declarative shadow DOM (<template shadowrootmode="open">) does appear in raw HTML, inside the template.

Iframes. Embedded content lives in its own document. Raw HTML only contains the <iframe src>; you fetch that URL separately if you need it. In a headless browser you access it through the frame, and cross-origin frames are separate pages with their own terms.

Lazy loading. Images with loading="lazy" still have their src in the HTML, so alt text and URLs survive a static fetch. Sections that load on scroll through JavaScript do not. In a browser, scroll in steps and wait for new content; on a static fetch, look for a paginated or feed URL instead.

Very short articles. Extractors may treat a short page as boilerplate and return nothing. Set a minimum length and fall back to the <main> or <article> element.

Fail loudly on empty output

The most expensive extraction bug is not a crash. It is an empty or near-empty result that your code accepts. Add a check before the text goes anywhere:

def usable(text: str, min_chars: int = 200) -> bool:
    words = text.split()
    return len(text) >= min_chars and len(words) >= 30

Tune the threshold per source. Anything below it becomes an error with the URL attached, not a blank document in an index or a prompt.

A static fetch with extraction built in

Live Web Data API, made by Teneo Labs, which runs this site, does the static fetch and the main-content step in one call and returns Markdown, text or HTML. When a static fetch finds no usable content, for example on a page that needs JavaScript, it returns a CRAWL_NOT_RENDERED error at 0 credits instead of an empty result; with render or renderFallback the page is read in a browser before extraction. The API docs list the output formats and options.

Test it yourself

Take one server-rendered article and one client-rendered page. Run Readability on the raw HTML of both, then on the rendered DOM of the second. Compare the three outputs side by side: length, headings kept, and anything that looks like navigation or a consent banner.