Crawl a JavaScript-heavy website

For that second kind, a default crawl sees almost nothing, and the fix is to turn on rendering. This guide shows when you need it and how to confirm it worked.

The problem: JavaScript-built content is invisible to an HTML crawl

A plain crawl reads the raw HTML the server returns. If your site is a single-page app or a heavy client-side framework build (React, Vue, Angular, and similar), that raw HTML often contains no real content and no links until JavaScript runs. So the crawler finds a handful of pages, or one, and stops, because there were no links in the HTML to follow.

How to tell your site needs rendering

• A default crawl finds far fewer pages than the site actually has.
• Pages come back with empty titles, no meta descriptions, and no links.
• You know the site is built as a single-page app or on a client-side framework.

Step 1: Run a default HTML crawl first

Start with a normal crawl. It is fast, and it gives you a baseline. If it captures the whole site, you do not need rendering at all, and you should leave it off for speed.

Step 2: Turn on JavaScript rendering

If the baseline looks thin, enable JavaScript rendering in the project's crawl settings. With rendering on, the crawler loads each page in a real browser engine, lets the JavaScript run, and reads the finished page, content and links included.

Step 3: Re-crawl and compare

Crawl again with rendering on and compare it to the baseline. If the rendered crawl finds many more pages and the titles and content fill in, rendering was the missing piece. You can confirm the difference with crawl history.

What rendering does, and the tradeoff

Rendering runs a full browser for every page, so it sees what a user's browser sees. The cost is speed and resources: a rendered crawl is slower and heavier than an HTML crawl. Use it when a site needs it, not by default.

JavaScript SEO tips

• Prefer server-side rendering or prerendering. Search engines can render JavaScript, but it is a delayed, resource-heavy second step, and content that is in the HTML from the start is the safest for SEO.
• Use real links. Make internal links proper <a href> elements. Links built as click handlers with no href are hard for crawlers to follow, even with rendering.
• Put important content in the rendered HTML, not behind an interaction a crawler will not perform.

Common issues

• Still missing content after enabling rendering: the links may not be real anchors, or content may load only after a user action like scrolling or clicking. Fix the markup so links and key content are present in the rendered page.
• The crawl is much slower: that is expected with rendering. Lower the page limit for a quick pass if you just need a sample.

Frequently asked questions

Related