JavaScript pages not being crawled
If a crawl found only a handful of pages, or came back with empty titles and no content, the site is almost certainly building its pages with JavaScript.
A default crawl reads raw HTML, and on a JavaScript site that HTML is nearly empty until scripts run.
The symptom
- The crawl found far fewer pages than the site has.
- Pages show empty or missing titles, descriptions, and links.
- The site is a single-page app or a client-side framework build.
Why it happens
A default crawl fetches the HTML the server sends and reads the links in it. On a client-side site, the server sends a mostly empty shell, and the content and links only appear after JavaScript runs in the browser. With nothing in the HTML to follow, the crawler cannot discover the rest of the site.
The fix: turn on JavaScript rendering
Enable JavaScript rendering in the project's crawl settings. The crawler then loads each page in a real browser engine, lets the JavaScript run, and reads the finished page, so the content and links appear.
Confirm it worked
Re-crawl with rendering on and compare it to the earlier crawl. If the page count jumps and titles and content fill in, rendering was the fix. Crawl history makes the before-and-after clear.
Still missing pages after rendering?
If rendering is on and pages are still missing, the problem is usually the links themselves:
- Links are not real anchors. If internal links are built as click handlers with no
href, there is nothing for a crawler to follow, even with rendering. Make them proper<a href>links. - Content loads only after an interaction, like clicking or scrolling. A crawler does not perform those actions, so put important links and content in the rendered page itself.
- Infinite scroll that loads content on scroll will not be fully seen. Provide real paginated links as well.
This is not just a crawler quirk. Search engines have the same trouble with links and content that are not really there until a user acts, so fixing it helps your SEO directly.
Frequently asked questions
Why is the crawler missing pages on my site?
The content and links are built with JavaScript and are not in the raw HTML. Turn on rendering so the crawler can see them.
Do I need JavaScript rendering?
Only if a default crawl misses content. If it captures everything, leave rendering off for speed.
It still misses pages with rendering on. Why?
Usually the internal links are not real <a href> anchors, or content only appears after an interaction. Fix the markup.
Does Google have the same problem with my site?
It can. Search engines render JavaScript as a delayed step and still struggle with links that are not real anchors, so the fix helps your rankings too.