How crawling works
Search engines find your pages by crawling. Crawlpit Monster runs the same process on your own machine, so it can show you your site the way a crawler sees it.
Understanding the process helps you read your report and fix the right things.
The crawl, render, index pipeline
It helps to separate three stages that search engines run and that shape your SEO:
- Crawl. Fetch a URL and read its HTML, including its links.
- Render. If the page needs JavaScript to build its content, run it in a browser engine to see the finished page.
- Index. Decide whether, and how, to store the page for search.
A page has to survive all three to rank. Crawlpit Monster mirrors the first two so you can see where a page falls out, and it reports the signals (robots, canonical, status) that decide the third.
It starts at your URL and follows links
A crawl begins at the start URL, usually the homepage. The crawler fetches it, extracts the links, queues the internal ones it has not seen, and works through the queue until it is empty or your limit is reached. This is why internal linking is so important: a page that nothing links to is hard for any crawler to find. Those appear in your report as orphan pages, and adding internal links is the fix.
Crawling versus rendering, and JavaScript SEO
This distinction causes more confused audits than any other.
- Crawling the HTML fetches the raw HTML the server sends. It is fast and, for server-rendered or static sites, contains all the content and links.
- Rendering loads the page in a real browser engine so JavaScript can run and build the content first. Sites made with client-side frameworks (React, Vue, Angular, and similar) often need this, because their raw HTML is close to empty until scripts run.
If a default crawl returns far fewer pages than the site has, or pages come back with empty titles and no links, the site is likely building content with JavaScript. Turn on rendering and crawl again. The tradeoff is speed, since rendering runs a full browser per page. See Crawl settings and JavaScript rendering.
Robots rules, canonicals, and indexability
The crawler reads the directives that decide whether a page can be indexed and reports what it finds:
- robots.txt can disallow crawling of paths. A page blocked here cannot be assessed normally, and a stray
Disallow: /shipped from staging can hide an entire site. - Meta robots and X-Robots-Tag can carry
noindex, which keeps a page out of search even if it is otherwise perfect. Anoindexleft on after a redesign is one of the most common and costly post-launch mistakes. - Canonical tags tell engines which URL is the primary version when several are similar. A canonical pointing at the wrong URL, or at a staging domain, quietly demotes the real page.
Because these are exactly the signals that decide indexing, they carry the most weight in your report and score.
Redirects and redirect chains
A redirect sends one URL to another. A single 301 is fine, but two problems show up in crawls:
- Redirect chains, where A goes to B goes to C. Each hop wastes crawl effort and dilutes signals. Point the first URL straight at the final one.
- Internal links to redirects, where your own pages link through a redirect instead of to the destination. Update the links to the final URL.
What counts as one site
The crawler stays within the site you point it at. It follows internal links and records external ones without crawling off into the wider web. How subdomains are treated depends on your scope setting.
Full crawls, and how this differs from crawl budget
Every crawl is a complete crawl, up to 50,000 pages, not a sample. Search engines ration how much of a site they crawl, which SEOs call crawl budget, and on large sites that means not every page gets crawled often. That limit is Google's, not yours: crawling your own site locally gives you the whole picture every time, which is exactly what an audit needs. Understanding your full site also helps you spend Google's crawl budget well, by cutting dead ends, redirect chains, and low-value pages.
Crawling responsibly
Only crawl sites you own or have permission to crawl. The crawler fetches real pages from a real server, so point it responsibly and avoid overloading sites you do not control. The Terms of use cover this in full.
Frequently asked questions
Does the crawler obey robots.txt?
It reads your robots.txt and meta robots tags and reports how they affect each page, so you can see what search engines are allowed to crawl and index. It is built for auditing sites you control.
Does it crawl external links?
It records external links so you can check them, but it does not crawl off your site into the wider web.
Do I need JavaScript rendering?
Only if your site relies on JavaScript to build content or links. If a default HTML crawl captures everything, leave rendering off for speed.
Will crawling affect my rankings or my server?
It has no effect on rankings; it only reads pages. It makes normal requests, so a reasonable crawl behaves like ordinary traffic.
Does it crawl subdomains?
That depends on your scope setting. Keep scope tight for one site, or widen it to include subdomains.
What is the difference between crawling and indexing?
Crawling is fetching and reading a page. Indexing is a search engine deciding to store it for search. A page can be crawled but not indexed, for example if it carries a noindex tag.