Run your first crawl

Rather than open pages one at a time, the crawler starts at a URL, follows every internal link it finds, records what each page contains, and returns one report covering the SEO issues across the whole site. Everything runs on your Mac, and the crawl only reads your pages, so it never changes anything on your site.

What a crawl actually does

Under the hood, a crawl is a loop. The crawler fetches your start page, reads the links in its HTML, adds the internal ones it has not seen to a queue, and fetches those next. It repeats until the queue is empty or it hits your page limit.

Along the way it stores, for every page, the response and status code, the title and meta description, the headings, the word count, the images and their alt text, the links in and out, the canonical and robots directives, any structured data, and the response time. Those stored facts are what every check in your report is built from.

This is the same discovery process search engines use, which is why a crawl surfaces the problems that actually cost you traffic: pages that cannot be indexed, pages nothing links to, broken links, and duplicate or missing tags.

Before you start

You need three things:

• Crawlpit Monster installed and open. If not, see Install on Mac.
• A site you own or have explicit permission to crawl.
• The site's URL. A live site, a staging site, or a local site all work.

Step 1: Create a project

A project holds one site and its full crawl history.

1. Click New project.
2. Name it, usually after the site or brand.
3. Enter the start URL.
4. Click Create.

Step 2: Choose the right start URL

The start URL decides where discovery begins, and a poor choice can skew a whole crawl. Follow four rules:

• Start at the homepage unless you deliberately want to audit one section. The homepage links out widely, so the crawler discovers the most from it.
• Use your canonical host. If your site lives at https://example.com, do not start at http://example.com or https://www.example.com if those redirect. Starting on a redirected version spends the first hop bouncing rather than crawling.
• Include the protocol, always https:// on a modern site.
• Match the trailing-slash form your site actually uses, so the crawler is not immediately redirected.

Step 3: Start the crawl

1. With the project open, click Start crawl.
2. The crawler begins at your start URL and follows internal links to discover the rest of the site.
3. A live view shows pages found and pages finished as it works.

Crawl time depends on three things: the number of pages, your connection speed, and whether JavaScript rendering is on. A small site finishes in seconds; a large one takes longer, up to 50,000 pages, and every crawl is a full crawl rather than a sample. You can keep working in the app while it runs, and clicking Stop keeps the partial results.

Step 4: Read the report

When the crawl finishes you land on the dashboard, which has four parts:

• The health score, a single weighted number for the whole site.
• Issue groups, counts of each kind of problem.
• The page table, every URL with its status and issues.
• The page detail view, one click deeper, with everything about a single URL.

The full tour is in The dashboard.

Step 5: Fix issues in priority order

The biggest mistake on a first audit is fixing the easy things first because there are a lot of them. Work by impact instead. This order clears the problems that cost the most traffic for the least effort:

The checks reference explains each issue and how to fix it in detail.

Step 6: Crawl again and compare

After you make changes, crawl again and compare the two runs. The health score should rise and the issue counts should fall. Because every crawl is saved, you can show a clear before and after. See Compare crawls with crawl history.

Common first-crawl mistakes

• Starting on a redirected URL, which fills the first results with redirects instead of pages.
• Expecting every page when the site builds links with JavaScript. Turn on rendering and re-crawl.
• Chasing low-impact issues first because they are numerous, while a noindex quietly hides a key page.
• Crawling a site you do not have permission to crawl. Only audit sites you own or are authorized to.

Frequently asked questions

Related