Crawl a localhost website

That means you can catch SEO problems while you are still building, before anything is deployed, which no cloud tool can do.

Why crawl localhost

Fixing SEO during development is cheaper than fixing it after launch. Crawling your local build lets you check titles, headings, internal links, and indexability as you go, and catch a mistake like a global noindex before it ever leaves your machine.

Before you start

Have your local development server running, and know the port it uses. Local sites typically run over http (not https) on a port like 3000, 5173, or 8000.

Step 1: Start your local server

Run your site locally as you normally would during development, for example on http://localhost:3000. Confirm it loads in a browser first.

Step 2: Create a project with the localhost URL

1. Click New project.
2. Name it, for example "Example (local)".
3. Enter the local start URL, including the port, for example http://localhost:3000.

Step 3: Crawl

Start the crawl. The crawler fetches your local pages and produces the same report it would for any site, entirely offline on your machine.

JavaScript development builds

Many local dev servers run JavaScript frameworks that build content in the browser, so a default HTML crawl may see very little. If your local build is a single-page app or a client-side framework, turn on JavaScript rendering. See Crawl a JavaScript-heavy website.

Notes for common setups

• Local sites usually use http, not https. Enter the URL exactly as it loads in your browser.
• Get the port right; frameworks default to different ports (for example 3000 or 5173).
• Static-site generators often serve a production-like build locally, which crawls cleanly with a plain HTML crawl.

What to check locally

Everything you would check on a live site, with two that matter most during development:

• Indexability, so a noindex meant only for development never ships.
• Internal links, so broken links are caught before deploy.

Common issues

• Nothing or very little found: the site probably builds content with JavaScript. Turn on rendering and re-crawl.
• Connection refused: the local server is not running, or the port is wrong.
• Wrong pages: confirm the start URL and port match the build you intend to audit.

Frequently asked questions

Related