Crawl settings
Most crawls need no configuration. When you do need to change something, here is every setting, what it does for your audit, and when to touch it.
Settings at a glance
| Setting | What it controls | Default | Change it when |
|---|---|---|---|
| Start URL | Where discovery begins | Your homepage | Auditing one section, or fixing a redirected host |
| Page limit | Max pages per crawl (up to 50,000) | High | Quick pass on a large site |
| JavaScript rendering | Whether pages are rendered in a browser engine | Off (HTML only) | Content or links are built with JavaScript |
| Access | Credentials for private sites | None | Staging, password-protected, or localhost |
| Scope | What counts as the site, including subdomains | The single site | A property spans subdomains |
Start URL
Where the crawl begins, usually the homepage. The crawler discovers the rest of the site from here, so choose a page that links out widely, use your canonical host, and include https://. Starting on a URL you redirect away from wastes the first hop and clutters the report with redirects.
Page limit
Caps how many pages a single crawl will fetch, up to 50,000. Because every crawl is a full crawl, the limit is mainly a time control on very large sites: lower it for a fast first pass, then run without the cap for the complete picture.
JavaScript rendering
By default the crawler reads the HTML a server returns. That is fast and complete for server-rendered and static sites. Client-side sites build their content and links with JavaScript after the HTML loads, so a raw-HTML crawl sees almost nothing. Rendering fixes that by loading each page in a real browser engine before reading it.
How to tell you need rendering
- A default crawl finds far fewer pages than you know exist.
- Pages come back with empty or near-empty titles, no meta descriptions, and no links.
- The site is built on a client-side framework, or is a single-page app.
The tradeoff
Rendering runs a full browser per page, so it is slower and uses more memory and CPU. Use it when a site needs it, not as a default. A good habit is to run a fast HTML crawl first; if the results look thin, switch rendering on and crawl again.
Access for private and pre-launch sites
Reaching sites the cloud cannot is the point of a local crawler. To crawl a site that is not publicly open:
- Staging or password-protected sites: provide the credentials or basic-auth details so the crawler can authenticate.
- Localhost: point the start URL at your local server, for example
http://localhost:3000, and crawl it before anything ships.
Step-by-step guides: Audit a staging or password-protected site and Crawl a localhost website.
Scope
Controls what the crawler treats as part of the site, including whether to include subdomains. Keep scope tight to audit one site cleanly. Widen it when a property genuinely spans subdomains, for example a blog on a subdomain you want in the same crawl.
Recommended presets
- Normal small-to-medium site: defaults. HTML crawl, high page limit.
- Large site, quick read: lower the page limit for a fast first pass, then run a full crawl with no cap.
- JavaScript-heavy site or single-page app: turn on JavaScript rendering.
- Staging or password-protected site: set access credentials, HTML crawl unless the build is JavaScript-heavy.
- Localhost during development: localhost start URL, rendering on only if the dev build is client-side.
Where settings are saved
Settings are saved per project, so each site keeps its own. Change them any time; the next crawl uses the new values.
Frequently asked questions
Do I need JavaScript rendering?
Only if your site relies on JavaScript to build content or links. If a default crawl captures everything, leave it off for speed.
Why did my crawl miss most of the site?
Usually JavaScript-built links that are not in the raw HTML. Turn on rendering and re-crawl. Robots rules and orphan pages are the other common causes.
Can I crawl a site behind a login?
Yes. Provide the credentials, or point at a localhost server for a site running on your machine.
Will a higher page limit slow the crawl?
Only if the site actually has more pages to fetch. The limit is a ceiling, not a target.
Should I always leave rendering on to be safe?
No. It is slower and heavier. Use it when a site needs it, which you can confirm by comparing an HTML crawl against a rendered one.