A crawl is slow or stalls
A crawl that is taking a while is usually working, not stuck. Big sites take time, and JavaScript rendering multiplies it. This page helps you tell the difference and speed things up.
Is it slow, or just big?
The live progress view is the first thing to check. If the pages-found and pages-finished counts are still moving, the crawl is working through the site, not stalled. A large site simply has a lot of pages to fetch.
Common causes
| Cause | What to do |
|---|---|
| Large site with many pages | Normal. Lower the page limit for a quick first pass |
| JavaScript rendering is on | Rendering runs a browser per page and is much slower. Turn it off if the site does not need it |
| The site itself responds slowly | High response times mean the server is slow to answer. See response time |
| A slow or unreliable connection | Crawling depends on your network |
| One very slow page | A single page timing out can hold things up briefly before the crawl moves on |
Telling stalled from working
If the progress counters have not moved for a long time, and the site is small, the crawl may be waiting on a slow or unresponsive page. If the counters are creeping up, it is working, just steadily.
How to safely stop and restart
Clicking Stop ends the crawl and keeps the partial results, so nothing is lost. To try again, start a fresh crawl. It is safe to stop and restart as often as you need.
Speeding up a crawl
- Turn off JavaScript rendering if the site does not need it. This is the biggest single speedup.
- Lower the page limit for a quick first pass on a large site, then run a full crawl when you have time.
- Check the site's response time. A slow server slows every request, and that is a site issue to fix. See response time.
Realistic expectations
A full crawl of a large site, up to 50,000 pages, takes real time, and turning on rendering can multiply it several times over, because every page runs a full browser engine. For a quick read on a big site, a smaller page limit gives you a fast sample.
Frequently asked questions
Why is my crawl slow?
The most common reasons are a large site, JavaScript rendering being on, or a slow-responding server. The progress view shows whether it is still working.
Can I stop a crawl safely?
Yes. Stop keeps the partial results, and you can start a new crawl any time.
Does JavaScript rendering slow the crawl?
Significantly. It runs a browser engine per page. Use it only when the site needs it.
How long should a crawl take?
Seconds for a small site, minutes or more for a large one, and longer again with rendering on.