Crawl history
SEO is iterative. You crawl, fix, and crawl again, and the only way to know a fix worked is to compare before and after. Crawlpit Monster saves every crawl, so a site's whole history is one click away.
Every crawl is saved
Each time you crawl a project, the full result is stored locally and kept until you delete it. Nothing is overwritten, so you build a timeline of the site: what it looked like last week, last month, and today. Because it is stored on your Mac, the history is yours and private, with no account and no upload.
Why history matters for SEO
Three things make saved crawls valuable:
- Proof a fix worked. Fixing a
noindexor a batch of broken links only counts if it actually cleared. A comparison shows it did. - Catching regressions. A redesign or a deploy can quietly reintroduce old problems. Comparing the latest crawl against the previous one surfaces them fast.
- A progress story. A health-score trend that climbs over weeks is the simplest way to see, and show, that the work is paying off.
Comparing two crawls
Open a project's history and pick two crawls to compare. The comparison shows what changed between them:
- Issues resolved: problems present in the earlier crawl and gone in the later one.
- Issues introduced: new problems that were not there before.
- Score change: how the health score moved.
- Pages added or removed: URLs the crawler found this time but not last time, or the reverse.
Reading a comparison
Group what you see into three buckets:
- Resolved confirms your fixes landed. Good.
- Introduced is where to focus next; something new caused these.
- Unchanged issues are still outstanding from before.
Start with anything introduced at the critical or high tier, since a regression there can undo real ranking gains.
Tracking the health-score trend
Beyond a single comparison, the score across all saved crawls is a trend line for the site. A steady climb means the work is compounding. A sudden drop points to a specific crawl to open and investigate. Watching the trend is more useful than fixating on any one number.
When something regresses
If a comparison shows new issues after a deploy, the culprit is usually one of a few things: a noindex or robots rule shipped by accident, a batch of new broken links from a content change, or a template change that dropped titles or headings. The comparison narrows it down; the page detail view confirms it.
Common mistakes
- Comparing across a settings change. If you turned on JavaScript rendering between crawls, the later crawl may find many more pages. That is a settings difference, not a regression. Keep settings steady when you want a clean before-and-after.
- Only ever looking at the latest crawl. The value is in the comparison. A single crawl tells you the state; two tell you the direction.
- Deleting old crawls too eagerly. History is cheap to keep and impossible to recreate after the fact.
Frequently asked questions
How long is crawl history kept?
Until you delete it. Crawls are stored locally and are not expired automatically.
Where is the history stored?
In the app's data folder on your Mac. Nothing is uploaded. See Where your crawl data is stored.
Can I compare crawls that are not next to each other?
Yes. You can compare any two saved crawls, for example this month against three months ago.
Does comparing crawls re-fetch the site?
No. A comparison uses the two saved crawls you already ran. To get a fresh picture, run a new crawl first.