Indexability
Indexability is whether a search engine is allowed to include a page in its index. It is the highest-stakes check in your report, because a page that cannot be indexed cannot rank, no matter how good it is.
This page explains what controls indexability, the subtle trap that catches even experienced SEOs, and how to fix each cause.
What this check flags
The crawler flags pages that are not indexable and, crucially, tells you why. A page can be excluded for several reasons, and the fix depends entirely on which one applies.
Why indexability matters most
Every other on-page improvement assumes the page can be indexed in the first place. A perfect title, clean headings, and rich content are worth nothing on a page carrying a stray noindex. That is why indexability is weighted heaviest in the health score and why it is the first thing to fix in any audit.
What controls indexability
| Signal | Where it lives | Effect |
|---|---|---|
noindex | Meta robots tag or X-Robots-Tag HTTP header | Tells search engines not to index the page |
Disallow | robots.txt | Blocks crawling of the path (not the same as removing it from the index) |
| Canonical tag | <link rel="canonical"> in the head | Points to a different URL as the primary version |
| Status code | HTTP response | Redirects and errors are not indexable as-is |
Crawl blocking is not the same as de-indexing
This is the nuance that catches people. robots.txt controls crawling, not indexing:
- A page disallowed in robots.txt is not crawled, but it can still appear in search results as a bare URL with no description if other pages link to it, because the engine sees the link but is not allowed to fetch the page to read its content.
- To reliably keep a page out of the index, use a
noindextag and allow crawling of the page, so the engine can actually fetch it and see thenoindex.
Putting both a Disallow and a noindex on the same page backfires: the engine cannot crawl the page, so it never sees the noindex, and the URL can linger in results. If you want a page gone from the index, let it be crawled and let the noindex do its job.
How to fix indexability issues
- A page that should rank is set to
noindex: remove thenoindexfrom the meta tag or theX-Robots-Tagheader. - A page you want indexed is blocked by robots.txt: remove or narrow the
Disallowrule. - A page is canonicalized to the wrong URL: point the canonical at the correct primary URL, or remove it if the page is unique.
- A page returns a redirect or error when it should be live: fix the status so it returns
200. - A page you want out of the index: add
noindexand make sure the page is crawlable so the tag is seen.
The post-launch trap
The single most common and costly indexability mistake is shipping a staging site's noindex or Disallow: / to production. Staging environments are usually blocked from search on purpose, and if that block goes live with the site, the entire site can vanish from search. After any launch or migration, crawl the site and check indexability first.
How Crawlpit Monster reports it
Indexability appears as an issue group on the dashboard, and the page table has an indexable column. For each non-indexable page, the report shows the reason, whether it is a noindex tag, a robots.txt block, a canonical pointing elsewhere, or a non-200 status, so you know exactly which fix to apply.
Common mistakes
- Using robots.txt to remove a page from Google. Disallow blocks crawling, not indexing. Use
noindexon a crawlable page instead. - Blocking and noindexing the same page, which stops the engine from ever seeing the
noindex. - A canonical pointing at a staging domain or the wrong URL, which quietly demotes the real page.
- Assuming a page is indexed because it looks fine. Only the indexability signals decide it; check them.
Frequently asked questions
Why is my page not indexed?
Usually a noindex tag, a robots.txt block, a canonical pointing to another URL, or a non-200 status. The report shows which one applies to each page.
What is the difference between noindex and robots.txt?
noindex tells engines not to index a page they are allowed to crawl. robots.txt Disallow blocks crawling, which does not reliably remove a page from the index. To de-index, use noindex and allow crawling.
Does noindex remove a page from Google?
Yes, once the engine crawls the page and sees the tag. The page must be crawlable for the noindex to be honored.
What does a canonical tag do to indexability?
It tells engines which URL is the primary version. A page canonicalized to a different URL is usually not indexed in its own right, which is correct for duplicates but a problem if the canonical is wrong.