Crawlability vs. indexability: the complete technical SEO guide

Crawlability and indexability get used as if they mean the same thing. They do not, and the confusion is expensive. People fix the wrong one, spend a week on a robots.txt file when the real problem is a noindex tag, and watch pages stay missing from Google the whole time.

The distinction is simple once you see it. Crawlability is whether a search engine can reach your pages. Indexability is whether it will store them. A page has to clear the first to have any shot at the second, and clearing the first guarantees nothing about the second.

This guide explains what each term means, the specific things that break each one, how to tell which stage is failing, and how to fix it.

What is the difference between crawlability and indexability?

Crawlability is whether a search engine can discover and fetch a page. Indexability is whether a search engine can analyze that page and store it in its index to be served in results.

Crawlability is about access. Indexability is about eligibility. Those are two different gates, and a page passes through them in order.

Google describes its process in three stages, and the official guide to how Google Search works is blunt about the catch: not all pages make it through each stage, and Google does not guarantee it will crawl, index, or serve any given page. Crawling comes first. Indexing comes second, and it is selective. Google analyzes what it fetched and decides whether the page is worth storing.

So the relationship is one-directional. A page must be crawlable to become indexable, because Google cannot store what it never fetched. But crawlable does not mean indexed. Plenty of pages get crawled and then left out of the index on purpose or on merit. Keep the two gates separate in your head and most technical SEO diagnosis gets easier.

What is crawlability?

Crawlability is a page's accessibility to search engine crawlers: whether Googlebot can find the URL and successfully fetch its content.

It has two parts, and a failure in either one blocks the page. Discovery is the first part, can Google find the URL at all. Retrieval is the second, can Google actually download it once found.

Discovery runs on links and sitemaps. Googlebot finds new URLs by following internal links from pages it already knows and by reading your XML sitemap. Retrieval runs on access, the server has to respond, and no rule can forbid the fetch. Both have to work.

What breaks crawlability

Crawlability fails for a handful of well-defined reasons:

• A robots.txt block. A Disallow rule tells Googlebot not to fetch a URL or path. This is the most common deliberate crawl block, and the most common accidental one when Disallow: / survives from staging.
• Orphan pages and weak internal linking. If nothing links to a page, discovery struggles. A page reachable only through a sitemap is crawled less and less reliably than one that is well linked.
• Server errors and timeouts. If the server returns 5xx errors or is too slow, Googlebot cannot retrieve the page even though it found the URL.
• Broken internal links and redirect chains. Links to dead pages or long redirect hops waste crawl budget and can strand content deep in the site.
• Crawl budget waste. On large sites, thousands of low-value or duplicate URLs consume the attention Googlebot would otherwise spend on pages that matter.

Most of these are discoverable with a crawl of your own site. A crawler that follows your internal links the way Googlebot does will show you what is reachable, what returns an error, and what nothing links to. Orphan page detection catches the discovery failures, and a broken link and redirect checker catches the retrieval ones.

What is indexability?

Indexability is whether a page, once crawled, is eligible to be stored in the search index and shown in results.

This is the second gate. Google has the page. Now it decides whether to keep it. That decision runs on explicit directives you set and on quality signals Google judges for itself.

The important thing is that indexability failures are invisible to a casual look. The page loads fine in a browser. It returns a 200 status. Nothing appears broken. Yet Google has quietly chosen not to index it, and only a deliberate check reveals why.

What breaks indexability

A crawled page can fail to get indexed for these reasons:

• A noindex directive. A noindex meta tag or X-Robots-Tag header tells Google not to index the page. This is the cleanest way to keep a crawlable page out of the index, and the most common accidental deindexer when it is left on after launch.
• A canonical pointing elsewhere. If the page's canonical tag names a different URL as the master version, Google indexes that other URL instead of this one.
• Duplicate or thin content. If the page duplicates another, or adds little unique value, Google may crawl it and decline to store it. In Search Console this shows as "Crawled, currently not indexed."
• Quality and trust signals. Indexing is selective. Low-value pages that do not appear to satisfy any user intent can be left out on merit alone.

Can a page be crawlable but not indexable, or the reverse?

Yes to both, and understanding how is the heart of this topic.

The two gates are independent enough that all four combinations happen in the wild, as the matrix above shows.

The common case is crawlable but not indexed. Google fetches the page without any trouble, then a noindex tag, a canonical pointing elsewhere, or a quality judgment keeps it out of the index. The page is perfectly accessible and still absent from search.

The counterintuitive case is the reverse: a page blocked from crawling that still gets indexed. If you disallow a URL in robots.txt but other sites link to it, Google can index the URL from those external signals without ever fetching its content, and it appears in results with no description.

How do you diagnose whether crawlability or indexability is failing?

You diagnose it by finding which stage the page stalled at, and Google Search Console tells you directly.

The Page indexing report labels every excluded URL with a reason, and those reasons map cleanly onto the two gates. Read the label first, then act.

Beyond the report, two tools sharpen the diagnosis. The URL Inspection tool in Search Console shows exactly how Google fetched and rendered a specific page, including whether it was blocked and what the indexed version looks like. And a crawl of your own site, checked against your sitemap and Search Console data, reveals the systemic patterns: the template shipping a noindex, the section nothing links to, the redirect chain eating crawl budget. A quick single-page check confirms one URL's crawl and index status fast when you are working on a specific page.

How to fix crawlability and indexability issues

Fixing follows the diagnosis, and the order matters: restore crawlability first, because indexability cannot be evaluated on a page Google cannot fetch.

Work the crawl gate, then the index gate.

For crawlability, open access and improve discovery. Remove any robots.txt rule blocking pages that should be public, fix server errors and slow responses, add internal links to under-linked and orphan pages, flatten redirect chains, and keep your sitemap clean so Google spends its crawl budget on pages that matter.

For indexability, align your directives with your intent. Remove stray noindex tags from pages that should rank, correct canonical tags that point to the wrong URL, consolidate or differentiate duplicate content, and strengthen thin pages so they earn their place in the index. Then use URL Inspection to request a recrawl and confirm the fix took.

Because these problems hide across templates and repeat at scale, the efficient path is a full crawl that flags both gates at once. Crawlpit Monster's on-page and technical audit reports each page's crawl status, indexability, blocking directives, canonicals, and broken links in one place, so you can see which gate every problem sits behind. This is item one of the broader technical SEO audit checklist, and the foundation everything else in an audit depends on.

Crawl first, index second, rank never without both

Here is the whole guide in one line. Crawlability gets your page fetched, indexability gets it stored, and a page has to win both before it can rank at all. Treat them as separate gates with separate fixes. When a page goes missing, resist the urge to guess. Read the Search Console status, decide whether the failure is access or eligibility, and fix the gate that is actually stuck instead of the one you assumed.

The fastest way to see both gates across your whole site is to crawl it. Point Crawlpit Monster at your site and let the on-page audit show you every page's crawl and index status in a single report. It runs on your own machine, reaches staging and password-protected sites, keeps each crawl so you can track fixes over time, and is free for up to 100 pages.