Audit a staging or password-protected site

Cloud tools crawl from their own servers, so a private environment is invisible to them. This guide shows how to audit a private site and what to check before launch.

Why audit before launch

Fixing SEO issues after launch means they were live, and possibly indexed, while broken. Auditing staging lets you catch a stray noindex, broken internal links, or missing titles before any of it ships. It is also the only place to catch the single most dangerous launch bug: a staging block that goes live with the site.

What you can crawl

• Staging and preview environments, often on a subdomain like staging.example.com.
• Password-protected sites using HTTP basic authentication.
• Pre-launch builds that are not yet publicly linked.

Before you start

Have two things ready: the staging site's start URL, and the access details for it, such as the basic-auth username and password.

Step 1: Create a project for the staging site

1. Click New project.
2. Name it so the staging version is obvious, for example "Example (staging)".
3. Enter the staging start URL.

Step 2: Provide access

If the environment is protected with HTTP basic authentication, supply the credentials in the project's access settings so the crawler can authenticate. Once it can reach the site, it crawls exactly as it would a public one.

Step 3: Crawl and review

Start the crawl and read the report on the dashboard, the same way you would for a live site.

What to check before launch

Focus your pre-launch pass on the issues that are expensive to ship:

• Indexability first. Staging is usually set to noindex or blocked in robots.txt on purpose. Confirm those blocks will not go live with the site, or your whole site could drop out of search. See indexability.
• Canonicals. Make sure no canonical tags point at the staging domain, which would demote the real pages after launch.
• Broken links and redirects, so nothing ships broken.
• Titles, meta descriptions, and headings on the key templates.

Pre-launch checklist

• No unintended noindex on pages that should rank.
• No Disallow: / that will carry over to production.
• Canonicals point at production URLs, not staging.
• Internal links resolve, with no 404s.
• Key pages have unique titles and descriptions.

How this differs from cloud tools

Cloud crawlers run on the vendor's servers, so they can only see publicly reachable pages. A local crawler runs on your machine, inside your access, so staging, private, and pre-launch sites are all reachable. This is a core reason to audit locally.

Common issues

• Authentication fails: double-check the credentials and that the environment uses basic auth. Login forms that set a session cookie work differently from basic auth.
• Few pages found on a JavaScript staging build: turn on rendering. See Crawl a JavaScript-heavy website.
• Everything shows as non-indexable: that is expected on staging. The point is to confirm those blocks do not ship.

Frequently asked questions

Related