SEO Audit Tool Shows 403 but Your Site Works in a Browser

By Ugur Saritepe · July 23, 2026

Your SEO audit tool reported 403 errors on half your pages, or the crawl died in a wall of 429s, yet the site opens instantly in every browser. In most cases nothing is broken: a bot gate blocked the tool, not Google. The real question is who exactly is blocked, and two checks answer it.

The two-minute self-diagnosis

Check 1: confirm the gate. Run this in a terminal against a page your tool flagged:

curl -sI https://yourdomain.com/

• 200:no gate on this page. Your tool's errors have a different cause, usually rate limiting partway through the crawl (cause 2).
• 403 or 429 with a header like cf-mitigated: challenge or x-vercel-mitigated: challenge: a challenge gate is confirmed (cause 1).

Check 2: ask Google directly.Open Search Console, paste the same URL into URL Inspection, and click Test Live URL. This fetch comes from real, IP-verified Googlebot, so it is the ground truth. If the live test returns your page, Google is not blocked and you are in false-alarm territory (causes 1 and 2). If it fails with “Blocked due to access forbidden (403)” or a fetch error, you have a real problem (cause 3). AI crawlers are a separate question either way (cause 4).

The four causes, in order of likelihood

1. A challenge gate is blocking the tool, and only the tool

Products like Cloudflare's Bot Fight Mode and Managed Challenge, Vercel's Attack Mode, DataDome, and Imperva sit in front of your site and challenge every client that does not look like a real browser. They do not trust the User-Agent string. They fingerprint the TLS handshake and HTTP/2 behavior and require the client to execute JavaScript, which plain HTTP crawlers cannot do. That covers most SEO audit tools.

30-second check: compare a plain request with one wearing a full Chrome User-Agent:

curl -sI https://yourdomain.com/ -o /dev/null -w "%{http_code}\n"
curl -sI -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" https://yourdomain.com/ -o /dev/null -w "%{http_code}\n"

Good(meaning: gate confirmed, mystery solved): both print the same 403 or 429. The gate is fingerprinting the connection itself, so the borrowed User-Agent changes nothing. We ran exactly this against our own site game-scout.app on July 23, 2026, with Vercel's challenge active: plain curl got 429, curl with a perfect Chrome User-Agent got 429, and both carried x-vercel-mitigated: challenge. The site loads normally in a real browser. Bad:the Chrome User-Agent suddenly gets a 200. That is not a challenge gate but a plain User-Agent filter, and those block by string matching, so check whether Googlebot's string is on the blocklist too.

These gates deliberately let verified search crawlers through. Vercel's Attack Mode documentation states that known bots like search engines are automatically allowed through the challenge, and Cloudflare's products treat Googlebot as a verified bot identified by IP range. Your audit tool enjoys no such exemption, which is why it sees a wall while Google sees your pages.

2. Rate limiting killed the crawl partway

If curl gets a clean 200 but the audit report is still full of errors, look at their shape. A crawl that starts fine and then collapses into 429s after the first few dozen pages hit a rate limit, not a bot gate. Firewalls commonly cap requests per minute from one IP, and an audit tool fetching ten pages per second looks exactly like a scraper.

30-second check:sort the tool's crawl log by time. Good: errors are spread randomly (look elsewhere). Bad: clean 200s at the start, then a block of 429s from one timestamp on. Fix:rerun the crawl at one or two URLs per second, or add a rate-limit exception for the tool's IPs. Googlebot paces itself and backs off on 429s, so an occasional rate-limited audit says nothing about your standing in Google.

3. Googlebot is actually blocked

This is the one case where the audit tool did you a favor. If the URL Inspection live test fails with “Blocked due to access forbidden (403)”, your firewall is turning away real Googlebot, and pages will start dropping out of the index as they get recrawled. Common causes: a country-blocking rule that catches Google's US crawl IPs, a custom rule that challenges all automated clients including verified bots, or a blocklist entry someone added during a scraping incident.

30-second check:open your firewall's event log (Security Events on Cloudflare, the Firewall tab on Vercel) and filter for the Googlebot user agent. Good: its requests show as allowed. Bad: rows of block or challenge actions against it. Fix: add an allow rule for verified bots, which both platforms validate by IP, for example Cloudflare's verified-bots allow rule. Never allowlist the string “Googlebot” by itself: any scraper can wear that name, and the verified-bot category exists precisely so you do not have to trust strings. If pages already vanished, fix the rule first, then see our guide on pages stuck in “Discovered - currently not indexed” for the recovery order.

4. AI crawlers are blocked, and that is a choice

GPTBot, ClaudeBot, and PerplexityBot do not get the search-engine exemption. Since July 1, 2025, Cloudflare blocks AI crawlers by default on new domains, and challenge gates stop them the same way they stop audit tools. The result is a site that is perfectly healthy in Google and invisible to AI assistants, because the systems that would read and cite your pages never get past the front door.

30-second check: open your bot-management settings and yourdomain.com/robots.txt and look for AI crawler names (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). Fix: there is no universal fix because this one is a policy decision, not a bug. If you want AI assistants to cite you, allow those crawlers in both places. If you do not want your content used by AI systems, blocking them is working as intended. Just make the choice deliberately instead of inheriting a default.

What to do next, in order

Stop at the first line that matches:

1. URL Inspection live test fails: fix the firewall rule for verified bots today (cause 3). This is the only genuinely urgent case.
2. Live test passes and curl shows a challenge header: false alarm (cause 1). Allowlist the tool's IPs if you need its crawls, otherwise close the audit report with a clear conscience.
3. Live test passes, curl gets 200, errors cluster late in the crawl: rate limiting (cause 2). Slow the crawler down.
4. Everything above passes: decide your AI-crawler policy (cause 4) while you are already in the firewall settings.

How to confirm the fix worked

For cause 3, rerun the URL Inspection live test after changing the rule; it should fetch the page within minutes, no waiting for a recrawl. Then watch the firewall event log for a day and confirm Googlebot rows show as allowed. For causes 1 and 2, rerun the audit after allowlisting or slowing down; the 403s and 429s disappear from the report. For cause 4, request one of your pages through the AI assistant you care about and see whether it can now read the content.

From experience: our own checker gets blocked too

We run an SEO checker, and bot gates block it exactly as described above. On July 23, 2026 we ran our full toolchain against our own site, game-scout.app, while Vercel's challenge was active. The checker reported Critical failures on checks that were actually fine, because every one of its requests got the 429 challenge instead of the page. Search Console for the same site, same week: 5,243 impressions and 117 clicks in the last 7 days. Google was never blocked for a minute. That is the false alarm in one picture: the tool saw a wall, Google saw a working site.

The other side: while triaging a site behind Cloudflare's Managed Challenge, we sent requests with a complete, byte-accurate Chrome header set. Still 403, still cf-mitigated: challenge, because the gate fingerprints the connection and demands JavaScript execution, not a plausible costume. We ended up building a real-browser transport for our own checker, and even vanilla headless Chromium gets caught by the strongest gates. The honest behavior we settled on applies to any tool you use: when a checker cannot get past a gate, it should say so, not report the challenge page as your site's health.

More symptom-by-symptom diagnosis guides live in our symptom guides section.

FAQ

Why does curl get a 403 but the site loads in my browser?

A bot gate in front of the site (Cloudflare Managed Challenge, Vercel Attack Mode, DataDome, Imperva) fingerprints each request: TLS handshake, HTTP/2 settings, and whether the client can run JavaScript. A browser passes all three; curl and most SEO crawlers fail, so they get a 403 or 429 challenge page no matter what User-Agent string they send.

Will Cloudflare or Vercel bot protection hurt my Google rankings?

Not by itself. Both platforms exempt verified search crawlers: Googlebot is identified by IP range, not by its User-Agent string, and is allowed through the challenge. Confirm it on your own site with a URL Inspection live test in Search Console. If that fetch returns your page, Google is unaffected regardless of what your audit tool sees.

Should I allowlist my SEO tool in the firewall?

Only if you rely on its crawls, and then allowlist by the tool's published IP ranges, never by User-Agent string alone, because any scraper can fake a User-Agent. If you do not want to open the gate, that is a valid choice: use Search Console as your source of truth and treat third-party audit errors on gated pages as noise.

Do blocked AI crawlers affect my Google rankings?

No. GPTBot, ClaudeBot, and PerplexityBot feed AI assistants, not Google's index, so blocking them changes nothing about ordinary search rankings. It does decide whether AI tools can read and cite your site. Since July 2025 Cloudflare blocks AI crawlers by default on new domains, so a site can be healthy in Google and invisible to AI assistants at the same time.