Cloudflare Blocking Googlebot? Find the Rule and Fix It

By Ugur Saritepe · July 27, 2026

Search Console reports “Blocked due to access forbidden (403)”, your site opens fine in a browser, and Cloudflare sits in front of it. One of five Cloudflare layers is rejecting Googlebot, and the Security Events log names which one. This guide walks you from that log to the exact rule, then fixes it without turning your protection off.

The five-minute self-diagnosis

Two checks split the whole problem space. Run both before touching any setting.

Check 1: the live test, your ground truth. In Search Console, open URL Inspection for an affected URL and click Test Live URL. This makes real Googlebot fetch the page right now. Good: the test says the page is available. Old 403s in the Pages report may simply predate a config change, and they clear as pages get recrawled. Bad: the live test itself fails with a 403. Googlebot is being blocked at this moment, and one of the five causes below is doing it.

Check 2: the event log, your suspect list. In the Cloudflare dashboard, open Security > Events and filter by user agent containing Googlebot. Each event shows the action taken and, critically, the security product that took it. That column is the map to this guide: a custom or IP access rule is cause 1, a bot product is cause 2, managed rules or rate limiting is cause 4. Two warnings: the free plan keeps only the last 24 hours of events, and the log can be sampled, so an empty result does not prove innocence.

Do not test with a spoofed user agent. Running curl -A "Googlebot" from your laptop tests the wrong thing: Cloudflare verifies real Googlebot by IP, so your spoofed request looks like a fake Googlebot and gets blocked even when the real one sails through. Getting blocked by that test proves nothing. The live test in Search Console is the only fetch that exercises the real path.

Related but different symptom: if the live test passes and only your SEO tools report 403s, Cloudflare is blocking the tools, not Google. That case is covered in our guide to audit tools showing 403 while the site works.

The five causes, in order of likelihood

1. A custom rule or IP block is matching Googlebot

The most common cause in every thread we researched: a rule added for a good reason that catches Googlebot as collateral. The classic is a country rule that challenges or blocks visitors outside your market. Googlebot crawls mostly from US IP addresses, so “block everything except my country” blocks Google too, and the error page it serves carries code 1020, access denied by a firewall rule. Blocklist entries added during a scraping incident do the same quieter damage.

30-second check: in the Security Events list from the self-diagnosis, look at events whose product is a custom rule or IP access rule. The event names the rule. Good: the blocked requests claiming to be Googlebot come from random hosting IPs, meaning they are fakes and your rule is doing its job. Bad: blocked requests from IPs that reverse-resolve to Google. Verify with:

host 66.249.66.1

Real Googlebot IPs resolve to googlebot.com, google.com, or googleusercontent.com, like crawl-66-249-66-1.googlebot.com (we ran exactly that lookup while writing this). Google also publishes the full method and its official IP list.

Fix: keep your rule and add an exception above it: a custom rule with the Skip action and the expression cf.client.bot, which is true only for bots Cloudflare has verified by IP. Cloudflare's verified-bots allow rule is the documented pattern. Never allowlist the string “Googlebot” by itself: any scraper can wear that name.

2. Bot Fight Mode or Super Bot Fight Mode is challenging the crawl

Cloudflare's bot products should not touch Googlebot: as of July 2026, Cloudflare's documentation states that verified bots have historically been excluded from default bot configurations across all plans. But Super Bot Fight Mode, on paid plans, has a configurable setting per bot category, including one for verified bots. If someone set that category to challenge or block, Googlebot is included.

30-second check: in your Cloudflare bot settings, read the action set for each category. Good: verified bots allowed, with only automated traffic challenged. Bad: a challenge or block action on the verified-bots category. Fix: set verified bots back to allow. On paid plans you can also add the same cf.client.bot skip rule from cause 1, because custom rules run before Super Bot Fight Mode.

The free-plan version, Bot Fight Mode, is blunter: it cannot be scoped or skipped with WAF rules at all, because it does not run on Cloudflare's ruleset engine. It should leave verified bots alone, but if your Security Events show it acting on requests that verify as real Googlebot, your only real options are turning it off or upgrading to the configurable version. Most Googlebot events under Bot Fight Mode turn out to be fakes failing the IP check, which is the feature working as intended.

3. Under Attack mode was left on

Under Attack mode answers every visit with a five-second browser check that requires JavaScript before the page loads. Cloudflare's own documentation calls it a last resort that temporarily pauses access to your site. It is easy to switch on during a bad night and forget, and a crawler that cannot pass an interactive browser check never reaches your content.

30-second check: your Cloudflare overview page shows whether Under Attack mode is active. Good: security level is normal. Bad: Under Attack has been on for weeks. Fix: switch it off, wait a minute, and re-run the Search Console live test. Keep it for actual attacks, measured in hours, not as a permanent setting.

4. Managed WAF rules or rate limiting caught the crawl

Two quieter candidates show up in the event log the same way. A managed WAF ruleset, tuned aggressively, can score a burst of crawler requests as an attack. And a rate limit sized for human browsing can trip when Googlebot fetches many URLs in a short window, returning 403 or 429 mid-crawl, which is why Search Console sometimes shows a mix of indexed pages and forbidden errors on the same site.

30-second check: Security Events again, product column reading managed rules or rate limit. Good: the only matches are on suspicious paths like /wp-login.php probes. Bad: events firing on your ordinary content URLs against IPs that verify as Google. Fix: both accept exceptions: the same verified-bots skip rule covers managed rules, and rate limiting rules can be skipped for verified bots the same way. Loosening the limit for everyone is the worse fix; skipping verified bots keeps the protection intact.

5. The 403 is coming from your origin, not Cloudflare

If the live test still fails and Security Events shows the request passing through with no block event, Cloudflare is delivering a 403 that your own server generated. Common sources: a .htaccessdeny list, a security plugin on your CMS with its own bot filter, or your hosting provider's firewall sitting behind Cloudflare.

30-second check: compare the error page itself. A Cloudflare block shows a Cloudflare-branded page with a Ray ID; a plain or server-branded 403 page came from your origin. Fix: hunt the same way one layer down: server access logs instead of Security Events, then remove or scope the deny rule. The reverse-DNS verification from cause 1 works there too.

What to do next, in order

1. Run the URL Inspection live test. If it passes, the 403s are stale: request indexing and let recrawls clear them.
2. Open Security > Events, filter for Googlebot, and read the product column. It points at cause 1, 2, or 4 directly.
3. Reverse-resolve the blocked IPs. Blocked fakes are the system working; only verified Google IPs being blocked need a fix.
4. Add the cf.client.bot skip rule rather than deleting the protection that caused the block.
5. No block events but still failing? Check Under Attack mode (cause 3), then your origin server (cause 5).

How to confirm the fix worked

Re-run Test Live URL on the page that failed. It should now come back available; that single result proves the path from Google to your content is open. Then click Request Indexingon your most important affected pages and let the rest recover through normal recrawling. In the Pages report, the “Blocked due to access forbidden (403)” count falls over days to a few weeks as pages get revisited. If pages went fully missing during the block, our guide on pages stuck waiting for a crawl covers the recovery order, and the other platform guides cover the hosting layers under your proxy.

From experience: the status code lies more than you would think

We run automated SEO scans against firewalled sites every day, and the single most useful thing that practice taught us is that challenge pages do not reliably announce themselves with an error code. Some gates serve their “checking your browser” interstitial with HTTP 200. Our scan engine identifies those pages by their titles, the “Just a moment...” and “Attention Required!” interstitials, before trusting anything else in the response, because every other signal in a challenge page describes the gate, not your site. When you are debugging by hand, look at what the response actually is, not just its status.

The second lesson came from our own tooling on July 23, 2026: our scanner reported critical failures against game-scout.app, one of our own sites, which runs on Vercel rather than Cloudflare. Its firewall was gating automated clients, so our tool saw blocks while the site was fine in every browser and stayed fully indexed. A 403 to a tool is not a 403 to Googlebot; treating tool output as ground truth would have sent us hunting a Google problem that did not exist. We ended up building a real-browser fetcher just to see gated sites the way a person does. The Search Console live test is the only verdict that counts.

FAQ

Does Cloudflare block Googlebot by default?

No. Googlebot is on Cloudflare's verified bots list, and Cloudflare states that verified bots have historically been excluded from default bot configurations on every plan. When Googlebot gets a 403, a rule someone added is responsible: a custom rule, an IP or country block, a bot-management setting, or Under Attack mode left on.

How do I allow Googlebot without letting every scraper in?

Use a WAF custom rule with the Skip action and the expression cf.client.bot, which matches only bots Cloudflare has verified by IP. Never allowlist the user-agent string Googlebot: any scraper can send that string, and Cloudflare's verified-bot check exists precisely so you do not have to trust it.

What is Cloudflare error 1020 and does it hurt SEO?

Error 1020 means a Cloudflare firewall rule denied the request. If that rule is matching Googlebot, Search Console records it as a 403-class block and the affected pages drop out of Google as they get recrawled. Find the responsible rule by searching Security Events for the Ray ID shown on the 1020 page.

How long until pages recover after unblocking Googlebot?

Days to a few weeks, because each page must be recrawled before its error clears. Confirm the fix with a URL Inspection live test, request indexing on your most important pages, and watch the 403 count in the Search Console Pages report fall as recrawls happen. The log retention on Cloudflare's free plan is 24 hours, so verify soon after fixing.

In your AI assistant

Ask your AI about your own site

TrafficTriage plugs into Claude, Cursor, and ChatGPT. Your assistant runs the 8 checks, reads your Search Console data, and can even check your site every morning. Free account, no API key, nothing to install.

Sign up freeSee how it works