Cloudflare blocks AI crawlers by default: are you blocking yourself?
Photo: Cloudflare office entrance area (2021), HaeB / Wikimedia Commons, CC BY-SA 4.0.
You worked hard writing GEO-ready articles to get cited by ChatGPT or Perplexity. But if your website runs on Cloudflare and you never rechecked the configuration, your own infrastructure may be quietly shutting those AIs out.
Since July 1, 2025 - dubbed "Content Independence Day" - Cloudflare, the internet infrastructure company serving roughly 20% of global websites, blocks unlicensed AI crawlers by default on every new domain. The reason: AI platforms take content for free to train and answer with, while referral traffic back to source sites collapses - in some cases 750 times lower than traditional search, per Cloudflare's published data. If your goal is to be cited by AI (GEO), you must actively check and allow the AI crawlers that matter, instead of letting the default block you by accident.
What is Content Independence Day?
- Default blocking for new domains: every new domain registered through Cloudflare automatically blocks unlicensed AI crawlers unless the owner opts in.
- Full owner control: you choose which AI crawlers may access the site and for what purpose (model training, answer inference, or search) - each type is distinguished.
- A "pay-per-crawl" service: site owners can charge AI companies per crawler visit instead of giving content away free.
Source: Cloudflare Blog, "Content Independence Day: no AI crawl without compensation!" (July 1, 2025), via MIT Technology Review.
Why does this matter for your GEO strategy?
GEO (Generative Engine Optimization) only works if AI can actually read your content. There are 2 separate blocking layers to check:
- The infrastructure layer (Cloudflare or another CDN): with "AI Crawl Control" blocking by default, bots such as GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and Google-Extended are refused before they even reach your robots.txt.
- The website layer (robots.txt): even with the infrastructure open, a
Disallowline for those bots in your own robots.txt still stops them from crawling.
If only 1 of the 2 layers is open, AI crawlers are still blocked at the other.
How do you check your website quickly?
- If you use Cloudflare: open the Dashboard → Bots or AI Crawl Control, review the list of blocked AI crawlers, and re-enable the bots you want (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended).
- Check robots.txt: open
yourdomain.com/robots.txtdirectly, find theUser-agentlines matching those bot names, and make sure there is noDisallow: /for the AI bots you want to allow. - Consider adding llms.txt: a summary of your brand, services and key pages written for AI - not required, but increasingly added so AI describes the brand accurately.
There is no one-size-fits-all answer. If the goal is AI citations and brand reach (a GEO strategy), open the door to the mainstream AI crawlers. If your content is proprietary and commercially valuable (research data, paid content), consider blocking or using pay-per-crawl to get paid fairly. See how to optimize for AI Overviews and What is GEO for the full picture.
Frequently asked questions
When did Cloudflare start blocking AI crawlers?
From July 1, 2025, dubbed Content Independence Day, Cloudflare began blocking unlicensed AI crawlers by default on every new domain registered through it, unless the site owner actively allows them.
Is my website blocking AI crawlers?
Check 2 places: your Cloudflare configuration (the AI Crawl Control section, if you use Cloudflare) and your website's robots.txt. If robots.txt has Disallow lines for GPTBot, ClaudeBot, PerplexityBot or Google-Extended, those AIs cannot crawl your content.
Should I block AI crawlers?
It depends on your goal. To be cited and named by ChatGPT, Perplexity and Google AI Overviews (a GEO strategy), you need to allow these crawlers. To protect proprietary content from uncompensated AI training, consider blocking or pay-per-crawl.