AI Search

Cloudflare's New AI Crawler Defaults Reshape Search Access

Cloudflare's new September 15 AI crawler defaults now block AI training and agent bots on ad-supported pages unless site owners opt in explicitly first.

By The Pull Desk·Today·5 min read
AI Search

Cloudflare's New AI Crawler Defaults Reshape Search Access

The short version

THE SHORT VERSION: As of September 15, 2026, Cloudflare's new AI crawler defaults block Training and Agent bots on ad-supported pages unless a site owner explicitly allows them — meaning B2B content can vanish from the AI systems buyers now use for research, regardless of how well it's optimized.

What happened

Cloudflare announced on July 1, 2026 that it would split AI bot traffic into three categories — Search, Agent and Training — and, starting September 15, 2026, flip the defaults so Training and Agent crawlers are blocked on any page that displays ads, unless the site owner turns them back on. Search bots, which drive referral traffic, stay allowed by default. Mixed-use crawlers that combine search and training get blocked entirely under the new Training default unless a site owner carves out an exception. Cloudflare also shipped BotBase, giving Enterprise customers searchable visibility into which bots are hitting their domains and how those bots are classified, plus a Pay Per Use marketplace with early partners Ceramic.ai and You.com. Details are at Cloudflare's blog.

Why AI crawler defaults matter now

From the publisher

MagnetizeX builds founder visibility systems for B2B firms.

See how →

Every AI Search post on this desk has covered citation behavior — who gets quoted, how often, by which model. This is the layer underneath all of that: if the crawler can't reach your page, none of it matters. Cloudflare sits in front of a significant share of the web, so a default-setting change here has internet-wide reach. A B2B blog or comparison page that runs on Cloudflare's free tier, displays any ad unit, and hasn't touched the dashboard since the old defaults may now be invisible to the same AI agents buyers are using to shortlist vendors — a self-inflicted visibility gap no amount of content optimization fixes.

  1. Check your Cloudflare bot settings today
    Log into the Cloudflare dashboard and review AI Crawl Control or bot-management settings for every domain that runs ads. If Training or Agent bots show as blocked and you want AI systems citing your content, switch them back on explicitly — the new default won't do it for you.
  2. Separate your ad-supported pages from your cornerstone content
    If your best buyer-education content sits on pages with ad units, decide deliberately whether you want AI agents and training crawlers reading it. A comparison page you want ChatGPT or Perplexity to cite needs different bot permissions than a monetized blog post you don't.
  3. Ask your web team which CDN and bot settings you're on
    Not every site runs Cloudflare, but most site owners don't know which CDN, WAF or bot-management layer controls their crawler exposure. Get a straight answer this week, because the same access question — search yes, training no, agent maybe — is coming to every major CDN.

By the numbers: Cloudflare says bots now outnumber human traffic on the internet for the first time, and Google's crawlers alone access roughly twice as much site content as other AI companies' crawlers.

"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."

— Matthew Prince, co-founder and CEO, Cloudflare (TechCrunch, July 1, 2026)

What to do this week

Spend 20 minutes this week in your CDN or hosting dashboard checking bot-permission settings on your three highest-intent pages — pricing, comparison, and "alternatives to" pages. If you're on Cloudflare, that's the AI Crawl Control panel; on another host, ask your webmaster directly which bots are allowed. Getting this wrong costs you AI visibility no amount of on-page optimization can recover.

Related in AI Search