Cloudflare begins blocking AI bots by default on millions of websites

Cloudflare ↘ 8.28% began enforcing new default rules on Tuesday that automatically block AI training and agent bots on millions of websites, marking one of the largest shifts yet in how publishers manage their relationship with the AI industry.

What Changes Today

Under the new policy, any new domain joining Cloudflare and all existing free-tier customers who did not adjust their settings before the deadline will have AI training and agent crawlers blocked by default on pages that display advertisements. Search crawlers — those that index content so users can find it later — remain allowed.gulfnews+3

The company first announced the changes on July 1, 2026, in what it called its second annual "Content Independence Day," giving site owners a roughly two-and-a-half-month window to configure preferences before the defaults took effect. Cloudflare sits in front of more than 20% of web domains, according to its own disclosures, giving the policy broad reach across the internet.cloudflare+2

A Three-Category System

Rather than treating all AI crawlers alike, Cloudflare now sorts automated traffic into three categories: Search, Training, and Agent. Website owners can block each category independently, and for each can choose to block on all pages, block only on ad-carrying pages, or allow access entirely.developers.cloudflare+2

A key wrinkle involves multi-purpose crawlers. Bots like Googlebot, Applebot, and BingBot that serve both search indexing and AI training functions will be subject to the most restrictive rule that applies. That means sites that have opted to block training crawlers — including those that previously toggled the legacy "Block AI bots" setting — could inadvertently block Google's Alphabet Inc. ↘ 1.94% crawler as well.cloudflare+3

The Publisher Tension

The policy reflects a growing frustration among publishers over AI companies consuming their content without sending meaningful traffic in return. Traditional search engines crawl and index pages, then direct users back to the source. Generative AI services often extract information and present answers directly, reducing visits to the original site.gulfnews

Cloudflare is also providing some customers with analytics showing how often individual AI operators crawl their sites compared with the referral traffic they generate. CEO Matthew Prince framed the effort not as an outright block on AI but as a way to let website owners distinguish between services that help people discover content and those that consume it for other purposes.marketscreener+2