Newsletter Subscribe
Enter your email address below and subscribe to our newsletter
[forminator_form id="25163"]

cloudflare+1cloudflarecloudflareCloudflare on Tuesday rolled out new tools that let website owners block AI companies from training on their content without sacrificing visibility in search engines — resolving a tension that has plagued publishers since generative AI reshaped the economics of the open web.
The feature, called Disallow AI Training, went live on September 15 alongside a new "Accountable" designation that Cloudflare created after weeks of talks with major crawler operators. Apple , Google Alphabet Inc. , and Microsoft each qualified as Accountable by meeting or committing to four requirements: a robots.txt-based training opt-out, a mechanism for opting out of AI summaries, URL-level transparency into how content is used, and an assurance that refusing training will not affect search rankings.cloudflare+1
The system replaces Cloudflare's older "Block AI Bots" toggle with three independent controls — Search, Training, and Agent — giving site operators granular authority over each type of automated traffic. Under Disallow AI Training, Accountable mixed-use crawlers such as Googlebot, Applebot, and Bingbot continue indexing pages for search, while dedicated training crawlers from Amazon, Anthropic, Meta, and OpenAI are blocked outright.businesswire+2
"This is how we make the Internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used," said Matthew Prince, Cloudflare's co-founder and CEO.businesswire
Existing customers who previously selected "Block" or "Block on pages with ads" for training will be migrated automatically to Disallow AI Training, and Cloudflare says most site owners need not take any action. New domains that monetize with advertising will default to allowing search while disallowing training and blocking agents on ad-carrying pages.gulfnews+3
The practical effect at each search engine varies. At Google, the setting works through a Disallow rule for Google-Extended, which Google's documentation says does not affect search ranking. At Apple, a parallel Applebot-Extended directive serves the same function. Microsoft's Bing does not yet support a robots.txt no-training preference; that capability is targeted for early 2027, and in the interim the existing NOARCHIVE meta tag remains Bing's training opt-out.searchenginejournal+1
Cloudflare framed the launch as the first step in a broader campaign. Mixed-use crawlers accounted for 36.6 percent of verified crawler traffic on its network — the single largest category — yet only 17 percent of sites restricted training while fewer than 1 percent blocked search. The company said its next focus is AI summaries, where a binary opt-in or opt-out is "too blunt." By early next year, Cloudflare aims to let publishers control how much of their content appears in AI-generated answers, set once on its platform rather than with each operator individually.cloudflare+1