Google tests global AI opt-out for publishers after UK order and Cloudflare ultimatum

Google Alphabet Inc. ↗ 3.92% is confronting a growing revolt from publishers, regulators, and infrastructure providers over its practice of using a single web crawler to serve both traditional search and its AI products — a strategy critics say gives the company an unfair advantage while draining the economic life from the open web.

The Crawler Problem

Googlebot has for years scoured the internet to power search results. But as columnist Parmy Olson reported in Bloomberg on Monday, the line between content collected for search and content used to generate AI answers has blurred, with the same bot now feeding both Google Search and its Gemini AI model. Google's crawler sees 3.2 times more of the web than OpenAI's and 4.8 times more than Microsoft ↗ 6.75% Corp.'s, according to Cloudflare ↘ 2.95% CEO Matthew Prince, whose firm acts as a traffic gatekeeper for roughly a fifth of the web.bloomberg+2

The bundling creates a dilemma for publishers: blocking AI training means losing the search visibility they depend on for revenue. "Humans will be a rounding error on the internet versus what their agents and their bots are doing," Prince said. AI agents now generate more than 57% of web traffic in 2026 compared to about 42% for humans, according to Cloudflare data — the first time machine activity has surpassed human activity online.tbsnews+1

Publishers Feel the Squeeze

The consequences are already visible. Media group Reach has reported a 50% drop in Google referral traffic as AI Overviews reduce the need for users to click through to original sources. Broader data tells a similar story: less than one-third of Google searches now result in a click to an external website, according to a SparkToro analysis published in June. Small publishers have lost as much as 60% of search referral traffic over two years, per Chartbeat data.searchengineland+1

Resistance Builds

Two forces are now pressing Google to change. In June, the UK's Competition and Markets Authority ordered Google to let publishers opt out of AI features while remaining in search results, a move the BBC called a "world first". Google confirmed it is testing the opt-out globally once UK trials conclude.www+3

Meanwhile, Prince's Cloudflare issued an ultimatum: from September 15, the firm will block mixed-purpose crawlers by default on ad-supported pages, potentially cutting Google off from millions of websites. Internal Google slides from 2024, disclosed in a US antitrust case, revealed the company had considered offering an opt-out before launching AI Overviews but chose not to because it was "evolving into a space for monetisation".tbsnews+3

Google still controls roughly 90% of the search market. Whether the company will fully separate its crawlers — rather than simply asking publishers to trust a toggle — remains the central unresolved question.