robots.txt

Google, OpenAI, Microsoft, Anthropic back AI opt-out rule

Four of the largest AI developers will honor a single robots.txt signal that lets websites refuse the use of their content for generative AI training. Publishers will no longer need to block each company's crawler one by one. Google Alphabet…