OpenAI, Google, Microsoft and Anthropic will honour one AI training opt-out. Should your website use it?

Every few weeks a client asks me some version of "is AI stealing my website?" The honest answer has been messy: each AI company had its own crawler name, its own rules and its own small print. Blocking them meant keeping a list, and you never knew if you'd caught them all.

Yesterday that got simpler. Not solved, but simpler. And it forces a question most business owners haven't thought through: do you actually want AI to stop learning from you?

What is the new AI training opt-out token?

As AI Weekly summarised The Verge's report, the four companies are backing one shared signal for publishers. It comes from an Internet-Draft called "Machine-Readable Training Preference Signal", written by Anthropic's standards lead Kevin Kelley and dated 30 October. It's an individual draft, so it has no formal IETF standing yet, but the four biggest labs have said they'll follow it.

In practice it is two lines in the robots.txt file that already sits at the root of your website:

The draft was shaped with input from organisations including Cloudflare, News Corp, the Financial Times, the BBC, the EFF, Creative Commons and Common Crawl, according to AI Weekly. A write-up of SiliconANGLE's coverage on DEV confirms the same token and timelines.

Does blocking AI training remove my website from ChatGPT or Google answers?

No, and this is where most people will get it wrong. The token is about training: what goes into the next model. It is not about fetching: the AI visiting your page today to answer someone's question.

• Search is untouched. According to Martin Cid Magazine's breakdown, Google says Google-Extended doesn't affect Google Search, and Microsoft says its settings don't change whether you appear in Bing.
• AI answers are a separate switch. Microsoft has a separate "grounding" preference, set page by page with a meta tag or HTTP header, for content its AI pulls in to answer questions in Bing, Copilot and Microsoft 365. Microsoft warns that saying no there significantly reduces your visibility in those answers.
• The past stays in. Changing robots.txt doesn't remove content already collected into datasets or trained models. It only looks forward.
• It's a request, not a lock. OpenAI describes the token as "a coordination mechanism rather than a security control". Polite crawlers follow it. Bad scrapers don't.

Why it matters to you

For a small business, the real question isn't "can I stop AI?" It's "what do I want AI to know about me?"

If people ask ChatGPT, Claude or Gemini "who's a good accountant in Valencia?" or "best AI coach for small business", the models answer from what they learned in training and from pages they fetch live. A business that blocks training today might, over time, be known less well by the next generation of models. That's a real trade-off, and I wrote about the bigger picture in your next customer won't look at your website, it will read it.

On the other side, if your product is your content (courses, photography, paid newsletters, recipes, templates), training on it for free is a different matter. For you, this token is the first opt-out that works across all four big labs with one line.

My rule of thumb:

• You sell a service and want clients to find you: leave training on for your public marketing pages. Being known by AI is free marketing.
• You sell content: block training on the paid or premium parts, and keep your sales pages and "about" open.
• Either way: don't block search crawlers or AI answer features by accident while trying to block training.

How to set your AI training preference in robots.txt

1. Look at what you have now. Type yourdomain.com/robots.txt into your browser. Note any lines mentioning GPTBot, ClaudeBot, Google-Extended or User-agent: * with Disallow: /. Some site builders and security plugins added AI blocks without telling you.
2. Decide page by page, not site-wide. Write two lists: pages you want AI to know (home, services, about, pricing, FAQ, blog) and pages you'd rather keep out of training (paid content, client areas, downloads).
3. Add the new group only if list two isn't empty. Use User-Agent: __usage_ai-training, then a Disallow: line for each folder you want out of training, or Disallow: / for everything. Remember the wildcard group doesn't count.
4. Keep your old rules. Don't delete existing bot-specific lines during the changeover. Google and Microsoft won't follow the new token until 2027.
5. Check nothing blocks search. Make sure there's no blanket Disallow: / under User-agent: * unless you truly want to vanish from search engines.
6. Set two reminders. 1 February 2027 (Microsoft starts honouring the token) and early 2027 to check Google's progress. If you use Wix, Squarespace, Shopify or WordPress and can't edit robots.txt directly, ask whoever manages your site.

Copy this prompt: your AI training policy in 10 minutes

Paste this into ChatGPT, Claude or Gemini with your current robots.txt. It helps you decide and writes the lines for you. Check the result before you publish it: a wrong Disallow line can hide your site from search.

You are a careful website advisor for a small business.
Help me decide which parts of my website AI companies may
use to train their models, and write the robots.txt lines.

MY BUSINESS: [e.g. physiotherapy clinic, online course creator]
HOW CLIENTS FIND ME: [e.g. Google, Instagram, referrals, ChatGPT]
WHAT I SELL: [service / content / products]
MY WEBSITE SECTIONS: [e.g. /services, /blog, /courses, /members]
MY CURRENT ROBOTS.TXT:
[paste it here]

Context: OpenAI, Google, Microsoft and Anthropic will honour
a shared robots.txt group "User-Agent: __usage_ai-training"
that only covers AI training. It doesn't affect search,
doesn't stop crawling, and "User-agent: *" doesn't cover it.

Then:
1. Ask me up to 5 questions if anything is unclear.
2. Recommend which sections to keep open to AI training and
   which to block, with one sentence of reasoning each.
3. Flag any existing line that might be hiding my site from
   search engines or AI answers by mistake.
4. Write the full updated robots.txt, keeping my old rules.
5. List what this will NOT protect, in plain English.

Mark anything you're unsure about as a guess.

Also today in AI

Comcast cuts call-centre jobs as AI agents take over

Comcast filed WARN notices for 912 jobs (634 in Philadelphia, 119 in New Jersey and 159 in Washington state), effective 31 December, while planning more than 1,100 new hires, according to AI Weekly. GadgetReview reports that Comcast, which lost 144,000 broadband customers last quarter, works with AI customer-service firms Cresta and Sierra, and that AI agents have removed more than 30 million service interactions this year and cut contact rates by roughly 20%.

What it means for you: if a giant can move a fifth of its customer contacts to AI, a small business can start with its top ten repeated questions. Write the answers once, put them on your site and in an assistant, and keep a human for the tricky ones. My post on companies rehiring the people AI replaced covers what goes wrong when you cut people first.

Google's first data-centre satellite is online

Google says its Project Suncatcher prototype, launched on a SpaceX Falcon 9 on 1 October, is now operational, and that its first test was Gemma 4 writing a haiku from orbit, as reported by AI Weekly. The satellite carries four TPUs but can power only one at a time, for about 20 minutes on a 1 kW budget. NewsBytes notes it is testing whether the chips survive radiation and extreme temperatures. Two more satellites with newer chips are planned for 2027.

What it means for you: nothing to do today. It's a signal of how hard AI companies are hunting for energy and computing power, which is why AI prices and usage limits keep shifting. Don't build your business on one tool's current price.

UK regulator investigates Meta's terms for publishers

The UK's Competition and Markets Authority opened a formal investigation on 30 October into whether Meta's updated commercial terms, which would let it use UK publishers' website content for its own products and AI features without a choice, abuse a dominant position, according to AI Weekly. Meta's revisions, due on 20 November, have been paused.

What it means for you: if you run ads or a business page on Facebook or Instagram, read the terms you accept. I unpacked what Meta's recent business terms mean for liability in Meta's new terms make you pay for its AI's mistakes.

FAQ

Sources: AI Weekly: OpenAI, Google, Microsoft, Anthropic back one AI opt-out token (30 Oct 2026, summarising The Verge); Martin Cid Magazine: One robots.txt line will opt your site out of AI training (31 Oct 2026); DEV Community, based on SiliconANGLE (31 Oct 2026); AI Weekly: AI news today (Comcast, Google Suncatcher, CMA and Meta, 30 Oct 2026); GadgetReview (Comcast, 26 Oct 2026); NewsBytes (Suncatcher). The rule of thumb and steps are my own advice.