What was announced?
Previously, publishers had to maintain individual directives for every AI bot to restrict training on proprietary content. On October 30, 2026, Google, Microsoft Bing, and OpenAI announced joint support for a single shared signal in robots.txt.
The framework stems from the AIPREF working group at the Internet Engineering Task Force (IETF). Google presented the plan in the Search Central Blog (by John Mueller), targeting early 2027 in its crawler docs. Microsoft Bing will honor the preference starting February 1, 2027, while OpenAI scheduled implementation in the coming months.
Industry outlets like Search Engine Roundtable and PPC Land view this as the first unified consensus among major search engines and AI labs in the generative era.
| Provider | Commitment to Signal | Target Timeline |
|---|---|---|
| Honors the signal; Disallow without explicit Google-Extended rule treats Google-Extended as blocked | Early 2027 | |
| Microsoft Bing | Honors training signal when crawling; separate page-level grounding controls | From Feb 1, 2027 / Cut-off March 1, 2027 |
| OpenAI | Honors the signal; bot-specific rules (like GPTBot) remain operational | Coming months |
How does the new directive work?
The implementation is based on the IETF draft draft-kelley-aipref-training-preference-signal-00, introducing a policy token named __usage_ai-training placed like a standard User-Agent:
User-Agent: __usage_ai-training
Disallow: /
Allow: /presse/In this example, training is disallowed across the domain, while the press directory is explicitly permitted. Two specifications in the draft are fundamental:
First, wildcard declarations like User-Agent: * explicitly do not apply to this token—it must be stated directly. Second, it is not a crawl block: bots may still fetch and index pages.
The draft defines AI training as updating parameters of generative models. Categorization, indexing, or ranking is unaffected. This remains an individual draft without formal standard RFC status; specifications may evolve.
What does this mean specifically for Google and Bing?
With Google, the signal introduces an important side effect: adding Disallow for the token without an explicit rule for Google-Extended causes Google to treat Google-Extended as blocked.
Google-Extended governs training for Gemini models and grounding in Gemini apps and Vertex AI. Grounding means generative systems reference and link live web documents. Opting out stops citations in Gemini, though regular Google search rankings remain untouched.
Microsoft separates training and grounding: training is set in robots.txt, while grounding is governed by page-level tags. With DisallowExtended, Copilot citations remain permitted while third-party APIs are excluded. Microsoft respects the signal from February 1, 2027; sites using NoArchive must migrate before March 1, 2027.
What does this mean for your business?
Our Assessment: A unified signal simplifies rights management, yet remains a voluntary commitment without technical enforcement. For most businesses, visibility in AI answer engines (GEO) offers greater value than restricting training data. Our Recommendation:
- 1Clarify objectives: Decide whether you strictly reject model pre-training or whether you also want to avoid citations in AI search interfaces as well.
- 2Audit robots.txt: Review your configuration for inherited rules or blocks against crawlers like Google-Extended or GPTBot.
- 3Explicitly permit Google-Extended: If blocking training broadly but retaining Gemini citations, declare an explicit Allow directive for Google-Extended.
- 4Prepare Bing migration: If previously relying on NoArchive, transition to the new metadata tags before March 1, 2027.
- 5Maintain calm: Since directives take effect in 2027, there is no need for rushed changes. With clean Next.js development or a structured website relaunch, crawler rules are managed centrally in code. Read more in our guide on Google AI search source links.
ACR Assistenz
High-performance Next.js platform & technical search engine optimization
Frequently Asked Questions
Referenced Sources & References
- Google Search Central Blog: Giving publishers more choice with an AI training preference (Oct 30, 2026) — Official announcement by John Mueller supporting the shared robots.txt training preference(Accessed: 01.11.2026)
- Google Search Central: Documentation on Google crawlers and fetchers — Technical guidance on Google-Extended and training preferences starting early 2027(Accessed: 01.11.2026)
- Microsoft Bing Blog: Your Content, Your Choice – Clearer Preferences for Training and Grounding (Oct 30, 2026) — Deadlines starting Feb 1, 2027 and granular page-level AI grounding controls(Accessed: 01.11.2026)
- IETF Datatracker: draft-kelley-aipref-training-preference-signal-00 (Oct 30, 2026) — Individual IETF draft defining the __usage_ai-training policy token(Accessed: 01.11.2026)
- OpenAI News: Helping websites express AI training preferences (Oct 30, 2026) — Commitment to adopting the standardized robots.txt signal in upcoming months(Accessed: 01.11.2026)
Curious about what your robots.txt says today about AI training and generative search citations? We audit your crawler directives and implement an optimal setup before the 2027 regulations arrive.
Alexander Weise is the founder of WeiseMedia on Mariendorfer Damm in Berlin. He engineers high-performance Next.js platforms, bespoke CRMs, and specialized digital flagships with an obsession for speed, compliance, and conversions.