Why the Cloudflare settings deserve a fresh review
Cloudflare's September 15, 2026 announcement introduced Disallow AI Training and changed how its Block options apply to mixed-use crawlers. Cloudflare says choosing Block can also stop search crawlers such as Googlebot and Bingbot. Its Disallow AI Training option is intended to preserve search access while expressing or enforcing training restrictions, depending on the provider.
This does not mean every existing domain suddenly lost search access. Cloudflare describes migration rules for existing preferences. Review the settings on your own domain, including any separate firewall rules, before changing them. The announcement linked below explains provider-specific limits and rollout details.
Choose the control for the use you want to restrict
OpenAI documents separate controls for OAI-SearchBot and GPTBot. Allowing the first while disallowing the second separates ChatGPT search discovery from model-training permission. ChatGPT-User handles user-requested visits; robots.txt rules may not apply to those visits.
Google-Extended is a robots.txt control token, not a separate HTTP crawler identity. It controls certain Gemini training and grounding uses. Google states that it does not affect inclusion or ranking in Google Search. Do not describe it as a switch that blocks only training across every Google AI product.
| Crawler or token | Documented purpose | What to review |
|---|---|---|
| OAI-SearchBot | ChatGPT search discovery | Search permission and server access |
| GPTBot | Content that may be used for model training | Training preference |
| ChatGPT-User | Visits initiated by a user | Server access; robots rules may not apply |
| Googlebot | Google Search crawling | Search access and page indexability |
| Google-Extended | Certain Gemini training and grounding uses | Content-use preference, separate from Search ranking |
Test a small policy before editing the live file
The example below permits OAI-SearchBot on public paths, excludes /private/ for that agent, and disallows GPTBot. It is an illustrative policy for these two agents, not a complete robots.txt file or a universal AI opt-out. Preserve your existing groups, sitemap declarations and path restrictions when adapting it.
Paste the example into the AI crawler checker. Test https://example.org/guide and then https://example.org/private/guide. The search-agent decision should change from allowed to disallowed; GPTBot should remain disallowed on both. These are expected rule-matching results, not observations of requests from either provider.
User-agent: OAI-SearchBot
Allow: /
Disallow: /private/
User-agent: GPTBot
Disallow: /Verify the deployed response and the access logs
After an approved policy change, open the deployed robots.txt and compare its content with the version you tested. Check an intended public page and a deliberately excluded path. Record the URL, time, selected agent and matching rule so that another person can repeat the check.
Next, inspect server or CDN responses and verified crawler logs. A successful rule simulation cannot establish whether a firewall challenges a request, a login blocks the page, or the response contains the expected content. A user-agent string alone is not proof of the visitor's identity. Use the provider's documented verification procedure when interpreting logs.
Keep private content behind authentication. A robots rule expresses crawler policy; it is not a security boundary. The AI readiness checker can add page diagnostics, but neither a score nor an allow decision guarantees indexing or citations. For a missing Google result, follow the indexing diagnostic workflow.