Blocking AI bots could also block Google: what changes on September 15

You publish a useful article. You want people to find it through Google, but you would rather keep it out of AI training. A setting called "Block AI bots" sounds like a reasonable place to start.

That choice deserves another look before September 15. Cloudflare says its training blocks will also cover crawlers it classifies as serving both search and training, naming Googlebot, Bingbot and Applebot. Customers can opt out of that change before the deadline. Cloudflare's announcement

For a small site, this is an awkward decision to revisit. Search might be how readers discover a project or an article they could never have known to look for by name. Wanting those readers while declining other uses of the writing seems reasonable enough.

The difficulty starts when one crawler serves several purposes and the controls operate at different levels. You can express a preference about how a company uses your writing. You can also refuse its requests altogether. Those choices have different consequences.

Two changes share the same date

Cloudflare separates the relevant activity into Search, which collects material for later answers; Agent, which visits on someone's behalf; and Training, which gathers material for models.

For new domains joining Cloudflare, the September 15 defaults will block Training and Agent activity on pages displaying ads. Search remains allowed by default. Separately, training blocks will expand to mixed-purpose crawlers, including through the legacy setting. That second change can affect an existing domain. Cloudflare's policy documentation

A headline about new domains and advertising is easy to dismiss if you run an established, ad-free blog. But those details concern the new-domain defaults. You still need to check whether the expanded training block applies to your existing configuration.

Cloudflare says the most restrictive applicable rule wins. A crawler classified as both Search and Training can therefore be blocked even when Search is allowed. Its announcement places the opt-out in Security settings. The mixed-purpose crawler change

I would save a screenshot of the current configuration before changing anything. A week later, when someone asks whether a traffic change coincided with a settings change, having the old values is much more useful than remembering that you clicked something about bots.

Google-Extended controls a different thing

Google documents a separate control called Google-Extended. Publishers can use it to restrict future Gemini model training and certain uses of their content to inform answers in Gemini Apps and Vertex AI. Google says this control does not affect inclusion or ranking in Google Search. It also has no separate HTTP user-agent string: Google uses existing crawlers to fetch the content. Google's crawler documentation

For a publisher who wants to decline those uses across the site, the corresponding group in an existing robots.txt file is:

User-agent: Google-Extended
Disallow: /

This is a product-specific preference. It is not a block on all Google AI features or all AI companies. It also cannot make a request pass through a firewall that has already refused it.

The two documents describe different controls. Google's token concerns permitted uses of content. Cloudflare's announcement concerns which requests its rules will block. Allowing a crawler to reach the page and setting limits on subsequent use are separate decisions.

That is the part I would explain to anyone responsible for both publishing and maintaining a site. Otherwise, a checkbox and a two-line text file can look like interchangeable ways of expressing the same wish.

A robots.txt rule cannot undo a firewall block

robots.txt relies on crawlers following its instructions. Google explicitly documents that limitation. It also explains that a URL blocked from crawling can still appear in search results when Google discovers it through links elsewhere. Google's guide to robots.txt

A refusal to serve the page has another consequence. Google says it does not index page URLs returning 4xx errors such as 403 Forbidden, and previously indexed URLs returning those errors are removed over time. It treats 429 separately. How Google handles HTTP errors

If traffic falls, check whether the crawler can still retrieve the article. Rewriting the introduction or changing its title won't repair a request that fails before Google gets the HTML.

There is also no basis for predicting that a particular site will vanish from search on September 15. This article describes an announced configuration change. The outcome for a site depends on its rules and the responses crawlers receive. A deadline is useful for planning a check; it cannot tell you what your traffic chart will look like.

What I would check on a small site

Start by writing down the outcome you want. For example: keep ordinary search discovery working, decline the Gemini uses covered by Google-Extended, and review access for other AI services individually. That is specific enough to compare with actual settings.

Then review Cloudflare's policy for the domain. Its documentation locates the controls under Security Settings > Configure AI bot policies, with options to allow a category, block it everywhere, or block it on ad-supported pages. Check the legacy Block AI bots setting too, if that is what you previously configured. Cloudflare's settings reference

If preserving search access is the intention, review the mixed-purpose opt-out before the announced date. Keep a note of what you selected and why. I would avoid changing several unrelated security settings in the same sitting, because it makes later troubleshooting unnecessarily difficult.

For evidence of blocking, inspect Security Events. Look at the action and the security feature responsible, together with the hostname, path, IP address and user agent. Cloudflare's event views can use sampling, so an empty filtered list is not proof that every request succeeded. Security Events documentation

Be careful with the crawler's identity. Anyone can send a request with Googlebot in the user-agent header. Google documents verification using its published IP ranges or reverse DNS followed by a forward lookup. A request from your laptop with that name attached does not establish what happens to genuine Google traffic. Verifying Google requests

Finally, inspect a few representative pages in Search Console and run the live test. Google distinguishes the saved index report from the current fetch. The live test checks access by Google-InspectionTool and does not guarantee indexing. Pair its result with evidence from real crawler requests, especially if your rules distinguish between bots. URL Inspection documentation

I would choose the homepage, a recent article and an older page that brings readers in. Record the results before the change and check those same URLs afterward. If something fails, you have a small, repeatable investigation instead of a general suspicion about SEO.

The choice should be understandable

I want publishers to have useful controls over automated access. I also think someone maintaining a personal blog should be able to understand what a training block will do without comparing several vendors' terminology.

If refusing training access also refuses a search crawler, the interface should say so beside the setting. I would like to see the affected crawler names there, while I am making the choice.