Blocking AI Crawlers Is a Policy Choice, Not a Citation Switch

A public service-business website can want search discovery without welcoming every use of its content. Start with the purpose you want to permit or restrict, then inspect the operator's corresponding control. Allowing a training crawler is not evidence that a business will earn recommendations, and a broad block can reach uses the owner wanted to preserve.

The useful decision is specific: which public pages should be available for search, what training preference should the site express, and how should user-requested visits be handled? The worksheet below separates those decisions before anyone changes a setting. It draws on operator documentation reviewed on September 22, 2026, rather than a traffic experiment or a customer configuration test.

Should a service business allow AI bots or block them?

A service business seeking public discovery should review selective participation before choosing a blanket block. That means matching each control to an intended use, while keeping private or restricted material outside the public-discovery decision. OpenAI explicitly documents a search-versus-training choice; Google's Search controls have a different scope. Neither operator's permission settings establish that an answer will recommend the business.

I would start with the pages a prospective customer needs: service descriptions, coverage areas, pricing explanations, and evidence behind the offer. Decide which of those pages you want found before discussing bot names. A rights restriction, infrastructure concern, or contractual obligation can change that decision, even for a business that otherwise wants more discovery.

The working map has 3 purposes: search discovery, training preference, and user-directed retrieval. These are categories for this worksheet, not a claim that every operator has exactly 3 products or settings. OpenAI's bot documentation makes the distinction concrete through OAI-SearchBot, GPTBot, and ChatGPT-User.

Keep the business goal beside the technical choice. "Customers should find our public service page" is a different instruction from "exclude our site from ChatGPT search answers." Asking a developer to "block AI" leaves that disagreement unresolved and gives the implementation no clear acceptance condition.

Which OpenAI controls separate search from training?

OpenAI documents independent choices for OAI-SearchBot and GPTBot: a webmaster can permit the search bot while disallowing the training bot. Its documentation also identifies ChatGPT-User as handling certain user actions rather than automatic web crawling. Review those separately, including the limits around navigational links and user-initiated requests, instead of treating the names as interchangeable.

Figure 1 is an original evidence diagram of those documented roles. It describes controls and exceptions, not measured outcomes or a configuration that has been tested on your domain.

Source: Original diagram of OpenAI's bot documentation, reviewed September 22, 2026. No operator artwork or customer data reproduced.

For the first 2 lanes, OpenAI gives a direct example: allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not train its generative foundation models. That supports a selective policy. It does not establish deletion of earlier copies, compliance by other organizations, or a recommendation benefit from allowing training.

OpenAI says sites opted out of OAI-SearchBot will not appear in ChatGPT search answers, although navigational links can remain. Preserve that exception when reporting a result: seeing a link after opting out is not, by itself, proof that the documented search policy failed.

The third lane needs its own review. OpenAI says ChatGPT-User is not used for automatic web crawling and that robots.txt rules may not apply because users initiate these actions. "May not apply" should stay that narrow; it is not a finding that every directive is always ignored, nor a reason to treat a public preference as a complete access-control system.

Can Google Search AI features be treated as a separate switch?

Google's documentation places AI Overviews and AI Mode within Search. A page must be indexed and eligible for a Search snippet to qualify as a supporting link. Googlebot manages Search crawling, while the listed preview controls limit information shown in Search. Those controls should therefore be reviewed for their Search-wide trade-offs rather than described as an AI-only switch.

Figure 2 separates the Search surface from training and grounding in some other Google systems. It preserves the boundary in the source without extending it into a claim about every Gemini feature or the implementation details of Google-Extended.

Source: Original diagram of Google Search Central's AI features guidance, reviewed September 22, 2026. The linked Google-Extended specification was not reviewed.

Google lists 4 controls in this passage: nosnippet, data-nosnippet, max-snippet, and noindex. This is a list of controls to investigate, not a recommendation to apply all 4. Their individual implementation semantics need their own documentation review before a production change; copying the list into a ticket would skip the decision about what information should remain available.

For supporting-link eligibility, retain both conditions: 1 indexed page and eligibility for a Search snippet. Eligibility still does not establish indexing or serving, which Google explicitly says are not guaranteed. A successful request to a page is an earlier observation than a selected supporting link.

Google directs owners to Google-Extended for limiting AI training and grounding in some other systems. That establishes a separate documentation path. The linked specification was outside this review, so this article does not provide token-level instructions or claim an exhaustive opt-out scope.

What changed in Cloudflare's mixed-use crawler policy?

Cloudflare's September 15, 2026 announcement distinguishes Disallow AI Training from Block and changes how Block treats mixed-use training crawlers. Its description of Disallow AI Training allows Accountable mixed-use crawlers for search while blocking other training crawlers. Read this as the vendor's stated policy and inspect the relevant account configuration before assuming how an existing rule now behaves.

The announcement describes 2 materially different intentions. Disallow AI Training publishes the applicable no-training preference through Bot Preference Sync and preserves search access for Accountable mixed-use crawlers. Block, along with Block on pages with ads, now applies to all training crawlers, including mixed-use crawlers, according to Cloudflare's explanation.

That is a reason to revisit the setting's meaning, not evidence that a particular site has lost search visibility. Cloudflare explains that the earlier treatment avoided applying those blocks to mixed-use crawlers because doing so could also affect discoverability. A label retained in a dashboard can require a new policy discussion when its scope changes.

Keep 2 kinds of evidence separate when considering the Accountable designation: capabilities available today and commitments to deliver capabilities. Cloudflare explicitly includes both. The designation therefore cannot stand in for a test that a promised feature has shipped or that a particular operator behaved as expected on your site.

For a service owner, the next action is a review request with the desired outcome attached. Ask the person responsible for the account to identify the current setting, relevant rules, and documented behavior. This article has not tested a Cloudflare customer account and does not establish universal availability or enforcement across other infrastructure.

What should go into a crawler-policy change record?

Write the intended outcome, affected public pages, current settings, proposed change, and rollback owner before implementation. Preserve configuration evidence separately from search observations and business results. The record below is an original operational worksheet: it helps an owner approve a bounded change, but it is neither a ready-made robots.txt file nor evidence that a test has already happened.

I would propose 1 purpose-specific change at a time, where practical, so the review record stays interpretable. That is an operational recommendation, not an experimentally proven optimum. If several settings must change together, record the bundle honestly instead of attributing later differences to whichever setting seems most interesting.

A request carrying a bot's user-agent string can help inspect a configuration response. It is not proof that the request came from the real operator. Preserve the limits of the access evidence alongside the response, especially when sharing a screenshot with someone who did not run the check.

Use "unknown" for an absent baseline. Zero enquiries recorded in a measured period and no reliable enquiry tracking are different conditions. An owner making a policy choice deserves to know which condition applies before interpreting an apparent change as improvement or damage.

How should you observe the result without claiming causality?

Check whether the intended configuration was applied, then observe access, answers, and business activity as separate records. Keep buyer questions and their context stable enough to make comparisons useful. A before-and-after difference can prompt investigation, but this worksheet provides no controlled experiment that isolates a crawler setting as the cause of an answer, referral, or sale.

Use an operational check after implementation, followed by weekly observations for 4 weeks as a proposed review window. The 4-week period is a planning choice, not an established indexing delay or promised time to impact. A serious access problem warrants attention immediately rather than waiting for the calendar to finish.

For answer observations, preserve the wording of the question and the full answer, not just a favorable excerpt. Include cited source URLs and retain runs where the business is absent. Record account or location context when available, and note missing context rather than implying that all runs were equivalent.

Then decide what the observation warrants. An unwanted access restriction can justify rollback; an inaccurate description can justify investigating its cited source. Neither requires claiming that opening a crawler will solve the problem. A policy owner can make a defensible decision while remaining uncertain about business impact.

How can Trovance help you review answers after a policy choice?

Trovance helps you observe how AI answers the questions your buyers ask and preserve the answer, its citations, and its sources. That gives the crawler-policy record something concrete to examine after the access decision. Start with a buyer question that matters to the business, then keep the answer evidence available for the person reviewing what to do next.

Use that preserved material to compare the description of your services with the public pages you want customers to find. Inspect the sources an answer cites, distinguish an absent mention from a wrong statement, and bring the relevant evidence into the next content discussion. The review can stay focused on what the answer says instead of relying on someone's recollection of a chat.

For a small team, that is a useful continuation of the worksheet: the policy decision has an owner, and the answer review has source material. Start your free trial to see how AI describes your brand and choose which buyer-question answer deserves a closer look.