Allowing Googlebot does not allow GPTBot

Allowing Googlebot does not allow GPTBot, and allowing GPTBot does not put you in ChatGPT's live search index: those are separate user agents with separate jobs. OpenAI documents four user agents: a search indexer, a user-initiated fetcher, a training crawler and an ads-safety validator, each a separate permission. Google's crawler and user-agent overview splits its own fleet the same way. A robots.txt written for search engines in 2019 can silently block the agent that decides whether an answer engine may quote you today.

Short version, before the templates. Allow the retrieval and user-fetch agents for every engine you want to be quoted in, treat training crawlers as a separate business decision, serve real HTML, and file schema and sitemaps under clarity work rather than ranking work. Cloudflare recorded GPTBot's share of crawl traffic rising from 2.2% to 7.7% between May 2024 and May 2025, and its agent-readiness study of the 200,000 most visited domains found that while 78% had a robots.txt, only 4% declared AI usage preferences and 3.9% served Markdown to agents that asked for it.

Which AI crawlers should you allow in robots.txt?

Allow every agent that performs live retrieval or a user-initiated fetch, because those put your page inside an answer this week. Training crawlers are a separate decision with different economics. Allowing search while disallowing training is defensible; doing it by accident is not.

Treat each operator's documentation as the source of truth for the exact string; a misspelled user agent is a silent no-op.

Read the operator column, not the brand. OpenAI, Anthropic and Perplexity each split retrieval from training, so one string tells you nothing about the other agents flying the same flag. The row teams get wrong is Googlebot: Google states that AI Overviews and AI Mode need no special optimization and no new machine readable files, only indexed, snippet-eligible pages, so a Googlebot restriction is an AI Overviews restriction.

The training rows are a real business decision rather than a formality. By June 2026 Cloudflare attributed 52% of crawler requests to AI training, enough bandwidth to deserve an explicit answer.

The top-10 instinct is worth retiring too. Ahrefs found only 38% of AI Overview citations also rank in the organic top 10 across 4 million AI Overview URLs and 863,000 SERPs. Most cited pages now arrive through fan-out queries, an argument for site-wide crawlability rather than one keyword's position.

What does a robots.txt template for AI crawlers look like?

Short, explicit, and ordered so named groups carry their own allowances:

User-agent: Googlebot

User-agent: Bingbot

User-agent: OAI-SearchBot

User-agent: ChatGPT-User

User-agent: PerplexityBot

User-agent: Perplexity-User

User-agent: Claude-SearchBot

User-agent: Claude-User

Allow: /

Disallow: /cart/

Disallow: /account/

# Training crawl. Delete this group if you are not allowing training.

User-agent: GPTBot

Allow: /

Disallow: /cart/

Disallow: /account/

User-agent: *

Allow: /

Disallow: /cart/

Disallow: /account/

Sitemap: https://example.com/sitemap-index.xml

Every group repeats the same two Disallow lines, and the repetition is not clutter: a named group replaces the wildcard group rather than adding to it, so drop those lines and that agent is allowed everywhere. Path matching is prefix-based and case sensitive. The Sitemap line is global.

Robots.txt is a request rather than a lock; enforcement now happens at the network edge. Cloudflare shipped a pay-per-crawl mechanism built on HTTP 402, then an August 2026 AEO launch. Check what your CDN blocks by default before blaming your file.

None of it matters if the HTML is empty. The crawler study Vercel ran with MERJ found the major AI crawlers do not execute JavaScript, while Google's Gemini leverages the infrastructure of Googlebot, which renders JavaScript for Search. A client-rendered page can pass your Googlebot check and still hand an AI crawler a shell.

Do you need an llms.txt file?

Not for retrieval. No major engine has publicly documented using llms.txt to decide what it fetches or cites, and Google says outright that no new machine readable files or markup are needed. Anyone selling the file as a citation lever is ahead of the evidence.

That does not make it useless, it makes it cheap. A short markdown file at your root listing canonical pages with one line each is a usable map for agents you point at it:

# Example Co

> One sentence on what the company does and who it serves.

## Docs

- [Getting started](https://example.com/docs/start): first-run setup in ten minutes.

## Proof

- [Benchmark method](https://example.com/benchmarks): how the numbers were produced.

Budget it accordingly: an hour once, then leave it alone. C-SEO Bench (NeurIPS 2025) tested the current conversational-SEO rewrite methods and found most of them largely ineffective and often harmful to citation rank across two tasks and six domains. That benchmark did not test files like llms.txt, but it is a fair warning about any tactic sold as a citation lever before anyone has tested it.

Which schema types are worth shipping?

Four carry their weight on most B2B sites: Organization on every page, Article or BlogPosting on editorial, Product or SoftwareApplication on pages you sell from, and BreadcrumbList for hierarchy. FAQPage earns a place only where the same questions and answers are visible in the same words.

Structured data declares what a page is. It does not vote for the page. Google's introduction to structured data covers what the markup is for, the search gallery lists the types Search supports, and the schema.org getting started documentation covers the vocabulary. Types outside that list are notes to nobody.

<script type="application/ld+json">

{ "@context": "https://schema.org", "@type": "Organization",

"@id": "https://example.com/#organization",

"name": "Example Co", "url": "https://example.com",

"description": "One sentence on what you do and who you serve.",

"sameAs": ["https://www.linkedin.com/company/exampleco", "https://github.com/exampleco"],

"knowsAbout": ["Topic one", "Topic two", "Topic three"] }

</script>

Two fields do the work, and one of them gets oversold. The @id gives you a stable identifier that Article, Person and Product blocks point back at, which keeps your markup coherent. Do not stretch it past that: entity markup is clarity work, not a retrieval lever. In a 2026 study of 443 entity-oriented retrieval configurations on a news benchmark, entity signals added at most 0.051 MAP once they could not peek at the answers.

The sameAs array ties one name to the profiles that corroborate it, and the W3C provenance vocabulary's Entity, Activity and Agent model is the same instinct one level up: record who made a claim. Schema must describe what a visitor sees: Google's structured data guidelines say not to mark up content that is not visible to readers of the page, and breaking them can bring a manual action that removes rich-result eligibility. Mirror visible FAQ answers word for word and use ISO dates.

What belongs in your XML sitemap?

Only canonical, indexable URLs you would be content to see quoted back at you. Tag archives, filtered parameters, paginated series, staging pages and anything carrying noindex dilute the file. Two fields per entry is the whole job:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<url>

<loc>https://example.com/guides/ai-crawlers/</loc>

<lastmod>2026-09-02</lastmod>

</url>

</urlset>

Keep lastmod honest. It is the field crawlers use to decide what to re-fetch, and the one most often falsified by a CMS that rewrites timestamps on deploy. Freshness signals only work while they are true, and Google's guidance on succeeding in AI search points back at ordinary indexing work, not a separate AI file format.

Split a large site into a sitemap index rather than one oversized file, and point robots.txt at the index, not the children:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap><loc>https://example.com/sitemap-pages.xml</loc></sitemap>

<sitemap><loc>https://example.com/sitemap-posts.xml</loc></sitemap>

</sitemapindex>

Then check what engines keep. AirOps' March 2026 analysis of 548,534 pages ChatGPT retrieved found only 15% were cited, and pages whose titles overlapped the query by at least 50% were cited 20.1% of the time against 9.3% for low overlap. That trait lives in the page, not the sitemap. The file gets you fetched; it does not get you quoted.

How do you verify the whole stack is working?

Four checks, in order, each with a visible failure.

1. Fetch the page with JavaScript disabled and read what comes back.

2. Grep server logs for the agent strings you allowed and confirm they arrive.

3. Run your JSON-LD through a validator and clear errors before adding types.

4. Ask the engines your buyer questions and log the sources each answer cites.

Only the last check measures the outcome. The first three confirm the plumbing: Seer found 87% of SearchGPT citations matched Bing's top results, so retrievability gates the answer without deciding it, and more than 4 in 10 AI citations point to sources brands do not control.

Measure the answer as a rate. A robots.txt or schema change is exactly the kind of small, real effect a five-run sample will bury: a 2026 variance-components study put run-to-run variance high enough to hide it, and work on quantifying uncertainty in AI visibility arrives at confidence intervals wide enough to say the same. A few weeks of answers, in more than one phrasing and on more than one engine, before and after, or you are reading weather.

How does Trovance verify your crawler and schema setup changed anything?

Trovance treats the technical floor as the first thing to check and the last thing to take credit for. You define the buyer questions that matter; it sweeps them across AI engines every week, preserving each answer run with its citations so you can compare the record across a deploy.

That record makes a robots.txt edit or a schema release falsifiable. If your pages appear in no answer and as no cited source, the diagnosis points at retrieval and the fix list is the one above. If they are quoted while the answer still recommends a competitor, the gap is proof rather than plumbing. Share of answer across your tracked questions tells you which of the two you are in.

From there the work gets specific. The Brain keeps the claims you are entitled to make with the evidence behind each, and recommended actions arrive as a ranked queue of content worth making, backed by answer evidence, rather than generic audit items. Drafts are written in your voice from your author voice guidance and research sources, a person reviews and approves everything before it publishes, and the next analysis cycle reruns the questions.

What Trovance will not tell you is that allowing GPTBot buys a citation. It does not, and no honest system reduces this to a single AI visibility score that implies otherwise. Crawler access and clean markup remove reasons to be excluded; they do not create a reason to be chosen. The platform shows you which problem you have before you fund a quarter against the wrong one.

What should you do this week?

Work in the order that makes each later step measurable. Read your robots.txt line by line and check that every named group repeats its own allowances. Fetch three important pages with JavaScript disabled and confirm the words you care about sit in the raw HTML. Add Organization and Article markup.

Then do the two things most teams skip. Trim the sitemap to pages you would defend in an answer, and record a baseline: five buyer questions, each in two or three phrasings, on two engines, once a week for a few weeks, logged with the sources each answer cited.

Be honest about what this buys. Access and markup are the floor, tactics sit above them, and the C-SEO Bench result is a standing reminder that most of what gets sold above the floor does not survive testing. If you would rather have the before-and-after measured than assembled by hand, start a free Trovance trial and watch the same questions across the change.

Fix the mechanics

• Technical SEO for AI search - the wider checklist this page sits in.

• AI crawlers don't run your JavaScript - a fetchable URL is not a readable page.

• Do you need an llms.txt file? - the longer answer behind that section.

• Schema markup and AI citations - what structured data does not buy.

Check whether it changed anything

• What an agentic browser sees on your site - the same audit from the agent's side.

• A named entity is not a retrieval signal - why entity markup is clarity work.

• How to measure AI agent traffic - reading crawler hits in your logs.

• Measuring AI search visibility without one score - telling whether a fix moved answers.