What Your Site Looks Like to AI Crawlers and Agents

The version of your website that an AI crawler evaluates is usually the raw HTML your server returns on the first request. A person sees something else: the page after scripts execute, fonts load, and an accordion opens. That gap is where agent-readiness problems start, and it is measurable in an afternoon. Vercel's crawler research with MERJ found that several major AI crawlers fetch HTML without executing JavaScript, while Google's crawler passes pages through its Web Rendering Service. Same URL, two different documents, two different verdicts about what your company does.

The short version: several documented AI crawlers receive only your HTML in document order, your HTTP status codes, your links, and any structured data you shipped in the source. They do not fire the events that reveal click-gated, scroll-gated, login-gated or form-gated content. Agentic browsers such as ChatGPT Atlas are a different case, and their rendering behavior is undocumented, which makes raw HTML a safe working assumption and nothing stronger. This piece stays with what crawler operators and researchers have actually published.

What does an AI crawler actually receive when it opens your page?

It receives a document, not an experience. The response body, the status code, the headers, the anchor text of every link, and any JSON-LD in the source are available on the first fetch. Everything else is conditional on machinery the client may never run.

You can identify much of the traffic doing this. Google publishes its crawlers and their user agent strings, and OpenAI documents four user agents covering search indexing, on-demand user fetches, and training collection. Cloudflare's 2025 look at who is crawling the web tracked the AI crawler figure moving from 2.2% to 7.7%, which is the scale at which this stopped being a curiosity for most sites.

Two facts about that first fetch are easy to miss. Index position and citation overlap heavily but not completely: Seer found 87% of SearchGPT citations matched Bing's top results across 500 citations, while Ahrefs found 38% of AI Overview citations rank in the organic top 10 across 4 million URLs, down from about 76% a year earlier. A page a conventional index cannot reach is a harder page for an assistant to find, and a page that ranks well is still no guarantee of a citation. The second fact is that no exotic file format saves you: Google states plainly that no new machine readable files or markup are needed for its AI features, only that pages be indexed and snippet-eligible.

Which parts of your site disappear before an agent ever sees them?

Four categories vanish, and they are the four categories modern marketing sites lean on hardest. Client-rendered content is first: if the words arrive through a script, a crawler that does not run scripts receives an empty shell where your value proposition should be.

Second, interaction-gated content: anything behind an accordion, a tab, a carousel past the first slide, or a load-more button exists only after an event the crawler did not fire. Third, form-gated and login-gated content, including gated PDFs. Fourth, text baked into images without alt attributes, which covers a surprising amount of pricing detail on B2B sites.

Cloudflare measured agent readability across the 200,000 most visited domains, which is a fair signal that the question is now being asked at web scale. The reason it is worth asking about your own site is mechanical. Building for a browser that runs everything produces pages whose substance arrives late, and a client that stops at the first response gets the shell. That is the default outcome of a normal modern build, so the absence of an obvious defect on your site proves nothing.

The wrong fix is serving agents a stripped-down page that humans never see. Google's spam policies treat that as cloaking. The working fix is server-rendering or pre-rendering the content that matters, so the substance sits in the first response for everyone who asks for it.

How does an agent decide the page is about you?

By resolving a string into an entity, which is a step you can help or hinder. Entity-oriented retrieval has been tested across 443 configurations, which is where the mechanics of resolving a name to an entity get examined in detail. A brand that appears under three name variants, or describes itself differently on every page, arrives at that step as an ambiguous token.

The mechanics of fixing it are ordinary. Use one canonical name consistently, and one plain-language description of what you sell and to whom, written in the HTML where a first fetch can reach it. Then add structured data saying what the page is. Google's own introduction to structured data explains what markup is used for, its gallery lists the supported types, and the schema.org getting started guide covers the vocabulary itself.

Entity clarity buys you a shot at being read correctly. It does not buy the answer. Profound's citation research found 57% of AI citations point to sources brands do not control, which means a large part of how you are described was written by review sites and forum roundups. Your own markup cannot overrule what those sources say about you.

How do you look at your own site the way an agent does?

Run the fetch yourself, in this order, on the five or six pages that carry your commercial argument.

1. Request the page with a plain HTTP client and read the response body as text. If your product description is absent there, a client that does not render will not find it.

2. Load the page in a browser with JavaScript disabled and compare what remains against what you meant to say.

3. Check the status codes and redirects on every path an agent might follow, including old URLs still cited in third-party articles.

4. Expand every accordion, tab, and carousel, then ask which of that content also exists in the source. Anything missing from the source is invisible.

5. Validate your structured data against the supported types, and confirm the same name and description appear on the homepage, the about page, and the pricing page.

6. Read your robots.txt and your edge rules, and check whether they block the documented agents you are trying to appear in front of. Access is now a product decision: Cloudflare's pay-per-crawl work uses HTTP 402 to make that choice explicit.

Then look at the logs. The user agent strings published by Google and OpenAI let you separate assistant fetches from human sessions on evidence. Undisclosed and spoofed clients will still get through that filter, so treat the number you can attribute as a floor on agent traffic, and keep the raw log so you can revise the figure when a vendor publishes a new string.

What the logs will not tell you is whether any of it mattered. A crawl is a fetch, not a reader, and the two have come apart: Ahrefs measured a 58% lower average clickthrough rate for the top-ranking page when an AI Overview is present. Being read well and being visited are now separate outcomes, and only one of them shows up in your session count.

What does not work, even though everyone recommends it?

Rewriting pages in the hope of being cited more is close to a coin flip. C-SEO Bench (NeurIPS 2025) tested conversational-SEO methods and found only 3 of 54 unilateral conditions produced statistically significant citation-rank gains; the benchmark ran across two tasks and six domains, so the result covers a wide spread of material. Treat any agent-readiness checklist that promises citation lifts from phrasing as unsupported until it shows its conditions.

What survived testing is less exciting and more durable. The GEO study found statistics, quotations and cited sources improved on baseline by 41% on its visibility metrics, while keyword stuffing produced no measurable gain in the same benchmark; the published version appeared in the KDD 2024 proceedings. Evidence in the source HTML beats persuasion in the copy, and some persuasion actively costs you: scarcity and exclusivity framing measurably reduces how often a model recommends a product.

Measurement deserves the same skepticism. Repeated runs matter because one measurement cannot separate a real change from ordinary run-to-run variation, and a 2026 variance-components study examines exactly that decomposition. A single before-and-after comparison after a rebuild therefore proves very little. If you want a defensible read on whether an agent's picture of you changed, you need repeated runs and a record of what each one said, which is where the IAB's presence, prominence, portrayal and persuasion framing earns its place.

How does Trovance show you what agents see and say?

Trovance starts from the answer. You define the questions your buyers actually ask, and the platform runs them repeatedly across AI engines, preserving each answer run as a snapshot with its full context: who was mentioned, who was cited, who was recommended, and which sources carried the answer. That record is what turns a one-off observation into an appearance rate you can defend.

The agent-eye view of your pages then has somewhere to land. When an answer describes your category without you, the snapshot shows whether its sources were yours, a competitor's, or third-party pages neither of you controls. Answer coverage across tracked questions separates a retrieval problem, where nothing of yours is reachable, from a proof problem, where you are read and passed over. Your Brand Core holds the claims you are entitled to make and the evidence behind each one, so a recommended action names the specific missing asset.

Drafts are produced from approved claims and a person reviews everything before it publishes. Nothing ships on its own. That is a deliberate constraint, because the failure mode of automated content is confident text that your team cannot defend when a customer asks where the number came from.

What Trovance will not promise is control over what any model says about you, a guaranteed citation, or a single visibility score that collapses these questions into one number. The engines are probabilistic, most of the sources are not yours, and vendor rendering behavior is undocumented. What the system does instead is close the loop: after you fix a page or publish an asset, the next analysis cycle reruns the same questions and shows whether the answers moved, so you are deciding against current evidence.

What should you do this week?

Start with the fetch, because everything else is downstream of it. Pull your five highest-intent pages as raw HTML and mark which ones lose their argument without JavaScript. Fix those before writing anything new; server-rendered substance is mechanical work with a short feedback cycle, and it is the cheapest thing on this list.

Then take the interaction-gated content out of hiding: move the pricing detail, the comparison table, and the FAQ answers into the source, where a first fetch will reach them. Confirm one name and one description across your key pages, and validate the structured data you already have before you add types you cannot maintain.

Last, set a baseline before you claim improvement. Record how often you appear in the answers to your real buyer questions, across multiple runs and several days, and keep the sources each answer used. If you want that baseline without running it by hand every week, start a free Trovance analysis and see which of your pages an agent can actually read.

See what agents see

• AI crawlers don't run your JavaScript - the rendering gap in more technical detail.

• Your homepage isn't an agent brief - the page-level audit to run first.

• A named entity is not a retrieval signal - why markup alone does not resolve you.

• Can AI find the right page in your PDF? - the same problem inside documents.

Measure what happens next

• A crawl is not an audience - why fetch counts are not demand.

• How to measure AI agent traffic - separating agent fetches from human sessions in your analytics.

• An agent-ready website can still miss the shortlist - readability is necessary, not sufficient.

• How cross-platform agents build product shortlists - what happens after your page is read.

FAQs