AI Crawlers Don't Run Your JavaScript. That's Only the First Problem.

Here is the version of this story you have probably already read: AI crawlers can't execute JavaScript, so your React site is invisible to ChatGPT, so switch to server-side rendering. All of that is true. It is also where nearly every article on the subject stops.

The trouble is that rendering is the most visible failure, not the only one, and the same research everyone cites for the rendering finding contains three others that get almost no coverage. If you fix rendering and stop, you will have solved one of four problems and will likely conclude the whole thing was overblown when your citations do not move.

What did the study actually find about JavaScript?

That none of the major AI crawlers execute it. Vercel and MERJ analyzed real crawler traffic across nextjs.org and the wider Vercel network, validating against both Next.js applications and a custom monolithic framework, and the conclusion was unambiguous.

The non-rendering list covers OpenAI's crawlers (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider, PerplexityBot, and Common Crawl's CCBot.

The detail that makes it concrete: ChatGPT's crawlers fetched JavaScript files in 11.50% of requests and Claude's in 23.84%, and neither ran them. They downloaded your scripts and read them as text.

The scale is not marginal. In the month measured, GPTBot generated 569 million requests across Vercel's network and Claude 370 million, together around 20% of Googlebot's 4.5 billion. Add AppleBot and PerplexityBot and the four crawlers reached nearly 1.3 billion fetches.

Which crawlers do render, and why does it matter?

Two, and the exceptions are more useful than the rule.

Gemini renders because it uses Googlebot's infrastructure, giving it the same Web Rendering Service capabilities that let Google index client-side applications. AppleBot renders through a browser-based crawler, processing JavaScript, CSS, and Ajax requests for full-page rendering.

This is why "test it in Google" produces false confidence. A client-rendered page can index perfectly in Google, appear in Gemini, and be a blank shell to ChatGPT, Claude, and Perplexity simultaneously. Semrush's index of 126 million prompts found citation patterns differ sharply by platform for similar reasons, with mention-to-citation overlap as low as 30% on Gemini. The engines disagree, and the disagreement is architectural rather than editorial.

One nuance worth carrying: content in the initial HTML response can still be read even when it is not conventional markup. Vercel noted that JSON data or delayed React Server Components in the first response may still be indexed, because models can interpret non-HTML content. The requirement is not "avoid JavaScript." It is "put the substance in the first response."

How do you check what an engine actually sees?

View source, not inspect element. This distinction is the whole test and it trips up experienced people.

Your browser's DevTools Elements panel shows the rendered DOM, after JavaScript has run. That is what a human sees. View Source shows the raw HTML response, before any script executes. That is what GPTBot, ClaudeBot, and PerplexityBot get.

Open your highest-value page, view source, and search for a distinctive sentence from your main content. If it is there, AI crawlers can read it. If it only appears in DevTools, they cannot.

Do this on the pages that carry buying decisions rather than your blog: pricing, product capability, comparison, security, integrations. Those are the pages an engine needs to shortlist you, and in most stacks they are the most likely to be dynamically assembled.

For a harder check, request the page with curl and no JavaScript engine at all, or fetch it as a specific bot user agent and read what comes back. The gap between that and what you see in a browser is your exposure.

What is the 404 problem nobody talks about?

Roughly a third of AI crawler activity is wasted on pages that do not exist.

This is the finding I keep pointing people to, because it is enormous and almost nobody cites it. ChatGPT spent 34.82% of its fetches on 404 pages. ClaudeBot showed 34.16%. ChatGPT spent a further 14.36% following redirects.

Googlebot, by comparison, spent 8.22% on 404s and 1.49% on redirects.

Vercel's analysis of those errors found the crawlers repeatedly attempting to fetch outdated assets, particularly from /static/ folders. The crawlers have not developed the URL selection and validation sophistication that Google built over two decades.

The practical consequence: if a meaningful share of AI crawl activity on your domain is burning on dead URLs, the pages you care about are being fetched less than the raw request count suggests. Redirect hygiene, current sitemaps, and consistent URL patterns are not housekeeping in this context. They are the difference between your crawl budget reaching your pricing page or a deleted asset path from a 2023 rebuild.

Do the crawlers even fetch when they say they do?

Often not, and this is the fourth failure.

Vercel reported that even when asking Claude or ChatGPT for fresh Next.js documentation, they frequently saw no corresponding fetch in the nextjs.org server logs. The models appeared to answer from cached or training data while presenting the response as current.

That has an uncomfortable implication for anyone measuring this work. You can server-render everything correctly, fix every redirect, publish accurate current facts, and still be described by a model using information from months ago. The fix worked. The model did not look.

It also means log analysis and answer monitoring measure different things. Your logs tell you whether crawlers can reach you. The answers tell you what models currently believe. Those two can diverge for a long time, and the gap between them is not a bug in your implementation. Measure the answer side across repeated runs rather than single checks: SparkToro and Gumshoe.ai found the odds of an identical brand list appearing twice were under 1 in 100.

So what should you actually fix, in what order?

Rendering first, because it is binary. Then the three failures that determine whether the fix ever gets used.

1. Server-render the substance. SSR, static generation, or ISR for main content, meta information, and navigation structure. Client-side rendering is fine for view counters, chat widgets, and social feeds. The test is whether a buyer or an engine needs it to evaluate you.

2. Fix URL hygiene. Maintain redirects, keep sitemaps current, use consistent URL patterns. Given the 34% figure, this is not a minor cleanup task.

3. Publish the facts engines look for and buyers ask about. 6sense found 58% of buyers contacted sellers earlier than they otherwise would have specifically because vendor websites failed to answer questions about pricing, security, and AI capabilities. If a human had to ask, a model could not find it either.

4. Measure both sides. Server logs confirm access. Answer monitoring confirms belief. Watching only one produces a confident and wrong conclusion.

There is a version of this work that fails despite doing everything right, and it is worth naming so you can rule it out. If your pages are server-rendered, your URLs are clean, your facts are current, and models still describe you badly, the problem is not technical. It is more likely evidence or reputation: Semrush's analysis of 126 million AI prompts found that AI narratives draw heavily on customer reviews, community discussions, publishers, and retailers rather than owned websites alone. Rendering gets you eligible. It does not get you chosen. The Princeton GEO study found citations, quotations, and statistics produced the largest visibility gains of nine tactics tested, up to 40%, which is the work that starts once a page can actually be read.

What does this mean for how you audit?

Stop auditing your site the way a person reads it.

The most useful habit I picked up from this research was mundane: fetch the page the way a crawler does, and read exactly what comes back. Not a score, not a checklist, the actual bytes. The first time I did it on a page I was proud of, the response contained a nav bar, a footer, and an empty div where the argument was supposed to be. Google had been indexing it fine for a year.

That is the check Trovance runs continuously rather than the once-a-quarter way I used to do it by hand, because rendering is not a state you fix and forget. A framework upgrade, a new component, a marketing page rebuilt in a different tool, and a page that was readable last month is a shell again.

The broader point survives the tooling, though. The engines that decide whether you appear in a buying conversation are reading a version of your site you have probably never looked at. Look at it.

See how AI sees your brand

The part that stays true

Rendering will improve. The crawler operators are early, their 404 rates prove it, and it would be strange if OpenAI and Anthropic did not eventually build rendering pipelines the way Google did.

What will not change is the underlying requirement. Retrieval systems reward content that is available, unambiguous, and cheap to extract. Server-side rendering satisfies that today because it puts the substance in the first response. Whatever the crawlers can do in three years, being legible on the first request will still be the cheapest way to be understood.

FAQs

Do AI crawlers execute JavaScript?

No. Vercel and MERJ's analysis of real crawler traffic found none of the major AI crawlers render JavaScript, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider, PerplexityBot, and CCBot. They fetch the raw HTML response and read what is in it.

Which AI crawlers can render JavaScript?

Two exceptions. Gemini renders because it uses Googlebot's Web Rendering Service infrastructure, and AppleBot renders through a browser-based crawler that processes JavaScript, CSS, and Ajax requests. Every other major AI crawler measured reads only the initial HTML response.

How do I test whether AI crawlers can read my page?

Use View Source rather than DevTools. The Elements panel shows the rendered DOM after JavaScript executes; View Source shows the raw HTML an AI crawler receives. Search for a distinctive sentence from your main content. If it appears only in DevTools, crawlers cannot see it.

Can a page rank on Google but be invisible to ChatGPT?

Yes, and this is common. Googlebot renders JavaScript through its Web Rendering Service, so a client-side rendered page can index and rank normally. ChatGPT, Claude, and Perplexity read raw HTML only, so the same page can be a blank shell to them.

Why do AI crawlers hit so many 404 pages?

Vercel found ChatGPT spent 34.82% of fetches on 404s and Claude 34.16%, versus 8.22% for Googlebot. The errors clustered on outdated assets, often in /static/ folders, suggesting AI crawlers lack the URL selection and validation maturity Google developed over many years.

Will server-side rendering get my brand cited?

No. Rendering makes you eligible to be read, not certain to be cited. Semrush's analysis of 126 million prompts found AI narratives draw substantially on reviews, community discussions, and publishers. Technical access is a prerequisite, while evidence and third-party reputation determine selection.

Should I use dynamic rendering to serve crawlers different content?

Serve the same content, not different content. Dynamic rendering that returns materially different material to bots than to users is cloaking under Google's spam policy. Server-side rendering, static generation, and ISR solve the problem without that risk by making one version readable to everyone.