AI Visibility Tests Miss How Buyers Actually Prompt

A new arXiv preprint examined 8,133 multi-turn human-LLM conversations and found that the final message usually did not contain the full request visible across the session. In the commercial sample, the final prompt carried a median 35.6% of the session's unique user-side content vocabulary. The public PRISM sample landed at 36.4%.

The more useful result was categorical. At least one explicit request dimension appeared earlier but not in the final prompt in 50.3% of commercial conversations and 44.8% of PRISM conversations. A prompt panel that replays only the endpoint can therefore test a different request from the one the person assembled.

That does not make isolated prompts useless.

It makes their claim narrower: they measure answers to isolated strings, under a stated model and run protocol. They do not automatically measure how a buyer's question behaves inside a conversation.

What did the conversation study actually measure?

The study measured where explicit pieces of a user's request appeared across turns. It did not measure private intent, answer quality, brand visibility, citations, or purchase behavior.

Benjamin Tannenbaum, a researcher at Aiso, used 670 English commercial multi-turn conversations in a discovery-replication design. The external validation used 7,463 PRISM conversations from 1,389 participants. PRISM contributed 2,308 values-guided, 2,279 controversy-guided, and 2,876 unguided conversations.

The analysis excluded assistant text and examined only user turns. That matters because a later message may respond to an assistant's suggestion, error, or framing. The paper can show where user-authored evidence appears. It cannot separate a stable preference from a correction or a newly formed preference.

The researcher tracked two kinds of representation. The lexical measure asked how much of the session's unique user-side content vocabulary appeared in one prompt. A second measure used nine frozen cue families: budget, location, use case, attributes, time, alternatives, corrections, comparisons, and evidence requests.

Those rules are intentionally literal. A phrase such as "under $900" can trigger a budget cue. The rules cannot know whether that budget still applies three turns later unless the transcript says so.

I downloaded the full eight-page preprint and checked the cohort audit rather than treating the abstract as the method. The commercial pool began with 2,605 rows and ended with 1,749 unique usable records, including 670 English multi-turn conversations. PRISM began with 8,011 released conversations; 488 failed the English criterion and 60 carried a human-text PII flag.

The sample is substantial. It is not a random census of B2B buying. The commercial records are proprietary, while PRISM is recruited and protocol-driven. That limits the conclusion before any marketer turns these percentages into a universal benchmark.

What does the final prompt leave out?

The final prompt often leaves explicit constraints, alternatives, corrections, or evidence requirements in earlier user turns. It can also add something new, which means the endpoint behaves like another state update rather than a neat summary.

In the commercial corpus, the final prompt contained at most half of the session vocabulary in 68.4% of conversations. PRISM reached 74.3%. Yet the study's length-matched null expected medians of 34.4% and 35.6%, close to the observed 35.6% and 36.4%.

That is an important refusal. The lexical result does not prove semantic drift or careless users. Final turns are simply short relative to whole sessions. The useful question is whether information needed to interpret the turn remains elsewhere.

The categorical result gets closer. A history-only dimension appeared in 50.3% of the commercial sample, with a 95% confidence interval of 46.5% to 54.1%. PRISM's 44.8% estimate had a participant-clustered interval of 43.5% to 46.2%.

Among conversations with at least one detected dimension, the endpoint reproduced the full set in only 26.1% of 456 commercial conversations and 26.2% of 4,534 PRISM conversations. That denominator is narrower than the full sample and needs to travel with the statistic.

The endpoint was not merely incomplete. It introduced a previously unseen dimension in 17.9% of commercial and 19.3% of PRISM conversations. Omission and addition happened together in 8.7% and 7.6%.

Source: The Prompt Is Not the Query.

For a buyer, that pattern is ordinary. "Which one would you choose?" may inherit the budget and security requirement from earlier turns while adding a preference for the fastest implementation. Replaying the last sentence alone removes the basis for the choice.

For a visibility platform, the mistake is more subtle. The panel may faithfully run the exact text it was given and still mislabel the unit. It tested an endpoint string, not the accumulated request state.

Why does conversation depth weaken an isolated-prompt panel?

As observed conversations became longer, the final prompt represented less of the session while more detected state remained in history. The gradient is descriptive because users were not randomly assigned to conversation lengths.

At two user turns, median final lexical coverage was 54.5% in the commercial sample and 55.6% in PRISM. At five or more turns, those values fell to 18.2% and 17.6%.

History-only dimensions moved the other way. They rose from 36.5% to 72.2% in the commercial data and from 26.3% to 71.0% in PRISM between two turns and five-plus turns.

The fixed-depth analysis followed the same 187 commercial and 1,283 PRISM conversations through at least five user turns. Commercial cumulative lexical coverage rose from 24.5% after turn one to 86.8% after turn five. PRISM rose from 18.2% to 100% under the paper's vocabulary measure.

I also checked Figure 2 against the surrounding text before using it. The caption says the source-specific curves converge at five or more turns and warns that depth was not randomized. Cropping away that sentence would turn an association into something that looks causal, so the caption stays in the image.

Source: The Prompt Is Not the Query.

Longer conversations may involve harder tasks, more uncertainty, different users, or more assistant influence. The study cannot tell us which factor created the depth pattern. It can tell us that a panel built from single strings needs to disclose what it leaves behind.

Does this prove that history changes AI answers?

No. The paper did not rerun any final prompt with and without history, so it cannot estimate an effect on facts, sources, brands, recommendations, or constraint satisfaction.

This boundary is easy to miss because the measurement problem makes a causal effect plausible. If the budget appears only in turn two, surely removing turn two could change the answer. Maybe. The model may infer the missing requirement, ignore it even when present, or follow a stale assumption from assistant history.

The paper says a direct test should hold the final turn and model fixed, then compare three conditions: full interleaved history, user-turn-only history, and the isolated final turn. That experiment would separate accumulated user specification from assistant-established framing.

History is not automatically helpful. It can contain irrelevant text, superseded constraints, and model errors. A complete transcript may preserve more observed state while also preserving more contamination.

The paper's sensitivity checks make the descriptive finding harder to dismiss. Selecting one PRISM conversation per participant left 43.5% with a history-only dimension and 26.2% with complete final state. Requiring substantive endpoints raised median lexical coverage to 40.0% in both sources, while history-only dimensions stayed at 50.5% commercially and 45.2% in PRISM.

Still, stable observational results are not answer-effect results. The next study needs paired runs. Until then, "history changes brand visibility" is blocked.

How should a team test conversational AI visibility?

A team should preserve the session as the experimental unit, define a history policy, and report isolated prompts as a separate condition rather than treating them as the buyer's complete request.

Start with a small panel of real buyer jobs. For each one, write an opening request and a governed sequence of follow-up turns. Each turn should have a reason: add a constraint, compare an alternative, correct an assumption, or request evidence. Do not manufacture a chat merely to create more prompts.

Then run three matched conditions:

Keep the model, date, system instruction, sampling settings, and final user message fixed. Repeat runs because one answer cannot establish a stable citation or recommendation pattern.

Code the output at the level you can observe: facts stated, sources cited, companies mentioned, comparison criteria, recommendations, and constraint satisfaction. A mention is not a citation. A citation is not a recommendation. None of them is a purchase.

The report should include the denominator for each rate. "Recommended in 7 of 30 full-history runs" is inspectable. "Conversation-aware visibility improved" hides the model, prompt family, condition, and run count.

A useful result may be that the brand appears only when a relevant use case from turn one is retained. That is not proof that the page needs more keywords. It may point to missing use-case evidence, weak comparison language, poor product fit, or an assistant framing effect.

This is why prompt count is a weak substitute for experimental design. One hundred isolated variations can look broad while sampling the same shallow state. Ten well-designed sessions across three conditions can reveal where the answer actually changes and where it does not.

How can Trovance test the question behind the prompt?

Trovance is useful here because a prompt panel alone can show answer variance without explaining whether the tested input preserved the buyer's request. The operational gap is the missing bridge between an observed answer and the evidence decision that follows.

A team can use Trovance to observe how AI systems explain, cite, compare, and recommend the company across defined conversation conditions. The raw answer, source trail, prompt, and history policy need to stay attached so an isolated endpoint is never confused with a full session.

Trovance then helps diagnose the evidence gap behind the result. If the company disappears when a budget constraint enters, the next step is not automatically another article. The gap may call for a pricing explanation, a comparison page, clearer qualification language, product work, or no action because the company is a bad fit.

When a proof-backed asset is justified, Trovance helps the team produce and publish the page that should exist, with claims, sources, and human review intact. A later matched rerun can test whether the market explanation changed without promising a citation, ranking, recommendation, or sale.

Use Trovance to connect conversational answer evidence to the asset your team should produce.

FAQs

Is an isolated prompt panel useless for AI visibility testing?

No. It estimates how a model answers a defined string under a defined protocol, which is useful for monitoring and comparison. The problem is the label. If buyers build constraints across turns, the panel should not claim to represent the full conversational request unless it also preserves and tests that state.

What is conversation-conditioned request state?

It is the explicit task specification and discourse context needed to interpret the current turn within the observed session. The paper includes goals, constraints, alternatives, corrections, comparison criteria, and evidence requests. It avoids calling this true intent because a transcript cannot reveal every private preference or determine whether an earlier requirement still applies.

Did the study analyze real B2B buying conversations?

The paper describes 670 English commercial multi-turn conversations from a proprietary governed corpus, but it does not publish the transcripts or establish that every session involved B2B buying. PRISM adds 7,463 public conversations from recruited participants across guided and unguided protocols. Treat both as relevant conversation evidence, not a buyer census.

Why is the lexical coverage result not the main finding?

Short final turns mechanically contain fewer unique words than whole sessions. The paper's length-matched null nearly reproduced the observed median coverage, so the authors read vocabulary as local information availability rather than semantic loss. The categorical cue results are more useful because they identify explicit request dimensions found earlier but absent at the endpoint.

Should every test include the entire assistant history?

No. Assistant history can contain stale assumptions, irrelevant text, and model errors. A better protocol compares the isolated final turn, user-turn-only history, and full interleaved history while holding the final message and model fixed. The differences show whether accumulated user state or assistant framing is associated with the observed answer.

How many repeated runs are enough for a visibility finding?

This paper does not establish an optimal run count. Choose a denominator before testing, repeat each condition enough to observe variance, and report the raw counts with model, date, prompt family, and settings. A rate such as 7 of 30 is more honest than a percentage detached from its sample and protocol.

What should a team do when conversation conditions change the answer?

Inspect which facts, citations, companies, criteria, or recommendations moved and which request dimension changed between conditions. Then trace the result to available evidence. The right response may be a proof page, a pricing clarification, a comparison, product work, or no content action. Do not turn every difference into an automatic publishing brief.

Related Resources

• There is no such thing as an AI visibility score

• A visibility gap is not a content brief

• The front page of the Internet is now an answer

• Your homepage is not an agent brief

• The most-quoted study in AI search, quoted correctly

• A crawl is not an audience

See how Trovance turns an observed answer into an evidence decision