A benchmark can contain 102,025 AI responses and still leave the most important strategy question open: do those responses represent your buyers, prompts, engines, and market?
That question starts where a generic score stops. If you first need the measurement layers themselves, use the retrieval, citation, comparison, and recommendation audit before judging any blended benchmark.
A new vendor-produced preprint reports 102 brands, 3,508 tracking runs, about 15,815 brand mentions, and 149,912 citations across five AI engines from March through May 2026. That is useful production telemetry. It is not a market census, an independent benchmark, or evidence that a recommended content change caused an outcome.
The right audit has six parts: sample, denominator, outcome, coding, reproducibility, and causal boundary. The response count sits inside the first part. It cannot do the work of the other five.
What does a large AI response count actually prove?
A large response count proves that the research system observed many response-level events under its own collection rules. It does not prove that the sampled brands or questions represent a category.
The Ranqo preprint's production cohort includes 102 brands and 3,508 completed runs. Its unit is a (brand, prompt, platform, run) tuple, so one brand can contribute many observations. The useful denominator for a market claim is therefore not only 102,025. It is also the number and kind of brands, prompts per brand, engines per plan, runs per cell, and dates covered.
The paper names the sampling problem directly. Its brands skew toward SaaS, retail execution, fintech, and Indian DTC. The full customer roster, inclusion flow, category counts, and missing-run breakdown are not public.
That does not erase the data. It limits where the average can travel.
I downloaded the 14-page preprint and checked the methods, limitations, disclosure, and appendices before using the headline. The unusually honest sentence is buried in the limitations: this is a convenience-sampled cohort.
A team can keep the result as a directional prior. It should not turn the 72.9% Tier 1 figure into a benchmark for an early-stage B2B company without checking whether the sampled brands, regions, categories, and prompts resemble its own market.
Which denominator can change an AI visibility result?
The prompt and engine denominator can change the result before any brand evidence changes.
The paper uses six prompt categories: discovery, problem/solution, comparison, use case, expert, and brand research. Comparison and brand-research prompts can name the brand. The other four are normally unbranded. In the first-run analysis, branded prompts produced 94% to 100% recognition across engines, while unbranded recognition ranged from 12% to 52%.
That gap is partly built into the question. Asking "what does Razorpay do" and asking "which payment platform fits an Indian marketplace" do not test the same job. A visibility average that mixes both needs to disclose their shares.
The paper does this better than most dashboards. It reports the unbranded result as primary and notes that branded prompts were about 9% of first-run responses. It also shows prompt-category mention rates ranging from about 11% for problem and use-case questions to 23% for discovery questions.
Before accepting a benchmark, ask for four denominators:
| Denominator | What must be disclosed | Why it changes the number |
|---|---|---|
| Brands | Inclusion rule, category, stage, region, exclusions | A convenience sample can overstate patterns common to its strongest segment |
| Prompts | Exact set, branded share, categories, generation method | Naming the brand raises mention probability by construction |
| Engines | Models, retrieval state, plan access, run counts | Engines do not expose the same source or answer behavior |
| Runs | Repeats per prompt-engine cell, dates, failures | One response can be noise; carried-forward failures can flatten change |
A headline without these denominators is a product metric wearing a lab coat.
Repeated-run uncertainty deserves its own treatment. Six signals that beat a single AI visibility score explains how to preserve that variance instead of burying it in one point estimate.
Are the engines comparable inside one benchmark?
The engines are comparable only after the benchmark preserves their different retrieval policies instead of hiding them inside one average.
In the Ranqo setup, Gemini and Perplexity were always grounded. Grok used always-on retrieval. ChatGPT and Claude used web search on a weekly cooldown per brand, then answered from model parameters between search passes.
Source: Kumar, Generative Engine Optimization at Scale.
The paper's cross-engine visibility score weights ChatGPT at 0.30, Gemini, Perplexity, and Claude at 0.20 each, and Grok at 0.10. It renormalizes those weights when a plan includes fewer engines.
That formula is transparent, which is good. It is still a product decision. The weights do not emerge from observed buyer usage, equal retrieval conditions, or one shared sample. A movement in the composite may reflect the heaviest platform, a retrieval week, a plan subset, or a real change in mention rate.
Report each engine first. Keep prompt, model class, search state, run date, and raw answer attached. Build an aggregate only when the decision requires one, then label the weighting rule beside it.
How should reported findings stay separate from planned tests?
Reported observations should sit in a different table, state, and sentence tense from experiments the researchers intend to run later.
The preprint does this visibly. Table 4 lists six reported findings, including the brand-stature ladder, source composition, and sentiment noise. Beneath them are seven v1.1 protocols for future work, including an RCT after recommendations, a schema test, entity-first sequencing, and a web-search on/off experiment.
Source: Kumar, Generative Engine Optimization at Scale.
The distinction matters because the prose occasionally moves faster than the evidence. The paper says schema does little on its own, then identifies the schema regression as Protocol P4. It recommends entity-first sequencing, while the test of entity-first citation rate is Protocol P5. It proposes re-auditing after 14 to 30 days, but the closed-loop treatment study is still P3.
Those are hypotheses worth testing. They are not findings from the reported production cohort.
A second benchmark shows why restraint matters. C-SEO Bench tested more than 1,900 queries and 16,000 documents across two tasks, six domains, and four models. Most rewrite methods were largely ineffective, and some lowered citation rank. The statistics transformation reduced rank in 19 of 24 settings.
That controlled result does not invalidate the Ranqo observations. It blocks the shortcut from "this page type appears often among citations" to "publishing this page type will raise your visibility."
The same boundary applies when a dashboard exposes an apparent absence: a visibility gap is not a content brief, because observation still needs diagnosis before production.
What should an AI visibility methods audit record?
A methods audit should record enough detail for a skeptical reader to reconstruct what the number means, even when the private data cannot be rerun.
Use this six-part contract:
| Audit field | Minimum evidence | Fail-closed question |
|---|---|---|
| Sample | Inclusion rule, exclusions, category mix, period | Who is missing from this cohort? |
| Denominator | Brands, prompts, engines, runs, failures | What changed before the outcome was calculated? |
| Outcome | Mention, citation, rank, recommendation, or business event | Which event was actually observed? |
| Coding | Rubric, classifier, validation set, agreement, error rate | Could the label be a model or coder artifact? |
| Reproducibility | Data access, code, prompts, model versions, SQL | Can an outside team rerun any material part? |
| Causality | Control, assignment, intervention, window, confounders | Is this an observation or a treatment effect? |
The Ranqo paper is stronger than a vendor blog because it discloses its platform weights, 10,000 bootstrap resamples, manual tier rubric, conflict of interest, and missing causal test. It is weaker than a reproducible public benchmark because customer data, analysis code, production prompts, and SQL are not released.
Its tier result needs special care. The three groups contain 11, 36, and 55 brands. One author assigned tiers using Wikipedia, press, funding, and regional leadership.
Those are proxies for web prominence, close to the construct being compared with visibility. A second-rater agreement check remains future work.
I also checked the C-SEO Bench appendix because negative results often live there. Its four-model tables preserve standard deviations and corrected significance tests. That is a better basis for a bounded experiment than a list of tactics copied from a correlation table.
How can Trovance turn a benchmark into a bounded test?
Trovance can turn a benchmark into a local evidence question instead of importing its average as a strategy.
A team can use Trovance to observe how AI systems explain, cite, compare, and recommend its company for one defined buyer question. The raw answers, prompts, engines, dates, and citations create the local denominator the external benchmark cannot supply. Trovance then helps diagnose the evidence gap behind the result: missing proof, stale facts, unclear comparison language, weak third-party support, or legitimate bad fit.
Observation alone does not resolve the gap. A score cannot tell the team whether it should publish a proof page, refresh an existing comparison, seek outside validation, clarify the product, or do nothing. Trovance helps produce and publish the proof-backed asset that should exist while keeping the claim, source, caveat, and publication decision with a person.
That is why a team would use Trovance here. The benchmark becomes a hypothesis generator. The company's own answer trail becomes the decision evidence. Start your free trial.
