6 min
Zach Chmael
In This Article
A six-dataset retrieval study found question-to-question matching raised average claim recall, but it did not test public websites or AI citations.
Updated
TL;DR
🧪 The experiment compared three retrieval pipelines across six datasets.
📈 Question-matched indexing reached 71.5% average claim recall, versus 53.6% for naive retrieval.
🎯 Average context precision reached 63.5%, versus a 42.3% baseline.
⚠️ Relevant-context noise sensitivity worsened to 21.0%, compared with 13.8% for naive retrieval.
🚫 The study tested retrieval depths of 1, 3, 5, and 10 chunks, not public AI search rankings or citations.
Can Likely Buyer Questions Improve AI Retrieval?
A controlled retrieval study found that representing evidence with likely questions raised average claim recall from 53.6% to 71.5%. Average context precision rose from 42.3% to 63.5%. The mechanism is useful: a question may be easier to match with another question than with the declarative paragraph holding its answer.
There is a catch. The experiment tested an internal RAG index across six datasets, one embedding model, and one generator. It did not test public websites, AI citations, or FAQ pages. Its best-performing method was also more sensitive to noise in relevant context, scoring 21.0% versus 13.8% for the baseline, where lower was better.
The practical answer is to map real buyer questions to sourced claims, then test whether the right evidence appears. Do not turn the paper into another unsupported rule about adding question headings everywhere.
What did the question-matching study test?
The study tested whether an internal retrieval system could find better evidence by matching a user's question against precomputed question-like embeddings instead of matching it directly against declarative document chunks.
Vake, Vičič, and Tošić compared Naive RAG, Hypothetical Document Embeddings (HyDE), and their proposed Hypothetical Prompt Embeddings (HyPE). The six benchmarks covered web passages, Wikipedia fact questions, mixed RAG tasks, multi-hop reasoning, general knowledge, and narrow-domain articles.
The dataset sizes were wildly uneven. MS MARCO contained 82,326 question-answer pairs, while Single-Topic RAG contained 80. RAGBench had 73,286 pairs, and Ragas-WikiQA had 232. The aggregate table gave each dataset equal weight rather than pooling every question.
I downloaded the 12-page paper and checked the dataset table, method, full result grid, and significance section before using the averages. Two datasets were split into chunks of up to 500 tokens with 50-token overlap. The other four arrived pre-segmented. Every pipeline used bge-m3 embeddings and Mistral-NeMo as the generator.
Experimental choice | What the authors used | What remains untested |
|---|---|---|
Retrieval representation | Chunk embeddings for Naive RAG; synthetic-answer embeddings for HyDE; generated-question embeddings for HyPE | Public search indexes and crawler behavior |
Retrieval depth | 1, 3, 5, and 10 chunks | Longer result sets and production latency |
Models | One embedding model and one generator | Cross-model stability |
Evaluation | RAGChecker retrieval and generation metrics | Independent human review and buyer outcomes |
How did question-to-question matching work?
HyPE generated several questions that each chunk might answer during indexing, embedded those questions, and linked every question vector back to the original chunk.
Source: Vake, Vičič, and Tošić.
At query time, the system embedded the live question and searched for nearby question vectors. It then returned the associated source chunks to the generator. The paper calls this prompt-to-prompt matching. Naive RAG used prompt-to-document matching, while HyDE generated a hypothetical answer at inference time and used document-to-document matching.
The indexing choice matters because true evidence often sounds nothing like the question used to find it. A product page may state, "Exports retain field-level provenance." A buyer may ask, "Can I trace each generated claim back to its source?" The concepts overlap, but their sentence forms and vocabulary differ.
HyPE paid the generation cost once per chunk during indexing. The paper did not report total index-build tokens, dollars, or elapsed time. It also treated every generated question equally. There was no question-quality score, rejection rule, or human approval step.
How much did question matching improve retrieval?
Across the six datasets, HyPE raised average claim recall by 17.9 percentage points over Naive RAG and raised average context precision by 21.2 points.
Source: Vake, Vičič, and Tošić.
Figure 6 holds retrieval depth at five chunks and compares cosine with Euclidean distance. Under cosine distance, HyPE reached 71.5% claim recall and 63.5% context precision. The corresponding Naive scores were 53.6% and 42.3%. Changing the distance metric barely moved any of those four values.
The strongest dataset-level gains appeared where baseline matching was weak. On Single-Topic RAG at k=1, precision rose from 28.7% to 68.8%. At k=10, recall rose from 36.8% to 81.4%.
Those are benchmark results, not forecasts for a marketing site. The equal-weight six-dataset average also hides the fact that dataset denominators ranged by more than a thousandfold.
Where did the retrieval method get worse?
The method got worse on one important retrieval condition and one generation condition, which is why the paper does not support a blanket question-matching rule.
On MS MARCO at k=1, HyPE precision was 68.7% versus 73.6% for Naive RAG. Recall was 50.2% versus 56.2%. The authors argue that MS MARCO's short, answer-focused passages already match search questions well, leaving less room for generated question vectors to help.
The larger warning appears in generator evaluation. HyPE improved average faithfulness from 52.2% to 69.3% and reduced the hallucination metric from 26.0% to 19.9%. Yet its relevant-context noise sensitivity worsened from 13.8% to 21.0%. The authors suggest repeated representations of useful chunks may also repeat corrupted evidence.
I checked the paired significance table rather than accepting the paper's broad language. The analysis used only six paired dataset observations. Several Holm-adjusted p-values were 0.063, 0.086, or 0.094, so the effect sizes and full dataset results matter more than a loose claim of universal significance.
Does this prove FAQ pages improve AI visibility?
No. The study did not publish FAQ pages, modify websites, invite crawlers, observe citations, or measure visibility in a public answer engine.
HyPE changes a private vector index. Its generated questions are machine representations linked to source chunks. A visible FAQ page changes the public document itself. Those are different interventions, controlled by different systems.
A team can still use buyer questions as a planning device. The safe inference is that question and answer phrasing can create a retrieval gap inside one tested architecture. The unsupported leap is that adding Who is this for? above a paragraph will raise ChatGPT citations.
This distinction also protects the reader from fake precision. Better retrieval is not the same event as a mention, citation, recommendation, shortlist, or sale. The paper measured none of those outcomes.
How should teams map buyer questions to evidence?
Teams should use buyer questions to inspect whether each important answer has a current claim, a primary source, and a public asset, then test the actual answer rather than counting question marks on the page.
Start with a small question map:
Buyer question | Required evidence | Useful public asset | Failure to diagnose |
|---|---|---|---|
Can this product work with our stack? | Current integration scope and limitations | Integration documentation | Missing product fact, stale page, or legitimate no |
How does it compare with an alternative? | Shared criteria, tradeoffs, and proof | Honest comparison page | Weak comparison language or unsupported claim |
Is the result safe to publish? | Source trail, approval state, and known limits | Methodology or governance page | Missing proof or approval boundary |
Who should not buy it? | Fit conditions and exclusions | Buyer-fit page or FAQ | Positioning ambiguity or legitimate bad fit |
Generated questions can help discover phrasing variants, but they should not invent the answer. Keep the source of truth attached. Reject questions that assume nonexistent features, false comparisons, or outcomes the evidence cannot support.
Then run the same buyer question across the relevant answer surfaces and preserve the raw response, sources, date, and model. If the evidence is absent, determine whether the gap is retrieval, proof, product clarity, reputation, freshness, or fit before commissioning a page.
How can Trovance connect buyer questions to missing evidence?
Trovance can connect buyer questions to missing evidence by preserving how AI systems explain, cite, compare, and recommend a company, then tracing each weak or missing answer back to the claim and proof that should exist.
A visibility score alone cannot show whether the model missed a clear fact, the company never published it, a third-party source contradicted it, or the buyer question exposed a real product limitation. Trovance diagnoses that evidence gap before recommending production.
A team would use Trovance here to keep the buyer question, observed answer, cited sources, diagnosis, and resulting asset in one trail. When content is justified, the platform helps produce and publish the proof-backed page, comparison, FAQ, or article that should exist. It does not promise that an external model will retrieve or cite it. Use Trovance to connect a buyer question to the evidence decision.
FAQs
What is question-to-question retrieval?
Question-to-question retrieval represents source chunks with likely questions they can answer, then matches a live question against those question-like representations. In the HyPE study, each generated question vector pointed back to the original chunk. The generator received the source text, not the generated question, when composing its response.
What are Hypothetical Prompt Embeddings?
Hypothetical Prompt Embeddings, or HyPE, are generated question embeddings created for each source chunk during indexing. The tested system stored several vectors per chunk and searched those vectors when a user asked a question. This moved synthetic generation out of the live query path, but added an upfront indexing cost.
Did HyPE beat naive RAG on every dataset?
No. HyPE produced stronger aggregate recall and precision across six datasets, but it trailed Naive RAG on MS MARCO at retrieval depth one. Its relevant-context noise sensitivity was also worse. The result depends on the dataset, retrieval depth, metric, embedding model, generator, and quality of generated questions.
Do question headings improve ChatGPT citations?
This study cannot answer that. It changed representations inside a private RAG index and did not alter public web pages or observe ChatGPT citations. Question headings may help human scanning or clarify a page, but a citation claim requires a direct public-search experiment with repeated runs, sources, dates, and controls.
How many datasets did the researchers test?
The researchers tested six datasets, ranging from 80 question-answer pairs in Single-Topic RAG to 82,326 in MS MARCO. Their aggregate table weighted the six datasets equally rather than pooling every question. That choice makes the mean a cross-dataset comparison, not a universal estimate across all sampled questions.
What was the main tradeoff in the results?
HyPE improved average claim recall, context precision, faithfulness, and the paper's hallucination metric. It also became more sensitive to noise inside relevant context. The authors suggest repeated representations may reinforce useful evidence and repeat corruption. Any operational use therefore needs source validation, question review, deduplication, and regression tests.
How should a content team use likely buyer questions?
Use likely buyer questions to map each answer to a current claim, primary source, and suitable public asset. Treat generated variants as discovery aids, not facts. After publication, test the real question across relevant systems and preserve raw answers and sources. Diagnose retrieval, proof, clarity, reputation, freshness, and fit separately.
Related Resources
Connect buyer questions to proof-backed production with Trovance
