TL;DR
🧪 The audit tested 4 models across 3 catalogs, using zero-shot top-10 recommendations.
🎬 Out-of-catalog rates were only 0–0.2% on MovieLens, but reached 8.3% on Amazon Toys and 8.4% on Yelp.
📉 Models reported mean confidence of 67–86 while 92–100% of recommended items existed in the catalog.
🚧 Confidence filtering cut hallucination by at most 0.7 points at a coverage cost of 18 points in that cell.
🧾 Use a 5-part evidence check: existence, sources, fit, limitations, and repeated behavior.
Treat an AI product recommendation as a lead to investigate, even when the model sounds certain. First confirm that the item exists, then check the sources, buyer fit, trade-offs, and repeated behavior. The confidence number cannot do those jobs for you.
A new CIKM 2026 audit makes the problem unusually concrete. Across 4 language models and 3 public catalogs, the models' stated confidence did a poor job of separating real catalog items from invented ones. Filtering by confidence reduced hallucinations by no more than 0.7 percentage points, while removing far more recommendations from view.
For marketers, this is less a model-ranking story than a measurement warning. A confident answer is still an answer. It is not product evidence, buyer research, or revenue attribution.
What did the recommendation audit actually test?
The audit tested whether four models could recommend real items from three fixed catalogs, and whether each model's stated confidence tracked catalog membership.
Srijith Ravikumar evaluated Mistral Large, Llama-3.3-70B, GPT-OSS-120B, and Claude Sonnet 4.6. Each model received a user's 30 most recent interactions, then produced a top-10 list at temperature zero. The catalogs contained 62,423 MovieLens titles, 890,874 Amazon Toys products, and 150,346 Yelp businesses.
The audit used 300 users per catalog, plus 100 separate users for calibration. It checked exact title matches first, then allowed a normalized fuzzy-match threshold of 90 to absorb formatting differences. Anything failing both checks counted as out of domain.
We read the full seven-page manuscript and its prompt appendix before using the headline numbers.
That appendix matters: the confidence prompt asked how sure the model was that each item was a good recommendation. The audit then tested that score against whether the item existed in the catalog. The paper treats the mismatch as part of the finding, not a minor footnote.
Why can a confident recommendation still fail?
A confident recommendation can fail because confidence, catalog existence, factual support, and buyer fit are different tests.
The audit found a sharp catalog effect. MovieLens produced an out-of-domain rate between 0% and 0.2%. Amazon Toys ranged from 4.5% to 8.3%, while Yelp ranged from 2.2% to 8.4%. Familiar movie titles were easier for these models to reproduce than long-tail toys or regional businesses.
Source: Srijith Ravikumar, CIKM 2026 author manuscript.
The confidence curves tell a second story. Across all 12 model-catalog cells, models were under-confident about catalog membership. They gave mean scores of 67–86 while their actual catalog-hit rates were 92–100%. On MovieLens, one cell had zero out-of-domain items but an expected calibration error of 0.223.
That does not mean a low score is secretly reassuring. It means the score answered the wrong operational question. A model can hedge about whether you will like a real product or sound assured about an invented one. Neither number verifies the product record.
Can confidence filtering make AI recommendations safe?
Confidence filtering did not make these zero-shot recommendations safe because it removed correct items almost as readily as hallucinations.
The paper swept 4 filtering levels, from alpha .05 to .20, across Amazon Toys and Yelp. The largest reduction was on Yelp with GPT-OSS: 8.4% to 7.7%. That 0.7-point improvement cost 18 points of coverage. The median reduction across 32 tested cells was below 0.1 points.
Source: Srijith Ravikumar, CIKM 2026 author manuscript.
One result moved backward. Claude's Yelp hallucination rate rose from 4.1% to 4.3% at alpha .10 because the threshold removed correct items faster than false ones.
We checked Table 3 against the abstract because "up to 0.7 points" sounds useful until the denominator appears. Here, confidence-based abstention kept fewer recommendations without creating a clean truth filter. A grounded recommender, another prompt, or a different confidence signal may behave differently. This study did not test every production system.
How should marketers audit an AI product recommendation?
Marketers should use a staged evidence check that separates what can be observed from what still needs human judgment or business data.
Check | Question | Evidence required | Boundary |
|---|---|---|---|
Existence | Is the named product, company, plan, or feature real and current? | Official catalog, product page, documentation, or registry | A real item can still be a poor fit |
Claim support | Do the answer's factual claims have reachable sources? | First-party documentation plus credible independent evidence where needed | A citation does not prove endorsement |
Buyer fit | Do the criteria match the buyer's job, constraints, and stage? | Stated use case, budget, requirements, and alternatives | Human judgment remains accountable |
Trade-offs | Does the answer include limits, exclusions, and credible substitutes? | Current comparison evidence and product boundaries | Missing caveats can distort a shortlist |
Stability | Does the result survive a defined repeat protocol? | Dated prompts, model/version context, repeated runs, and raw answers | Repetition shows variance, not causality |
Business effect | Did the recommendation influence qualified demand or revenue? | Analytics, cohort, attribution method, and observation window | Public view counts and model mentions do not prove acquisition |
This order matters. If the product does not exist, stop. If it exists but the claim has no source, flag the claim. If the evidence is sound but the fit is wrong, accept the exclusion instead of turning every disappointing answer into a content assignment.
The same discipline applies when the answer favors your company. A flattering recommendation can still use stale pricing, omit a limitation, or cite somebody else's summary. Accuracy is useful even when it costs you the top spot.
What does this study leave unresolved?
The study leaves buyer preference, grounded production systems, live catalog drift, and business outcomes unresolved.
These were zero-shot models used directly as recommenders. The paper did not test systems that retrieve a current catalog before generating, constrain output to valid item IDs, or validate availability after generation. It also did not measure whether a buyer clicked, converted, retained, or regretted the choice.
Catalog membership is a narrow oracle. It can tell you whether Product X appears in the dataset. It cannot tell you whether Product X fits a two-person marketing team, complies with a buyer's security rules, ships the claimed feature, or beats the available alternatives.
Prompt wording also moved confidence differently by model. On one Yelp ablation, a catalog-anchored prompt cut Llama's expected calibration error from 0.224 to 0.141. The same change pushed Claude's error to 0.434 while accuracy stayed at 96%. A verbal confidence score is partly a property of how the question was asked.
How can Trovance audit a recommendation before it becomes a content action?
Trovance can preserve the recommendation and its sources, diagnose the missing evidence, and help the team produce the proof-backed asset that should exist.
The operational gap here is not a lack of confidence scores. It is the missing trail between a buyer question, the model's explanation, visible citations, the company's current proof, and the action a marketer takes next. Observation alone cannot tell you whether the issue is an invented item, stale fact, weak comparison, missing proof, legitimate bad fit, or a model-specific variance pattern.
A team would use Trovance to observe how AI systems explain, cite, compare, and recommend the company, then check the evidence behind that answer. If a real gap exists, Trovance helps turn it into an approved comparison, product clarification, FAQ, proof page, or source-backed article. It does not guarantee a citation, recommendation, ranking, or sale.
Scan my AI visibility to inspect the answer and diagnose what evidence, if any, should change.
FAQs
Can an AI confidence score prove a product recommendation is accurate?
No. A confidence score is produced by a particular model, prompt, and scoring method. It may describe recommendation quality, certainty about facts, or something less clear. Verify the named item, factual claims, sources, fit criteria, and limitations separately before treating the recommendation as accurate or useful.
Did the study test ChatGPT shopping recommendations?
No. It tested four named instruction-tuned models as zero-shot recommenders against MovieLens, Amazon Toys, and Yelp datasets. It did not audit the normal ChatGPT shopping interface, its retrieval stack, current inventory feeds, sponsored results, or user behavior. The reported rates should stay attached to the study's exact setup.
Why were hallucination rates higher for some catalogs?
The paper found near-zero out-of-domain recommendations for canonical MovieLens titles and higher rates for long-tail toys and regional businesses. The author proposes differences in pretraining visibility as an explanation, but the experiment does not isolate that cause. Catalog size, naming, regional concentration, and model training may all matter.
Does low model confidence mean a recommendation is probably false?
Not in this audit. The models were systematically under-confident about catalog membership, often giving middling scores to items that existed. A low score may reflect hesitation about buyer taste rather than item existence. Use a current catalog or official product record to verify existence instead of interpreting the model's mood.
Can repeated prompts prove an AI recommendation is stable?
Repeated prompts can show whether an answer varies under a defined protocol. They cannot prove that the result represents every buyer, model version, location, account state, or future run. Record the prompt, date, model context, sources, and denominator, then treat stability as one measurement dimension rather than proof of correctness.
Should a missing recommendation always trigger new content?
No. The gap may come from weak fit, stale product facts, inaccessible evidence, limited third-party support, an unfair comparison, or ordinary model variation. Check the raw answer and current proof first. Publish only when missing or unclear public evidence is the diagnosed constraint and the team can support the new claim.
Can AI recommendation visibility be tied to revenue?
Only with business analytics beyond the model output. A defensible claim needs a defined cohort, attribution method, observation window, baseline, and downstream events such as qualified visits, sign-ups, pipeline, retention, or sales. A mention, citation, recommendation, public view count, or confidence score does not establish acquisition or revenue by itself.
Audit how AI systems explain and recommend your company with Trovance
