Treat an AI product recommendation as a lead to investigate, even when the model sounds certain. First confirm that the item exists, then check the sources, buyer fit, trade-offs, and repeated behavior. The confidence number cannot do those jobs for you.
A new CIKM 2026 audit makes the problem unusually concrete. Across 4 language models and 3 public catalogs, the models' stated confidence did a poor job of separating real catalog items from invented ones. Filtering by confidence reduced hallucinations by no more than 1.65 percentage points, and the filtered lists were shorter but no cleaner.
For marketers, this is less a model-ranking story than a measurement warning. A confident answer is still an answer. It is not product evidence, buyer research, or revenue attribution.
What did the recommendation audit actually test?
The audit tested whether four models could recommend real items from three fixed catalogs, and whether each model's stated confidence tracked catalog membership.
Srijith Ravikumar evaluated Mistral Large, Llama-3.3-70B, GPT-OSS-120B, and Claude Sonnet 4.6. Each model received a user's 30 most recent interactions, then produced a top-10 list at temperature zero. The catalogs contained 62,423 MovieLens titles, 890,874 Amazon Toys products, and 150,346 Yelp businesses.
The audit used 300 users per catalog, plus 100 separate users for calibration. Deciding whether a title was in the catalog turned out to be the hard part: on the same outputs, the choice of string matcher moved the reported rate by an order of magnitude. The author validated the matcher against 201 human judgments and adopted an LLM judge that compares each title with eight candidate catalog entries. Anything it could not match counted as out of domain.
We read the full seven-page manuscript before using the headline numbers.
The prompt matters: the confidence question asked for a general rating of each recommendation, not for a probability that the item was real. The audit then tested that score against whether the item existed in the catalog. The paper treats the mismatch as part of the finding, not a minor footnote.
Why can a confident recommendation still fail?
A confident recommendation can fail because confidence, catalog existence, factual support, and buyer fit are different tests.
The audit found a sharp catalog effect. MovieLens produced an out-of-domain rate between 0.6% and 2.7%. Yelp ranged from 11.6% to 38.7%, while Amazon Toys ranged from 49.3% to 61.0%. Familiar movie titles were easier for these models to reproduce than long-tail toys or regional businesses.
The confidence scores tell a second story. Each model held a near-constant confidence level while catalog accuracy swung by about 60 points. Across the 12 model-catalog cells, mean scores ran 67–86 while catalog-hit rates ran from 39.0% to 99.4%. The catalog set the direction of the error: all four models were under-confident on MovieLens, where 97.3–99.4% of their recommendations were in the catalog, and all four were over-confident on Amazon Toys, where only 39.0–50.7% were. In all, 7 cells were under-confident and 5 over-confident.
That does not mean a low score is secretly reassuring. It means the score answered the wrong operational question. A model can hedge about whether you will like a real product or sound assured about an invented one. Neither number verifies the product record.
Can confidence filtering make AI recommendations safe?
Confidence filtering did not make these zero-shot recommendations safe because it removed correct items almost as readily as hallucinations.
The paper swept 4 filtering levels, from alpha .05 to .20. Across all 48 combinations of model, catalog, and level, the largest reduction was 1.65 points and the median was 0.02 points. On Amazon Toys, where roughly half the recommendations were not in the catalog, discarding the least-confident tenth removed well under a point.
Some results moved backward. Mistral's Amazon Toys hallucination rate rose from 49.3% to 50.0% at alpha .20 because the threshold removed correct items faster than false ones. In all, 18 of the 48 combinations left the filtered list more hallucinated than the unfiltered one.
We checked the paper's Table 4 against its abstract because "at most 1.65 pp" sounds useful until the denominator appears. Here, confidence-based abstention kept fewer recommendations without creating a clean truth filter. A grounded recommender, another prompt, or a different confidence signal may behave differently. This study did not test every production system.
How should marketers audit an AI product recommendation?
Marketers should use a staged evidence check that separates what can be observed from what still needs human judgment or business data.
Check | Question | Evidence required | Boundary |
|---|---|---|---|
Existence | Is the named product, company, plan, or feature real and current? | Official catalog, product page, documentation, or registry | A real item can still be a poor fit |
Claim support | Do the answer's factual claims have reachable sources? | First-party documentation plus credible independent evidence where needed | A citation does not prove endorsement |
Buyer fit | Do the criteria match the buyer's job, constraints, and stage? | Stated use case, budget, requirements, and alternatives | Human judgment remains accountable |
Trade-offs | Does the answer include limits, exclusions, and credible substitutes? | Current comparison evidence and product boundaries | Missing caveats can distort a shortlist |
Stability | Does the result survive a defined repeat protocol? | Dated prompts, model/version context, repeated runs, and raw answers | Repetition shows variance, not causality |
Business effect | Did the recommendation influence qualified demand or revenue? | Analytics, cohort, attribution method, and observation window | Public view counts and model mentions do not prove acquisition |
This order matters. If the product does not exist, stop. If it exists but the claim has no source, flag the claim. If the evidence is sound but the fit is wrong, accept the exclusion instead of turning every disappointing answer into a content assignment.
The same discipline applies when the answer favors your company. A flattering recommendation can still use stale pricing, omit a limitation, or cite somebody else's summary. Accuracy is useful even when it costs you the top spot.
What does this study leave unresolved?
The study leaves buyer preference, grounded production systems, live catalog drift, and business outcomes unresolved.
These were zero-shot models used directly as recommenders. The paper did not test systems that retrieve a current catalog before generating, constrain output to valid item IDs, or validate availability after generation. It also did not measure whether a buyer clicked, converted, retained, or regretted the choice.
Catalog membership is a narrow oracle. It can tell you whether Product X appears in the dataset. It cannot tell you whether Product X fits a two-person marketing team, complies with a buyer's security rules, ships the claimed feature, or beats the available alternatives.
Prompt wording also moved confidence differently by model. On one Yelp ablation, a catalog-anchored prompt raised Llama's mean confidence from 75.4 to 85.6 and cut its out-of-catalog rate from 12.9% to 1.5%, though the new prompt also changed the list itself. The same change lowered Claude's mean confidence from 67.2 to 53.0 while its catalog-hit rate rose to 91.1%, and pushed its calibration error to 0.380. A verbal confidence score is partly a property of how the question was asked.
How can Trovance audit a recommendation before it becomes a content action?
Trovance can preserve the recommendation and its sources, diagnose the missing evidence, and help the team produce the proof-backed asset that should exist.
The operational gap here is not a lack of confidence scores. It is the missing trail between a buyer question, the model's explanation, visible citations, the company's current proof, and the action a marketer takes next. Observation alone cannot tell you whether the issue is an invented item, stale fact, weak comparison, missing proof, legitimate bad fit, or a model-specific variance pattern.
A team would use Trovance to observe how AI systems explain, cite, compare, and recommend the company, then check the evidence behind that answer. If a real gap exists, Trovance helps turn it into an approved comparison, product clarification, FAQ, proof page, or source-backed article. It does not guarantee a citation, recommendation, ranking, or sale.
Start a free Trovance trial to inspect the answer and diagnose what evidence, if any, should change.
