8 min
Zach Chmael
In This Article
Mentioning the right companies, products, and concepts can improve topical coverage without making a page more useful for retrieval.
Updated
TL;DR
🧪 The preprint evaluated 443 retrieval configurations on the TREC Robust04 newswire collection.
📉 Across 437 unsupervised configurations, none beat the BM25 baseline MAP of 0.292 under clean supervision.
🎯 A highly discriminative entity run reached only 19.7% of relevant candidate documents at a cutoff of 20 entities.
🌊 A broad LLM-labeled run reached 97.1% of relevant documents but covered non-relevant documents at a similar rate.
🔗 Across 193 entity-selection strategies, relevant and non-relevant coverage correlated at r = 0.954.
A Named Entity Is Not A Retrieval Signal
Adding more companies, products, people, and category terms to a page does not automatically make the page easier to retrieve for the right question. A July 2026 arXiv preprint tested 443 entity-oriented retrieval configurations on TREC Robust04. Once entity selection was kept independent of the relevance judgments used for evaluation, the best gain over BM25 was 0.051 MAP points.
The paper did not test ChatGPT, brand pages, AI citations, or buyer behavior. It tested document reranking in a historic newswire collection. Still, it makes one distinction marketers badly need: an entity can be conceptually related to a question without helping a retrieval system distinguish a relevant document from an irrelevant one.
That is the gap between mentioning the category and proving why your company belongs in a specific answer.
What did the entity-retrieval study actually test?
The study tested whether entity signals improve document reranking when the system must identify useful entities without borrowing the relevance judgments used to score the result.
Shubham Chatterjee's 34-page preprint uses TREC Robust04, a newswire retrieval collection. It evaluates six neural entity-aware architectures, 437 unsupervised document-ranking configurations, and 193 entity-selection strategies. BM25+RM3 is the lexical baseline.
The entity-ranker experiments use five-fold cross-validation over title queries. Document ranking is scored with MAP, nDCG@20, and P@20. Entity coverage is measured at cutoffs of 10, 20, and 50 inside each query's BM25 top-1000 candidate pool.
I downloaded the full paper and checked the ranking tables instead of treating the abstract as the method. The setup matters because the paper compares two different questions.
Under derived supervision, the entity run is built from document relevance judgments. That asks whether a model can use an entity signal once the benchmark has already aligned it with relevance. Under clean supervision, entity selection cannot use those judgments. That asks whether the whole pipeline can recover the signal under conditions closer to deployment.
Those are both legitimate experiments. They are not interchangeable.
Why did the strongest entity result collapse under clean supervision?
The strongest result collapsed because most of its apparent advantage came from the entity labels supplied by the benchmark, not from a change in the document candidate pool.
With derived entity supervision, QDER reached 0.608 MAP. Under clean supervision, the same architecture reached 0.294, close to the BM25 baseline of 0.292. DREQ moved from 0.697 to 0.293 MAP across the same supervision boundary.
The paper then separates the entity side from the document side in a four-condition DREQ test. Changing the document-side filter moved MAP by at most 0.003 points. Changing the entity run moved it by roughly 0.405. Recall@1000 stayed fixed at 0.7772 in all four conditions.
The broad sweep lands in the same place. None of 437 unsupervised entity-oriented configurations beat BM25 under clean supervision. Their mean MAP was 0.231, their median was 0.241, and their maximum was 0.289 against BM25's 0.292.
This is not proof that entity-aware retrieval is broken. The best clean configuration reached 0.343 MAP, matching the official Robust04 best system cited by the paper and beating many neural rerankers in its comparison. The paper's complaint is about evaluation: a capacity result was too easy to read as an end-to-end result.
What is the difference between a related entity and a useful signal?
A related entity fits the topic; a useful retrieval signal helps distinguish relevant documents from non-relevant ones in the collection and linking environment being tested.
The paper calls the first property Conceptual Entity Relevance. An entity such as SOC 2 can be conceptually relevant to a query about enterprise software security. That says the concept belongs in the discussion. It does not say that mentioning SOC 2 separates one qualified vendor from every other vendor that uses the same term.
The second property is Observable Entity Relevance. It asks whether the linked entity appears more often in relevant candidate documents than in non-relevant ones for that query. This is conditional on the corpus, query, candidate pool, and entity linker. Change the environment and the observation may change.
That distinction exposes the coverage problem. A binary-derived entity run achieved a discrimination ratio of 154.9 at a cutoff of 20, yet it reached only 19.7% of relevant candidate documents. The LLM conceptual-relevance run reached 97.1%, but its discrimination ratio was 1.09, meaning relevant and non-relevant coverage were nearly alike.
Source: Entities as Retrieval Signals.
The best leakage-free compromise in this analysis was an LGBM selector with 94.9% relevant coverage and a 1.20 discrimination ratio. Even that point remained outside the paper's desired region of broad relevant coverage and strong discrimination.
For content, the safe inference is modest. Category vocabulary can establish topical fit. It cannot substitute for facts that separate one company, product, use case, or constraint from another.
Why does adding more entities fail as a content rule?
Adding more entities fails as a general content rule because wider coverage can pull relevant and irrelevant material into the same signal set.
Across all 193 entity-selection configurations, relevant-document coverage and non-relevant-document coverage had a Pearson correlation of 0.954 with p < 0.001. In this benchmark, methods that reached more relevant documents tended to reach more non-relevant documents too.
Source: Entities as Retrieval Signals.
The paper also tests a familiar instinct: use inverse document frequency to favor rarer entities. That did not clean up the LLM-labeled run. Its bait rate rose from 29.2% to 31.2%, its top-1 bait rate rose from 9.6% to 18.5%, and relevant coverage fell from 97.1% to 91.2%.
Rarity was not the same as usefulness here. The binary label rule discarded a common partition containing 877,047 entity-query pairs. Under the paper's diagnostic labels, 83.6% of those discarded pairs were discriminative, while 16.3% were bait.
I checked Figures 1 and 2 against the surrounding text before cropping them. Figure 1's left panel uses a log-scale discrimination axis, and Figure 2 reports correlation inside one benchmark. Removing either fact would make the pictures sound more universal than they are.
The paper gives us a refusal, not a recipe: do not count entities and call the page retrieval-ready.
What should a marketing team audit instead of entity count?
A marketing team should audit whether each important claim carries evidence that helps a buyer distinguish the company for a defined job, constraint, or comparison.
Start with the buyer question, not a list of category nouns. If the question is "Which data platform fits a healthcare team that needs regional hosting and audit-ready access controls?", a page needs more than the names of HIPAA, SOC 2, AWS, and Azure. It needs current, inspectable statements about hosting regions, control scope, exclusions, and where the proof lives.
Use a simple evidence table:
Audit field | Weak page signal | Stronger evidence to inspect |
|---|---|---|
Buyer job | Category name appears repeatedly | Named workflow, user, and decision context |
Constraint | Compliance term is mentioned | Scope, status, date, exception, and source artifact |
Comparison | Competitor names appear | Explicit fit boundary and criterion-by-criterion facts |
Proof | Adjective or badge | Report, documentation, customer artifact, or measured result with method |
Freshness | Undated claim | Owner, effective date, and update path |
This is not a claim that the stronger page will win retrieval. The paper never tested brand content or answer engines. It is a better evidence standard because each statement can be checked by a person, extracted by a system, and corrected when it changes.
Then observe the answer. Which facts did the AI system state, and which source did it cite? Did it compare the right alternatives? Did it recommend the company for the right reason, or merely repeat category language?
A missing mention may point to weak evidence, inaccessible documentation, stale facts, poor positioning, a product gap, or legitimate bad fit. Entity count cannot tell those apart.
How can Trovance turn a vague entity gap into useful evidence?
Trovance helps a team move from "we are missing from this answer" to a source-backed decision about what, if anything, should be published.
The team can use Trovance to observe how AI systems explain, cite, compare, and recommend the company for a defined buyer question. That answer trail matters here because a concept may be present while the distinguishing evidence is absent. A score or entity count cannot show whether the system found the right proof, repeated a generic category description, or excluded the company for a valid reason.
Trovance then helps diagnose the evidence gap behind the result. The issue may be a missing comparison, an unsupported security claim, stale documentation, unclear fit language, inaccessible proof, or no content problem at all. When a public asset is justified, Trovance helps the team produce and publish the proof-backed page that should exist, with claims, sources, and human review attached.
That is why a team would use Trovance here: to connect an observed answer to the specific evidence decision, rather than turning a retrieval paper into an entity-stuffing checklist. Use Trovance to trace the answer back to the evidence your team should produce.
FAQs
Does this study prove named entities do not help AI search?
No. The study evaluates entity-oriented document reranking on TREC Robust04, not AI answer engines, brand websites, citations, or recommendations. It shows that entity coverage and discrimination diverged in that benchmark. The useful lesson is to avoid treating entity mentions as proof of retrieval value without observing the actual system and query.
What is Conceptual Entity Relevance?
Conceptual Entity Relevance means an entity is semantically related to the query. A compliance framework can be relevant to an enterprise-security question even when many documents mention it. The label captures topical fit, not whether the entity distinguishes relevant documents from non-relevant ones in a particular corpus and linking environment.
What is Observable Entity Relevance?
Observable Entity Relevance asks whether a linked entity appears more often in relevant candidate documents than in non-relevant candidates for a given query. It is tied to the collection, entity linker, candidate pool, and judgments. An entity can make conceptual sense while providing little observed discrimination in that specific setup.
Should marketers stop naming products, standards, and competitors?
No; specific names can clarify scope, make claims checkable, and help readers understand fit. The mistake is counting those names as an outcome. Each entity should support a real statement about a buyer job, constraint, comparison, proof artifact, or boundary. Mentioning a standard is weaker than documenting what the standard covers.
Is rare terminology a stronger retrieval signal?
Not automatically. In this study, adding IDF filtering to an LLM-labeled entity run increased the reported bait rate and reduced relevant-document coverage. That result belongs to one retrieval benchmark, so it is not a universal content rule. It does show why rarity and usefulness need separate measurement rather than assumption.
How should a team test whether evidence changes an AI answer?
Define the buyer question, preserve the model, date, prompt, and run protocol, then record facts, sources, comparisons, and recommendations across repeated runs. After publishing a justified evidence asset, rerun the same protocol. Report raw counts and changes without calling correlation causation or treating one answer as a stable market result.
What content should be produced after an entity gap appears?
Only the asset supported by the diagnosed gap. That may be a comparison, documentation update, proof page, FAQ, case study, or no new content. Check whether the issue is evidence, positioning, product truth, access, freshness, reputation, or bad fit before drafting. A missing entity alone is not a content brief.
Related Resources
See how Trovance connects observed answers to source-backed evidence decisions.
