Adding more companies, products, people, and category terms to a page does not automatically make the page easier to retrieve for the right question. An arXiv preprint, first posted in April 2026 and revised in July, tested 443 entity-oriented retrieval configurations on TREC Robust04. Once entity selection was kept independent of the relevance judgments used for evaluation, the best gain over BM25 was 0.051 MAP points.
The paper did not test ChatGPT, brand pages, AI citations, or buyer behavior. It tested document reranking in a historic newswire collection. Still, it makes one distinction marketers badly need: an entity can be conceptually related to a question without helping a retrieval system distinguish a relevant document from an irrelevant one.
That is the gap between mentioning the category and proving why your company belongs in a specific answer.
What did the entity-retrieval study actually test?
The study tested whether entity signals improve document reranking when the system must identify useful entities without borrowing the relevance judgments used to score the result.
Shubham Chatterjee's 34-page preprint uses TREC Robust04, a newswire retrieval collection. It evaluates six neural entity-aware architectures, 437 unsupervised document-ranking configurations, and 193 entity-selection strategies. BM25+RM3 is the lexical baseline.
The entity-ranker experiments use five-fold cross-validation over title queries. Document ranking is scored with MAP, nDCG@20, and P@20. Entity coverage is measured at cutoffs of 10, 20, and 50 inside each query's BM25 top-1000 candidate pool.
I downloaded the full paper and checked the ranking tables instead of treating the abstract as the method. The setup matters because the paper compares two different questions.
Under derived supervision, the entity run is built from document relevance judgments. That asks whether a model can use an entity signal once the benchmark has already aligned it with relevance. Under clean supervision, entity selection cannot use those judgments. That asks whether the whole pipeline can recover the signal under conditions closer to deployment.
Those are both legitimate experiments. They are not interchangeable.
Why did the strongest entity result collapse under clean supervision?
The strongest result collapsed because most of its apparent advantage came from the entity labels supplied by the benchmark, not from a change in the document candidate pool.
With derived entity supervision, QDER reached 0.608 MAP. Under clean supervision, the same architecture reached 0.294, close to the BM25 baseline of 0.292. DREQ moved from 0.697 to 0.293 MAP across the same supervision boundary.
The paper then separates the entity side from the document side in a four-condition DREQ test. Changing the document-side filter moved MAP by at most 0.003 points. Changing the entity run moved it by roughly 0.405. Recall@1000 stayed fixed at 0.7772 in all four conditions.
The broad sweep lands in the same place. None of 437 unsupervised entity-oriented configurations beat BM25 under clean supervision. Their mean MAP was 0.231, their median was 0.241, and their maximum was 0.289 against BM25's 0.292.
This is not proof that entity-aware retrieval is broken. The best clean configuration reached 0.343 MAP, matching the official Robust04 best system cited by the paper and beating many neural rerankers in its comparison. The paper's complaint is about evaluation: a capacity result was too easy to read as an end-to-end result.
What is the difference between a related entity and a useful signal?
A related entity fits the topic; a useful retrieval signal helps distinguish relevant documents from non-relevant ones in the collection and linking environment being tested.
The paper calls the first property Conceptual Entity Relevance. An entity such as SOC 2 can be conceptually relevant to a query about enterprise software security. That says the concept belongs in the discussion. It does not say that mentioning SOC 2 separates one qualified vendor from every other vendor that uses the same term.
The second property is Observable Entity Relevance. It asks whether the linked entity appears more often in relevant candidate documents than in non-relevant ones for that query. This is conditional on the corpus, query, candidate pool, and entity linker. Change the environment and the observation may change.
That distinction exposes the coverage problem. A binary-derived entity run achieved a discrimination ratio of 154.9 at a cutoff of 20, yet it reached only 19.7% of relevant candidate documents. The LLM conceptual-relevance run reached 97.1%, but its discrimination ratio was 1.09, meaning relevant and non-relevant coverage were nearly alike.
Source: Entities as Retrieval Signals.
The best leakage-free compromise in this analysis was an LGBM selector with 94.9% relevant coverage and a 1.20 discrimination ratio. Even that point remained outside the paper's desired region of broad relevant coverage and strong discrimination.
For content, the safe inference is modest. Category vocabulary can establish topical fit. It cannot substitute for facts that separate one company, product, use case, or constraint from another.
Why does adding more entities fail as a content rule?
Adding more entities fails as a general content rule because wider coverage can pull relevant and irrelevant material into the same signal set.
Across all 193 entity-selection configurations, relevant-document coverage and non-relevant-document coverage had a Pearson correlation of 0.954 with p < 0.001. In this benchmark, methods that reached more relevant documents tended to reach more non-relevant documents too.
Source: Entities as Retrieval Signals.
The paper also tests a familiar instinct: use inverse document frequency to favor rarer entities. That did not clean up the LLM-labeled run. Its bait rate rose from 29.2% to 31.2%, its top-1 bait rate rose from 9.6% to 18.5%, and relevant coverage fell from 97.1% to 91.2%.
Rarity was not the same as usefulness here. The binary label rule discarded a common partition containing 877,047 entity-query pairs. Under the paper's diagnostic labels, 83.6% of those discarded pairs were discriminative, while 16.3% were bait.
I checked Figures 1 and 2 against the surrounding text before cropping them. Figure 1's left panel uses a log-scale discrimination axis, and Figure 2 reports correlation inside one benchmark. Removing either fact would make the pictures sound more universal than they are.
The paper gives us a refusal, not a recipe: do not count entities and call the page retrieval-ready.
What should a marketing team audit instead of entity count?
A marketing team should audit whether each important claim carries evidence that helps a buyer distinguish the company for a defined job, constraint, or comparison.
Start with the buyer question, not a list of category nouns. If the question is "Which data platform fits a healthcare team that needs regional hosting and audit-ready access controls?", a page needs more than the names of HIPAA, SOC 2, AWS, and Azure. It needs current, inspectable statements about hosting regions, control scope, exclusions, and where the proof lives.
Use a simple evidence table:
Audit field | Weak page signal | Stronger evidence to inspect |
|---|---|---|
Buyer job | Category name appears repeatedly | Named workflow, user, and decision context |
Constraint | Compliance term is mentioned | Scope, status, date, exception, and source artifact |
Comparison | Competitor names appear | Explicit fit boundary and criterion-by-criterion facts |
Proof | Adjective or badge | Report, documentation, customer artifact, or measured result with method |
Freshness | Undated claim | Owner, effective date, and update path |
This is not a claim that the stronger page will win retrieval. The paper never tested brand content or answer engines. It is a better evidence standard because each statement can be checked by a person, extracted by a system, and corrected when it changes.
Then observe the answer. Which facts did the AI system state, and which source did it cite? Did it compare the right alternatives? Did it recommend the company for the right reason, or merely repeat category language?
A missing mention may point to weak evidence, inaccessible documentation, stale facts, poor positioning, a product gap, or legitimate bad fit. Entity count cannot tell those apart.
How can Trovance turn a vague entity gap into useful evidence?
Trovance helps a team move from "we are missing from this answer" to a source-backed decision about what, if anything, should be published.
The team can use Trovance to observe how AI systems explain, cite, compare, and recommend the company for a defined buyer question. That answer trail matters here because a concept may be present while the distinguishing evidence is absent. A score or entity count cannot show whether the system found the right proof, repeated a generic category description, or excluded the company for a valid reason.
Trovance then helps diagnose the evidence gap behind the result. The issue may be a missing comparison, an unsupported security claim, stale documentation, unclear fit language, inaccessible proof, or no content problem at all. When a public asset is justified, Trovance helps the team produce and publish the proof-backed page that should exist, with claims, sources, and human review attached.
That is why a team would use Trovance here: to connect an observed answer to the specific evidence decision, rather than turning a retrieval paper into an entity-stuffing checklist. Use Trovance to trace the answer back to the evidence your team should produce.
