A citation can point to a real page and still fail the sentence beside it. Before publishing, check four separate things: the URL works, the nearby claim is directly supported, every source adds necessary evidence, and the conclusion follows from the evidence as a whole. When one check fails, narrow the claim, replace the source, rebuild the argument, or hold the draft.
A new fact-checking shared task shows why those checks cannot collapse into one score. One system improved citation precision from 0.223 to 0.337 and recall from 0.240 to 0.339, yet its reference-based entailment score fell from the baseline's 0.298 to 0.245.
I use "citation audit" here to mean those separate editorial checks, not a promise that one automated score can certify a draft as true.
The paper tested 1,158 claims, with an average of 7.8 premise articles per claim. It did not test brand blogs, open-web research, or live marketing teams. Treat it as a production design lesson, not a promised accuracy rate.
What does a citation audit need to verify?
A citation audit needs to verify the address, the nearby claim, the source's contribution, and the article's argument as separate things.
The first check is mechanical: does the URL exist, and was it among the sources approved for the draft? The SourceMinds auditor removes any URL outside the supplied premise set. That catches invented or out-of-scope links, but it cannot tell you whether an approved source supports the prose.
The second check is sentence-level support. For every factual sentence, find the exact passage that supports it. If the evidence is narrower than the sentence, narrow the sentence. If no passage supports it, remove the claim or retrieve better evidence.
The third check is necessity. SourceMinds defines citation precision strictly: every cited URL must be needed for the sentence. A pile of links is not stronger evidence when three links merely repeat the one source that matters.
The fourth check is article-level alignment. Do the selected facts justify the conclusion, or did the draft assemble true fragments into an unsupported story? That is where the system lost ground even as its citation metrics improved.
I recommend treating any failed check as a decision point: narrow the sentence, replace the evidence, rebuild the argument, or hold the draft.
What did the 1,158-claim test actually measure?
The test measured generated fact-checking articles built from a supplied claim, verdict, metadata, and bounded evidence set.
The official CLEF 2026 Task 3 dataset is derived from WatClaimCheck. Its 1,158 claims came with 7.8 premise articles on average, plus reference articles averaging 2,105 words. Those reference articles were used in evaluation.
Evidence at readable size. Swipe or scroll horizontally to read all columns; keyboard users can focus the image and use arrow keys.
Source: Hasan et al., SourceMinds at CheckThat! 2026.
Licensed CC BY 4.0. Cropped from Table 1 for legibility; open the full-resolution crop.
The submitted pipeline split premise documents into overlapping two-sentence chunks. Dense retrieval shortlisted up to 60 candidate chunks, a cross-encoder reranked them, and source-balanced selection kept 10 chunks with no more than 2 per URL.
A planner converted that evidence into structured JSON. The writer was prompted to produce a 250–450-word article, followed by conditional self-critique and a rule-based citation audit using natural language inference.
This review checked the paper's prompts, thresholds, test-set description, and official result table. The system had the answer label before writing and worked inside a supplied evidence set. A marketer searching the public web starts with a harder problem: the best source may be missing, stale, contradictory, or inaccessible.
Why can citation scores rise while support gets worse?
Citation scores can rise while support gets worse because sentence-level links and article-level reasoning measure different failures.
SourceMinds beat the shared-task baseline on citation precision, citation recall, evidence coverage, and the four-metric mean. Precision rose to 0.337 from 0.223. Recall rose to 0.339 from 0.240, coverage to 0.394 from 0.329, and the mean to 0.329 from 0.272.
Evidence at readable size. Swipe or scroll horizontally to read all columns; keyboard users can focus the image and use arrow keys.
Source: Hasan et al., SourceMinds at CheckThat! 2026.
Licensed CC BY 4.0. Cropped from Table 2 for legibility; open the full-resolution crop.
Entailment moved the other way. SourceMinds scored 0.245, below the baseline's 0.298. The metric compared the generated article with a ground-truth reference in both directions, so it rewarded reconstruction of the reference's evidence-to-verdict reasoning.
The paper's own interpretation is blunt: local grounding did not imply global alignment. The draft could attach reasonable sources sentence by sentence and still fail to reproduce the argument connecting evidence to the verdict.
The full result table matters because the mean score makes an easier headline and a less useful editorial diagnosis. Averaging four dimensions hides the exact failure an editor needs to catch before publication.
How should a team audit an AI-assisted draft?
A team should audit an AI-assisted draft by moving from claims to passages before reviewing prose polish.
Start with an atomic claim list. Split compound sentences when one citation supports only half the statement. For each claim, record the exact supporting passage, canonical source URL, source date, scope, and any condition the sentence must preserve.
Then run four gates:
| Gate | Question | Fail-closed action |
|---|---|---|
| URL validity | Is the source real, accessible, and approved? | Replace or remove the citation |
| Claim support | Does a passage directly support this exact sentence? | Narrow, rewrite, or remove the claim |
| Citation necessity | Does each source add distinct evidence? | Remove decorative or redundant links |
| Argument alignment | Does the conclusion follow from the cited claims? | Replan the article before editing prose |
Automated checks can help with the first three. SourceMinds matched each uncited sentence against its top 3 evidence chunks and attached a citation only above an entailment threshold of 0.5. For pruning, it used the same 0.5 threshold and a marginal-score rule of 0.05.
Those numbers are system settings, not universal standards. One NLI model made the call, and the authors flag long passages, indirect support, and partially relevant context as weaknesses. A human still needs to read the passage and decide whether the public wording is fair.
When should the draft fail closed?
The draft should fail closed whenever a consequential claim lacks direct, inspectable support.
Hold the asset if the original source cannot be opened, the cited passage does not contain the claimed fact, a denominator or time period is missing, or the conclusion depends on evidence the draft never presents. Do the same when sources conflict and the article quietly chooses one without explaining why.
Self-critique is not enough by itself. In SourceMinds, the same Qwen2.5-32B-Instruct model handled planning, writing, and critique. The authors note that the critic may inherit the writer's reasoning patterns instead of catching them.
The system only triggered critique when a draft had fewer than 3 citations, fewer than 2 distinct URLs, or a substantial uncited factual sentence. Those are sensible warning signs. They are not proof that a draft passing the heuristic is ready.
A failed audit should produce a specific next state: retrieve a better source, reduce the claim, rebuild the argument, or kill the article. "Add a citation" is often the wrong fix.
How can Trovance turn a citation failure into better evidence?
Trovance can connect a failed citation audit to the evidence gap that caused it, instead of treating the problem as a missing hyperlink.
A team can use Trovance to observe how AI systems explain, cite, compare, and recommend the company for a defined buyer question. That answer trail shows which claims and sources are shaping the market explanation. The next step is diagnosis: is the draft missing first-party proof, relying on a stale page, stretching a source beyond its scope, or trying to publish around a product or positioning gap?
A score cannot resolve that question. Neither can a citation checker that only confirms the URL exists. Trovance routes the next action that the evidence supports: proof, owned content, technical repair, distribution, monitoring, or no action. Human review keeps responsibility for the claim, source choice, caveat, and final publication.
That is why a team would use Trovance here: the citation failure becomes a governed evidence decision with a source trail and approval boundary, not another round of plausible copy. Explore Text Trovance if you want to bring the weak citation and source trail into that decision.
