AI Assistants Repeat Zombie Stats, and Unanimity Doesn't Help
In a one-day API sample, 40% of 1,285 AI-cited statistics reached a matching study and 40% could not be checked; agreement across assistants did not improve sourcing.
Trakkr Research asked five AI assistants 200 number-seeking questions and traced 1,285 cited statistics up to five pages deep. It found 40% reached a matching study, 8% reached a study but the number or context had changed, 40% stopped at an uncheckable source, and 11% led nowhere.
ChatGPT sourced the most (57%) and quoted the fewest statistics per answer (1.3); the other four ranged from 38% to 46% and were too close to rank. The cited page contained the figure 82% of the time, but only 21% of first citations were original research; more than half were company blogs.
API answers with web search were collected on one day, and an AI judge was corrected with 150 hand checks.
Key takeaways
- On 28 of 71 questions where all five assistants gave a percentage, four or five different figures appeared; the typical gap between highest and lowest was 16 points, and 31 questions spanned 20 points or more.
- Agreement did not signal reliability: of 10 questions where all five gave the same figure, five were traced and two were sourced; of 29 statistics quoted by at least three assistants, 11 were sourced.
- Four statistics quoted by all five assistants had no study found behind them, including 44% of salespeople giving up after one follow-up and 90% of social media users following a brand.
- When trails reached a study, 17% had changed; the digits moved in about one quarter of changes, while context such as what was measured, the year, source or who was asked usually shifted.
- First-page type predicted success: research-firm and government pages reached matching studies 61–62% of the time, company blogs 37%, and statistics roundups 35%; even research firms failed 38% of the time, often behind forms.
What people think
Analysis details
8 comments analyzed. Analyzed .
The sample selects active discussion branches and includes available parent comments. It may omit parts of the full thread. Selected themes do not measure community opinion or how common a view is.
Perplexity's citation promise and blog spam
Perplexity's design requiring each sentence to trace to a source is framed as a trust advantage, but a user's experience over a year ago, not re-evaluated since, found it consistently citing AI blog spam as a primary source, leading to a cancelled subscription. A system prompt to ignore unreliable sources and require multiple sources reportedly helps, though consequential claims still warrant reading the sources. The open question is how the system enforces source quality.
Provenance layer remains an unbuilt ideal
The desired use of AI is to automate the drudge work of tracing statements to references, documentation and verification, but current products prioritize faster answers that avoid due diligence. A proposed layer over an LLM would track accessible documents, extracted facts, queries and artifacts, with receipts for how conclusions were reached. Legaltech may already require such provenance, and the open question is whether a general implementation exists.
Critical reading is the transferable skill
Teaching students to be critical of LLM output is a valuable skill, not a reason to exclude AI from schools. The same skepticism applies to human-written material, so the argument reframes prompting as an exercise in evaluating claims rather than merely talking to a machine.
Citogenesis turns repetition into fact
A 70% PHP claim from an AI-written php2shell announcement was repeated by major tech outlets, making the original source impossible to trace. If consensus is treated as evidence, models will absorb the claim; offhand mentions are not sources, and a source should take epistemic responsibility for a claim.