AI Citations StudyKey Takeaways
We asked 13 AI models for a short paragraph with 3 APA citations on 30 topics, in two wordings, and checked every reference. Of the 1,136 citations from the standard request, only 45% were fully correct.
21% could not be found as cited, 13% were real papers with a wrong DOI or other detail, and 9% had a DOI that opens a different paper. 63% of the answers contained at least one bad citation.
The spread between models is huge: 81% fully correct for GPT-6 Luna against 11% for Gemini 2.5 Flash.
Telling a model to "only cite sources you are certain exist" barely moved the average (45% to 47%). Letting two models search the web helped GPT-5 Mini a lot and Gemini 2.5 Flash less. Always check the DOI yourself.
AI writing tools are happy to add citations when you ask. The question is whether the sources are real, and whether the link in the reference leads to the paper it names.
We tested 13 of the models available in Writingful, checked every citation against doi.org, Crossref and OpenAlex, and repeated the exercise inside the product so you can see what a user sees.
We make Writingful, so we have an interest in AI writing being useful. We also have an interest in you not publishing fake references, which is why we ran this.
What we tested
For each of 13 models we sent 30 requests in each of two wordings, one answer per request, through the providers' APIs with default settings and no web search. That is 780 answers, plus 60 more with web search switched on for two models (below).
The 30 topics cover health, education, technology, business and the environment, plus five deliberately obscure ones. Every request asked for the same thing.
The standard request (condition A):
Write a 120-word paragraph for a blog post about {topic}. Support it with 3 citations in APA 7 format. After the paragraph, list the 3 references in full under the heading "References", with a DOI or URL for each.
The careful wording (condition B) is the same text plus:
Only cite sources that you are certain exist. If you are not certain a source exists, say so instead of citing it.
Example topics: "how sleep affects student exam performance", "intermittent fasting and weight loss", "AI coding assistants and developer productivity", "email marketing return on investment", "coral reef bleaching", and the obscure ones: "radiation tolerance in tardigrades", "medieval Icelandic grain trade", "left-handedness and creativity", "standing desks for call-centre workers" and "freelance translators' use of machine translation tools".
Models tested (names as shown in Writingful): GPT-6 Luna, GPT-5.4 Mini, GPT-5 Mini, GPT-4.1 Mini, GPT-4.1 Nano, GPT-4o Mini, Gemini 3 Flash, Gemini 2.5 Flash, Gemini 2.5 Flash Lite, Grok 4.3, Claude Haiku 4.5, DeepSeek Flash and DeepSeek V4 Pro. All runs were on 14 October 2026.
How we checked each citation
Every reference was pulled apart into author, year, title and DOI or URL, then checked automatically:
- DOI: looked up on doi.org. A DOI that does not exist is flagged. A DOI that exists is compared with the claimed title, first author and year.
- Title search: if the DOI is missing or wrong, we searched Crossref and OpenAlex for the title, author and year to see whether the paper exists under another identifier.
- Links: every URL was requested to see whether it loads.
Each citation then got one of five results:
| Result | What it means |
|---|---|
| Fully correct | The paper exists with the title, first author and year as written, and the DOI or link given leads to it. |
| Real paper, but wrong DOI or details | The paper exists (or the DOI leads to the right authors and year), but the DOI or link is wrong or dead, or the title, year or journal is not what was written. |
| DOI opens a different paper | The DOI exists but belongs to an unrelated work, and the cited paper could not be found. |
| Not found as cited | No record matches the title, authors and year together. Some of these are real papers with a garbled title or journal. |
| Could not check | Books, reports and web pages with no matching record in Crossref or OpenAlex. We report them but do not count them as wrong. |
The results
Across the standard request, 45% of the 1,136 citations were fully correct. Leaving out the 13% we could not check, that is 52% of the citations we could check. 63% of the answers had at least one bad citation, so most paragraphs a user would paste into a post carried a reference that does not stand up.
GPT-6 Luna was best at 81%. Gemini 2.5 Flash was worst at 11%. Each model wrote about 90 citations across 30 topics.
View the numbers
| Item | Value (%) |
|---|---|
| GPT-6 Luna | 81 |
| DeepSeek Flash | 79 |
| DeepSeek V4 Pro | 76 |
| Gemini 3 Flash | 55 |
| Grok 4.3 | 54 |
| GPT-5.4 Mini | 53 |
| GPT-5 Mini | 46 |
| GPT-4.1 Mini | 42 |
| Claude Haiku 4.5 | 36 |
| GPT-4.1 Nano | 20 |
| GPT-4o Mini | 18 |
| Gemini 2.5 Flash Lite | 16 |
| Model | Fully correct | Real paper, but wrong DOI or details | DOI opens a different paper | Not found as cited | Could not check | Fully correct with careful wording |
|---|---|---|---|---|---|---|
| GPT-6 Luna | 81% | 9% | 2% | 5% | 3% | 80% |
| DeepSeek Flash | 79% | 9% | 1% | 2% | 9% | 78% |
| DeepSeek V4 Pro | 76% | 9% | 2% | 2% | 11% | 79% |
| Gemini 3 Flash | 55% | 12% | 7% | 11% | 16% | 51% |
| Grok 4.3 | 54% | 13% | 4% | 17% | 11% | 81% |
| GPT-5.4 Mini | 53% | 15% | 7% | 9% | 16% | 48% |
| GPT-5 Mini | 46% | 10% | 3% | 0% | 41% | 43% |
| GPT-4.1 Mini | 42% | 17% | 9% | 24% | 8% | 43% |
| Claude Haiku 4.5 | 36% | 24% | 10% | 18% | 12% | 47% |
| GPT-4.1 Nano | 20% | 14% | 18% | 40% | 8% | 21% |
| GPT-4o Mini | 18% | 14% | 18% | 45% | 6% | 18% |
| Gemini 2.5 Flash Lite | 16% | 9% | 19% | 34% | 22% | 15% |
| Gemini 2.5 Flash | 11% | 7% | 11% | 60% | 12% | 12% |
| All 13 models | 45% | 13% | 9% | 21% | 13% | 47% |
Each row is about 90 citations. Percentages are of all citations a model wrote under the standard request, except the last column, which is the careful wording.
What stands out
- Price and size do not predict it. GPT-6 Luna, one of the cheapest models in the list, topped the table, and it and both DeepSeek models were above 75%, while Claude Haiku 4.5 got 36%. Gemini 2.5 Flash got 11% and had no matching record for 60% of its references.
- The DOI is the weak point. Of the 1,761 DOIs the models wrote, 56% led to the paper cited. 28% did not exist at all and 17% opened a different paper.
- Obscure topics are worse. On the five obscure topics 22% of citations were fully correct, against 49% on the mainstream topics, and 35% were not found at all.
- The careful wording was a lottery. It lifted Grok 4.3 from 54% to 81%, partly because it declined to cite in some answers. It did almost nothing for GPT-4.1 Nano, GPT-4o Mini and Gemini 2.5 Flash, which kept producing references that did not check out.
What happens when the model can search the web
We repeated the standard request for two models with their providers' web search tool switched on. GPT-5 Mini went from 46% fully correct to 73%, and its not-found rate fell to 1%. Gemini 2.5 Flash went from 11% to 27%, but almost half of its references were web pages we could not verify, and many of the links were Google redirect links that no longer load.
Search helps. It does not remove the need to check. The plain model calls we tested are how a basic writing tool talks to a model, with no search; the Simple Writer in Writingful works that way. The Advanced Writer feeds the model reference pages from Google results, which we did not test here.
What the failures look like
These are real references from the study, copied as written.
A real paper with a DOI that opens something else
Goyal, M., Singh, S., Sibinga, E. M. S., et al. (2014). Meditation programs for psychological stress and well-being: A systematic review and meta-analysis. JAMA, 312(3), 357–368. https://doi.org/10.1001/jama.2014.8885
GPT-5.4 Mini wrote this. The paper exists, but that DOI opens an article titled "FDA Sets Social Media Rules". A reader who clicks the link would conclude the source is fake, when the reference itself is simply wrong.
A DOI that belongs to a test record
Fishkin, R., & Høgenhaven, T. (2015). The art of SEO: Mastering search engine optimization (3rd ed.). O’Reilly Media. https://doi.org/10.5555/12345678
GPT-4.1 Mini wrote this. Prefix 10.5555 is a testing prefix, and this DOI opens a joke record titled "Toward a Unified Theory of High-Energy Metaphysics: Silly String Theory".
A plausible reference that matches nothing
Anderson, J., Smith, R., & Patel, K. (2022). Sedentary behavior and lumbar pain in office settings. Journal of Ergonomics, 45(3), 112-125. https://doi.org/10.1080/12345678.2022.1234567
Grok 4.3 wrote this one in answer to the standard request. The authors are generic, the journal volume looks right, and the DOI is a run of consecutive digits. We found no record of the paper.
The same test inside Writingful
Numbers are one thing. We also wanted to show what a user sees, so we ran one request through the Simple Writer with six models selected and looked at every reference. This is what we typed into the form. The title was "How sleep affects student exam performance", content type Blog Post, 500 words, keywords "sleep, exam performance, students", and the additional instructions field held:
Support the claims with 3 citations in APA 7 format. List the 3 references in full under the heading "References", with a DOI or URL for each.
All six articles were ready within a minute and cost 29 credits in total. Here they are in the history list:
Writingful has a Compare view that shows two articles side by side. This is GPT-4.1 Nano on the left and Grok 4.3 on the right, with the reference lists in view:
We then checked all 18 references the same way as in the study:
| Model | Reference as written (author, year, title) | DOI as written | What the DOI opens | Result |
|---|---|---|---|---|
| Claude Haiku 4.5 | Czeisler (2013), Sleep and circadian rhythms in humans | 10.1016/j.sleh.2013.03.002 | Nothing: DOI not found | Real paper, wrong DOI or details |
| Claude Haiku 4.5 | Walker (2008), Practice with sleep makes perfect: Sleep-dependent motor skill learning | 10.1016/S0896-6273(02)00746-8 | "Practice with Sleep Makes Perfect" | Fully correct |
| Claude Haiku 4.5 | Williams (2010), Understanding the association between adolescent sleep and alcohol use | 10.1111/j.1365-2869.2009.00798.x | "Cardiorespiratory response to spontaneous cortical arousals during sta…" | DOI opens a different paper |
| GPT-4.1 Nano | Pilcher (1996), Effects of sleep deprivation on performance: A meta-analysis | 10.1093/sleep/19.4.318 | "Effects of Sleep Deprivation on Performance: A Meta-Analysis" | Fully correct |
| GPT-4.1 Nano | Walker (2010), Overnight alchemy: Sleep-dependent memory evolution | 10.1038/nrn2782 | "Inhibition: too much of a good thing?" | Real paper, wrong DOI or details |
| GPT-4.1 Nano | Carskadon (2019), Sleep's role in student learning and academic achievement | 10.1016/j.sleep.2019.05.002 | "The efficacy and safety of rotigotine transdermal patch for the treatm…" | DOI opens a different paper |
| GPT-4o Mini | Lo (2016), Cognitive performance, sleepiness, and mood in partially sleep deprived adolesce | 10.5665/sleep.5582 | Nothing: DOI not found | Real paper, wrong DOI or details |
| GPT-4o Mini | Tzischinsky (2017), Sleep and academic performance in children and adolescents: A meta-analysis | 10.1016/j.smrv.2015.12.002 | "A new view of “dream enactment” in REM sleep behavior disorder" | DOI opens a different paper |
| GPT-4o Mini | Walker (2020), Why We Sleep: Unlocking the Power of Sleep and Dreams | none | No DOI given | Could not check |
| GPT-6 Luna | Curcio (2006), Sleep loss, learning capacity and academic performance | 10.1016/j.smrv.2005.11.001 | "Sleep loss, learning capacity and academic performance" | Fully correct |
| GPT-6 Luna | Hershner (2014), Causes and consequences of sleepiness among college students | 10.2147/NSS.S62907 | "Causes and consequences of sleepiness among college students" | Fully correct |
| GPT-6 Luna | Okano (2019), Sleep quality, duration, and consistency are associated with better academic per | 10.1038/s41539-019-0055-z | "Sleep quality, duration, and consistency are associated with better ac…" | Fully correct |
| Gemini 2.5 Flash Lite | Curcio (2006), Sleep loss, learning, and memory: An updated review | 10.1016/j.smrv.2005.11.004 | Nothing: DOI not found | Not found as cited |
| Gemini 2.5 Flash Lite | Lund (2010), Sleep patterns and predictors of disturbed sleep in a large population of colleg | 10.1016/j.jadohealth.2009.05.011 | "How Adult Specialists Deal with the Principles of a Successful Transit…" | Real paper, wrong DOI or details |
| Gemini 2.5 Flash Lite | Walker (2017), Why we sleep: Unlocking the power of sleep and dreams | none | No DOI given | Could not check |
| Grok 4.3 | Curcio (2006), Sleep loss, learning capacity and academic performance | 10.1016/j.smrv.2005.11.001 | "Sleep loss, learning capacity and academic performance" | Fully correct |
| Grok 4.3 | Gaultney (2010), The prevalence of sleep disorders in college students: Impact on academic perfor | 10.1080/07448481.2010.483708 | "The Prevalence of Sleep Disorders in College Students: Impact on Acade…" | Fully correct |
| Grok 4.3 | Hershner (2014), Causes and consequences of sleepiness among college students | 10.2147/NSS.S62907 | "Causes and consequences of sleepiness among college students" | Fully correct |
GPT-6 Luna and Grok 4.3 got all three references right. The others each had at least one that did not hold up, and GPT-4.1 Nano's article, in the editor, also included a quote attributed to a named sleep researcher that we could not find in any source:
"Sleep is not a luxury; it is a critical time for cognitive restoration," emphasizes Dr. Mary Carskadon, a sleep researcher at Brown University.
Carskadon is a real researcher. We searched for the quote and found nothing. Treat quotes from AI the way you treat its citations: unverified until you find the source.
The two reference lists above look equally convincing. Only one of them is right. That is the whole problem: a fabricated citation and a correct one look identical until you click.
What to do about it
Check every reference before you publish. It takes two minutes.
- Click the DOI. If doi.org says "not found", or the page is about something else, the reference is wrong.
- Search the title on Crossref, OpenAlex or Semantic Scholar. A real paper shows up with its true DOI, so you can fix the reference.
- Check the authors and year on the real record. We saw real papers cited with the wrong year or journal.
- Read the abstract. A real paper can still not say what the sentence claims. The check we ran only tells you the paper exists, not that it supports your point.
- Find sources first. Give the model the facts and the sources you want it to use, and ask it to write around them.
Choose the model and the mode with care
If you need citations, the model matters more than the wording of the request. A model that writes a reference from memory will write an answer that looks right. A tool that lets the model search, and shows you the pages it used, is a better starting point, and you still have to open the pages.
Google's own guidance on generative AI says it is critical to manually fact-check and review AI-generated content before publishing, and that this includes metadata and other text that can appear in search results. Read it in Google's guidance on AI-generated content.
We changed our own tools after seeing this. The free paragraph generator is now set to write without citations, links or named sources even when a request asks for them, so you add real sources yourself.
Limits of this study
- One request, one wording per topic. We ran each prompt once per model, with default settings. Another run would give different answers, and the averages are what to read, not any single result.
- 30 topics, APA only. A different format or domain (law, medicine, code) may behave differently.
- No web search by default. Most of the 13 models were tested the way a plain writing tool calls them: no search, no tools. Search results are shown separately and only for two models.
- Checking has errors. Crossref and OpenAlex do not index everything, books and reports often have no DOI, and some titles are paraphrased. We tuned the matching to count a paper as found when the title, author and year agreed, and we corrected one flaw during the work (reports with "review" in the title were being treated as journal articles). In a check of 120 references we had marked "not found" against a second index (Europe PMC, which leans towards medicine), 2 matched a real paper with the same title, first author and year, so a small share of "not found" results are real papers our first pass missed.
- "Not found" is not proof of invention. It means no record matches as cited. Some are real papers with a garbled title or journal. We checked a handful by hand and found both kinds.
- Models change. These results are a snapshot from October 2026.
Get the data
The full list of 2,402 citations, with the model, the wording, the reference as written, the DOI, what the DOI actually opens and our result, is a CSV you can download: AI citation study data (CSV). The prompts are on this page in full. You are welcome to check our work, and to tell us where we got a result wrong.
