LinkedIn is all a-twitter with discussion about Fabricated citations: an audit across 2·5 million biomedical papers in The Lancet. They checked references from 2.5 million articles—but only articles with a PMID (a PubMed ID, sort of like a DOI for articles listed in PubMed), so they excluded 23% of the references. They did some automated checking—including getting an LLM to validate them (which seems strange since we often blame LLMs for bogus references, but I digress). They show an increasing—and to many, alarming—rise in fabricated references starting in 2024.
The authors go on to cite an article from 2015, showing that preparing accurate references has always been difficult. They go on to suggest that “reference verification is not standard in peer review. Automated reference verification can close this gap.”
I have seen many editors responding to this article, and others like it in recent weeks, lamenting that checking every single reference in a paper is largely impossible, perhaps because no tool has made it fast enough to fit into an editorial workflow. I think that RefRunner can change that. (For those paying careful attention, citation wizard has become RefRunner!)
The bulk of the references in a typical paper—and all of them that exist in Crossref and OpenAlex—can be verified with RefRunner in under 5 minutes. That’s less time than an editor spends getting the journal software to send the paper out for review. RefRunner not only verifies DOIs included in the references, but also attempts to find DOIs for every reference, and most of the time, if a reference exists in a public database, RefRunner will find it there.
RefRunner is an automated reference verification tool that authors can depend on to dramatically reduce the drudgery and increase the accuracy of their references. For editors, it offers two ways to make their lives easier. One is quick, easy, and free: Recommend that your authors check references with RefRunner before submitting. I think that most authors—regardless of whether they have a completely bogus paper fabricated by an LLM—want their references to be correct without spending hours verifying each one. The other way is to integrate RefRunner into your review and publication pipeline. As I’ve pointed out, checking just at submission isn’t sufficient; pre-prints get published, papers get retracted, and, I’d like to hope, incorrect metadata gets corrected (like this article that doesn’t include the author’s name; I am the author!).
Some references are just hard to find—conference papers, books (which have multiple editions produced by publishers whose names change weekly), newspaper articles, movies (I tried for a while to integrate IMDB before I came to my senses). For those, RefRunner provides links to Google Scholar, Google, and others so you can save some keystrokes running those down if you feel obliged. If it’s your paper, and you’re citing a conference paper from 3 years ago, maybe seeing that it’s difficult to verify the paper exists is the push you need to see if your colleague with that Really Great Idea ever managed to get it published. . . or maybe it wasn’t such a good idea after all.
What RefRunner Easily Catches
I’ve written earlier about finding and fixing hallucinated references. Here are examples of how automated reference verification helps authors who are genuinely doing their best to create accurate references to papers.
Real Articles with Wrong Metadata
Here the author dropped the last character of the DOI, an honest mistake. RefRunner flags the discrepancy and shows you the corrected version, making it easy to see that just the DOI was wrong.
Legitimate References Missing a DOI
Google Scholar contributes to this problem. It makes it easy to find an article and get something resembling a reference, but it’s missing the italics required by APA, and the DOI. Finding the DOI is several more clicks. Sometimes authors just don’t have time. RefRunner will take most anything resembling the reference, find the DOI, and perfectly format it using the same formatting engine used by Zotero and most other popular legacy reference tools. Here we see that the case of the title has also been corrected to APA’s preferred sentence capitalization.
Updating Pre-prints after Publication
This article was cited from its pre-print year, before it was assigned a volume, issue, and page numbers. Authors aren’t likely to recheck every DOI that they have already validated every time they revise and resubmit the paper, but with RefRunner, it’s easy and gives you peace of mind.
Identifying Retracted Papers
Though sometimes a retracted paper makes a splash, it’s just as likely that a retraction could miss an author’s attention. For an editor, who might not be deeply attuned to every sub-field covered by the journal, retractions are easy to miss. RefRunner checks retraction status every time and makes it clear when it happens. Unlike legacy tools that pull their data from your private static database, that you maintain, RefRunner checks every time.
What an Editor Can Do Now, with Minimal Effort
Recommend that your authors check their references with RefRunner before submitting. They’ll be glad they did, and so will you.
Try it out yourself. If you’ve got a paper of your own approaching submission, let RefRunner find your references for you by searching the in-text citations that you thought were merely placeholders. What’s really interesting though (well, it has been for me!), is to try it with 5–10 papers and see it in action. After checking dozens of papers, I still find it hard to believe how well it works. Once you’re convinced it will help improve citations and papers at your journal, you can formally integrate it into your pipeline.