Last month, arXiv announced they will begin banning authors for one year if they submit manuscripts with unchecked AI generation, particularly hallucinated references.
We decided to test how this new policy was working. Using Reviewer3’s reference checker, we sampled 200 papers submitted to arXiv on June 10th, well after the
ban announcement, and found some concerning results:
- 55/200 (27%) had at least 1 hallucinated reference.
- 29/200 (14%) had more than 1.
- In total, we identified 117 hallucinated references.