This morning I flicked on MSNBC’s “Morning Joe” and heard Dr. Vin Gupta assert that one was 19 times more likely to contract the COVID-19 virus indoors than out. I Googled and found the study he was talking about — a review of 110 cases in Japan. Out in preprint.
And I thought: This is what the end of peer review sounds like.
Peer review is failing hard during the pandemic — the latest being The Lancet and New England Journal of Medicine’s publication of obviously flawed and probably fraudulent research — and we’re hearing a ton of excuses. Almost all of them — from the rush to get out new information to the acceleration of research itself to the supersaturation of academic peer reviewers to the lack of double-blind review to the willingness to cut research corners as response to the virus becomes more and more politicized — are framed as benign but forgivable. The current system, the excuse makers say, is breaking under pressures it was never designed to withstand.
Like corrections in a newspaper, these excuses are designed to reassure us that everything else besides these anomalies is accurate. That the system itself is fine — or at least (as the cliché goes) the worst system for detecting scientific flaws … except for all the others. Carry on.
But what’s being said more and more openly — beyond the usual open-access and preprint-archives ghettos — is that the pandemic is exposing the fatal flaws of peer review itself. Some choice readings:
- Adam Marcus and Ivan Oransky (the latter the head of Retraction Watch) in WIRED on how “problematic research is not nearly as rare as the powers that be in science would have us think, and efforts to self-police haven’t been particularly effective.”
- Simine Vazire, an academic and former editor of the journal Social Psychological and Personality Science, in WIRED on how “peer-reviewed journals have been disastrously ineffective at preventing … bad science to get into the hands of policymakers and practitioners.”
- Columbia’s Andrew Gelman, rather devastatingly, on how the problem with peer review is the peers.
- Remco Heesen and Liam Kofi Bright asking point blank “Is Peer Review a Good Idea?” — and finding not a lot of reasons to say “yes.”
Take 45 minutes to read those four pieces and you will find it difficult to have faith in peer review as a sole signifier of quality. Another study found that peer review caught only one deficiency per manuscript in about 25 categories. Dr. Gupta seems comfortable commenting to millions within that error bar.
Pace Alex Danco, what will dethrone peer review is not just scientists doing post-pub reviews on Twitter, but when the media stop regarding peer review as the gold standard of value. And when the media get burned repeatedly reporting on peer-reviewed research, they’ll stop paying so much attention to peer review. That day is probably closer than we think, if Dr. Gupta and the copious reporting of preprint research by others in the media are bellwethers. (The New York Times’ subhead on its story about those Lancet and NEJM retractions: “Two major study retractions in one month have left researchers wondering if the peer review process is broken.”)
Meanwhile, most research communicators continue to be addicted to peer review and the sugar high of media coverage for new papers just out in a peer-reviewed journal. Which also means that research comms becomes crisis comms when postpublication review descends like locusts on those papers to find their flaws. These communicators have yet to find a new paradigm of value for what they do. While the pandemic changes everything else, I doubt they even see the need to find it.