Science covered in the news is “pretty likely to be overturned,” according to a 2017 PLoS ONE study (which, admittedly, was covered in STAT.)
So a new article in WIRED won’t surprise anyone: the findings of a big-journal, splashy-headline, lots-of-coverage recent study are in dispute. This time, it’s a study in Science magazine that claimed planting 1 trillion trees would be “one of the most effective carbon drawdowns to date.”
Other scientists have panned the study’s math as wildly overselling the sequestration potential of tree planting — by a factor of 5, according to one comment Science published to the paper. (Reforestation, say the study’s critics, would still be important for carbon sequestration, but it’s by no means a magic bullet and shouldn’t detract from the need to reduce global CO2 emissions in our quest to avoid the worst impacts of climate change.)
So there’s peer review, and then, if we’re lucky, the real peer review after publication, on Twitter and elsewhere, eliciting increasingly puckered defenses on the part of the study’s authors. Better than nothing, but the problem remains: this process comes after the headlines for the study have made their impression — an impression that’s hard to erase. (Enter obligatory Andrew Wakefield MMR-autism study reference here.)
What’s the solution? The Nature Conservancy scientist Stephen Wood emailed me about the WIRED piece and replied to my thank you with this gem:
I was wondering after I read that piece: what if popular media couldn’t cover a paper until it’s been out for 4 months? As it stands, media coverage of papers is basically just promotion because it’s timed to come out when the paper does, and therefore before the scientific community has a chance to interpret. What if coverage didn’t happen until after people wrote rebuttals in journals and the authors responded? Practicality aside, I think that’d be an improvement.
I think Steve’s idea is fantastic, as a thought experiment if nothing else. Some of the questions it raises:
- Could this “peer review” happen in preprint instead? If so, how would the scientific community become alerted to new papers that need wide peer review? I would imagine Michael Eisen would have a lot to say about this.
- Would journals still put as much promotional muscle behind new studies four months out, especially the ones that have been heavily critiqued? Might they even retract acceptance during that period?
- What would science communicators do if you took away their ability to promote a paper on publication? Would their promotion efforts simply time-shift four months hence? Or might they shift their focus to explaining how the paper fits into the literature, using the fruits of the wider peer review?
- What if your organization simply delayed promoting your new research for four months on its own? What would change for you? Would it be like fighting with two hands tied behind your back? Or might the practice even yield benefits — a greater degree of certainty in your staff’s findings, for instance, or a recalibration toward promoting your staff’s overall expertise and insight?
Science magazine (and science in general) publishes dud papers — a lot of them. The billion-trees study might indeed be one, and no journal’s peer review process is going to catch everything that a wider peer review process might. The problem comes when the study gets lots of media coverage and a non-evidence-based idea out of the study gets stuck in the public conversation and imagination, quickly becoming impossible to dislodge through the normal scientific round.
Would such a step be at all practical, especially with journals still promoting their studies? Or how else might we begin to curtail these errors and help reeducate people about science as a set of evolving models of the world, rather than a world of discoveries where journalists and the public seem to think one new paper can routinely cancel 20 years of literature?