July 2, 2019

Replication is the New Debunking

Reproducibility and replicability of research is hard to communicate to the public. It faces all of the hurdles any myth debunking faces, including:

Reproducibility and replicability of research faces all of the public communication hurdles any myth debunking faces, including:

Media are “far more likely to report on initial studies than follow-up research”:
The vast majority of discussion of reproducibility and replicability is “dominated by academic sources and science-focused outlets” (see discussion on page 36);
It’s hard to “unstick” an idea (e.g., vaccines cause autism) once stuck;
Some leading journals are reluctant to publish replications of studies that made headlines for those journals.

Yep, life is unfair.

My hypothesis: successful debunking through replication is a function of a) the breadth of the replicator’s network of dissemination multiplied by b) the trust the target audience has in that network (if the network itself isn’t already the target audience).

Broad network times trust of network.

So: this new ELife meta-review of 3,000 randomized controlled trials published in JAMA, the Lancet and the New England Journal of Medicine reveals 396 “medical reversals” — low-value medical practices that “are either ineffective or that cost more than other options but only offer similar effectiveness.” We’re talking about practices such as

Taking fish oil to reduce heart disease;
Teenage girls carrying around a lifelike doll to deter the allure of getting pregnant;
Keeping your house free of dust mites, mice and cockroaches if your child has asthma attacks;
Using a step- or calorie counter to help you lose weight; and
Having surgery on a torn knee meniscus instead of physical therapy.

The meta-review has gotten good media coverage. But, as we’ve seen before, media coverage isn’t enough to guarantee reversal of these myths.

And I would predict that many of these myths could be reversed — because health communications has a rich knowledge base on how to build such networks and such trust.

When you don’t have a strong network or trust, you fail at debunking. For instance: Debunking the myth of how the reintroduction of wolves worked magic on the ecosystem at Yellowstone National Park.

The lifestyle optimization guru Tim Ferris recently emailed about this video showing how reintroducing wolves into Yellowstone National Park induced a virtuous “trophic cascade” that — from reducing elk population that the wolves predated, which allowed willow trees to grow more abundantly, which provided beavers with more food, which generated more dams and altered the flow of Yellowstone’s rivers — supposedly benefitted the entire Yellowstone ecosystem, including altering river flow.

The video had already received more than 40 million views before Ferris’ recommendation. It’s a beautiful story. It also turns out to be completely unscientific — “absolutely a fairytale,” said one scientist. Its debunking received widespread coverage last year.

But the myth lives on — because it’s a beautiful story; because people love wolves; because conservation has a weak network with weak trust effects; and maybe, just maybe, because some in conservation find it useful.

Takeaway: If you want to debunk a myth, it’s never just about the science. You need a broad network that trusts you, is willing to disseminate your findings, and has the trust of your target audience. More on how researchers can build that kind of network in coming weeks.

Media are “far more likely to report on initial studies than follow-up research”:
The vast majority of discussion of reproducibility and replicability is “dominated by academic sources and science-focused outlets” (see discussion on page 36);
It’s hard to “unstick” an idea (e.g., vaccines cause autism) once stuck;
Some leading journals are reluctant to publish replications of studies that made headlines for those journals.

Life is unfair.

My hypothesis: The key to debunking through replication is a) the breadth of the replicator’s network of dissemination multiplied by b) the trust the target audience has in that network (if the network itself isn’t the target audience).

Taking fish oil to reduce heart disease;
Teenage girls carrying around a lifelike doll to deter their risk of getting pregnant;
Keeping your house free of dust mites, mice and cockroaches if your child has asthma attacks;
Using a step- or calorie counter to help you lose weight; and
Having surgery on a torn knee meniscus instead of physical therapy.

The meta-review has gotten good media coverage. But, as we’ve seen before, media coverage isn’t enough to guarantee reversal of these myths.

My hypothesis: to stick, the replications will need a strong, broad network of health providers who have the trust of patients who’ve already been bombarded with messages to the contrary for many of these myths. And I would predict that many of these myths could be reversed — health communications relies on such networks and such trust.

It’s a beautiful story. It also turns out to be completely unscientific — “absolutely a fairytale,” said one scientist. Its debunking received widespread coverage last year.

But the myth lives on — because it’s a beautiful story; because conservation has a weak network with weak trust effects; and maybe, just maybe, because some in conservation find it useful.