How researchers get heard

Another Stink in the Scientific Air

The science-media industrial complex is broken. I’ve been writing to you about it for nearly two years. Nutrition studies and COVID-19 have exposed it. Clickbait-y studies far too tempting to be true are published in volume, publicized at high volume, covered by the press at maximum volume and walked back only with the most massive of pushbacks. Then another day arrives and we go through it all over again.

I have no excuse for why I continue to be stunned by just how wretched the junk this system produces can be, and that I remain obsessed with calling it out. I also believed in Santa Claus until age 9. Clearly, I find it hard to abandon hopeless causes.

Ponder this: A study in the journal Climatic Change that finds that Gen Z and millennials in the United States who are “concerned about climate change” are on the verge of giving up having kids — or having as many as they might have wanted — because of that concern:

  • Almost all (96.5%) of the 607 anonymous 27–44 year old U.S. residents surveyed for the paper were “very” or “extremely concerned” about “the well-being of their existing, expected, or hypothetical children in a climate-changed world.”
  • Nearly 60% of the respondents were “very” or “extremely concerned” about the carbon footprint of having children.
  • The study also included extensive quotes from interviews with the survey respondents. Here’s one from a 31-year-old woman: “Climate change is the sole factor for me in deciding not to have biological children. I don’t want to birth children into a dying world [though] I dearly want to be a mother.”

You can imagine the headlines. Oh, never mind — here’s one: “Climate ‘apocalypse’ fears stopping people having children – study,” said the Guardian’s. Similarly breathless reporting of the paper made its way into the Daily Mail, the Independent, Gizmodo, Global News Canada and Matthew Schneider-Mayerson, lead author of the study, told the Guardian that “It is an unprecedented window into the way that [some people] are thinking and feeling about what many consider to be the most important decision in their lives.”

That bracketed “some people” turns out to be really important, though. Because you only have to read three pages into the paper to realize the survey respondents are basically ringers — first recruited through posts on the Facebook and Twitter pages of “ten well-known climate thinkers, activists and organizations,” then culled further through an additional screener survey to make sure those who remained belonged to the “Alarmed America” segment of the Yale Program on Climate Change Communications “Global Warming’s Six Americas.”

So: The respondents were already pretty primed to think climate change is and will be apocalyptic and that bringing a child into that apocalypse might be ethically questionable. That’s not all people, or even most.

Only Michael J. Coren of Quartz read the paper and bothered to call another climate change opinion expert for perspective — Stanford’s Jon Krosnick, who basically called the paper scientific malpractice. Krosnick said the findings were based on “a preposterous methodology with no projectability to any population.” Cohen reports Krosnick told him the study had problems including “the composition of respondents, demographic definitions, recruitment methods, and framing of the questions.”

Krosnick isn’t just criticizing the composition of the study’s respondents — which, as Cohen reports, turned out to be predominately white (88%), liberal (70%), and well-educated (52% with a graduate or professional degree, and 41% with a bachelor’s degree).

He’s also pointing out a more fundamental error in the study’s approach: You can’t ascertain what motivates someone’s behavior simply by asking them. Psychology, he tells Cohen, has shown over the last 100 years of experiments that people assign motivations to their actions or mindsets ex post facto and often without realizing they’re doing so.

“It turns out that vast majority of what we do and think is governed by unconscious processes,” Krosnick tells Cohen. There are ways to design experiments to get around this phenomena, but the Climatic Change study didn’t employ them.

Unlike the case of Eric Feigl-Lang, this study didn’t get much pushback at all on Twitter. Why not?

Because with climate, so many of us have a worldview of emergency that, regardless of the depth of the existing evidence supporting that worldview, also demands constant care and feeding with fresh evidence. Are the new studies flawed? No matter — we already “know” the reality behind them to be true. That is an atmosphere not of science, but of propaganda. Science (and science communication) that seeks to exploit these kinds of worldviews is no better than science communication that purports to hand down expertise on tablets from mountaintops.

I admit to being a bit obsessive about this phenomenon. It worries me that I’m becoming a bit of a Javert about it (you know, Javert from Les Misérables). Then I reread this great piece by the statistician Andrew Gelman, who reminds us that the science-media industrial complex makes it really hard to get changes much less retractions from journals that have published obviously flawed studies and that pursuing these issues in the face of such power will make you look like a bit of a Javert.

“It’s fine to be obsessive,” Gelman reminds us. “That’s part of what it’s like to be a scientist, or a journalist, or even a citizen.”

Another brawl in the square, another stink in the air …

Credit: Inge Gjellesvik