My blood runs cold every time I hear researchers say: “We hope our study won’t be interpreted as saying (insert horrible unintended conclusion not warranted by the study’s findings).”
Because it means a) they don’t have clear messaging for their study, b) they’ve overframed their study, or c) they have clear messaging but haven’t enforced it.
Which means it was preventable.
I use the verb “enforce” because researchers and research-driven organizations have that power, despite what they think about distorting headlines and news coverage of their researcher. If they’re worried at all about misinterpretation, they can say to a reporter right from the start: here’s what my study doesn’t mean, here’s what I did find, and here’s the difference.
Unfortunately, too many researchers and research communicators play with fire in promoting their studies — happy to exploit a current issue by pushing the relevance of their findings to that issue beyond what’s defensible, but then full of buyer’s remorse when the media follow their cue and overframe the study’s findings.
For example: Are social media the cause of declining mental health in U.S. adolescents?
“Social media may not make teens depressed after all: study” says a headline and photo caption for a story in The New York Post, reporting on a new study in Computers in Human Behavior based on annual questionnaires completed by 500 adolescents from ages 13-20 for eight years.
Unfortunately, the headline and caption are a complete distortion of the study — as was at least one other major news outlet headline covering the study.
But when you look more closely at the study, you quickly see that it doesn’t rule out that social media could be harming the mental health of youth. It just rules out that time spent on social media is a factor.
Lest you think this is an example of once again journalists distorting science, the economist Alex Tabarrok at Marginal Revolution made the same mistake with his headline and his conclusion.
So what caused this distortion?
First, some text in this study — “but is social media the real culprit? Or are we engaged in a moral panic, perhaps not understanding the root of the problem?” certainly sets the table for misunderstanding. That language overframes the reach of the study’s findings. We are not going to find out from this study the answer to the question “Is social media the real culprit?”
Second, Sarah Coyne of BYU, the study’s lead author, plays with fire in her framing to the press. For instance, she’s quoted by the Post as saying
“We are in the middle of a mental health crisis in the US, and social media and cellphones are often blamed…. I wanted to look at long-term impact and look at it across teenagers’ adolescence to see what’s going on…. Like everyone else, you assume social media is the bad guy.”
To her credit, Coyne is quoted later in the Post piece that it isn’t time spent on your phone, but what you do during that time that that matters. “Two people can spend the exact same amount of time on social media, but it might have a negative effect on one and a positive effect on the other, depending on what they are doing,” she tells the Post. It’s just we need to move past “screen time” as a crude indicator of risk.
But why play with the fire of “social media/mental health crisis”? Because it’s hot. Because it’s why reporters will call you about your study in the first place.
Third: headlines are perforce really short. And to work, they have to grab your attention. And headlines are not written by reporters except at very small outlets — they’re written by headline writers who may only glance at the first few paragraphs of the piece.
As a researcher, you might not be able to prevent headlines that distort your findings. But you must do everything in your power to prevent them, by enforcing your messaging about what the study doesn’t say as well as what it does say.
In another article on the study, Yahoo! Lifestyle senior editor Beth Greenfield wrote:
Coyne, a mother of five, stresses that the findings should not be taken as a green light for adolescents to enjoy limitless time on Instagram, Snapchat or other popular platforms. “One of my biggest fears about the study,” she says, is that it somehow gets misinterpreted, paving the way for “unfettered access to social media for, say, my 11-year-old.”
Too late. And now we understand why.
Takeaway: If you’re concerned about misinterpretation, don’t overframe your findings, and message them for what they don’t say as well as well as what they do.