In response to my essay last week about the drive to end “statistical significance” as a way to judge the results of scientific experiments and studies, a list member writes (shared here with permission):
I’m glad the movement to reform the use of statistical significance is gaining momentum. But an equally serious and more overlooked issue has to do with causal inference. Most scientific results only show associations (think: correlations) and can not, at least under reasonable assumptions, be used for causal inference. Yet in so many cases, causal language slips in. Recent studies about the benefits of parks/greenery for mental health come to mind, showing an association between the two. But the conclusion often drawn from this is that people had access to more parks, they would be happier. But it’s devilishly hard to tell if, for example, happier people tend to prefer to live near parks, reversing the direction of causality. Or if there is some other factor that both makes people happy and leads them to live near parks. I’m using this as an illustration only – not a critique of any particular study (some of these studies do control for socioeconomic status etc, which is at least a start).
But the use of causal language for studies that only show associations is endemic in the scientific literature. Basing policy on associations is not necessarily a good idea. The example of ice cream sales being correlated with shark attacks comes to mind – really, it’s the weather that leads more people to go to the beach and then both eat ice cream and be attacked by sharks. This seems silly but in fact this sort of problems exists in virtually any scientific setting, not least in ecology or social science. To me, this undermines conclusions drawn from scientific studies just as much as the simple and dichotomous use of p = 0.05. (Note: as long as a study avoids any reference to causality, or discusses the limitations of its methods in establishing causality, it is on firm ground. But associations are far less interesting and useful than actual causality, especially for policy.)
Here’s a good example from conservation science. Earlier studies had no way of estimating the causal impact of protected areas, whereas the study here did. As you can see, the results vastly differ.
As a supplement, the list member followed up with a new study in Biological Conservation that shows people living in closer proximity to protected areas in France exhibited increased pro-environmental behaviors such as voting for the Green Party, support for WWF and a French bird conservation group, and participation in a backyard bird watch citizen science program.
Of course, people already displaying pro-environmental behaviors could also prefer to live in proximity to protected areas…
For a more sophisticated example, see this recent PNAS study on the correlative “association” between residential green space in childhood and lower risk of psychiatric disorders into adulthood. The language dances in fascinating ways between care about what the study and the literature have established and what policy should be building — more green space in urban areas, now — upon all these correlative results. (One also wonders if the negative result would have been published, at least in PNAS.)
I’ve written about causation vs. correlation before, and it won’t surprise you I think my list member is right: “Associations are far less interesting and useful than actual causality, especially for policy.” But that implies policy can tell the difference, or is told it. The culture of research in some disciplines is such today that incentives for calling correlation on yourself are minimized, compared with the rewards for publishing a sexy finding. Multiple that single study by 20 or 50, and now you have correlation as the baseline for a line of inquiry (e.g., the link between living in proximity to green space and improved mental health).
Now you have a social phenomenon, not just an overblown study. Now you have science. Because we’re way beyond the famous link between ice cream consumption and polio, aren’t we? Because what is science, but the set of practices and results scientists say it is?
Too often, the near enemy of science is science.