Some modelers (and scientists) seem to have forgotten this, or never learned it. In the wake of the recent US elections — and the apparent disparity between the heavy odds modelers gave for Biden and Democratic US Senate candidates to win and the results — their response is best illustrated thus:
Here’s Exhibit A for Silver and his defenders: the lead data viz for FiveThirtyEight’s presidential election coverage:
It’s technically accurate. The election’s outcome is in fact depicted by one of those blue balls between the “TIE” and the rightward “+100” lines. So, Silver and others say, the site “got it right.”
But of course that’s not the message many people took away from the graphic or FiveThirtyEight’s other data viz, no matter Silver’s pre-election attempts to warn them via long text articles that Trump could still win with a “normal-sized” systemic polling error. Multiple long articles do not equal the reassurance of this FiveThirtyEight forecast
which ran under the header
and was cited incessantly.
FiveThirtyEight’s modelers understand that data viz trumps text — or at least they did before the election. Read the explanation from FiveThirtyEight’s Anna Wiederkehr of why the site redesigned its election graphics for 2020. The designers wanted to make sure people who only read its top-line forecast numbers wouldn’t think FiveThirtyEight had bungled its forecast — as that audience did in 2016. They knew people needed to get the message about the model’s uncertainty in the visualization itself.
But they didn’t. And now Silver is blaming humans for not reading his caveating articles. For being human.
Silver’s reaction is still all too typical for scientists in general. The computational biologist C. Brandon Ogbunu writes in WIRED that modelers (for elections or for pandemics) need to be clear that their in silico forecasts “say little about what the real world actually looks like” because the inputs might be systematically biased.
That disparity — between the model and the real world — would seem to be…problematic, and worth communicating forcefully.
Yes, but the real problem, he argues, is that “we often build our expectations from models around our emotional needs,” and overreact when reality deviates from modeling.
So…stop being so invested in the models we’re selling you about things you deeply care about!
Stop being so…human.
The computer scientist Jessica Hullman writes an excellent post on Statistical Modeling, Causal Inference, and Social Science reviewing many of the options for getting humans to focus less on an election model’s forecasts and more on its assumptions and uncertainty. Spoiler alert: It’s going to be really difficult! I like the way she puts it — it’s like having the FiveThirtyEight mascot Fivey Fox tell you “Don’t really trust anything here!” as soon as you hit the site.
FiveThirtyEight and The Economist aren’t going to do that; they’re too busy selling to advertisers the audiences that keep refreshing their forecasts. As Hullman phrases it, humans are hard-wired to want certainty and answers. Scolding them for pursuing what your business model is built upon is both hypocritical and self-defeating. (To his credit, Andrew Gelman, one of the statisticians behind the Economist’s forecast, is reviewing what went wrong with their modeling in public — or more accurately, semi-public — on his blog.)
Effective data visualization is instantly, contagiously insightful and explanatory — that the basis of its power. According to Hullman, research has yet to tell us whether we can add anything to a powerful data visualization that flags its uncertainty for everyone without destroying the visualization itself. We need that research.
In the meantime, I agree with her: The answer isn’t “don’t do modeling.”
But the answer also isn’t “do modeling, sell its forecast and then expect your audience overnight to become highly numerate while overcoming its strong drive for quick answers.”
Your audience is human. It’s your job as a communicator to account for that.