It’s the latest episode of Science+Story: The Podcast — I talk with Daniël (a social psychologist at Eindhoven University of Technology) about his experience organizing and paying “red teams” (an independent team of investigators, a concept borrowed from software) to find flaws and biases in new research, a practice which would have obvious benefits for communicating new research.
But isn’t that what peer review is for? Are you really asking that question? As Daniël puts it:
The peer review system as we have it is — well, experts look at it, but experts only have certain expertise. So in an extreme way, if you really want to make sure that your paper is solid and you have an extensive red team — so people with multiple expertise — you will get criticism on the statistical aspects and the theoretical aspects, on the methodological aspects. And your reviewers might be experts in one or two of these things, but typically not everything. So it’s always possible to miss something. And some journals work with dedicated statistical reviewers in addition to the content reviewers. But that’s, for example, already very rare. So, yeah, you don’t cover all your bases during peer review.
So an intense, open way to improve peer review is interesting enough to me. But something Andrew Gelman said recently also made me want to talk to Daniël — about how the need to get through peer review has actually created a culture in science that is all about avoiding or challenging criticism instead of improving by it. Daniël sees science culture similarly to Gelman, which is why he got interested in red teaming:
I think that we just see that there is no place for the critics. I love critics. Like, I think criticism in science is really important. But it’s not appreciated. People very often get upset if you criticize their work. So there seems to be no place where these people can do their work without being told, like, “Oh, you’re so harsh,” or “You’re so, you know, aggressive,” or something … So that’s always an interest — how can I make sure that there is room for criticism in science? Because I think we need a bit more of it. So that’s the driving idea … behind more of the work we do. And then for this idea I remember that we were just discussing with a group of close colleagues. And we just thought, “This sounds like too fun not to try. It might just work.” And I think it surprised us a little bit when we just tried it. it surprised us a little bit how well it worked, like, how interesting it was, what people brought up, if you just have a bunch of experts to go through your work. It’s just amazing to see. And yeah, you learn a lot … But I think this place for critics that I think is really important, we should want to have these people on board in science, and we don’t reward them as much as we should I think.
We also cover whether red teaming might slow research down too much, how red teaming and research funding might evolve going forward and whether Daniël thinks fast COVID-19 research has gotten better over the last year.