TrueTuesday: mistakes & failure Jan 22, 2019
Mevrouw Noemie AUBERT BONN
Being a scientist is so much more than just a title. We all share an environment, a culture and most importantly a community. Yet despite the fact that scientific findings and knowledge are covered on all possible media, the factors that affect, characterize, and influence this community and its environment are rarely discussed and often even taboo. Every six weeks, Ph.D. student Noémie Aubert Bonn targets such a topic together with 5 experts for TrueTuesday. The second one is mistakes and failure.
In science, most of what leaves the lab to end up in papers, the news, or public stories are ‘successes’. The failures, mistakes, and even negative findings generally remain invisible to the outside world. Although it may seem intuitive that the ‘best side of science’ is the one we show publicly, the invisibility of failures might have more serious consequences than we think.
First and foremost, being aware that research does not always work as planned or confirm hypotheses helps science consumers remain realistic in their expectations. Whether these consumers are the public, students, or fellow researchers, keeping a realistic mindset is a key ingredient for critical thinking.
The absence of mistakes and failure in literature may also influence the expectations that researchers set upon themselves. Young researchers, in particular, may underestimate the possibility of failure and undertake overambitious plans in which there is no room for mistakes. That being said, overambitious plans also promise unrealistic outcomes, and the overabundance of unrealistic promises in science may slowly become the norm—if not the expectation—in grants application, making it harder to value reality.
In turn, this self-imposed pressure may escalate into personal harm, such as feelings of ‘imposter syndrome’ — as discussed with Christine Liu previously — or other pervasive stresses which can push brilliant scientists away from science.
Finally, the invisibility of non-successes may also influence knowledge itself. The non-publication of negative findings, for instance, is thought to be one of the highest threats to current research integrity. But also, and maybe most worryingly, the ‘perfect’ picture of science may discourage students and researchers from being open about their mistakes and those of others, and may lead to concealments, lack of transparency, or even misconduct. But what do the experts have to say on this matter?
MELANIE STEFAN ON A CV OF FAILURE
You certainly have a few versions of your CV, each supplemented with new successes. But what about a CV of failure? In 2010, Dr. Melanie Stefan, now lecturer at Edinburgh Medical School: Biomedical Sciences, mentioned the idea in the Journal Nature. Let’s ask her about it.
Q: How did you come up with the idea of building a CV of failure?
A: I like following sports, especially football. It struck me that failures in professional sports are immediately visible and discussed by a wide audience, whereas failures in academia are usually kept hidden and not discussed at all. I thought it would be helpful to be more open about failures to show that it is just a normal part of the academic career path.
Q: How can a CV of failure help scientists?
A: Personally, it has helped me understand that I am not the only one who fails, and that even the most successful colleagues have had moments of rejection. I also hope it makes successful senior scientists reflect on their own history, and encourages junior colleagues who face difficult times.
Q: Do you keep updating your own CV of failure? Do you still agree with your idea from 2010?
A: I have kind off been neglecting my CV of failures. There are just too many to keep track of! But I do keep a list of rejected grant applications so I can go back and learn from them. Also, my thinking has evolved, thanks to conversations I have had with others. Many pointed out that "CVs of failure" presuppose a position of power and privilege. A lot of great scientists who fail early on do not have the support structures (networks, money, institutional support, etc.) to keep trying until they succeed. So, reading about other people's early failures and later successes might actually be discouraging instead of helpful. The other thing is that we need to have a conversation about what "failure" & "success" even mean. In academia, we have a narrow view of success and we measure it using very specific, yet arbitrary metrics. But there are different ways of being successful, and therefore different views on what constitutes failure.
IRENE INGARDI ON NIGHTS FILLED WITH FAILURE
FuckUp Nights (FUN) are exactly what their name indicates: public gatherings where people come to talk about their failures. Irene Ingardi coordinates FUN in Brussels. Let's ask her about it.
Q: Describe a typical FuckUp Night.
A: FUN typically happen in a sort of informal place, a nice outdoor gathering in the case of Mexico (the country where FUN started), in the case of Brussels a café or vibrant cultural centre. The format is quite easy to get: 3-4 people go on stage to tell the story of their biggest professional failure in 6:40 minutes and 10 images. After each talk, the audience asks questions.
Q: It seems a bit weird that people go on scene to expose their mistakes. How do you find your speakers? How is their experience afterwards of FUN?
A: Back in 2014 Kira and I were just two weirdos introducing ourselves in networking events with formulas like: “I work at Antiheroes, we organize events where people share their biggest profes-sional FuckUp. Any chance you have a story?” It’s a bit easier now to introduce the subject, I normal-ly target stories I’ve heard of or projects I like that could have a story, I get in touch and meet to break the failure story ice. We’ve had enthusiastic reactions after the FUN stage experience, people feeling “liberated” and enjoying the no filter discussion.
Q: Most events are based on success. What do you think that an event based on failures can bring that one on success cannot?
A: An event disclosing stories of failure triggers lots of attention and curiosity. First, speakers don’t get on stage to promote how brilliant they are but to expose some weakness or a story that often brought them “down to earth”. It rebalances a discourse of overnight success & meritocracy that is very present in Western societies and that leaves behind many shades of what it means to be hu-man. It brings in an authentic and a more transparent conversation on the real path to reach a goal (focus on process rather than outcome) and heals some of the negative emotions that haunt us when scared to “fail”.
JULIA ROHRER ON DISAGREEING WITH YOURSELF
Are researchers allowed to disagree with themselves? Tal Yarkoni, Christopher Chabris, and Julia Rohrer are interested in this phenomenon. Collecting statements from researchers who have lost confidence in their own findings, they founded the Loss-of-Confidence Project (https://psyarxiv.com/exmb2). Let’s hear what Julia can tell us about it.
Q: How did the idea of such a project come about?
A: In 2016, Dana Carney published a document on her website in which she disclosed details about her seminal study on Power Pose and concluded that she no longer believed in the effects. This caused quite a stir and members of the community thought it would be helpful if more scientists wrote such statements. Hence, the Loss-of-Confidence project was born. Multiple people contributed to the initial draft of the project, and one year later, I joined to coordinate the collection of statements.
Q: How was the project received by the scientific community? Was it difficult to convince researchers to share statements?
A: The reactions were overwhelmingly positive. However, we received only a handful of statements; fewer than one would expect given the number of people who stated that they intended to volunteer. There are strong disincentives against admitting errors, and it takes time to write such a statement—time you could use to work on new projects! The discrepancy between the enthusiastic public reaction and the hesitancy to actually submit a statement highlights that we need to have an open debate about how we treat mistakes and about how much priority we assign to self-correction.
Q: How do you think the Loss-of-Confidence project can make a change in scientific culture?
A: It looks like scientists are not quite able to gracefully accept criticism from others, let alone admit errors. That’s just silly: Everybody makes mistakes; sweeping them under the rug is immature. If some researchers read about the project and think “looks like others screwed up, too” or “wow, they admitted an error and their career didn’t suddenly end”—that would already be a great result.
BORIS BARBOUR ON THE CREATION OF PUBPEER
PubPeer is a public platform where anyone can comment, criticize, or question pub-lished scientific literature. The process is straightforward, you simply access www.pubpeer.com, search for a paper, and raise your concerns to the authors. Dr. Boris Barbour is part of the brains behind Pub-Peer. Let’s ask him about the story and current use of PubPeer.
Q: In a few sentences, what is PubPeer, and how did the idea of creating such a system come about?
A: PubPeer is a website with a dynamically created page per scientific publication. This page allows anybody to comment/discuss/question/praise the paper, anonymously if they wish. The founder Brandon Stell was genuinely frustrated that the discussions of journal clubs were never shared publicly nor with the authors.
Q: Scientific articles are already peer-reviewed, so why would we need a system like PubPeer?
A: Experience shows that peer review is far from perfect. People have less and less time to do the reviewing properly, and of course, many journals don't review in depth or at even at all (so-called predatory journals). PubPeer also allows discussion of preprints.
Q: I heard that PubPeer has helped uncover cases of misconduct, but what if it’s an honest mistake? Are we putting the authors at risk by commenting on their article on PubPeer?
A: It is true that questions on PubPeer have led to the discovery of a surprising amount of poor prac-tice and misconduct. However, there are no accusations of misconduct or fraud on PubPeer (see our FAQ). Instead, there are comments/questions about the science or data, to which the authors are encouraged to respond. We believe authors should provide after-sales service for work they chose to make public.
GEERT MOLENBERGHS ON CORRECTING YOUR OWN MISTAKES
Geert Molenberghs is Professor of Biostatistics at UHasselt and KU Leuven. As a fellow of the American Statistical Association and director at L-BioStat, Prof. Molenberghs’ contribution to the field of statistics is widely recognized. But even an excellent researcher can also make mistakes. Prof. Molenberghs had the courage to highlight some of his mistakes by publishing three errata. Considering that he has roughly 550 peer-reviewed papers, adding up to about 750 publications (when considering other categories such as proceedings, book chapters, edited book, and entire books), three errata seems to be not that much. Yet publishing an erratum always requires courage as well as a strong sense of integrity. So let’s hear about his experience.
Q: Tell us a bit about what happens from the moment you notice a mistake until the moment an erratum is issued.
A: When an error in a proof or calculation is discovered, the error is corrected, written up, and submitted as such to the original journal. It is usually published as ‘Correction’ or ‘Erratum.’
Q: Did you feel that these errata had a negative impact on your career/reputation?
A: Not at all! Errors are an inevitable part of being human, and while we should do every-thing to minimize them, they happen, and therefore should be corrected. One learns a lot from mistakes so we should not be led to believe that errors are equal to failure.
Q: What would you say to colleagues who are scared of publicly correcting a mistake?
A: The most important thing is that people know about the error and the correct result. In some cases, it can lead to much more insight and even additional results. Do everything to avoid them, but if they happen, come out immediately and try to learn from them to the max!
Be sure out to check out our social media next week Tuesday to discover what our new TrueTuesday topic will be and don’t forget to check out the article about mental health.