Andrei Tarkovsky’s Stalker (1979)
Charlie Munger said: “Show me the incentive and I will show you the outcome”. The issue with Academic Research, in my opinion, is that the outcomes have drifted very far away from the original goal, which I believe to be the advancement of scientific understanding, and training of the new generation of researchers. Academic research was meant to be about breaking new ground, keeping to honesty and good scientific conduct, being clear and upfront about uncertainties, errors, and mistakes, and improving our common understanding of science, all the while training the new generation to follow these goals and principles.
Recently, I have bumped into multiple cases where I believe the correctness of the results, the honesty of the people writing them, or the lack of curiosity once they are told that their results are wrong, incorrect, faulty, or misleading, has been unsatisfying. Simply put, researchers are not too interested in learning that their papers or reports are wrong, and/or misleading others who don’t happen to know that the results are — known to the authors, and a few select others to be partially — incorrect.
The issue is, once you published the paper, and got the promotions and fame, it doesn’t matter that the results are wrong, and known to be wrong or misleading, and potentially doing harm to the advancement of science. It’s not your problem. It’s someone else’s problem. In fact, when I challenged the authors of one such paper, considered state-of-the-art, and known to the authors to have incorrect evaluation, one of the author’s response was along the lines of acknowledging the issues, but refusing to retract the paper, and instead asking if it’s bothering me in publishing my paper.
The incentives are wrong. In my opinion, it’s not all about the papers — mine or others’. It’s about scientific integrity, about being honest with each other, it’s about caring about the results, it’s about advancement of our common scientific understanding through seeking of truth and correctness. It’s about communicating when something is wildly wrong, and either retracting the relevant incorrect claims, or notifying the community of the known serious issues. It’s about caring for what we all consider to be the common understanding of what is the truth, and cultivating an environment where the new generation grows up to learn what the proper scientific conduct is, and that it is not acceptable to seriously deviate from it.
Unfortunately, I am seeing more and more PhD students who are less and less interested in correctness, precision, and what I’d consider proper scientific conduct. They have learned from those successful in the field what does and does not matter. I remember when someone once told me that my approach for a particular algorithm was wrong (about strongly connected components discovery), and how upset I was that I had no idea. I went home that day and I immediately fixed my tool to use the right approach (i.e. to use Tarjan’s algorithm). I felt deep shame that I had no clue what I was doing, and that I messed up. In contrast, recently talked with a PhD student who wrote a tool that was meant to perform well in certain contexts. When I explained the student that the approach to the evaluation was incorrect, they didn’t follow up at all. I think they were surprised that I later followed up, demonstrating that indeed the evaluation was not careful enough, and the tool is less useful than it seems from the paper’s evaluation. It was a strange experience — when I was a PhD student and someone sat down and showed me that what I was doing was potentially sloppy, I was super worried and very curious to find out what’s going on. I still remember this moment when a reviewer of my dissertation challenged a graph and I was stressed for a week before I figured out they misread the graph, because I didn’t label it clearly. Or when in a presentation I accidentally left out performing so-called ‘restarts’ as a major advancement in the history of SAT solvers, and someone in the audience rightfully pointed it out. I felt embarrassed for having made such a mistake.
I am not sure how to fix this problem. Seemingly, researchers are less and less keen on correctness, precision, and scientific curiosity. They are not incentivised to do so. If they were afraid of having their careers damaged due their singular focus on publishing papers, they’d not behave this way. I sometimes wonder if the system has become so damaged, so many PhD students have grown up to be professors in this environment, that much of what I wrote here seems alien, even repulsive, to many. It makes me sad.
Post navigation
Instagram Engineering
Shopify Engineering