Reproducing papers
It turns out that a good fraction of results documented in peer-reviewed scientific papers cannot be reproduced.
This comes as a huge surprise to everyone except the scientists, grad students and post-docs who actually performed these experiments and wrote them up.
This is not failure of ethics. Scientists are not trying to publish fabricated results. It is a failure of process and documentation. If you teleported into the lab while the experiment was being done, you would actually see the same results that were written up.
But then going to another lab and trying to set it up just so, and get the exact same result? Good luck with that.
Ask any grad student and they’ll tell you just how delicate these setups are. They work for months just to get a stable setup, and then when it works, they try not to breathe too hard on it and pump it for as much data as possible and publish a paper. They aren’t going for reproducability, just producability.
Why?
Because the paper is the ultimate goal of all this. Grad students need them to graduate with a PhD. Professors need them for tenure and funding. The department needs them for prestige.
Like I’ve argued before, the paper as a unit of dessimination of modern scientific results is outmoded.
A paper doesn’t even contain all the information and data required to reproduce a result. Because if it did, it would be the size of a book. It usually has just the high-level points and a summary of the data.
Instead, what if the paper was replaced by a publicly-visible activity log? A long series of small steps, each one small enough to understand and debug and… reproduce.