What's described here is roughly true of any deeply quantitative scientific research that depends on optimization and statistics, although for a number of reaso

What's described here is roughly true of any deeply quantitative scientific research that depends on optimization and statistics, although for a number of reasons, it's far worse in ML than other fields. I worked on a lot of "reproductions" in my career: given a paper, implement the paper and get the same results as the authors. As I got better, I learned techniques on how to be more effective, for example often running optimization many times and seeing how sensitive the optimization was to random seeds. Often times I'm at the point of inspecting individual steps in the data load/augmentation pipeline (buggy 90 degree image rotations are very common, as are swapped axis rectangle crops) None of this should be surprising when we recognize that the people producing these papers are incentivized to move as quickly as possible, not produce stable systems for people with less ML skill. Any time a paper includes the exact source code, training data, and the shortcut to get training working, it saves me roughly 2-3 weeks at least. And don't get me started on reproduction attempts that ultimately lead to papers being retracted. It's painfully common to fail to reproduce the author's work, then get access to their source, and find basic bugs that question the results.