> I'm about to tell you something which can sometimes be harder to believe than conspiracy theories about academia: you've got a bug in your code. If this is t

> I'm about to tell you something which can sometimes be harder to believe than conspiracy theories about academia: you've got a bug in your code. If this is the case in a majority of the instances where someone fails to reproduce the software backing a paper then there may be another issue at play. Someone re-implementing a paper is "just" taking a written description of something and transcribing it to code. If a mistake, that can completely ruin the results of the work, can be made this easily it should be fairly straight forward to see that the original implementer could also have made a mistake that could have thrown off their results. Does the world of academia have any tools to prevent this? It seems like this could effect a lot of the research being done today. Given the following: 1. Most research being done today utilizes some software to generate their results. 2. This software often encodes some novel way of implementing some analysis method. 3. The published paper from research with a novel method of analysis will need to describe their method of analysis. 4. Future researchers will find this paper, attempt to implement the described analysis method, and publish new results from this implementation. We can see we are wasting a lot of resources reimplementing already written code. We may also not be implementing this code correctly and may be skewing results in different ways. > Debugging research code is extremely difficult, and requires a different mind set - the attention to detail you need to adopt will be beyond anything you've done before. With research code, and numerical or data-driven code in particular, bugs will not manifest themselves in crashes. Often bugged code will not only run, but produce some kind of broken result. It is up to you to validate that every line of code you write is correct. No one is going to do that for you. And yes, sometimes that means doing things the hard way: examining data by hand, and staring at lines of code one by one until you spot what is wrong. This is a very common mindset that many academics have but I don't understand why this is the case. A paper seems like a fantastic opportunity to define an interface boundary. If your paper describes a new method to take A and transform it into B then it should be possible for you to write and publish your `a_to_b` method aside your paper. You could even write unit tests on your `a_to_b`. If a future researcher comes along and finds a way to apply your `a_to_b` to more things they could modify your code and, just by rerunning your tests, should be able to verify their implementation actually works. If a future researcher decided to use you `a_to_b` you could write some code to automatically generate a list of papers to reference. If we are seriously spending this much time treading the same water then it should be possible to dramatically improve the quality and throughput of academics by providing some tool like this to them. I know someone will say "but you gain so much knowledge re-implementing XYZ" and to them I'd say that you don't need to read the code while reading the paper. You could write the code yourself and just utilize the unit tests provided by the author to make sure you fully understand each edge case.