> well, you don't. unless you reimplement the code from the methods and see if it's reproducible. except that nobody does this. but let's say they did and found

> well, you don't. unless you reimplement the code from the methods and see if it's reproducible. except that nobody does this. but let's say they did and found completely different results, that are also new and amazing. who is right? the answer is: NOBODY KNOWS, because nobody is publishing their code! No one is going to spend time auditing the code to find a mistake either. In academia the way these things are, and should be, validated is by comparing the output to known results. In simulation this means do my simulations match experimental reality. In data analysis this means you check that you can verify something that's known to be true about a particular dataset. That kind of validation in itself doesn't happen very often (it's often missing for example in computational chemistry). But in all honesty having the code, wouldn't (and doesn't) help. I'm not arguing that the code shouldn't be open source, but that if you're presenting "new and amazing" results the way you back those up is not by saying you checked your code really well, but by showing that your method and implementation are consistent with known facts while presenting something new.