> people claiming that their non-software engineering grade code invalidates the results of their study. But that's exactly the problem. Are you familiar with

> people claiming that their non-software engineering grade code invalidates the results of their study. But that's exactly the problem. Are you familiar with that bug in early Civ games where an overflow was making Ghandi nuke the crap out of everyone? What if your code has a similar issue? What if you have a random value right smack in the middle of your calculations and you just happened to be lucky when you run your code? I'm not that familiar with Monte Carlo, my understanding is that this is just a way to sample the data. And I won't be testing your data sampling, but I will expect that given the same data to your calculations part (eg, after the sampling happens), I get exactly the same results every time I run the code and on any computer. And if there are differences I expect you to be able to explain why they don't matter, which will show you were aware of the differences in the first place and you were not just lucky. And then there is the matter of magic values that plaster research code. Researchers should understand that the rules for "software engineering grade code" are not there just because we want to complicate things, but because we want to make sure the code is correct and does what we expect it to do. /edit: The real problem is not getting good results with faulty code, is ignoring good solutions because faulty code.