4 ms·

> "If you are publishing articles with no source code, how is that even science?" - We have a methods section in the paper, just like in the science fair. yes

> "If you are publishing articles with no source code, how is that even science?" - We have a methods section in the paper, just like in the science fair. yes but the pipeline goes METHODS -> (CODE) -> RESULTS. you can read the methods and they check out. fine. you can't see the code. the results show something new and amazing. how do you know this is because of the methods used or because somebody flipped a minus sign in the code? well, you don't. unless you reimplement the code from the methods and see if it's reproducible. except that nobody does this. but let's say they did and found completely different results, that are also new and amazing. who is right? the answer is: NOBODY KNOWS, because nobody is publishing their code! so astrophysicist number 3 comes around, wants to know who is right, and has only these methods to go on, but no code. he has to start from scratch. because the universe hates us, he will find a third set of completely different new and amazing results that aren't even of the same datatype as the first two--but nobody knows who is right, because everyone is hiding their code and claiming it's based on the same methodology. > doing the same thing twice is pointless. No that's science. What is pointless is doing the same thing twice, in the hope you're doing the same thing as the last guy, because you don't know what the last guy did. Also, hoping that if you make errors (like the last guy undoubtedly did, because hey this is unreviewed closed source code we're talking about), that they'll be different errors. Except you can't know, because nobody publishes their code. All of this reminds me a little bit of the parable of the Stone Soup. And if the above behaviour is causing people stress and grey hairs about their job security, well I don't wish that upon anybody, but hoarding source code isn't the way forward either.

new299 14y ago
> well, you don't. unless you reimplement the code from the methods and see if it's reproducible. except that nobody does this. but let's say they did and found completely different results, that are also new and amazing. who is right? the answer is: NOBODY KNOWS, because nobody is publishing their code! No one is going to spend time auditing the code to find a mistake either. In academia the way these things are, and should be, validated is by comparing the output to known results. In simulation this means do my simulations match experimental reality. In data analysis this means you check that you can verify something that's known to be true about a particular dataset. That kind of validation in itself doesn't happen very often (it's often missing for example in computational chemistry). But in all honesty having the code, wouldn't (and doesn't) help. I'm not arguing that the code shouldn't be open source, but that if you're presenting "new and amazing" results the way you back those up is not by saying you checked your code really well, but by showing that your method and implementation are consistent with known facts while presenting something new.
johnminter 14y ago
> you can read the methods and they check out. I'm a materials scientist specializing in microscopy and image analysis, not an astrophysicist. In my area of interest, there is rarely sufficient information published to confidently repeat results. Indeed, the current reproducible research community had origins in computer vision (and was then embraced by the biostatistics community). I applaud the expectation of openness - especially if funded by government grants. Too often, we taxpayers pay for research that get trapped behind paywalls. I agree the the academic astrophysics community has it tough. This is symptomatic of the entire academic community that is producing too many Ph.D.s desiring too few academic positions. Guess what - it will only get worse when the unsustainable government spending and debt causes the education bubble to burst.

new299 14y ago

> well, you don't. unless you reimplement the code from the methods and see if it's reproducible. except that nobody does this. but let's say they did and found completely different results, that are also new and amazing. who is right? the answer is: NOBODY KNOWS, because nobody is publishing their code! No one is going to spend time auditing the code to find a mistake either. In academia the way these things are, and should be, validated is by comparing the output to known results. In simulation this means do my simulations match experimental reality. In data analysis this means you check that you can verify something that's known to be true about a particular dataset. That kind of validation in itself doesn't happen very often (it's often missing for example in computational chemistry). But in all honesty having the code, wouldn't (and doesn't) help. I'm not arguing that the code shouldn't be open source, but that if you're presenting "new and amazing" results the way you back those up is not by saying you checked your code really well, but by showing that your method and implementation are consistent with known facts while presenting something new.

johnminter 14y ago

> you can read the methods and they check out. I'm a materials scientist specializing in microscopy and image analysis, not an astrophysicist. In my area of interest, there is rarely sufficient information published to confidently repeat results. Indeed, the current reproducible research community had origins in computer vision (and was then embraced by the biostatistics community). I applaud the expectation of openness - especially if funded by government grants. Too often, we taxpayers pay for research that get trapped behind paywalls. I agree the the academic astrophysics community has it tough. This is symptomatic of the entire academic community that is producing too many Ph.D.s desiring too few academic positions. Guess what - it will only get worse when the unsustainable government spending and debt causes the education bubble to burst.