6 ms·
I have argued in the past that data and code need to be submitted with academic papers: http://blog.jgc.org/2013/04/the-importance-of-open-code.html http://blog
by jgrahamc 14y ago
I have argued in the past that data and code need to be submitted with academic papers: http://blog.jgc.org/2013/04/the-importance-of-open-code.html http://blog.jgc.org/2013/04/the-importance-of-open-code.html This is just the latest example of why, IMHO, code needs to be open.
- carbocation 14y agoMore specifically for people not following jgrahamc's full story, he and co-authors wrote a paper making the argument that data and code need to be submitted with academic papers, and the pre-eminent scientific journal, Nature, saw fit to publish their paper. So the idea is getting attention; hopefully it will also gain traction, too.
- jgrahamc 14y agoInformation about the Nature paper: http://blog.jgc.org/2012/02/case-for-open-computer-programs.html http://blog.jgc.org/2012/02/case-for-open-computer-programs.... My co-authors and I chose Nature as the target for the paper because of its reputation and because I believe in going to 11 (http://en.wikipedia.org/wiki/Up_to_eleven http://en.wikipedia.org/wiki/Up_to_eleven).
- latj 14y agoI didnt realize this was something that was still up for debate. In my experience source code has been a requirement since I entered the game- especially for high impact journals.
- carbocation 14y agoAs late as at least 2010, source code wasn't required for Nature or NEJM. People could request it from you, but it wasn't required to actually be submitted up front.
- jurassic 14y agoYes! We need to see more material under Matt Might's CRAPL license. I feel a strong urge to write off any result that doesn't have accompanying code with it. http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
- anigbrowl 14y agoGreat idea, awful awful name - CRAP license, as in 'Community Research/Academic Programming' = CRAP. Why would your make your concept the butt of jokes from the outset?
- michael_miller 14y agoAcronyms don't have to be positive to be successful. CRUD is an example of this.
- abstractbill 14y agoCRUD is often used as a pejorative though.
- rflrob 14y agoIt's right there on the page: "It's not the kind of code one is proud of." By releasing it as CRAP, you're acknowledging that the code is imperfect, reducing the perceived barrier. If academics feel like you can only release code if it's pristine, it'll never get released, because for the most part, the incentives aren't there to make clean code in an academic setting.
- anigbrowl 14y agoThis is to badly miss the point. What matters in this context is accuracy, not elegance. If I brute-force detection of the first 10000 prime numbers (for some non-CS context) by testing their divisibility from n to 1 it's horribly inefficient and programmers may laugh, but it's accurate, simple, and easily reproducible. Given that a definition of a prime number as divisible only by itself and 1, it may be better to present a brute-force algorithm than to get into a sideline discussion about validating some new-fangled method like the Sieve of Erasthostenes, that upstart. Science aims at proof rather than efficiency. What matters is the quality of the result, and its reproducibility. If you can get to the same result in a much more timely fashion that's awesome, but that's an engineering achievement rather than a scientific one. I don't think that anyone wants to put out code labeled as CRAP in an attempt to mollify engineers.
- deleted 14y ago[deleted]
- twog 14y agoI completely agree. This is the exact problem that we are trying to solve at my current startup, http://Banyan.co http://Banyan.co
- jgrahamc 14y agoI wish you luck, but I'm not sure a startup is the answer. What's needed is education of scientists and to gain their acceptance. Then a startup will have a market.
- justhw 14y agoYou've got a beautiful site, but please fix your SSL installation. Firefox did not let me see it without adding it as an exception.
- twog 14y agoThanks for the heads up. We rolled out a deploy that looks like it broke a few things this morning. Im on it right now.
- jacquesm 14y agohttp://jacquesmattheij.com/Hey+CS+Paper+writers+please+show+your+code http://jacquesmattheij.com/Hey+CS+Paper+writers+please+show+... So did I... however, showing code and data should be mandatory for acceptance of papers. Errors are all too easy to miss and since papers draw conclusions from the data a paper without data should actually be simply refused for being incomplete. Why publications would accept a paper for review and eventual publication without giving reviewers the ability to actually check the results is a mystery to me. The current system basically amounts to 'you're going to have to believe me'. I singled out CS papers because there the code is pretty much the essence of the paper but of course the same goes for every other field.
- jd 14y agoMost importantly -- when researchers know their data and methodology will be out in the open they'll have a big incentive to make it look clean and presentable. They know they risk getting called out on excluding certain segments of the data set, so they'll have to at the very least add a small remark in the spreadsheet justifying their decision. It also makes it really easy for other researchers to pick other start and end years to test if the result only holds for the original data. Which again, encourages researchers to explain justify the input data they've chosen. In addition, researchers are likely to discover mistakes while cleaning up the excel sheet, data sources and code before publishing it. Just like we find mistakes in our work while refactoring and cleaning up code before we push it to github. So even when nobody ever looks at the data and code we can expect the quality of the research to improve significantly: just because the code is there to be looked at. I think the case in favor of making data & code public for academic research is pretty overwhelming.
- lmg643 14y agoI think having data open and available is good for everyone. The more I learn about this particular incident, the more I feel the scandal is not (a) that famous harvard professors made a mistake, or (b) that microsoft excel is more error prone than alternatives (which seems like nonsense to me because the more complex the replacement the more error prone it will be)... To me the scandal is that you can be a tenured professor in economics and produce work that amounts to simple averages of widely available data and call it a research paper, and that people take you seriously, and presidential candidates use you as a reference, and your department doesn't bat an eyelash. The fact that they screwed up seems incidental - mistakes happen. It seems like the kind of back-of-the envelope work that any old blogger would be capable of doing, we just don't have a way to take good ideas, no matter where they come from, seriously. no matter how much we profess to try - at heart, the consensus is still status-driven, and pedigrees matter.
- vacri 14y agoI think having data open and available is good for everyone. Not necessarily everyone. Collecting good data is hard, long, and tedious, and the 'glory' part is the analysis. People get accolades for making analyses, not for good groundwork.
- curveship 14y agoFor anyone interested, Herndon, Ash and Polin did indeed release their data and code. It can be found here: http://www.peri.umass.edu/fileadmin/pdf/working_papers/working_papers_301-350/HAP-RR-GITD-code.zip http://www.peri.umass.edu/fileadmin/pdf/working_papers/worki... . Kudos to them.
- pseut 14y agoThat is the AER's usual policy [1]; R&R's paper was published in the May issue (Papers and proceedings) which is for all intents and purposes a completely different and almost non-peer-reviewed journal [2]. I suspect (without any knowledge) that there will some pressure to distinguish the AER and the papers and proceedings more after this episode. [1] http://www.aeaweb.org/aer/data.php http://www.aeaweb.org/aer/data.php [2] http://www.aeaweb.org/aer/pandp_faq.php http://www.aeaweb.org/aer/pandp_faq.php
- lanaius 14y agoThe argument against this, and I make this as a jaded and struggling graduate student so accept that bias, is that there is no acceptable outlet for negative results. There is also no acceptable outlet for those who consistently produce negative results and so, as I'm sure everyone understands, there is tremendous pressure to produce positive and interesting results. We hear about the times that this is forced due to manipulation and falsification and this is never okay, but there are a great many times where it falls into a nebulous gray area. We double and triple check our code, verify simulation results, and run down the checklists but that never guarantees that we do not have such "excel column errors". If data and code are open this is fantastic for science and progress because errors do not replicate and permute but it can be terrible for individual scientists and graduate students. As always, it comes down to money. If we are forced to correct or even worse retract a scientific publication, the community (both local and worldwide) bares down on you like a knife. The stakes are so high that not only is falsification lethal, in many cases honest mistakes can be lethal as well. Conversely one cannot take eternity flipping every stone to bulletproof every possible single problem and it can be very hard to identify which stones to check! This is not a defense of closed work and closed data, but it's a realization that opening data and code is not a simple and straightforward process. There are severe and deeply embedded cultural problems that make post-publication sharing difficult.
- pseut 14y agoI don't know your field, but the claim that there's no acceptable outlet for negative results isn't always true. There are usually outlets for interesting results, it's just that negative results tend to be less interesting than positive ones. Negative results that blow up accepted wisdom, though... An example from economics: Meese and Rogoff [1] has (by google scholar) about 3000 citations, which is massive for econ and is one of the key references in Intl Macro. They showed that all of the models in use at the time of publication sucked. [1] Empirical exchange rate models of the seventies: do they fit out of sample? links to the paper here: http://scholar.google.com/scholar?cluster=17091735262456791337&hl=en&as_sdt=0,16 http://scholar.google.com/scholar?cluster=170917352624567913...
- danielweber 14y agoOne problem is that often the data is highly valued -- the team may have spent years building it. And, in academic circles, prestige comes from papers, not data sets. So the team wants to mine papers out of that data set for years.
- donaldc 14y agoOne problem is that often the data is highly valued -- the team may have spent years building it. That was clearly not the case with the flawed paper this OP is concerning itself with. So, not a valid excuse in this case. At any rate, the team should learn to write papers as they build up the data set. That way they always have a little more data than anyone else working with it. Because without release of the data on which a given paper is based, there's no way to know if the paper is actually valid.
- danielweber 14y agoA good data set can have incorrect analysis written about it. "Clearly not the case" doesn't apply.