19 ms·
“Extremely angry with the state of academic CS research right now”
- tango12 11y agoMaybe it can be thought of as a tooling problem? Say, a plugin that allows a one-click publish code + data from Matlab, and then it all goes up on a well-indexed page so that others can download/run it.
- simonster 11y agoI doubt the problem is that academic CS researchers don't know how to publish their code, but rather that the disincentives are usually stronger than the incentives.
- cbhl 11y agoIs there even code to publish? I am under the impression a lot of papers from Bugzilla data are of the form "we imported the data into Excel and had a hand-crafted one-off spreadsheet".
- jcrites 11y agoIn that case, yes: the spreadsheet itself consists of data and analysis over data (aggregations over columns and rows, etc.) so the spreadsheet itself would ideally be version controlled and published. The idea isn't to ask researchers to formalize what they make more than before, but to include fully reproducible details in the publication. A spreadsheet is totally fine because you can see how it works, reproduce the result, and tweak the inputs/methods to build on it.
- KKKKkkkk1 11y agoFor a computer scientist, reproducibility means more work that they aren't paid to do. If I ask the Mozilla team to implement new feature X, the response will be either (a) point me to a donation link, or (b) We're open source, so why don't you implement the feature yourself? The computer scientist's response is the same.
- icebraining 11y agoIt's more like Mozilla writing a new version of Firefox that only runs on their own machines.
- jcrites 11y agoReproducibility for the computer scientist means including any code written and data collected or relied on in the scientific publication itself. In practice, getting there from here isn't literally zero work, since some actual human action is needed to bundle the code and data, but that effort ought to be negligible overall, especially if we make it a standard part of the scientific process.
- semi-extrinsic 11y agoTrust me, it's currently far from zero work to submit code with a research paper. I was recently the corresponding author on a software paper sent to a journal that at least verifies the code compiles and runs and produces expected output. Since the poor person testing the softwares submitted is permanently in the ninth circle of dependency hell, across all platforms and libraries imaginable, it took about fifteen emails back and forth plus an OS reinstall before everything checked out. And they said that wasn't anything extraordinary.
- jcrites 11y agoHow about a platform centered around Linux containers (or maybe one of several OS containers or VM images), as the repository image? I'm not saying the work is zero now, but maybe we can get there. If a researcher is developing on a platform where their repository is expressed as a container-like image, then they should be able to publish it for anyone to run exactly as-is. The container repo includes the data, the operating system, and any languages and libraries, with an init system that optionally builds the results.
- semi-extrinsic 11y agoYes, I think we need to go in this direction. The problem is that the container system is yet another tool for researchers to learn. The first step is to get everyone using VCS and nightly testing. Many are still at the point of clumsily written, old Fortran code that gets emailed around and exists in N different variants. (Not that there is anything wrong with Fortran.) Many are at the point where if you email them a link to a git repo to clone, they're clueless about what to do.
- visarga 11y agoOn this topic I'd like to mention the machine learning papers site GitXiv, which is wonderful. It publishes papers alongside the Github repo containing the code. http://gitxiv.com/ http://gitxiv.com/
- jcrites 11y agoThanks! That's awesome. I have long thought that a platform for scientific publication where code and data are included would be valuable, and I'm glad to learn it exists. Going further, imagine the code is like a container that anyone can run to reproduce the findings, including statistical analysis and summary on raw data sets -- the key findings should "build". Perhaps the platform also provides continuous build during "development" (research) so that researchers can work privately and then publish their "repo" publicly along with their paper. An easy way to clone and reproduce the build after publication: "fork my research" As an extreme version of the idea, imagine if the actual paper itself (TeX) and all the data within it are also built as part of the repository; any graphs in the paper are rendered from data in the repo, any numbers are data accesses, etc. This probably wouldn't be helpful to researchers, but it would promote scientific reproducibility and aid everyone building on a researcher's work. Tremendous work goes into authoring the papers themselves, sometimes with methods or tricks that are private; laying it all out publicly would greatly help students of science. Going even further: to avoid cherry picking of positive results, review boards expect experimental criteria to be published (at least privately to them) in advance, for research that involves capital-E experiments. Perhaps this includes analysis code at least in prototype form; like test driven development, the acceptance criteria are written first. When the paper is ready for review, the reviewers can compare the initial prototype analysis logic to the final form. Perhaps the board also expects all data and trials collected during experiments to be made available in the repository, whether positive or not. All collected data should be in the platform, in the most raw form it was originally recorded, as well as all steps of summary and analysis. I wonder if a process and platform like this could contribute to the integrity and quality and reproducibility of scientific research. People funding research ought to ask for it, especially public funded research, and the whole repo is made open eventually if not initially. Perhaps as part of the platform's value prop to researchers (on whom it is imposing probably more burdens than benefit, for sake of public benefit), the hosting is free and funded by a foundation, or steeply discounted. (OK, it won't pay for LHC scale data sets, but otherwise ...) So using it to host your data, code, and paper is free, at least up to a point. I would be interested to contribute time and resources toward building or supporting a platform like this.
- santaclaus 11y ago> They publish LaTeX-templated Word docs as paywalled PDFs. Somewhat tangential, but do CS academics actually write papers in Word? During my grad school days I did not encounter a single paper 'typeset' in Word. Writing was usually done with a LaTeX and and a makefile in a git repo.
- cbhl 11y agoThe IEEE Transactions on Education prefer submissions to be in Word format, which suggests that is how they typeset it.
- semi-extrinsic 11y agoThere are several IEEE Transactions that don't like LaTeX, and even one that uses a submission web form that is "Best viewed in Netscape Navigator". I don't understand why IEEE don't mandate a common, sane submission system.
- neffy 11y agoBecause the nice thing about IEEE standards is that we have so many of them ;)
- mcguire 11y agoI know some faculty members who use Word, and I know some ex-grad students who spent some time converting the format.
- a3_nm 11y agoIt depends on the subcommunity. I have seen LaTeX essentially everywhere in CS, but I recall some exceptions. I'm not sure where, but I think it was applied stuff, probably close to some application domain (e.g., bioinformatics).
- ginger_beer_m 11y agoI work in bioinformatics and I can confirm that most biologists feel very uncomfortable in the idea of having to touch LaTeX. As a result, in our collaboration project, I'd have to survive with using Word+endnote. It's a fairly painful experience..
- moron4hire 11y agoWhat it will take is creating a new system of research and development that ignores the traditional academic system. Because this has been a problem for a while and they clearly are not hearing the message. The reason academic research works is because it takes risk on potential failures, because it's only donated or grant money anyways. But academic institutions fetishize academic papers, which is the problem from the article. We need to legitimize research outside of the academic institution. Pay people to do research on their own time, if they perform it in an open-source, reproducible fashion. Incentivize it based on the reproducibility factor, but avoid attaching a profit motive. Look at what Xerox PARC and Bell Labs were able to do before the penny-pinching bean counters took over. Otherwise, it just sounds like "we want all this risky work done, and we don't want to pay for it."
- dinkumthinkum 11y agoThose academics at Bell labs, you mean the ones that published fundamental papers? I'm not sure that's an example of academics not writing papers.
- moron4hire 11y agoThe end result was not papers. The point was products. The papers were a byproduct. For most academic institutions, the entire point is the paper, whether it's useful to anyone or not.
- PeterisP 11y agoThe useful end result was definitely the papers. The products offer a proof of concept (that it works) and a demonstration (how the details add up). Then in future, others throw away the product (or observe it a bit) and make the next nice things (products for industry adaptations, or further research) mostly based on their papers.
- nycticorax 11y agoMaybe I've drunk the kool-aid, but saying "academic institutions fetishize academic papers" seems wrong to me. It's like saying that developers fetishize working applications. It's not a fetish, it's the whole point of the thing. The output of research is research papers. Yes, sharing raw data and code are both good things, and should be promoted. But no one is going to take the time to look at either unless the paper presents good evidence for some novel result.
- xixi77 11y agoHow is this different from any other academic research? What he is asking about is neither openness nor reproducibility (which are, indeed, very important). He is asking that researchers produce code that he can put into production. Not only they have negative incentives to do so (for one, providing such code will surely result in a stream of all kinds of support requests), it would actually work against the reproducibility objective. The purpose of the code written is usually very simple: to produce results of the paper, not to provide a tool other people can use out of the box. Even when such a tool is nominally provided (for example, when a statistics paper is accompanied by an R package), there are good reasons to be very careful with it: for example, the paper may include assumptions on valid range of inputs, and using the package without actually reading the paper first would lead to absurd results -- which is something that has happened. The way to use academic research results is to (1) read and understand the paper, (2) reproduce the code -- ideally, from scratch, so that his results are (hopefully) unaffected by authors' bugs, (3) verify on a test problem, and (4) apply to his data. Using an out of the box routine skips steps 1-3, which are the whole point of reproducibility.
- ilyanep 11y ago> He is asking that researchers produce code that he can put into production. That's not how I read his comments at all. What he seems to be asking for is the ability to take the code you used to produce the pretty graphs and tables in your paper and re-run it, maybe tweak it himself and use it on a slightly different dataset. He wants to be able to see that your results extend to more than just the toy synthetic dataset you made up, and also be able to verify that some bug in your verification code didn't make the results seem more successful. Finally, he wants to be able to compare apples-to-apples by knowing all the details of your procedure that you didn't bother putting into the paper. (Impersonal you, of course)
- kardos 11y ago> What he seems to be asking for is the ability to take the code you used to produce the pretty graphs and tables in your paper and re-run it, You're assuming such code exists. If the graphs were produced by hand (eg, typing things into MATLAB to create the plot and then saving the figure), then there is no code to hand off. Now the code request has risen to "redo all that work".
- thaw13579 11y agoTwo points to consider here. First, it's not fair to criticize the whole field of academic CS research. Not everyone's work can be accurately represented by a github repo. Second, even when it can, expecting the "Run author's [whatever] against an up-to-date dataset." step to work is asking a quite a lot. Typically there are a infrastructural assumptions baked in (file paths, legacy code dependencies, etc) and manual steps to get to plotted results. In an ideal world with enough resources, every lab would have technical staff to help with this process, but most researchers unfortunately don't have bandwidth to spend on this problem.
- _yosefk 11y agoA lot of researchers do make their results available as code and a whole lot of those who don't actually publish lies, as many people in academia from grad students to professors who implemented research papers and observed that the algorithms don't really work discovered. File paths and manual steps can be worked around, mobody said it should build and run painlessly forever after having been published on any future system with no tweaking and with the researcher indefinitely maintaining the program. If however nothing remotely close to a working state can be published it's not a great sign.
- thaw13579 11y agoYes, many do release code, and it is a great thing! But not everyone's research is represented by code (even within CS). Sometimes research also depends on data that cannot be shared publicly, obscure user interfaces, commercial libraries, includes yet to be published work, etc. In those cases, pushing out without the dependecies may also give the impression of poor research, because it appears to be broken.
- ef4 11y ago> In an ideal world with enough resources, every lab would have technical staff to help with this process, but most researchers unfortunately don't have bandwidth to spend on this problem. And they shoot themselves in the foot constantly by not prioritizing non-crappy software. Then think it's normal that a new student needs months of close support before they can start doing anything interesting. They think it's normal to spend weeks trying to get bitrotted code running again, or just give up and throw it out losing months or years of hard-won incremental improvements. I'm not even talking about supporting reproducibility by outsiders -- they can't even achieve reproducibility within their own labs, because they don't follow practices that are baseline in industry (version control, configuration management, standardized environments, etc).
- modeless 11y agoThe state of machine learning research these days seems pretty good. Essentially all research is published on ArXiv and there is a lot of code released too (though there could certainly be more). I think openness has been a big contributor to the recent explosion in popularity and success of machine learning. When talking to academics about this, machine learning would be a great field to hold up as an example.
- deleted 11y ago[deleted]
- pmalynin 11y agoI'd say the opposite as a member of a group at my university who review ML papers. First off right now there seems to be a drive to explain many phenomena in ML in particular why neural networks are good at what they do. A large body of them reaches a point of basically "they are good at modeling functions that they are good at modeling". The other type of papers that you see, is researchers drinking the group theory kool-aid and trying to explain everything through that. At one point we got 4 papers from 4 different groups that tried to do exactly that. All of them are flawed, either in their mathematics or assumptions (that will most likely never be true, like assumptions of linearity and your data sey being on a manifold). Actually speaking of math, many papers try to use very high level mathematics (functional analysis with homotopy theory) to essentially hide their errors as nobody bothers to verify it.
- archgoon 11y ago> All of them are flawed, either in their mathematics or assumptions (that will most likely never be true, like assumptions of linearity and your data set being on a manifold). Although I can see global linearity being unlikely in most cases; why is local linearity unlikely?
- modeless 11y agoI'm talking about the openness of the papers and code, which is the subject of the article. You are talking about something different.
- rudi-c 11y agoI took a graduate course on software engineering just a month ago and read many of those paper which use Mozilla's data. It's a very popular dataset in the field since it is both open and large. I'm sure Mike Hoye meant to criticize a part of that field, not academic CS research as a whole. My impression of the field was there was a severe mismatch of skillset. The set of people with the scientific background to carry proper experiments, and the funding to do so, is very disjoint from the set of people who understand the field. That made a lot of the papers feel "off". Almost like reading text generated by a machine: individual sentences make perfect sense, the whole doesn't seem to go in a relevant direction. As someone who's done a fair bit of practical software engineering, seeing academics study software engineers feels like seeing a WW2 veteran trying to understand how youngsters use snapchat. It feels very awkward for the youngster, just as it does for the software engineer. Which I imagine is one reason why Mike is pissed off. There is some irony that businesses are much more scientific in this particular subfield than academia, because business incentives require the results to be reproducible and meaningful, over a longer period of time.
- sklogic 11y agoWhat percentage of the CS research based on any data at all? It is mostly purely theoretical. Still, it would be nice if they published their proofs as a Coq code.
- moyix 11y agoI think you're conflating a very narrow slice of CS (theoretical computer science) with the larger field. There's a huge amount of CS research that relies on gathering and analyzing data, building systems, etc. Theoretical computer science is actually a very small slice of the research pie.
- jroesch 11y agoAt least in programming languages, systems, and formal verification project code is both available and often evaluated along side the publication. For example FSCQ(https://github.com/mit-pdos/fscq-impl https://github.com/mit-pdos/fscq-impl) from MIT, Verdi (https://github.com/uwplse/verdi https://github.com/uwplse/verdi) from UW, Dune(http://dune.scs.stanford.edu/ http://dune.scs.stanford.edu/) from Stanford, and many more. These are just the areas I work in, its very hard to making sweeping statement about CS as a full field since it is very diverse sub-community to sub-community.
- sklogic 11y agoMy perception might be skewed, of course. In the broad range of practical areas I worked in I rarely came across any data-driven papers. Mind giving some examples?
- alantrrs 11y agoWe're working on this exact problem at Empirical http://empiricalci.com http://empiricalci.com We're building a way for CS researchers to run, share and compare experiments including their whole research environment. We just applied to Apply HN and would appreciate your feedback: https://news.ycombinator.com/item?id=11586948 https://news.ycombinator.com/item?id=11586948
- askyourmother 11y agoThere are other issues right now, like impartiality and bought research, like Microsoft at Cambridge University. I recall the professors at our uni treating the undergrads with contempt as wasting their precious lucrative research time.
- drjesusphd 11y agoPrecious? Yes. Lucrative? Hardly.
- Ericson2314 11y agoThis is why I believe NixOS is so important. It allows one to completely freeze the entire development environment on one hand, but also does so in a declarative, well abstracted manner (vs VM image let's say) so tweaking/porting is actually feasible. Until well get to a point where building/installing/administrating is not hours of bullshit, research (and free software) will suffer.
- kaeluka 11y agoThe problem here is (EDIT: IMHO) a social one. The question is not "Why does nobody USE NixOS?", it's "Why does nobody WANT NixOS?" And the answer to that is that the incentives are set up such that reproducibility is a waste of time. As a CS researcher, I want to be idealistic. But the field is competitive and I'm not sure how much idealism I can afford. There's considerable effort to bring artifact evaluation into the academic mainstream (I'm actually helping out at OOPSLA this year [1]), and I think this is a good way forward. [1] http://2016.splashcon.org/track/splash-2016-artifacts http://2016.splashcon.org/track/splash-2016-artifacts
- Ericson2314 11y agoI'm not trying to argue the wrong-headed institutional incentives are irrelevant. It is true that once one gets over the learning curve, Nix* gives you artifact reproducability for free, but that still leaves the problem of the learning curve, and caring about reproducability in the first place. Thank you for yours, and others', work making artifact evaluation a priority. Concurrently, people (including myself) are trying to do something about that learning curve. Hopefully both efforts will massively succeed within a decade :).
- kaeluka 11y agoSorry if I've come on too strong. I might've gotten a bit defensive. I've done work in the past that is hard to reproduce and that's not an aspect of my work I'm proud of -- to say the least. Btw, I'm excited about NixOS. It sounds like you're involved. Thank you. I haven't used it, but I'm hoping to find the time soon.
- snissn 11y agoI would extrapolate that to all of academia. If you want to work on something that's useful, insightful and will make the world a better place, the ivory tower of academia is great if you want to live in denial in an echo chamber.
- rudi-c 11y agoThat's a strong claim. While I'm sure the sentiment is understandable for some fields of academia, others have produced significant results that have been adopted by the industry, sometimes very quickly. Machine Learning has a lot of theoretical results produced by academia, but the more practical techniques (decision trees, SVMs, neural networks, etc) also all came from academia. The engineers scaled the algorithms to run on bigger datasets, but the initial work is still driving those systems. Graphics research has seen comparable contributions between industrial research labs and academia. Sure, a higher proportion of papers from academia don't end up practical, but the number of papers that are very practical makes that quite irrelevant. It's to be expected since you can't predict what works in advance, and industrial research labs just don't bother publishing negative results, not that they don't get any. Many programming languages and compiler techniques came from papers. I could go on but you see the point - it depends on the field.
- snissn 11y agoBut that doesn't say anything about the person in academia who writes a speculative generic research paper about a way too simple implementation of a decision tree. I don't want to demean that accomplishment, but for me it would be hard to have sustained excitement from that. Also I think that a counter argument is that just because the initial work of X, Y and Z algorithm were sketched out in academia, they still very much so would have been scaled in industry once the actual need has been found for them.
- alexholehouse 11y agoReproducible is a major issue across science in general, but the difference is there's no reason why one shouldn't be able to easily re-run a defined analysis on a more recently updated data set to ask if conclusions drawn previously still hold. I actually published a side-project paper on this (in biological sciences) last year [1] - what was scary was there was such a lack of discussion surrounding this idea, despite the fact that large databases of biological data are CONSTANTLY changing and updating. [1] Holehouse, A. S. & Naegle, K. M. Reproducible Analysis of Post-Translational Modifications in Proteomes-Application to Human Mutations. PLoS One 10, e0144692 (2015). (http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0144692 http://journals.plos.org/plosone/article?id=10.1371/journal....)
- shanacarp 11y agoworse - most are private
- cderwin 11y agoThe other difference is that as far as I know, computer science is the only discipline for which industry has solved the problem of reproducibility; it's one thing to be asked to design a method to run reproducible studies of humans, it's another to ask researchers to run `git remote add https://github.com/user/repo https://github.com/user/repo && git push --set-upstream`. That's not asking for any support, or other effort on the researcher's part, and I frankly don't understand how the CS academic community doesn't have this as a standard when it'd be so easy to implement.
- jpolitz 11y agoIt is a real issue. The programming languages and software engineering communities have been working on evaluating software artifacts for a few years now: http://www.artifact-eval.org/ http://www.artifact-eval.org/ This doesn't address open access, but it does make sure the software is usable by a non-author to reproduce the results of the paper.
- kaeluka 11y ago@ open access: In my specific subfield of academic CS, a huge success story was ECOOP (one of the biggest conferences in the field) going open access last year. I'm hoping that there'll be other conferences following suit. My money is on open access being commonplace in 10-15 years.
- dgacmu 11y agoHelp me convince people to do a few things a bit better: http://da-data.blogspot.com/2016/04/stealing-googles-coding-practices-for.html http://da-data.blogspot.com/2016/04/stealing-googles-coding-... (tl;dr of my linked blog post: Apply a slightly lighter weight form of some industry engineering practices in CS research coding. I think it's feasible. It doesn't solve all of the problems, because as discussed elsewhere in this thread, some of them are incentive-related and I'm not going to claim to have answers to everything. :) (a) Convince more research groups to do their research on GitHub by default -- ideally, in open repositories. They get good hosted SCM, the world gets a better chance of seeing their code. (b) Create more incentives, like the USENIX Community Award, for research that puts out its code. I'd say that in the systems community, a pretty decent chunk of the papers at SOSP, OSDI, and NSDI have code releases (of varying degrees of usability) accompanying them, though that's not a scientific count. Mozilla could throw $1k to help create community-award-style incentives in the conferences they're interested in. Win-win. You get engaged with the community, you create some incentive for people to do the right thing, and you can use it as an onroad to deeper engagement with the winning authors (i.e., you can try to bring them in for internships. :).
- atishay811 11y agoIts not just the researchers' fault. Maybe the industry should help. Mozilla is a major stakeholder of the web platform that makes distribution easy. Lets make sure web is the best platform to do all the research. * Provide great scientific and matrix manipulation libraries within the browser. WebAssembly isn't going to solve this. Why would the academia rewrite everything? * Provide tools that help research being open. Uploading your code to Github isn't a solution. The real solution is making it easy to use and link other person't research. Can we make research as accessible as a javascript file that you include in your html file to run. And it shouldn't cost the creator to host/maintain it. Offline web sucks(still) and it costs money to host your servers. * Provide incentive to use the web for everything. A great one would be an easy to use and debug toolset and an easy set of methods to get data in and out, an editing environment that can be setup with one click. The closest is iPython Notebook. And it takes work to get there. Sharing should be default and easy. If it isn't we are no one to complain.
- wyattpeak 11y ago>Uploading your code to Github isn't a solution Why not? This sounds like exactly the solution. It's no burden on the researcher - they don't have to alter their research methods to fit a new system, they just dump the code, and leave it up to other users to reimplement. Building web tools to allow research sounds like precisely the wrong solution, at least in the short to medium term. Research funding doesn't go far enough as it is, you're not going to get researchers changing their processes entirely for no gain. What if they need custom hardware, or access to tools and libraries that haven't been implemented? Sharing _is_ default and easy. That's why Github has exploded.
- robotresearcher 11y agoYou forgot to consider IP issues. Schools vary a lot on their policies by but for many the code belongs to the school and can not be open sourced without permission which requires extra work. Funding sources have their own IP deals to consider too. It all sounds so easy until you look at the actual constraints. Professors are usually smart and experienced and they have thought about this stuff a lot. If it was as easy as you thought, it wouldn't be a problem. I publish all code as a matter of lab policy. I chose where to set up my lab partly so that I was able to do this. Not everyone has this luxury or makes this a priority.
- jokoon 11y agoI really wonder, what's the difference between computer science and mathematics? It seems like CS is a subset of math.
- yarper 11y agoIn a word - specificity. The same question arises in a number of older fields of study, for example "is chemistry the same as physics?"
- mcguire 11y agoHere's a guess: the short answer is that the physicist would say 'yes', but would be unable to solve problems that the chemist would consider trivial.
- mcguire 11y agoThere is considerable overlap, but neither is a proper subset of the other. If anything, math is a subset of CS, except for the part that isn't.
- spot 11y agomaking research reproducible with publications that actually contain their code (in addition to prose and graphs, all connected together) is one of the goals of Beaker Notebook, which has a publication server where you can share such results, and then with one click open them up in the cloud, modify them as you see fit, and then run them again (if you have seen Beaker before, this last part, the cloud hosted version, is new as of last week). http://beakernotebook.com http://beakernotebook.com and https://lab.beakernotebook.com/#/publications/featured https://lab.beakernotebook.com/#/publications/featured
- diimdeep 11y agoWow, looks impressive and that it comes from hedge fund is cool too. It's like Jupyter but much more robust, multiple languages in single notebook and ability to share data across languages out of box is really nice.
- tedmiston 11y agoFor an engineering community manager, that was an awfully rant-y way to make the point: "It's frustrating that researchers don't publish their code." Not to say I disagree with the frustration... but it's also not something new. It's been this way for decades. I'd much rather hear about who is doing work in this space and what they're working on. Here are the ones I know of: 1. The Center for Open Science (https://cos.io https://cos.io) is one such org trying to fix this with the Open Science Framework [1]. 2. GitHub also recognizes the need for citable code and gives special discounts for research groups, in fact, Mozilla is one they work with [2]. Two smaller related startups are: 3. Datazar (https://datazar.com https://datazar.com) - A way to freely distribute scientific data. 4. Liquid (https://getliquid.io https://getliquid.io) - A scientific data management platform. Somewhat like "Excel for scientific data as a Service". Also, a related HN thread from some years ago: "We need a GitHub of Science" https://news.ycombinator.com/item?id=2425823 https://news.ycombinator.com/item?id=2425823 --- 1: https://osf.io https://osf.io 2: https://github.com/blog/1840-improving-github-for-science https://github.com/blog/1840-improving-github-for-science
- asimuvPR 11y agoShould also include Gitlab as option to Github. I have no association with them. Just a user of both platforms.
- lgas 11y agoDo they also give discounts to researchers or otherwise show awareness and support of this issue? Might be good to call attention to that fact.
- asimuvPR 11y agoResearcher here: I Use Gitlab because I get everything for no cost as a regular user. With Github I had to pay and get limited features.
- sytse 11y agoGitLab.com and GitLab CE are free to use. GitLab CE is used by many universities. If you use GitLab EE you don't have to pay for student users.
- gravypod 11y agoPersonally, I'd feel code with light commenting would be better suited for getting a "practicable" point across to readers then a paper will. You won't need to put so much weight on describing some of your practices, you can just how how your solution is better.
- tachim 11y agoCodalab is designed to solve exactly this problem: https://worksheets.codalab.org/worksheets/0x818930127c4d47de84c1ceaadf04d014/ https://worksheets.codalab.org/worksheets/0x818930127c4d47de....
- mikeskim 11y agoI would not be surprised if most academic CS research is not reproducible. This is true for many other fields outside of CS, and I've seen it first hand in machine learning. It's a problem but it's also just how things are.
- einarvollset 11y agoAs a recovering CS academic, I have to say - boo-fucking-hoo. Tell me exactly where the funds to do this extra level of support is supposed to come from? Is Mozzila paying for those grad student hours? No? Well then.
- nfd 11y agoWell, tuition costs for even public universities have been rising a lot faster than inflation. Many universities these days are basically hedge funds with a school front. And a lot of universities get state funding. We're pushing a generation of kids without rich uncles to bail them out (myself being one of them; I start university in fall) towards massive, crippling debt. The places they're going aren't hurting for money. Is it that unreasonable to ask for open access?
- argonaut 11y agoState funding has been falling dramatically. The actual researchers are hurting for money. Grad students are paid a pittance, so rather understandably, most grad students want to graduate ASAP. Making your code nice and publishing it does not help you graduate early. It also does not help you get grants or funding. You could pressure universities / departments to make reproducibility a requirement for graduation, but I don't see why they would follow along, because this is not going to help the university get more funding or publish more papers (prestige). Now, if organizations and companies were willing to put money behind software reproducibility (maybe some sort of fellowships or million-dollar research grants to labs), then the incentives would be aligned.
- jcrites 11y agoI would like to make it a simple requirement of receiving any public funding. No one is asking for the code to be polished - I would actually rather it was not modified whatsoever from the state in which the research was conducted. The goal is reproducibility and integrity, so I want to see the actual code that was used, not some polished subsequent version. Every publication involving data and code or analysis should publish them to a degree that makes validation possible in at least as detailed a way as portrayed in "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff" (http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/ http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04dea...)
- intrasight 11y agoFrom reading this discussion, it seems that IP rules are inconsistent, and therefore part of the solution needs to be to perhaps have funding agencies enforce a standard license. With that barrier eliminated, scientists wouldn't expend valuable time negotiating with their school's layers.
- skybrian 11y agoIt seems like politely writing to the researchers and asking if they still have the code lying around might have good results. (If nothing else, it lets them know someone cares.)
- clintonb 11y agoA colleague was working on a replication study. We got the code from the original researcher and another researcher who did a follow-on study. The code barely runs, and the results seem off. I spent days debugging to no avail. Just because researchers provide code does not mean it is well-written code, or that it necessarily works.
- jcrites 11y agoThen I'd be skeptical about those results. Releasing the code allows others to judge the likely accuracy and integrity of the results. A lot of things can go wrong in complex, multi-step computational processes. If care and rigor has not been put into them, then I'd have no reason to believe in the integrity of the output. I want the general public right to judge that, as well as build on it when it's useful and valuable. Every publication involving data and technical analysis should publish them to a degree that makes validation possible in at least as detailed a way as portrayed in "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff" (http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/ http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04dea...)
- chrisbennet 11y agoFunny, as students we were expected to solve the exam problem and "show your work"... I think the root a the problem is that the goals of the researchers are not aligned with the goals of Science. This isn't a criticism of the researchers but instead of the "game" they are forced to play. For example,the goal of Science is to move the ball(knowledge) down the field for the benefit of mankind. We don't reward researchers for doing that, at least not very well. We reward researchers for writing papers full stop - not for making their research easy to reproduce or build on.
- jilljennV 11y agoWhat about this promising recent link? https://gym.openai.com https://gym.openai.com
- msiemens 11y agoI guess I can see why many researchers are not inclined to publish their code. I've worked as a research assistant in the institute of microelectronics of my university for a year and the code quality was somewhere between mediocre and downright terrible. And that's not to mention the absence of sane software engineering standards like bug tracking or code reviews.
- mehrdada 11y agoFund the research you want to see in the world. Incentivize it as you think is right. Don't whine.
- raverbashing 11y ago> But that never happens. Because CS researchers don't publish code or data. They publish LaTeX-templated Word docs as paywalled PDFs. Correct Publishing in pseudocode made sense some 30y. ago Today if you're publishing in pseudocode I personally think it's someone who is doing that because can't write a Hello World in C Oh and also let's stop the "research comes only from universities" idea
- emeryberger 11y agoJean Yang @ CMU (together with Sam Tobin-Hochstadt @ Indiana) posted a response to this on her website: http://jxyzabc.blogspot.com/2016/05/myth-cs-researchers-dont-publish-code.html http://jxyzabc.blogspot.com/2016/05/myth-cs-researchers-dont... Releasing code is a widespread practice in the programming languages and software engineering communities, and one that is getting stronger (see http://artifact-eval.org http://artifact-eval.org). If you are a CS researcher, please fill out this survey of open source and data in computer science: https://docs.google.com/forms/d/1U6I3B2XAqGJDYq101e14PS78jhGl4pTHy6eDx2zx0j4/formResponse https://docs.google.com/forms/d/1U6I3B2XAqGJDYq101e14PS78jhG...