3 ms·
> "We open-sourced our code" but no one can get it to work This is absolutely par for the course, open source means no warranty. It looks like these researche
by zozbot234 4y ago
> "We open-sourced our code" but no one can get it to work
This is absolutely par for the course, open source means no warranty. It looks like these researchers are being subjected to unwanted scrutiny simply because they're women, which is quite unfair and very much not OK.
- dogleash 4y ago> This is absolutely par for the course, open source means no warranty. This isn't about open source it's about scientific research. It 1) should be scrutinized and 2) if it's not reproducible, called out as such. The whole point of presenting research is about advancing the state of the art. If someone else can't reproduce your results, then it's not science. It's just marketing.
- zozbot234 4y agoThere's a whole lot of hard-to-reproduce research out there. Why is these women's work - in EDA of all things, which has been far from friendly to open source historically - being singled out for such strong and apparently unjustified criticism? It's just not fair.
- red-iron-pine 4y agoBecause re-doing huge studies often isn't feasible or cost-effective for a bunch of reasons. 10+ year studies across a few hundred thousand cases isn't going to happen for many studies. But this is recreating an ML environment and seeing if it generates a similar-ish outcome. Making someone's code work in a similar env is critical, and the article implies that there were a lot of "released at last minute" and "relearned every time" hurdles that implied that the ladies' work might have been dubious.
- gwd 4y agoTo echo this: I did my PhD, which involved adding a new feature to another software project, verifying that it worked, and presenting benchmarks. But for Reasons, the code ended up being based on a development branch between v2 and v3 of the upstream project. (I used to try to rebase every week, but I ended up spending too much of my time rebasing, and eventually gave up.) After going into industry, every so often someone would email me asking for my code. I'd mail them a link to a tarball of the Mercurial tree at the point I ran my benchmarks, offering to help get it working. I never heard from anybody a second time -- and nobody ever criticized my work for being un-reproducible. (In fact, eventually I received an ACM SigOps Hall of Fame Award for the work.) (And since compilers get more paranoid every year, and the snapshot was taken in 2006, the chance of anyone getting it working continues to approach zero.)
- UncleEntity 4y ago> After going into industry, every so often someone would email me asking for my code… I never heard from anybody a second time… I’m wondering if they wanted the code to clarify what you were doing or to try to port it to the current codebase? I look at code all the time for stuff I’m interested in without any plans to use and/or to see if there’s anything I can steal for whatever I’m currently poking at. Code I can read. Mathematical algorithmic notation of code, not so much.
- yjftsjthsd-h 4y ago> This is absolutely par for the course, open source means no warranty. No warranty, sure, but if the original authors have a working program and what they publish doesn't work then did they really meaningfully open source it? Like, if you ask an Android OEM for their kernel source and they send you a tarball that when compiled can't actually boot their phone, is that above board? (None of this is specific to the particular authors here, BTW; I will happily criticize literally anyone claiming to do Science or Open Source without publishing enough that others can actually reproduce their work.)
- mandevil 4y agoThe team I work for recently open-sourced a version of our SaaS code. Thanks to docker, we are pretty sure that it will work for you if you follow the regular install instructions- containerization is awesome for that. The instructions on how to set up a local dev environment, without docker? That's a lot harder, has a ton more moving pieces, and if we don't actively maintain and test it regularly (to catch when libraries rot out from underneath us) I suspect that in 6-12 months those instructions will not work for anyone, because this is an incredibly complex beast doing machine-learning related things with a whole lot of dependencies and moving pieces (DB, API, Web interface, caching layer, etc.) so there are a lot of potential failure points. This was the result of a team of 3-4 SW engineers and a UX engineer working on open-sourcing our existing, working, generating actual revenue SaaS code base for 5-6 months (not as our only task but as a very important one). So it was, say, somewhere around 2 man years of effort to take something that I know exists and works and turn it into a reasonable Open Source project. There is a huge difference in the amount of work required between simple "open sourcing" and actual effective Open Sourcing, and I suspect that few if anyone involved in "Open Science" is doing that work absent large funding sources specifically requiring and supporting with money the good kind. Just a simple fork of the repo? I think the result for us would have been the same as what happened here, and I know that our SaaS project works- I can look at my paycheck and see that.
- deleted 4y ago[deleted]