4 ms·

> This is absolutely par for the course, open source means no warranty. This isn't about open source it's about scientific research. It 1) should be scrutiniz

> This is absolutely par for the course, open source means no warranty. This isn't about open source it's about scientific research. It 1) should be scrutinized and 2) if it's not reproducible, called out as such. The whole point of presenting research is about advancing the state of the art. If someone else can't reproduce your results, then it's not science. It's just marketing.

zozbot234 4y ago
There's a whole lot of hard-to-reproduce research out there. Why is these women's work - in EDA of all things, which has been far from friendly to open source historically - being singled out for such strong and apparently unjustified criticism? It's just not fair.
- red-iron-pine 4y ago
  Because re-doing huge studies often isn't feasible or cost-effective for a bunch of reasons. 10+ year studies across a few hundred thousand cases isn't going to happen for many studies. But this is recreating an ML environment and seeing if it generates a similar-ish outcome. Making someone's code work in a similar env is critical, and the article implies that there were a lot of "released at last minute" and "relearned every time" hurdles that implied that the ladies' work might have been dubious.
- gwd 4y ago
  To echo this: I did my PhD, which involved adding a new feature to another software project, verifying that it worked, and presenting benchmarks. But for Reasons, the code ended up being based on a development branch between v2 and v3 of the upstream project. (I used to try to rebase every week, but I ended up spending too much of my time rebasing, and eventually gave up.) After going into industry, every so often someone would email me asking for my code. I'd mail them a link to a tarball of the Mercurial tree at the point I ran my benchmarks, offering to help get it working. I never heard from anybody a second time -- and nobody ever criticized my work for being un-reproducible. (In fact, eventually I received an ACM SigOps Hall of Fame Award for the work.) (And since compilers get more paranoid every year, and the snapshot was taken in 2006, the chance of anyone getting it working continues to approach zero.)
  - UncleEntity 4y ago
    > After going into industry, every so often someone would email me asking for my code… I never heard from anybody a second time… I’m wondering if they wanted the code to clarify what you were doing or to try to port it to the current codebase? I look at code all the time for stuff I’m interested in without any plans to use and/or to see if there’s anything I can steal for whatever I’m currently poking at. Code I can read. Mathematical algorithmic notation of code, not so much.

zozbot234 4y ago

There's a whole lot of hard-to-reproduce research out there. Why is these women's work - in EDA of all things, which has been far from friendly to open source historically - being singled out for such strong and apparently unjustified criticism? It's just not fair.

red-iron-pine 4y ago
Because re-doing huge studies often isn't feasible or cost-effective for a bunch of reasons. 10+ year studies across a few hundred thousand cases isn't going to happen for many studies. But this is recreating an ML environment and seeing if it generates a similar-ish outcome. Making someone's code work in a similar env is critical, and the article implies that there were a lot of "released at last minute" and "relearned every time" hurdles that implied that the ladies' work might have been dubious.
gwd 4y ago
To echo this: I did my PhD, which involved adding a new feature to another software project, verifying that it worked, and presenting benchmarks. But for Reasons, the code ended up being based on a development branch between v2 and v3 of the upstream project. (I used to try to rebase every week, but I ended up spending too much of my time rebasing, and eventually gave up.) After going into industry, every so often someone would email me asking for my code. I'd mail them a link to a tarball of the Mercurial tree at the point I ran my benchmarks, offering to help get it working. I never heard from anybody a second time -- and nobody ever criticized my work for being un-reproducible. (In fact, eventually I received an ACM SigOps Hall of Fame Award for the work.) (And since compilers get more paranoid every year, and the snapshot was taken in 2006, the chance of anyone getting it working continues to approach zero.)
- UncleEntity 4y ago
  > After going into industry, every so often someone would email me asking for my code… I never heard from anybody a second time… I’m wondering if they wanted the code to clarify what you were doing or to try to port it to the current codebase? I look at code all the time for stuff I’m interested in without any plans to use and/or to see if there’s anything I can steal for whatever I’m currently poking at. Code I can read. Mathematical algorithmic notation of code, not so much.