5 ms·

Any open source code released? I am very interested to have a try!

virmundi 9y ago
I doubt it. This is academia. When discussing releasing the code to prove that the researchers did what they said, I was told that I was unreasonable and should just trust people. If that worked, my house would run on cold fusion and my hybrid would fly. Edit: science requires reproducing. Opening the code is literally enabling reproduction. When researchers refuse to do that, they refuse to be scientists. I find this especially irksome when there are public funds used NG the researchers.
- phreeza 9y ago
  Not true for the ML scene, code is very frequently released. cf. http://www.gitxiv.com/ http://www.gitxiv.com/
- slaymaker1907 9y ago
  It can be somewhat tricky though if maybe they want to open source, but the code simply isn't ready yet to be released. The presumption there being that they will eventually open source their code.
  - mijoharas 9y ago
    They could release it under the crapl[0]. [0] http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
- chriswarbo 9y ago
  Thankfully this is starting to change; the UK research councils already require open access to papers as a requirement for funding, and they're currently looking at open access to data (which would include code). It's currently a bit fuzzy what open data requirements would be, so before enforcing the requirement they're trying to pin it down with regard to things like subject/patient/etc. confidentiality, industrial agreements, what to do about massive datasets (e.g. TB+), etc. There is hope!
- mmarx 9y ago
  > science requires reproducing. Opening the code is literally enabling reproduction. When researchers refuse to do that, they refuse to be scientists. Yet refusing to open the code is not refusing to enable reproduction.
- dagw 9y ago
  I'm as annoyed by this as the next person, but to be fair opening the code isn't necessary for reproduction, it just makes it easier. If their paper is properly written there should be enough information there to recreate what they've done. If there isn't that and they refuse to provide that information when asked then you have every right to complain.
  - mattkrause 9y ago
    If… There is a really interesting case study in the Collins Parser. Starting around 1999, Michael Collins published some really exciting results on parsing (interpreting the grammatical structure of nature language). However, people could not replicate them--a "clean room" implementation of Collins' models didn't work nearly as well as the paper claimed it should. Dan Bickel identified a number of apparently trivial implementation decisions that, when taken together accounted for Collins' models improved performance. There's a nice tech report describing this process here: http://repository.upenn.edu/cgi/viewcontent.cgi?article=1026&context=cis_reports http://repository.upenn.edu/cgi/viewcontent.cgi?article=1026...
    - YeGoblynQueenne 9y ago
      Kind of amazing that a single guy put in the effort to rev-eng such a complex implementation. Brilliant paper that, thanks :)
- YeGoblynQueenne 9y ago
  The thing is that anything ever published in a paper represents a claim: the researchers claim that they have achieved something. You may choose to trust the claim to be true, or not- or you may try to reproduce the results if you need to do so for some reason (for instance, because you want to use them in your own work). Until you have convinced yourself that something is true, you can't be expected to believe it just because someone else -anyone else- says it is. This is true even for theoretical papers with mathematical proofs and the like. I don't think this is a very controversial opinion. I've often seen reports in the (mostly popular) press starting with "such and such team claims to have solved such-and-such longstanding problem in a new paper released..." etc. In the popular press we tend to see claims made in papers reported as an absolute fact: "These Danish boffins trained a deep managerial neural pixie network to recognise the sex of starfish" etc. The point that the paper reports the team's own results as the team understands them and that other teams may have a different interpretation of the same results, is often lost in the translation. The best outlets often include a few opinions from researchers not involved in the work- I tend to trust those better.
  - mijoharas 9y ago
    One thing I've noticed more with science journalists (which I greatly approve of) is that they seem as a whole to have taken to heart the "get a quote from a researcher who wasn't involved in the study" mantra, so that it's rare to see an article without a "says x who was not involved in the research" line.
  - deleted 9y ago
    [deleted]

I doubt it. This is academia. When discussing releasing the code to prove that the researchers did what they said, I was told that I was unreasonable and should just trust people. If that worked, my house would run on cold fusion and my hybrid would fly. Edit: science requires reproducing. Opening the code is literally enabling reproduction. When researchers refuse to do that, they refuse to be scientists. I find this especially irksome when there are public funds used NG the researchers.

phreeza 9y ago
Not true for the ML scene, code is very frequently released. cf. http://www.gitxiv.com/ http://www.gitxiv.com/
slaymaker1907 9y ago
It can be somewhat tricky though if maybe they want to open source, but the code simply isn't ready yet to be released. The presumption there being that they will eventually open source their code.
- mijoharas 9y ago
  They could release it under the crapl[0]. [0] http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
chriswarbo 9y ago
Thankfully this is starting to change; the UK research councils already require open access to papers as a requirement for funding, and they're currently looking at open access to data (which would include code). It's currently a bit fuzzy what open data requirements would be, so before enforcing the requirement they're trying to pin it down with regard to things like subject/patient/etc. confidentiality, industrial agreements, what to do about massive datasets (e.g. TB+), etc. There is hope!
mmarx 9y ago
> science requires reproducing. Opening the code is literally enabling reproduction. When researchers refuse to do that, they refuse to be scientists. Yet refusing to open the code is not refusing to enable reproduction.
dagw 9y ago
I'm as annoyed by this as the next person, but to be fair opening the code isn't necessary for reproduction, it just makes it easier. If their paper is properly written there should be enough information there to recreate what they've done. If there isn't that and they refuse to provide that information when asked then you have every right to complain.
- mattkrause 9y ago
  If… There is a really interesting case study in the Collins Parser. Starting around 1999, Michael Collins published some really exciting results on parsing (interpreting the grammatical structure of nature language). However, people could not replicate them--a "clean room" implementation of Collins' models didn't work nearly as well as the paper claimed it should. Dan Bickel identified a number of apparently trivial implementation decisions that, when taken together accounted for Collins' models improved performance. There's a nice tech report describing this process here: http://repository.upenn.edu/cgi/viewcontent.cgi?article=1026&context=cis_reports http://repository.upenn.edu/cgi/viewcontent.cgi?article=1026...
  - YeGoblynQueenne 9y ago
    Kind of amazing that a single guy put in the effort to rev-eng such a complex implementation. Brilliant paper that, thanks :)
YeGoblynQueenne 9y ago
The thing is that anything ever published in a paper represents a claim: the researchers claim that they have achieved something. You may choose to trust the claim to be true, or not- or you may try to reproduce the results if you need to do so for some reason (for instance, because you want to use them in your own work). Until you have convinced yourself that something is true, you can't be expected to believe it just because someone else -anyone else- says it is. This is true even for theoretical papers with mathematical proofs and the like. I don't think this is a very controversial opinion. I've often seen reports in the (mostly popular) press starting with "such and such team claims to have solved such-and-such longstanding problem in a new paper released..." etc. In the popular press we tend to see claims made in papers reported as an absolute fact: "These Danish boffins trained a deep managerial neural pixie network to recognise the sex of starfish" etc. The point that the paper reports the team's own results as the team understands them and that other teams may have a different interpretation of the same results, is often lost in the translation. The best outlets often include a few opinions from researchers not involved in the work- I tend to trust those better.
- mijoharas 9y ago
  One thing I've noticed more with science journalists (which I greatly approve of) is that they seem as a whole to have taken to heart the "get a quote from a researcher who wasn't involved in the study" mantra, so that it's rare to see an article without a "says x who was not involved in the research" line.
- deleted 9y ago
  [deleted]