4 ms·

Two points to consider here. First, it's not fair to criticize the whole field of academic CS research. Not everyone's work can be accurately represented by a

Two points to consider here. First, it's not fair to criticize the whole field of academic CS research. Not everyone's work can be accurately represented by a github repo. Second, even when it can, expecting the "Run author's [whatever] against an up-to-date dataset." step to work is asking a quite a lot. Typically there are a infrastructural assumptions baked in (file paths, legacy code dependencies, etc) and manual steps to get to plotted results. In an ideal world with enough resources, every lab would have technical staff to help with this process, but most researchers unfortunately don't have bandwidth to spend on this problem.

_yosefk 10y ago
A lot of researchers do make their results available as code and a whole lot of those who don't actually publish lies, as many people in academia from grad students to professors who implemented research papers and observed that the algorithms don't really work discovered. File paths and manual steps can be worked around, mobody said it should build and run painlessly forever after having been published on any future system with no tweaking and with the researcher indefinitely maintaining the program. If however nothing remotely close to a working state can be published it's not a great sign.
- thaw13579 10y ago
  Yes, many do release code, and it is a great thing! But not everyone's research is represented by code (even within CS). Sometimes research also depends on data that cannot be shared publicly, obscure user interfaces, commercial libraries, includes yet to be published work, etc. In those cases, pushing out without the dependecies may also give the impression of poor research, because it appears to be broken.
ef4 10y ago
> In an ideal world with enough resources, every lab would have technical staff to help with this process, but most researchers unfortunately don't have bandwidth to spend on this problem. And they shoot themselves in the foot constantly by not prioritizing non-crappy software. Then think it's normal that a new student needs months of close support before they can start doing anything interesting. They think it's normal to spend weeks trying to get bitrotted code running again, or just give up and throw it out losing months or years of hard-won incremental improvements. I'm not even talking about supporting reproducibility by outsiders -- they can't even achieve reproducibility within their own labs, because they don't follow practices that are baseline in industry (version control, configuration management, standardized environments, etc).
- thaw13579 10y ago
  Indeed, this is all too common and such a waste. I imagine it would help to have a manual of "modern" best practices for research--maybe this exists? A lot of non-CS researchers could benefit as well.
  - drjesusphd 10y ago
    Yes, I would like to see such a thing also.
- majewsky 10y ago
  > I'm not even talking about supporting reproducibility by outsiders -- they can't even achieve reproducibility within their own labs, because they don't follow practices that are baseline in industry (version control, configuration management, standardized environments, etc). True that. Four years ago, when I was writing my thesis in computational physics, I attended a research group meetup. One session gathered all the students and had them showcase their research. When there was still some time left at the end, I asked the audience who was using version control systems for the programs they write, and only 5% or so raised their hands. I then immediately ran them through a Git tutorial, and people were amazed by what is possible.
morgante 10y ago
If your paper has results, it has code of some sort which can be put into a github repo. That's the bare minimum. If you don't know how to make code agnostic to file paths or dependencies, that's too bad, but fortunately a field practitioner picking your code up will know how to work around those issues. At least they're not starting from scratch on trying to rewrite your code.
- eastWestMath 10y ago
  As a theorist, not it goddamn doesn't.
  - harveywi 10y ago
    What about an agda proof?
  - jononor 10y ago
    How do you validate your theories?
    - eastWestMath 10y ago
      I prove them.
- deleted 10y ago
  [deleted]

_yosefk 10y ago

A lot of researchers do make their results available as code and a whole lot of those who don't actually publish lies, as many people in academia from grad students to professors who implemented research papers and observed that the algorithms don't really work discovered. File paths and manual steps can be worked around, mobody said it should build and run painlessly forever after having been published on any future system with no tweaking and with the researcher indefinitely maintaining the program. If however nothing remotely close to a working state can be published it's not a great sign.

thaw13579 10y ago
Yes, many do release code, and it is a great thing! But not everyone's research is represented by code (even within CS). Sometimes research also depends on data that cannot be shared publicly, obscure user interfaces, commercial libraries, includes yet to be published work, etc. In those cases, pushing out without the dependecies may also give the impression of poor research, because it appears to be broken.

ef4 10y ago

> In an ideal world with enough resources, every lab would have technical staff to help with this process, but most researchers unfortunately don't have bandwidth to spend on this problem. And they shoot themselves in the foot constantly by not prioritizing non-crappy software. Then think it's normal that a new student needs months of close support before they can start doing anything interesting. They think it's normal to spend weeks trying to get bitrotted code running again, or just give up and throw it out losing months or years of hard-won incremental improvements. I'm not even talking about supporting reproducibility by outsiders -- they can't even achieve reproducibility within their own labs, because they don't follow practices that are baseline in industry (version control, configuration management, standardized environments, etc).

thaw13579 10y ago
Indeed, this is all too common and such a waste. I imagine it would help to have a manual of "modern" best practices for research--maybe this exists? A lot of non-CS researchers could benefit as well.
- drjesusphd 10y ago
  Yes, I would like to see such a thing also.
majewsky 10y ago
> I'm not even talking about supporting reproducibility by outsiders -- they can't even achieve reproducibility within their own labs, because they don't follow practices that are baseline in industry (version control, configuration management, standardized environments, etc). True that. Four years ago, when I was writing my thesis in computational physics, I attended a research group meetup. One session gathered all the students and had them showcase their research. When there was still some time left at the end, I asked the audience who was using version control systems for the programs they write, and only 5% or so raised their hands. I then immediately ran them through a Git tutorial, and people were amazed by what is possible.

morgante 10y ago

If your paper has results, it has code of some sort which can be put into a github repo. That's the bare minimum. If you don't know how to make code agnostic to file paths or dependencies, that's too bad, but fortunately a field practitioner picking your code up will know how to work around those issues. At least they're not starting from scratch on trying to rewrite your code.

eastWestMath 10y ago
As a theorist, not it goddamn doesn't.
- harveywi 10y ago
  What about an agda proof?
- jononor 10y ago
  How do you validate your theories?
  - eastWestMath 10y ago
    I prove them.
deleted 10y ago
[deleted]