7 ms·

I also think code used to produce results should be published as well.

SalmoShalazar 11y ago
I agree with this in theory... but do you really want my garbage, ad hoc, messy code that I used to pipe output from one program to another?
- bravura 11y ago
  Yes. Especially because if you are forced to publish your code, it will be better.
- return0 11y ago
  Absolutely yes. Science isnt supposed to be clean and tidy, but reproducible.
- nv-vn 11y ago
  Considering that a mistake in such a program could completely skew results, yeah that seems pretty important.
- lacker 11y ago
  Yes, because if it doesn't even compile then I will call BS on your results.
  - skybrian 11y ago
    See, that's exactly what you shouldn't do if you want to encourage academics to release the source code. Academic code sometimes only has been made to work on one machine (the author's). Someone reproducing the results should still have access to the source, even if they have to hack it to work on their system.
    - deleted 11y ago
      [deleted]
- themartorana 11y ago
  I don't know... Did you do it right? ;) But seriously.
- CompuHacker 11y ago
  It's better than nothing.
- cube00 11y ago
  Definitely yes, then I can see if somewhere in that garbage, ad hoc, messy code why I can't replicate the result myself.
- skybrian 11y ago
  I'm not actually recommending this license, but it seems relevant to mention: The CRAPL: An academic-strength open source license http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
  - marvy 11y ago
    "the Community Research and Academic Programming License"
- mrcsparker 11y ago
  Yes, please. To be fair, a lot of people do publish their working code with the paper. It helps people like me understand how to do the work. Bad code is fine, good code is better, great code is great. Stick to fine and I can do my job and you can write papers knowing that your work will be used by many.
- zmmmmm 11y ago
  If it's that garbage, how do you know your results are correct? It should be at least non-garbage enough that someone else could theoretically check it, or you're not doing science at all.
  - jjoonathan 11y ago
    You don't. But there are 5x-100x more bugs lurking in the methodology so triage is the name of the game. A much bigger problem is that grantsmanship strongly incentivizes against verifying your methodology and being diligent in your construction of null hypotheses.
- nilkn 11y ago
  To be honest, I'm surprised anybody even takes research seriously which depends heavily on a custom program whose source code is kept hidden.
- euske 11y ago
  This is like saying "I don't want to file a tax because my handwriting is bad". Well, if you're embarrassed by your terrible handwriting, maybe it's a time to put some effort to improve it.
  - dinkumthinkum 11y ago
    Ok I don't mean this in a mean way but you don't seem to appreciate what CS research is like. It is not about handwriting.
    - Sacho 11y ago
      Filing taxes is not about handwriting either - I guess you missed the analogy. Research isn't about writing good code, but publishing your code is useful(for reproducibility, which is part of research), and if you're embarrassed by your code quality, then you should put some effort into it.
lacker 11y ago
That would also have the benefit of making sure the code actually works. From my time in CS grad school I'm pretty sure there are many published papers where the author never had code that achieved the results they described.
- taneq 11y ago
  Or code that worked on the specific test case described in the paper but was flaky as hell on anything else.
  - albinofrenchy 11y ago
    The most disillusioning thing for me as an undergrad was to try to replicate results from a paper on a relatively complicated problem, and not achieving anywhere near the papers level of results. I got access to the source code, and the super complicated algorithm added almost nothing to the results, the glossed over / hand waved past data normalization worked so well that there wasn't a need for any further classification. This paper was pretty well received and cited, despite it basically not work.
    - andreasvc 11y ago
      I am afraid that what you describe is a pretty fundamental limitation of peer review. Currently it is not expected of reviewers to try to replicate experiments, but only to check if the methodology of the paper seems sound and the relevant previous work is cited. Should this be expected of reviewers? Should the authors provide all code and data so that the results can be replicated at the push of a button? That seems a pretty tall order. Seems like we'll be stuck with a modus operandi of lots of papers being published, peer review only catching obvious errors, and replication being very rare.
      - albinofrenchy 11y ago
        I get that, and I don't even think that that should be the reviewers job. But for CS in particular, I do wish more people published source code and build procedures. I think a central hub for public comments, with source, and available PDF would be incredibly useful for CS as a field. For most papers I had access to, to get the source I'd have to go through at least one person. So I tried for quite a while to replicate the result before even trying to get access. If it was available, on say github, I'd have grabbed the source code as a resource to understand the paper, and with something like publicly available commenting, I think it'd have been a non-issue. The problem is, I can see how such a system would be incredibly constructive to the field and the community, but it could be a liability to the authors, and would make publishers meaningless, so I doubt it happens anytime soon.
dinkumthinkum 11y ago
I think you are providing a large barrier unnecessarily. Research projects and publishing a paper is more complicated than that. It is good when the researcher can do that but it is not always practical. Really, a lot of the comments to this and the downvotes seem to betray a lack of understanding of what cs research is like, which is understandable I guess. CS research is not like creating physics experiments, there are methods, techniques, and algorithms. When there is an experimental result, CS researchers do not treat those as sacrosanct and do have an informed skepticism but we are generally able to determine and discern the bulk is what is important from the content of the paper itself, in most cases. Yeah, if you look at the experimental results section from papers in SOSP, you could come up with all kinds of complaints probably all day long but there is more to it and the community than is being taken for granted here.
studentrob 11y ago
Wow, I never considered that this might fall under the same umbrella. It seems very reasonable because it isn't very hard to publish that now. Even if the shared code wouldn't run without a ton of requirements and very specific setup, it's still something. I know we all hate to release code that doesn't compile or run outside our own setup because we then feel we will be responsible for helping someone else getting it to run on their machines to which we have no physical access. Not to mention it is always a mess figuring that stuff out. But other research fields do not feel compelled to hold the hands of their researchers to replicate experiments. Presumably, whoever's code is easiest to run will more likely serve as the basis for other future research, thereby enhancing career growth of the original publisher. I'm not a researcher by the way, and this just dawned on me, so if this is obvious to all of you, then I apologize for the block of text.
Balgair 11y ago
Have you worked with biologists? If they could figure out how to use git, let alone github, I would be astounded. No forgetting, these are incredibly smart people, they just are bench-top scientists, not laptop ones. If you want their code, it will be, at best, a copy-paste of badly hardcoded Matlab, years in the making, with for-loops for the sake of for-loops alone, and without ANY comments or documentation whatsoever. Honestly, it would take less time for you to write it yourself and compare, rather than to try to peek under the hood of a spaghetti mess. Honest to god and on the grave of my mother this happened to a friend. He was going over some old Fortran 88 code, filled with GOTO statements. The code, at best, was a rat's nest. Only a deep and long fight could get it into your brain. At about 3 am after a long day in the lab, she finally gets to a line in the code that says GOTO LINE 12345 with the comment 'HAHA MADE YOU LOOK' . Her boss bought her a new computer after she threw that one off the roof. That experience is universal with Biologist code.
- AnthonyMouse 11y ago
  > Honestly, it would take less time for you to write it yourself and compare, rather than to try to peek under the hood of a spaghetti mess. That's not actually the point. The point is reproducibility. If you write your own code and get a different result, why? If the reason is the code then having the other code, no matter how terrible it is, allows you to figure out what the difference is. And then somebody is wrong and you know why and can publish that result.
  - timr 11y ago
    Exactly. IMO, this is a big problem in some areas of computational biology right now. Code to a famous project gets released, and instead of independently validating the results, other researchers (rationally) choose to just build on top of the famous project. It's faster, easier, and the "collaboration" leads to social prestige for everyone involved. The end result is that empires are built, reproducibility is (paradoxically) harmed, and we're someday going to end up finding out that some big, high-profile projects were built on pillars of sand.
- cheald 11y ago

I agree with this in theory... but do you really want my garbage, ad hoc, messy code that I used to pipe output from one program to another?

bravura 11y ago
Yes. Especially because if you are forced to publish your code, it will be better.
return0 11y ago
Absolutely yes. Science isnt supposed to be clean and tidy, but reproducible.
nv-vn 11y ago
Considering that a mistake in such a program could completely skew results, yeah that seems pretty important.
lacker 11y ago
Yes, because if it doesn't even compile then I will call BS on your results.
- skybrian 11y ago
  See, that's exactly what you shouldn't do if you want to encourage academics to release the source code. Academic code sometimes only has been made to work on one machine (the author's). Someone reproducing the results should still have access to the source, even if they have to hack it to work on their system.
  - deleted 11y ago
    [deleted]
themartorana 11y ago
I don't know... Did you do it right? ;) But seriously.
CompuHacker 11y ago
It's better than nothing.
cube00 11y ago
Definitely yes, then I can see if somewhere in that garbage, ad hoc, messy code why I can't replicate the result myself.
skybrian 11y ago
I'm not actually recommending this license, but it seems relevant to mention: The CRAPL: An academic-strength open source license http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
- marvy 11y ago
  "the Community Research and Academic Programming License"
mrcsparker 11y ago
Yes, please. To be fair, a lot of people do publish their working code with the paper. It helps people like me understand how to do the work. Bad code is fine, good code is better, great code is great. Stick to fine and I can do my job and you can write papers knowing that your work will be used by many.
zmmmmm 11y ago
If it's that garbage, how do you know your results are correct? It should be at least non-garbage enough that someone else could theoretically check it, or you're not doing science at all.
- jjoonathan 11y ago
  You don't. But there are 5x-100x more bugs lurking in the methodology so triage is the name of the game. A much bigger problem is that grantsmanship strongly incentivizes against verifying your methodology and being diligent in your construction of null hypotheses.
nilkn 11y ago
To be honest, I'm surprised anybody even takes research seriously which depends heavily on a custom program whose source code is kept hidden.
euske 11y ago
This is like saying "I don't want to file a tax because my handwriting is bad". Well, if you're embarrassed by your terrible handwriting, maybe it's a time to put some effort to improve it.
- dinkumthinkum 11y ago
  Ok I don't mean this in a mean way but you don't seem to appreciate what CS research is like. It is not about handwriting.
  - Sacho 11y ago
    Filing taxes is not about handwriting either - I guess you missed the analogy. Research isn't about writing good code, but publishing your code is useful(for reproducibility, which is part of research), and if you're embarrassed by your code quality, then you should put some effort into it.

lacker 11y ago

That would also have the benefit of making sure the code actually works. From my time in CS grad school I'm pretty sure there are many published papers where the author never had code that achieved the results they described.

taneq 11y ago
Or code that worked on the specific test case described in the paper but was flaky as hell on anything else.
- albinofrenchy 11y ago
  The most disillusioning thing for me as an undergrad was to try to replicate results from a paper on a relatively complicated problem, and not achieving anywhere near the papers level of results. I got access to the source code, and the super complicated algorithm added almost nothing to the results, the glossed over / hand waved past data normalization worked so well that there wasn't a need for any further classification. This paper was pretty well received and cited, despite it basically not work.
  - andreasvc 11y ago
    I am afraid that what you describe is a pretty fundamental limitation of peer review. Currently it is not expected of reviewers to try to replicate experiments, but only to check if the methodology of the paper seems sound and the relevant previous work is cited. Should this be expected of reviewers? Should the authors provide all code and data so that the results can be replicated at the push of a button? That seems a pretty tall order. Seems like we'll be stuck with a modus operandi of lots of papers being published, peer review only catching obvious errors, and replication being very rare.
    - albinofrenchy 11y ago
      I get that, and I don't even think that that should be the reviewers job. But for CS in particular, I do wish more people published source code and build procedures. I think a central hub for public comments, with source, and available PDF would be incredibly useful for CS as a field. For most papers I had access to, to get the source I'd have to go through at least one person. So I tried for quite a while to replicate the result before even trying to get access. If it was available, on say github, I'd have grabbed the source code as a resource to understand the paper, and with something like publicly available commenting, I think it'd have been a non-issue. The problem is, I can see how such a system would be incredibly constructive to the field and the community, but it could be a liability to the authors, and would make publishers meaningless, so I doubt it happens anytime soon.

dinkumthinkum 11y ago

I think you are providing a large barrier unnecessarily. Research projects and publishing a paper is more complicated than that. It is good when the researcher can do that but it is not always practical. Really, a lot of the comments to this and the downvotes seem to betray a lack of understanding of what cs research is like, which is understandable I guess. CS research is not like creating physics experiments, there are methods, techniques, and algorithms. When there is an experimental result, CS researchers do not treat those as sacrosanct and do have an informed skepticism but we are generally able to determine and discern the bulk is what is important from the content of the paper itself, in most cases. Yeah, if you look at the experimental results section from papers in SOSP, you could come up with all kinds of complaints probably all day long but there is more to it and the community than is being taken for granted here.

studentrob 11y ago

Wow, I never considered that this might fall under the same umbrella. It seems very reasonable because it isn't very hard to publish that now. Even if the shared code wouldn't run without a ton of requirements and very specific setup, it's still something. I know we all hate to release code that doesn't compile or run outside our own setup because we then feel we will be responsible for helping someone else getting it to run on their machines to which we have no physical access. Not to mention it is always a mess figuring that stuff out. But other research fields do not feel compelled to hold the hands of their researchers to replicate experiments. Presumably, whoever's code is easiest to run will more likely serve as the basis for other future research, thereby enhancing career growth of the original publisher. I'm not a researcher by the way, and this just dawned on me, so if this is obvious to all of you, then I apologize for the block of text.

Balgair 11y ago

Have you worked with biologists? If they could figure out how to use git, let alone github, I would be astounded. No forgetting, these are incredibly smart people, they just are bench-top scientists, not laptop ones. If you want their code, it will be, at best, a copy-paste of badly hardcoded Matlab, years in the making, with for-loops for the sake of for-loops alone, and without ANY comments or documentation whatsoever. Honestly, it would take less time for you to write it yourself and compare, rather than to try to peek under the hood of a spaghetti mess. Honest to god and on the grave of my mother this happened to a friend. He was going over some old Fortran 88 code, filled with GOTO statements. The code, at best, was a rat's nest. Only a deep and long fight could get it into your brain. At about 3 am after a long day in the lab, she finally gets to a line in the code that says GOTO LINE 12345 with the comment 'HAHA MADE YOU LOOK' . Her boss bought her a new computer after she threw that one off the roof. That experience is universal with Biologist code.

AnthonyMouse 11y ago
> Honestly, it would take less time for you to write it yourself and compare, rather than to try to peek under the hood of a spaghetti mess. That's not actually the point. The point is reproducibility. If you write your own code and get a different result, why? If the reason is the code then having the other code, no matter how terrible it is, allows you to figure out what the difference is. And then somebody is wrong and you know why and can publish that result.
- timr 11y ago
  Exactly. IMO, this is a big problem in some areas of computational biology right now. Code to a famous project gets released, and instead of independently validating the results, other researchers (rationally) choose to just build on top of the famous project. It's faster, easier, and the "collaboration" leads to social prestige for everyone involved. The end result is that empires are built, reproducibility is (paradoxically) harmed, and we're someday going to end up finding out that some big, high-profile projects were built on pillars of sand.
cheald 11y ago