7 ms·

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the resea

This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research, but I'll clean up all the code, rerun experiments to make sure it all still works like before, and add a bunch of documentation. But no, still not good enough -- it has to run at the press of a button in your particular programming environment. If we don't know how to write a script (or couldn't be bothered to spend the time writing one) to check that the data is on disk and, if not, crawl a website to download some huge dataset in one click, test our code on your OS, your CPU/GPU/TPU/..., etc., we were being "insincere" with our open-sourcing efforts.

liquidify 7y ago
If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.
- draw_down 7y ago
  I'd say that depends on the readers an awful lot.
- throwaway287391 7y ago
  What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering firms with millions to invest in making it work properly. He's mad that individual machine learning researchers don't always do a great job of making it happen for their cutting-edge research which was a difficult enough engineering task to get running on their machine.
  - gcommer 7y ago
    Reread the original post: > I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.
    - throwaway287391 7y ago
      You're right, I misread that part.
      - jfoutz 7y ago
        throwaway287391 it's hilarious how much you value your own opinion. You feel it's important your interlocutor knows you know you were corrected, while remaining anonymous. This is like, the funniest thing i've seen all week. You should run for office. _edit_ Mmmm delicious downvotes. How many alts do you have throwaway287391? i don't think dang would or should expose you. But it is even more funny to me that you would log in with an alt (2?!), specifically to downvote. Anyway.I'm sure with some practice you'll get better at trolling little buddy.
        throwaway287391 7y ago
        I don't have much to say to most of this, but I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma or whatever the threshold is.
        jfoutz 7y ago
        > I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma I'll take you at your word. You want to just drop in and drop bombs and pretend that the information you are presenting is in some sense useful. Other folks pointed out that you are wrong. But you also want the reputation. You want the respect for revising your opinion. You're you from your perspective, but from everyone else's perspective you're not you, you're the basest troll that has ever posted on HN, because you have no rep, no history, not provenance. It's funny to me, because you want it both ways. It's pretty hilarious. I shouldn't feed the trolls. I know better. Enjoy being a complete waste of time to everyone that glances at your writing!
- randcraw 7y ago
  Agreed. If your results are not reproducible by others, the publication should issue a retraction and withdraw your work. Code that works is essential.
  - Volt 7y ago
    I hope you're a scientist.
  - dodobirdlord 7y ago
    Nonsense! The "reproducible" in "reproducible result" doesn't mean anything except that what is described in the paper can be reproduced. Code that works is nice and all, but totally irrelevant to the question of reproducibility. Would you claim that that the Higgs paper must be retracted because CERN won't ship you the LHC? They've got ATLAS and CMS to demonstrate reproducibility, and they're wildly different!
- p1esk 7y ago
  Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll help you.
  - throwawayjava 7y ago
    ...and university/company sometimes makes you wipe out the git commit log, so you can't always just use "HEAD at time of submission". Pro tip: at paper submission time, md5sum your code and also git tag it in your private repo. When you release the code, if you have to / want to release with a clean history, make submission-time your initial commit and then make the current state of the repo your second commit. I've never encountered an institution that won't allow that level of history in the code release, even places that are pretty hard core about wiping pre-release history.
  - caenorst 7y ago
    And don't you think it would have been more efficient to write it down clean at first with good comments and code practice ? Those good practices are here for a reason and not just for annoying you for the sack of "beauty". You would save more time for yourself when you wanna expand the paper further or just reuse some part of it, have better reputation for your work, and gain some citation because people will have the tendency to use your code as a starting point.
    - inimino 7y ago
      "Writing it down clean at first" implies that you know what is going to work when you start. If that's the case, it's not research.
    - patrick5415 7y ago
      No, it would not be more efficient. I’ve tried this route and it hands down consumes way more time. Until it’s clear that something will get re-used, it’s most efficient to leave it dirty. The software types love to sht on researchers for their poor quality code. Yet a few threads over, there’s always a discussion about how more tests don’t make it onto the kanban board (or whatever you guys are using these days) when crunch time approaches. It’s not just us.
Veedrac 7y ago
If it was just one thing or the other, perhaps he wouldn't have been so critical of it. People are going to understand if you require a few paths to be changed or some files to be downloaded manually into specific relative locations. The ask is only that somebody should be able to use the code without doing their own novel research, and be able to verify it's doing what it is claimed to be doing. If this means your download links need to point to the right location, the code has to run in a location other than /home/hellokitty/ml, and that it needs to meet standards of readability—well, good, that's what the scientific process is about.
caenorst 7y ago
I have spent enough time to debunk papers to know that a lot of researchers are shamelessly cheating or make mistakes (as EVERYBODY does). Reviewing your code should be part of the peer review process. Sorry but I don't trust your research just based on yours words and cherry picked images. Also research is incremental, so producing a proper code on which other can work on top of should be part of the CONTRIBUTION.
- ur-whale 7y ago
  I fully agree, and would actually go a lot further. A very large number of research papers I've read in my life were so poorly written as to being near impossible to understand. The time it took to infer the actual meaning of certain parts of certain papers was vastly larger than just reading the code would have been. It has become very clear to me that being a strong CS researcher does not necessarily imply mastery of language or even the very basic ability to explain one's ideas. As a matter of fact, with very few exception (eg Feynmann), the two skills seem to be at odds with one another. On the other hand, code can be read and understood much more readily, especially when it can be experimented with, if only to insert printfs in it to understand what it does. The culture of scientific publication in CS must change. The "I'm pressured by my advisor and therefore have no time to publish my code" argument is complete bollocks. The code should come first and your advisor should pressure you to publish that first (assuming it actually works). If someone wants to read your poorly worded explanation of how it works, fine, publish a paper. Show me the code first.
namibj 7y ago
I'd much prefer some code with a license that doesn't make looking at it a hazard for writing (A)GPL later, than to have to read the paper many times until I figure out how they meant to combine loads of prior research that is only mentioned to exist on the side, without any info on whether it was used in this or not. Being able to compile in some way and, possibly after some bug-fixing/compiler-pleasing yielding useful results that approach/relate to those in the paper, greatly increases my confidence in the researcher and his work. Pretty code is nice, but it's weighed separately, not affecting the appreciation for the original research work itself. Only the existence of code with an AGPL or less restricted license affects how I perceive the paper/original research.
KirinDave 7y ago
Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actually describe your algorithms correctly and don't misrepresent a piece of code and a paper?" Or should everyone just decide you can find your own research funding if it's not going to help anyone?
- p1esk 7y ago
  Pardon me, maybe I just misunderstood the whole idea of "reproducible" but it is only not reproducible if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.
  - throwawayjava 7y ago
    > if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results. This is the actually important definition of "reproducible". "has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description. The "must be able to apt install or it's not real science" is a particularly dangerous path to go down. It's how you end up with e.g. shit loads of well-engineered LISP or FORTRAN code with no actual scientific insights or knowledge transfer. Which has actually happened in the past. A lot.
    - telchar 7y ago
      I don't think people in this thread are insisting on a downloadable package. Just that the algorithm published in the write-up accurately describe the work that was actually done to achieve the stated results. If even that threshold can't be met, IMO the authors are polluting the literature with bad work.
ur-whale 7y ago
You might want to read up on docker.
hoseja 7y ago
Your paper has to be well documented and clean. The code is often more important than your paper.

liquidify 7y ago

If there are bugs in your code that would have prevented you from being able to do what you claim to do, then to the readers of your paper, it seems more likely that you might not have performed the experimentation that you claim to.

draw_down 7y ago
I'd say that depends on the readers an awful lot.
throwaway287391 7y ago
What? This has nothing to do with bugs in the code that would've affected the original author's ability to run the original experiments described in the paper. The user I replied to is talking about things like hard-coded data paths and wanting to be able to run the code at the press of a button on any platform. The expectation for this level of cross-platform compatibility is difficult to achieve for engineering firms with millions to invest in making it work properly. He's mad that individual machine learning researchers don't always do a great job of making it happen for their cutting-edge research which was a difficult enough engineering task to get running on their machine.
- gcommer 7y ago
  Reread the original post: > I have to fix bugs before the code can run. I'm very curious how did the author run that code with the bugs.
  - throwaway287391 7y ago
    You're right, I misread that part.
    - jfoutz 7y ago
      throwaway287391 it's hilarious how much you value your own opinion. You feel it's important your interlocutor knows you know you were corrected, while remaining anonymous. This is like, the funniest thing i've seen all week. You should run for office. _edit_ Mmmm delicious downvotes. How many alts do you have throwaway287391? i don't think dang would or should expose you. But it is even more funny to me that you would log in with an alt (2?!), specifically to downvote. Anyway.I'm sure with some practice you'll get better at trolling little buddy.
      - throwaway287391 7y ago
        I don't have much to say to most of this, but I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma or whatever the threshold is.
        jfoutz 7y ago
        > I'll point out that I didn't and couldn't have downvoted you as none of my throwaways have >=500 karma I'll take you at your word. You want to just drop in and drop bombs and pretend that the information you are presenting is in some sense useful. Other folks pointed out that you are wrong. But you also want the reputation. You want the respect for revising your opinion. You're you from your perspective, but from everyone else's perspective you're not you, you're the basest troll that has ever posted on HN, because you have no rep, no history, not provenance. It's funny to me, because you want it both ways. It's pretty hilarious. I shouldn't feed the trolls. I know better. Enjoy being a complete waste of time to everyone that glances at your writing!
randcraw 7y ago
Agreed. If your results are not reproducible by others, the publication should issue a retraction and withdraw your work. Code that works is essential.
- Volt 7y ago
  I hope you're a scientist.
- dodobirdlord 7y ago
  Nonsense! The "reproducible" in "reproducible result" doesn't mean anything except that what is described in the paper can be reproduced. Code that works is nice and all, but totally irrelevant to the question of reproducibility. Would you claim that that the Higgs paper must be retracted because CERN won't ship you the LHC? They've got ATLAS and CMS to demonstrate reproducibility, and they're wildly different!
p1esk 7y ago
Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll help you.
- throwawayjava 7y ago
  ...and university/company sometimes makes you wipe out the git commit log, so you can't always just use "HEAD at time of submission". Pro tip: at paper submission time, md5sum your code and also git tag it in your private repo. When you release the code, if you have to / want to release with a clean history, make submission-time your initial commit and then make the current state of the repo your second commit. I've never encountered an institution that won't allow that level of history in the code release, even places that are pretty hard core about wiping pre-release history.
- caenorst 7y ago
  And don't you think it would have been more efficient to write it down clean at first with good comments and code practice ? Those good practices are here for a reason and not just for annoying you for the sack of "beauty". You would save more time for yourself when you wanna expand the paper further or just reuse some part of it, have better reputation for your work, and gain some citation because people will have the tendency to use your code as a starting point.
  - inimino 7y ago
    "Writing it down clean at first" implies that you know what is going to work when you start. If that's the case, it's not research.
  - patrick5415 7y ago
    No, it would not be more efficient. I’ve tried this route and it hands down consumes way more time. Until it’s clear that something will get re-used, it’s most efficient to leave it dirty. The software types love to sht on researchers for their poor quality code. Yet a few threads over, there’s always a discussion about how more tests don’t make it onto the kanban board (or whatever you guys are using these days) when crunch time approaches. It’s not just us.

Veedrac 7y ago

If it was just one thing or the other, perhaps he wouldn't have been so critical of it. People are going to understand if you require a few paths to be changed or some files to be downloaded manually into specific relative locations. The ask is only that somebody should be able to use the code without doing their own novel research, and be able to verify it's doing what it is claimed to be doing. If this means your download links need to point to the right location, the code has to run in a location other than /home/hellokitty/ml, and that it needs to meet standards of readability—well, good, that's what the scientific process is about.

caenorst 7y ago

I have spent enough time to debunk papers to know that a lot of researchers are shamelessly cheating or make mistakes (as EVERYBODY does). Reviewing your code should be part of the peer review process. Sorry but I don't trust your research just based on yours words and cherry picked images. Also research is incremental, so producing a proper code on which other can work on top of should be part of the CONTRIBUTION.

ur-whale 7y ago
I fully agree, and would actually go a lot further. A very large number of research papers I've read in my life were so poorly written as to being near impossible to understand. The time it took to infer the actual meaning of certain parts of certain papers was vastly larger than just reading the code would have been. It has become very clear to me that being a strong CS researcher does not necessarily imply mastery of language or even the very basic ability to explain one's ideas. As a matter of fact, with very few exception (eg Feynmann), the two skills seem to be at odds with one another. On the other hand, code can be read and understood much more readily, especially when it can be experimented with, if only to insert printfs in it to understand what it does. The culture of scientific publication in CS must change. The "I'm pressured by my advisor and therefore have no time to publish my code" argument is complete bollocks. The code should come first and your advisor should pressure you to publish that first (assuming it actually works). If someone wants to read your poorly worded explanation of how it works, fine, publish a paper. Show me the code first.

namibj 7y ago

I'd much prefer some code with a license that doesn't make looking at it a hazard for writing (A)GPL later, than to have to read the paper many times until I figure out how they meant to combine loads of prior research that is only mentioned to exist on the side, without any info on whether it was used in this or not. Being able to compile in some way and, possibly after some bug-fixing/compiler-pleasing yielding useful results that approach/relate to those in the paper, greatly increases my confidence in the researcher and his work. Pretty code is nice, but it's weighed separately, not affecting the appreciation for the original research work itself. Only the existence of code with an AGPL or less restricted license affects how I perceive the paper/original research.

KirinDave 7y ago

Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actually describe your algorithms correctly and don't misrepresent a piece of code and a paper?" Or should everyone just decide you can find your own research funding if it's not going to help anyone?

p1esk 7y ago
Pardon me, maybe I just misunderstood the whole idea of "reproducible" but it is only not reproducible if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results.
- throwawayjava 7y ago
  > if you tried to implement the ideas described in the paper yourself, and I mean really tried, and contacted the authors for help, and still failed to achieve the claimed results. This is the actually important definition of "reproducible". "has an install.sh" is really nice, but it's far more important that an informed reader can recreate the artifact for themselves from the written description. The "must be able to apt install or it's not real science" is a particularly dangerous path to go down. It's how you end up with e.g. shit loads of well-engineered LISP or FORTRAN code with no actual scientific insights or knowledge transfer. Which has actually happened in the past. A lot.
  - telchar 7y ago
    I don't think people in this thread are insisting on a downloadable package. Just that the algorithm published in the write-up accurately describe the work that was actually done to achieve the stated results. If even that threshold can't be met, IMO the authors are polluting the literature with bad work.

ur-whale 7y ago

You might want to read up on docker.

hoseja 7y ago

Your paper has to be well documented and clean. The code is often more important than your paper.