7 ms·

I spent a couple months implementing a promising optimization technique that was published in Nature. I assumed that I was missing something or had a bug in my

I spent a couple months implementing a promising optimization technique that was published in Nature. I assumed that I was missing something or had a bug in my code because I could not reproduce any of the results. It turns out, one guy in my research group knew the author. He requested the source code for me to the program used to generate the data for publication. That program did not even come close to doing what the paper claimed due to some very serious bugs. When I brought up these issues, nobody really seemed to care much. It wouldn't be worth anyone's time to try to correct the issue or write to the journal.

BugsJustFindMe 8y ago
> it wouldn't be worth anyone's time to...write to the journal. I'm sorry, what? It almost sounds like everyone in your lab is scum. Hopefully YOU at least spent the 60 seconds to write an email?
- aluren 8y ago
  It's not at all uncommon that implementations provided with papers do not actually do what the paper says it does. You're often lucky when there is an implementation at all. But most of the time it's just that running the exact same implementation under the same conditions requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another. Each deviation from the original paper is liable to subtly affect the results in nondeterministic ways. That's why no one really gets surprised or even cares, life is too short to call out all the bad software written in academia.
  - RidingPegasus 8y ago
    > requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another Containerisation is fairly mature and simple to use. Many in other fields struggle with these exact same issues and are able to create reproducible environments just fine. I find it amazing that those publishing don't include their implementation, all that work locked away on a rusty hdd.
    - sk5t 8y ago
      In the software dev world--but outside of BSD--I would argue that containerization is extremely immature. Outside the software dev world, it is not easy to use.
    - Fomite 8y ago
      Containerization is not simple to use for many users of scientific code. Hell, I'm pretty sure in my department, the first response would be "What's that?"
      - thomasz 8y ago
        Containers may be a bit difficult, but every researcher should be capable of using a VM and distributing it.
        Fomite 8y ago
        I'm excited if I can get my colleagues to use version control.
        thomasz 8y ago
        Git is way more complicated than VMware.
        user5994461 8y ago
        Use SVN just like everybody else.
        gpm 8y ago
        Do you think that they would use a VCS that was less invasive and more transparent to their workflow, like dropbox? I'm thinking of making a VCS that simply runs in the background and - Automatically records every file save (effectively a git commit without a message) - Allows adding messages through tagging (like git tag) - Handles 'branching' just by asking you make a copy of the directory with a different name, properly understands how to diff/merge/etc copied files/directories that have since diverged. - Has built in support for large files
  - xorcist 8y ago
    If a research result requires specific versions of libraries to replicate, wouldn't that completely invalidate any scientific claim being made? Any change in precision or numerical methods that affects results surely must be well within the error margins.
    - aluren 8y ago
      I was more referring to changes in the API where the input and output suddenly have to be in different formats in the middle of a pipeline, causing a crash. What can also happen is that somehow the old format is still valid and gets processed all the same, thus yielding nonsensical results. Sometimes a lab devises their own format which no one else uses, and the specification may be updated without notice between the moment they publish and the moment you try things out. Most people have no idea about things like 'backwards compatibility', 'unit tests', 'containers', etc. Code is just a tool to them and the fact that they had to write some is annoying them in itself.
- neuromantik8086 8y ago
  Grad students / postdocs / human lab rats aren't scum, the incentives just aren't in place to promote good behavior (such as calling other researchers out on their bullshit). If you're trying to acquire a vaunted tenure track job, you can't afford to piss off $senior_tenured_researcher_at_prestigious_institution, since $senior could blacklist you so that you won't get hired at the incredibly small set of universities out there. Sometimes things work out despite pissing off major powers (Carl Sagan technically had to "settle" for Cornell due to being denied tenure at Harvard, in no small part because of a bad recommendation letter from Harold Urey [0]), but not often. Even if you do manage to get a tenure track job, you pretty much have to keep your head down for 7 years in order to secure your position. And once you have tenure, you still get attacked vociferously. Look at what happened when Andrew Gelman rightly pointed out that Susan Fiske (and other social psychologists) have been abusing statistics for years. Rather than a hearty "congratulations", he was called a "methodological terrorist" and a great humdrum came about [1]. When framed against these circumstances, it should be evident that there is literally nothing to gain and everything to lose from sending out a short e-mail pointing out that someone's model doesn't work. [0] https://www.reddit.com/r/todayilearned/comments/15m8om/til_carl_sagan_was_denied_tenure_at_harvard/ https://www.reddit.com/r/todayilearned/comments/15m8om/til_c... [1] https://www.businessinsider.com/susan-fiske-methodological-terrorism-2016-9 https://www.businessinsider.com/susan-fiske-methodological-t...
  - adrianrocamora 8y ago
    I'm a researcher myself and I guess this is one of those "does the end justify the means?" scenarios... Out bad research and its perpetrators and science loses out on a scientist that actually wants to do good work. Or don't and then watch yourself rationalize worse decisions later on for the sake of your research, slowly becoming as corrupt as they were and realizing that a lot of your cited work could potentially be as bad (or worse) as the ones you helped get published. I really believe we need a better way. Privately funded / bootstrapped OPEN research comes to mind as a potential solution to bring some healthy competition to this potentially corrupt system. Mathematicians are starting to do this, I think computational researchers have the potential to be next.
  - chithanh 8y ago
feral 8y ago
> It wouldn't be worth anyone's time to try to correct the issue or write to the journal. I don't see how a research scientist could say this; something seems culturally wrong there. Nature is a pretty serious publication. You spent months on it, maybe someone else will. How can it not be worth your time to 'try to correct the issue or write to the journal'?
- aluren 8y ago
  -Nature is full of bogus, like all journals, if not more. The reason for its high impact factor isn't due to consistently solid papers all around, it's due to a few very high impact, foundational papers and a crowd of minor impact/shaky ones that were sneaked in by big shots and friends of the editors. Look up retraction rates by journal and prepare to be surprised. -I don't think you realize the step it takes to actually write to Nature and say: "Excuse me, that paper you published is wrong and you should retract it." You're liable to alienate all of the authors as well as piss off the editorial board for pointing out that they let a mistake slip through. You should be prepared to face vigorous backlash, and be completely confident in your own results if you don't want the repercussions to overwhelm you. Many communities related to specific fields are small and niche, so making enemies with one team often means that you've actually cut yourself off a good part of that community. You should be prepared for awkward moments in conferences when you meet each other, incendiary questions when giving talks and scathing "anonymous" peer-reviews for your future submitted work. It's far easier to assume good faith, give the authors the benefit of the doubt ("yeah the implementation is buggy but what software isn't? The main idea is probably sound") and move on. -I don't know about other fields, but in mine, there's an unofficial accepted consensus on a set of very high-profile papers that happen to be either complete bunk or utterly useless. Again, we're talking about Science, Nature, etc. These papers made a lot of noise when they came out years ago, and now no one in their right mind would base their work on them. Again, you can't just knock on Science's editorial board and go "Excuse me, that hyped paper from a very big shot is useless and you shouldn't have published it", so it's just something people in the community know and whisper among themselves. I guess it may seem strange to outsiders though.
  - hnaccy 8y ago

BugsJustFindMe 8y ago

> it wouldn't be worth anyone's time to...write to the journal. I'm sorry, what? It almost sounds like everyone in your lab is scum. Hopefully YOU at least spent the 60 seconds to write an email?

aluren 8y ago
It's not at all uncommon that implementations provided with papers do not actually do what the paper says it does. You're often lucky when there is an implementation at all. But most of the time it's just that running the exact same implementation under the same conditions requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another. Each deviation from the original paper is liable to subtly affect the results in nondeterministic ways. That's why no one really gets surprised or even cares, life is too short to call out all the bad software written in academia.
- RidingPegasus 8y ago
  > requires setting up a very specific environment, installing specific versions of libraries, using some niche software and converting data from one byzantine format to another Containerisation is fairly mature and simple to use. Many in other fields struggle with these exact same issues and are able to create reproducible environments just fine. I find it amazing that those publishing don't include their implementation, all that work locked away on a rusty hdd.
  - sk5t 8y ago
    In the software dev world--but outside of BSD--I would argue that containerization is extremely immature. Outside the software dev world, it is not easy to use.
  - Fomite 8y ago
    Containerization is not simple to use for many users of scientific code. Hell, I'm pretty sure in my department, the first response would be "What's that?"
    - thomasz 8y ago
      Containers may be a bit difficult, but every researcher should be capable of using a VM and distributing it.
      - Fomite 8y ago
        I'm excited if I can get my colleagues to use version control.
        thomasz 8y ago
        Git is way more complicated than VMware.
        user5994461 8y ago
        Use SVN just like everybody else.
        gpm 8y ago
        Do you think that they would use a VCS that was less invasive and more transparent to their workflow, like dropbox? I'm thinking of making a VCS that simply runs in the background and - Automatically records every file save (effectively a git commit without a message) - Allows adding messages through tagging (like git tag) - Handles 'branching' just by asking you make a copy of the directory with a different name, properly understands how to diff/merge/etc copied files/directories that have since diverged. - Has built in support for large files
- xorcist 8y ago
  If a research result requires specific versions of libraries to replicate, wouldn't that completely invalidate any scientific claim being made? Any change in precision or numerical methods that affects results surely must be well within the error margins.
  - aluren 8y ago
    I was more referring to changes in the API where the input and output suddenly have to be in different formats in the middle of a pipeline, causing a crash. What can also happen is that somehow the old format is still valid and gets processed all the same, thus yielding nonsensical results. Sometimes a lab devises their own format which no one else uses, and the specification may be updated without notice between the moment they publish and the moment you try things out. Most people have no idea about things like 'backwards compatibility', 'unit tests', 'containers', etc. Code is just a tool to them and the fact that they had to write some is annoying them in itself.
neuromantik8086 8y ago
Grad students / postdocs / human lab rats aren't scum, the incentives just aren't in place to promote good behavior (such as calling other researchers out on their bullshit). If you're trying to acquire a vaunted tenure track job, you can't afford to piss off $senior_tenured_researcher_at_prestigious_institution, since $senior could blacklist you so that you won't get hired at the incredibly small set of universities out there. Sometimes things work out despite pissing off major powers (Carl Sagan technically had to "settle" for Cornell due to being denied tenure at Harvard, in no small part because of a bad recommendation letter from Harold Urey [0]), but not often. Even if you do manage to get a tenure track job, you pretty much have to keep your head down for 7 years in order to secure your position. And once you have tenure, you still get attacked vociferously. Look at what happened when Andrew Gelman rightly pointed out that Susan Fiske (and other social psychologists) have been abusing statistics for years. Rather than a hearty "congratulations", he was called a "methodological terrorist" and a great humdrum came about [1]. When framed against these circumstances, it should be evident that there is literally nothing to gain and everything to lose from sending out a short e-mail pointing out that someone's model doesn't work. [0] https://www.reddit.com/r/todayilearned/comments/15m8om/til_carl_sagan_was_denied_tenure_at_harvard/ https://www.reddit.com/r/todayilearned/comments/15m8om/til_c... [1] https://www.businessinsider.com/susan-fiske-methodological-terrorism-2016-9 https://www.businessinsider.com/susan-fiske-methodological-t...
- adrianrocamora 8y ago
  I'm a researcher myself and I guess this is one of those "does the end justify the means?" scenarios... Out bad research and its perpetrators and science loses out on a scientist that actually wants to do good work. Or don't and then watch yourself rationalize worse decisions later on for the sake of your research, slowly becoming as corrupt as they were and realizing that a lot of your cited work could potentially be as bad (or worse) as the ones you helped get published. I really believe we need a better way. Privately funded / bootstrapped OPEN research comes to mind as a potential solution to bring some healthy competition to this potentially corrupt system. Mathematicians are starting to do this, I think computational researchers have the potential to be next.
- chithanh 8y ago

feral 8y ago

> It wouldn't be worth anyone's time to try to correct the issue or write to the journal. I don't see how a research scientist could say this; something seems culturally wrong there. Nature is a pretty serious publication. You spent months on it, maybe someone else will. How can it not be worth your time to 'try to correct the issue or write to the journal'?

aluren 8y ago
-Nature is full of bogus, like all journals, if not more. The reason for its high impact factor isn't due to consistently solid papers all around, it's due to a few very high impact, foundational papers and a crowd of minor impact/shaky ones that were sneaked in by big shots and friends of the editors. Look up retraction rates by journal and prepare to be surprised. -I don't think you realize the step it takes to actually write to Nature and say: "Excuse me, that paper you published is wrong and you should retract it." You're liable to alienate all of the authors as well as piss off the editorial board for pointing out that they let a mistake slip through. You should be prepared to face vigorous backlash, and be completely confident in your own results if you don't want the repercussions to overwhelm you. Many communities related to specific fields are small and niche, so making enemies with one team often means that you've actually cut yourself off a good part of that community. You should be prepared for awkward moments in conferences when you meet each other, incendiary questions when giving talks and scathing "anonymous" peer-reviews for your future submitted work. It's far easier to assume good faith, give the authors the benefit of the doubt ("yeah the implementation is buggy but what software isn't? The main idea is probably sound") and move on. -I don't know about other fields, but in mine, there's an unofficial accepted consensus on a set of very high-profile papers that happen to be either complete bunk or utterly useless. Again, we're talking about Science, Nature, etc. These papers made a lot of noise when they came out years ago, and now no one in their right mind would base their work on them. Again, you can't just knock on Science's editorial board and go "Excuse me, that hyped paper from a very big shot is useless and you shouldn't have published it", so it's just something people in the community know and whisper among themselves. I guess it may seem strange to outsiders though.
- hnaccy 8y ago