8 ms·

> It wouldn't be worth anyone's time to try to correct the issue or write to the journal. I don't see how a research scientist could say this; something seems

by feral 8y ago

> It wouldn't be worth anyone's time to try to correct the issue or write to the journal. I don't see how a research scientist could say this; something seems culturally wrong there. Nature is a pretty serious publication. You spent months on it, maybe someone else will. How can it not be worth your time to 'try to correct the issue or write to the journal'?

aluren 8y ago
-Nature is full of bogus, like all journals, if not more. The reason for its high impact factor isn't due to consistently solid papers all around, it's due to a few very high impact, foundational papers and a crowd of minor impact/shaky ones that were sneaked in by big shots and friends of the editors. Look up retraction rates by journal and prepare to be surprised. -I don't think you realize the step it takes to actually write to Nature and say: "Excuse me, that paper you published is wrong and you should retract it." You're liable to alienate all of the authors as well as piss off the editorial board for pointing out that they let a mistake slip through. You should be prepared to face vigorous backlash, and be completely confident in your own results if you don't want the repercussions to overwhelm you. Many communities related to specific fields are small and niche, so making enemies with one team often means that you've actually cut yourself off a good part of that community. You should be prepared for awkward moments in conferences when you meet each other, incendiary questions when giving talks and scathing "anonymous" peer-reviews for your future submitted work. It's far easier to assume good faith, give the authors the benefit of the doubt ("yeah the implementation is buggy but what software isn't? The main idea is probably sound") and move on. -I don't know about other fields, but in mine, there's an unofficial accepted consensus on a set of very high-profile papers that happen to be either complete bunk or utterly useless. Again, we're talking about Science, Nature, etc. These papers made a lot of noise when they came out years ago, and now no one in their right mind would base their work on them. Again, you can't just knock on Science's editorial board and go "Excuse me, that hyped paper from a very big shot is useless and you shouldn't have published it", so it's just something people in the community know and whisper among themselves. I guess it may seem strange to outsiders though.
- hnaccy 8y ago
  > I guess it may seem strange to outsiders though. Of course it seems strange. The output of the scientific/academic community is often presented to people as being a source of truth and scientists do little to dissuade this. To learn that it's all rotten to the core is upsetting.
  - aluren 8y ago
    It's not rotten to the core, it's just imperfect, like all human communities are. Nepotism, conflicts of interests, human error, unconscious biases, etc. don't magically disappear just because we're in academia. As I said below: "All in all, I'd say we're doing fine. We're just not the ethereal source of truth that some people hold us to be, the very same people who, after claming that 'God is dead', are very quick to replace Him with His sillicon-based equivalent around which we would act like priests, except with lab coats in lieu of clerical garments." It's all about trust, really. Many people choose to trust state-of-the-art medical research when they do vaccinations on their kids; some do not. You can indeed choose not to trust us as a 'source of truth', and it seems a fair chunk of the US political landscape is being led down that path; we'll see where it goes.
    - lvh 8y ago
      Most of those other systems don’t claim to be designed to prevent them intrinsically via process, though.
      - aluren 8y ago
        We never claimed to be completely free of bias, or immune to nepotism. We're human. I think you're making it to be a bigger deal than it actually is. Most of these things are just noise. We just ignore it. If someone rises to prominence with outrageously fake results, rest assured that they will get shot down quickly by concurrents (all the incentive to publish fake results lies within highly competitive fields). If you know what you're doing, and work in a lab where people know what they're doing too, you can still do pretty good science. I would guess than in all communities, academic or not, there's a nonzero amount of bogus, over-hype, nepotism, politics and what not, and a subset of people who actually get things done and make the whole field advance. After a while, the test of time truly determines what was actually useful from what is bunk. That's how it's always been.
deleted 8y ago
[deleted]
BeetleB 8y ago
>I don't see how a research scientist could say this; something seems culturally wrong there. I can tell you haven't spent time as a grad student in a top university in the US. There is almost no incentive for a research scientist to pursue this. They're very busy and stressed out, and this will not help them in any way. Your argument that one person wasted a lot of time, and so others will be spared that pointless effort is a sound one, but you have to realize that in some (sub)disciplines, scientists view a good bulk of the research as problematic and a waste of time. Why go through the trouble for this particular case? Also, I doubt a simple email to Nature will change much. There would likely be a somewhat lengthy process, which will suck more of your time. And to be brutally honest, the chances are higher that the code did produce those results, but the grad students/post docs have been modifying the code for their next batch of research. Even something as basic as version control is unheard of in much of scientific research. Definitely a cultural problem, as you describe it.
deleted 8y ago
[deleted]

by feral 8y ago

aluren 8y ago

-Nature is full of bogus, like all journals, if not more. The reason for its high impact factor isn't due to consistently solid papers all around, it's due to a few very high impact, foundational papers and a crowd of minor impact/shaky ones that were sneaked in by big shots and friends of the editors. Look up retraction rates by journal and prepare to be surprised. -I don't think you realize the step it takes to actually write to Nature and say: "Excuse me, that paper you published is wrong and you should retract it." You're liable to alienate all of the authors as well as piss off the editorial board for pointing out that they let a mistake slip through. You should be prepared to face vigorous backlash, and be completely confident in your own results if you don't want the repercussions to overwhelm you. Many communities related to specific fields are small and niche, so making enemies with one team often means that you've actually cut yourself off a good part of that community. You should be prepared for awkward moments in conferences when you meet each other, incendiary questions when giving talks and scathing "anonymous" peer-reviews for your future submitted work. It's far easier to assume good faith, give the authors the benefit of the doubt ("yeah the implementation is buggy but what software isn't? The main idea is probably sound") and move on. -I don't know about other fields, but in mine, there's an unofficial accepted consensus on a set of very high-profile papers that happen to be either complete bunk or utterly useless. Again, we're talking about Science, Nature, etc. These papers made a lot of noise when they came out years ago, and now no one in their right mind would base their work on them. Again, you can't just knock on Science's editorial board and go "Excuse me, that hyped paper from a very big shot is useless and you shouldn't have published it", so it's just something people in the community know and whisper among themselves. I guess it may seem strange to outsiders though.

hnaccy 8y ago
> I guess it may seem strange to outsiders though. Of course it seems strange. The output of the scientific/academic community is often presented to people as being a source of truth and scientists do little to dissuade this. To learn that it's all rotten to the core is upsetting.
- aluren 8y ago
  It's not rotten to the core, it's just imperfect, like all human communities are. Nepotism, conflicts of interests, human error, unconscious biases, etc. don't magically disappear just because we're in academia. As I said below: "All in all, I'd say we're doing fine. We're just not the ethereal source of truth that some people hold us to be, the very same people who, after claming that 'God is dead', are very quick to replace Him with His sillicon-based equivalent around which we would act like priests, except with lab coats in lieu of clerical garments." It's all about trust, really. Many people choose to trust state-of-the-art medical research when they do vaccinations on their kids; some do not. You can indeed choose not to trust us as a 'source of truth', and it seems a fair chunk of the US political landscape is being led down that path; we'll see where it goes.
  - lvh 8y ago
    Most of those other systems don’t claim to be designed to prevent them intrinsically via process, though.
    - aluren 8y ago
      We never claimed to be completely free of bias, or immune to nepotism. We're human. I think you're making it to be a bigger deal than it actually is. Most of these things are just noise. We just ignore it. If someone rises to prominence with outrageously fake results, rest assured that they will get shot down quickly by concurrents (all the incentive to publish fake results lies within highly competitive fields). If you know what you're doing, and work in a lab where people know what they're doing too, you can still do pretty good science. I would guess than in all communities, academic or not, there's a nonzero amount of bogus, over-hype, nepotism, politics and what not, and a subset of people who actually get things done and make the whole field advance. After a while, the test of time truly determines what was actually useful from what is bunk. That's how it's always been.

deleted 8y ago

[deleted]

BeetleB 8y ago

>I don't see how a research scientist could say this; something seems culturally wrong there. I can tell you haven't spent time as a grad student in a top university in the US. There is almost no incentive for a research scientist to pursue this. They're very busy and stressed out, and this will not help them in any way. Your argument that one person wasted a lot of time, and so others will be spared that pointless effort is a sound one, but you have to realize that in some (sub)disciplines, scientists view a good bulk of the research as problematic and a waste of time. Why go through the trouble for this particular case? Also, I doubt a simple email to Nature will change much. There would likely be a somewhat lengthy process, which will suck more of your time. And to be brutally honest, the chances are higher that the code did produce those results, but the grad students/post docs have been modifying the code for their next batch of research. Even something as basic as version control is unheard of in much of scientific research. Definitely a cultural problem, as you describe it.

deleted 8y ago

[deleted]