7 ms·

I was chatting with a colleague last summer, and asking him for more details about a method he had published that I was having trouble reproducing. After pointi

by ovis 10y ago

I was chatting with a colleague last summer, and asking him for more details about a method he had published that I was having trouble reproducing. After pointing me to a few papers that didn't include the relevant details, hefinally told me that "if we told everyone about how it was done, anyone could do it." While at the time I was pretty upset with him, perhaps it's the competitive nature of science and funding that also gives people a mild incentive to be secretive. In this case, simply publishing code would have resolved the questions.

hannob 10y ago
> I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it." That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get away with such malicious behavior.
- wycx 10y ago
  That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...
  - michaelmrose 10y ago
    At that point its actually not even science and shouldn't be funded
    - toss1941 10y ago
      I remember my teachers in high school making such a big deal about the scientific method on how important it was, how experiments must be reproducible to be useful, but today you barely hear it mentioned one way or another.
      - SilasX 10y ago
        Wait, you're telling me that schools teach comforting lies about society that don't match how the real world actually works?
        dTal 10y ago
        It's no more a "comforting lie" than your driving instructor telling you to check your blind spot. Not everyone does it, and bad things happen as a result, but all the more important to teach it. I'm actually surprised to hear of high school teaching good scientific practice. I don't remember ever being taught that. Widely may it spread.
        SilasX 10y ago
        Hold on -- there's a category confusion here: "check your blindspots" isn't a comforting lie; it's a command. Converting it to "checking your blindspots will avoid collisions with careless drivers" would make it no longer a lie.
        dTal 10y ago
        I was pointing out the category confusion. Science teachers don't purport to tell you what scientists actually do (history or sociology teachers might, I suppose, but they don't usually "make such a big deal" out of the scientific method). They purport to teach science. As you say - it's not a comforting lie, it's a command.
        SilasX 10y ago
        The "comforting lie" I was referring to was "this is how scientists actually do it and why you can trust the results".
        dTal 10y ago
        Erm, yes. We established that. And I assert that this is not what science teachers teach.
    - wycx 10y ago
      The way you get funded is by being able to do things that others can't do, or by being better at some technique than everyone else. Giving away all your hard won tricks by putting all of them in a methods section takes away your advantage come funding time.
      - FeepingCreature 10y ago
        So what you're saying is science is dead and we're going back to mysticism and alchemy.
      - michaelmrose 10y ago
        The problem is that it is far easier and cheaper to produce stuff that looks like science but isn't so if you fund stuff without being able to prove it/reproduce it then you will probably end up funding 90% bullshit while starving real science for money. Any real scientists ought to recognize that secrecy as a strategy is terrible for science as a whole even if it is very temporarily good for them.
      - bradjohnson 10y ago
        What's the point of funding something that won't help anyone because the author won't tell you how they did it? How is any conclusion that they've come to useful if nobody can verify if it's correct?
        2xlbuds 10y ago
        At that point it's just like funding magicians to come up with new tricks.
        notzorbo3 10y ago
        Who says the author(s) won't tell anybody? They only need to keep it secret from the competition, not the hand that feeds them.
        Too 10y ago
        This can definitely be the problem. I've seen studies where 50% of funding came from university and 50% from a private company. University required study to be published in their database, company wants the end result and don't want media to know what they are working with. So the report is obfuscated just enough to be legible for publication without giving away too much data from the company. Especially for final masters degree projects this is very common as the students don't get paid by the university at all, so many try to find a company to sponsor. But the students still need the uni to publish the report for them to get their final degree so you get this conflict of interest again. Most of these reports are just written with the end goal of getting a degree, not of creating solid research, this really needs the stricter universities not letting through all that crap, for now they shouldn't really be trusted the same way as proper research papers.
    - coldtea 10y ago
      Lots of insights/technologies you use today were produced by scientists working in such and even more secretive conditions, even with public funding involved (e.g. nobody at the Manhattan project would broadcast their discoveries to the world and same for many other fields, not necessarily war related).
  - yummyfajitas 10y ago
    That's not how science works. During my scientific career I gave code and data away to any and all who wanted them. That includes critics, competitors, and anyone else who was curious. I felt my results were solid enough (and that my skill at producing more results was good enough) that this wouldn't hurt me. This was just how things worked in my field (physics). Interestingly, my experimental colleagues rarely felt that they couldn't reproduce results they saw in a paper.
  - dTal 10y ago
    That might be how something works, but it's too kind to call it "science".
  - comicjk 10y ago
    That's an absurd degree of cynicism. I just finished my PhD in chemical engineering, and I did the reverse of this. In fact, I successfully migrated the younger members of my research group to a completely open-source software stack for chemical simulations, so that in principle anyone with a Linux box could reproduce our results. We published all our code and plain text versions of the data. I'm not saying you're wrong about the incentives - scientists are often incentivized toward secrecy - but I deny that we have to follow such incentives.
    - soVeryTired 10y ago
      It differs depending on the field. Biology has a reputation for being extremely competitive. I've heard of PIs telling their students what they're forbidden to reveal at conferences for fear of getting scooped. As a CS grad student it was a completely different story: I was delighted just to get someone who was willing to listen to me talk.
      - Fomite 10y ago
        It also differs based on study. Am I worried about getting scooped on the results of a major, multi-year cohort study that would take thousands of dollars to replicate? Nope. Am I worried about being scooped on the math modeling study that is perfectly reproducible based on two slides from a presentation? Hell yes.
      - briandear 10y ago
        I agree. I was on the team at Texas A&M that cloned the domestic cat -- (circa 2001) and secrecy was very important (notwithstanding the very real physical threats we had from anti-cloning nutcases.) Even the lab location was secret. The lab itself wasn't hidden, but nobody but us knew what was going on there. It was simply called "Reproductive Sciences." There was some very high incentives to be "first" -- which we were. Unfortunately for me, as an undergraduate research fellow, my name didn't make it into the Nature paper.. but wow what an experience!
    - wycx 10y ago
      I am writing a grant proposal right now. The success rate is ~19%. Personnel (i.e. publication record) of the team is weighted at 40% of the proposal. If you don't have an H-index of >50, there is no incentive to share with other that which gives you an advantage in the science section.
djsumdog 10y ago
Code/design/engineer are considerably different than say biology, because you can (and should) publish artefacts. The actual code/algorithms (not pseudo code) needs to be shared and archived with the journal. There has been some work for archiving code with some journals and some allow video uploads as well for segments from the actual experiments.

by ovis 10y ago

hannob 10y ago

> I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it." That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get away with such malicious behavior.

wycx 10y ago
That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...
- michaelmrose 10y ago
  At that point its actually not even science and shouldn't be funded
  - toss1941 10y ago
    I remember my teachers in high school making such a big deal about the scientific method on how important it was, how experiments must be reproducible to be useful, but today you barely hear it mentioned one way or another.
    - SilasX 10y ago
      Wait, you're telling me that schools teach comforting lies about society that don't match how the real world actually works?
      - dTal 10y ago
        It's no more a "comforting lie" than your driving instructor telling you to check your blind spot. Not everyone does it, and bad things happen as a result, but all the more important to teach it. I'm actually surprised to hear of high school teaching good scientific practice. I don't remember ever being taught that. Widely may it spread.
        SilasX 10y ago
        Hold on -- there's a category confusion here: "check your blindspots" isn't a comforting lie; it's a command. Converting it to "checking your blindspots will avoid collisions with careless drivers" would make it no longer a lie.
        dTal 10y ago
        I was pointing out the category confusion. Science teachers don't purport to tell you what scientists actually do (history or sociology teachers might, I suppose, but they don't usually "make such a big deal" out of the scientific method). They purport to teach science. As you say - it's not a comforting lie, it's a command.
        SilasX 10y ago
        The "comforting lie" I was referring to was "this is how scientists actually do it and why you can trust the results".
        dTal 10y ago
        Erm, yes. We established that. And I assert that this is not what science teachers teach.
  - wycx 10y ago
    The way you get funded is by being able to do things that others can't do, or by being better at some technique than everyone else. Giving away all your hard won tricks by putting all of them in a methods section takes away your advantage come funding time.
    - FeepingCreature 10y ago
      So what you're saying is science is dead and we're going back to mysticism and alchemy.
    - michaelmrose 10y ago
      The problem is that it is far easier and cheaper to produce stuff that looks like science but isn't so if you fund stuff without being able to prove it/reproduce it then you will probably end up funding 90% bullshit while starving real science for money. Any real scientists ought to recognize that secrecy as a strategy is terrible for science as a whole even if it is very temporarily good for them.
    - bradjohnson 10y ago
      What's the point of funding something that won't help anyone because the author won't tell you how they did it? How is any conclusion that they've come to useful if nobody can verify if it's correct?
      - 2xlbuds 10y ago
        At that point it's just like funding magicians to come up with new tricks.
      - notzorbo3 10y ago
        Who says the author(s) won't tell anybody? They only need to keep it secret from the competition, not the hand that feeds them.
        Too 10y ago
        This can definitely be the problem. I've seen studies where 50% of funding came from university and 50% from a private company. University required study to be published in their database, company wants the end result and don't want media to know what they are working with. So the report is obfuscated just enough to be legible for publication without giving away too much data from the company. Especially for final masters degree projects this is very common as the students don't get paid by the university at all, so many try to find a company to sponsor. But the students still need the uni to publish the report for them to get their final degree so you get this conflict of interest again. Most of these reports are just written with the end goal of getting a degree, not of creating solid research, this really needs the stricter universities not letting through all that crap, for now they shouldn't really be trusted the same way as proper research papers.
  - coldtea 10y ago
    Lots of insights/technologies you use today were produced by scientists working in such and even more secretive conditions, even with public funding involved (e.g. nobody at the Manhattan project would broadcast their discoveries to the world and same for many other fields, not necessarily war related).
- yummyfajitas 10y ago
  That's not how science works. During my scientific career I gave code and data away to any and all who wanted them. That includes critics, competitors, and anyone else who was curious. I felt my results were solid enough (and that my skill at producing more results was good enough) that this wouldn't hurt me. This was just how things worked in my field (physics). Interestingly, my experimental colleagues rarely felt that they couldn't reproduce results they saw in a paper.
- dTal 10y ago
  That might be how something works, but it's too kind to call it "science".
- comicjk 10y ago
  That's an absurd degree of cynicism. I just finished my PhD in chemical engineering, and I did the reverse of this. In fact, I successfully migrated the younger members of my research group to a completely open-source software stack for chemical simulations, so that in principle anyone with a Linux box could reproduce our results. We published all our code and plain text versions of the data. I'm not saying you're wrong about the incentives - scientists are often incentivized toward secrecy - but I deny that we have to follow such incentives.
  - soVeryTired 10y ago
    It differs depending on the field. Biology has a reputation for being extremely competitive. I've heard of PIs telling their students what they're forbidden to reveal at conferences for fear of getting scooped. As a CS grad student it was a completely different story: I was delighted just to get someone who was willing to listen to me talk.
    - Fomite 10y ago
      It also differs based on study. Am I worried about getting scooped on the results of a major, multi-year cohort study that would take thousands of dollars to replicate? Nope. Am I worried about being scooped on the math modeling study that is perfectly reproducible based on two slides from a presentation? Hell yes.
    - briandear 10y ago
      I agree. I was on the team at Texas A&M that cloned the domestic cat -- (circa 2001) and secrecy was very important (notwithstanding the very real physical threats we had from anti-cloning nutcases.) Even the lab location was secret. The lab itself wasn't hidden, but nobody but us knew what was going on there. It was simply called "Reproductive Sciences." There was some very high incentives to be "first" -- which we were. Unfortunately for me, as an undergraduate research fellow, my name didn't make it into the Nature paper.. but wow what an experience!
  - wycx 10y ago
    I am writing a grant proposal right now. The success rate is ~19%. Personnel (i.e. publication record) of the team is weighted at 40% of the proposal. If you don't have an H-index of >50, there is no incentive to share with other that which gives you an advantage in the science section.

djsumdog 10y ago

Code/design/engineer are considerably different than say biology, because you can (and should) publish artefacts. The actual code/algorithms (not pseudo code) needs to be shared and archived with the journal. There has been some work for archiving code with some journals and some allow video uploads as well for segments from the actual experiments.