6 ms·

I'm glad you stood your ground. But what I don't get is the mindset of everyone who wanted to publish the potentially-faulty data. The moment anyone tries to bu

I'm glad you stood your ground. But what I don't get is the mindset of everyone who wanted to publish the potentially-faulty data. The moment anyone tries to build on the result, its faultiness will be known. Is that not more embarrassing?

danieltillett 11y ago
The reason is pressure to get the papers out as fast as possible or else you won’t get any more grants. It should be embarrassing to publish garbage, but for lots of people who have succeeded in the current system it appears not to be. Everyone involved was rather sheepish afterwards except the post doc. He had felt much the same as me, but didn’t feel he could raise the issue with his boss. Privately he was very grateful I had stood up to his boss, as he didn’t feel he could do the same and was under enormous pressure to pump out the data. All round not good.
- 88e282102ae2e5b 11y ago
  Sometimes I wonder if I'm in a bubble. My PI thanked me when I told him I had discovered that out our data was systematically biased and it would take weeks to correct the issue.
  - danieltillett 11y ago
    Yes most PI’s are happy to avoid the embarrassment and only want to publish good data. The problem is not that most scientists are not trying to do the right thing, but that they are under enormous pressure to pump out the results. It only takes a few falling to this pressure to destroy the public’s faith in science - the consequence of this are catastrophic. We must solve this problem or we will not have science.
    - netcan 11y ago
      I know very little about academia. But, in the context of today's world of information and collaboration, it always seems to me that it should be a better time than any to (A) have a better than publication and (B) put a lot more into reproducing results independently. I mean what is the whole publication, review, reproduction system if not an early system of crowdsourcing. To take a simplistic example: a publication norm which includes instructions for reproduction, ideally requiring as little resources as possible.
      - 88e282102ae2e5b 11y ago
        Instructions for reproduction are mandatory already. That said, you probably can't pick up any paper and reproduce the experiments exactly without asking the authors a question or two, at least in biology. Systems are getting increasingly complex as well. In my lab, it literally takes several weeks for people to learn how to do our assay, and that's with constant feedback from an experienced user. Every step is published but that still doesn't help when things don't go according to plan. Also, not every lab has the same equipment. Even if we provided free training, a lab would still need to drop $300k to get all the machines required, which are not common. Not every result is worth reproducing either. If someone publishes a paper that shows several lines of evidence for the same thing and they've done all reasonable controls, and it doesn't disagree with any existing models, why would you reproduce that? Outside of deliberate fraud, it's a pretty solid bet that it's true, and you can save enormous amounts of time and money by proceeding to build off of it instead of reproducing it first. And that's really what it comes down to: no one's paying you to do this kind of work. Governments would have to fund it, because you're fundamentally asking for twice as much work to be done, which will cost twice as much money (and time). But take heart: when someone DOES make an outrageous claim, it's very common for labs to try to reproduce it (or really, disprove it). See [1] and [2] for recent examples. [1] "STAP stem cell controversy ends in suicide for Japanese scientist" http://www.latimes.com/science/sciencenow/la-sci-sn-stap-stem-cell-suicide-20140805-story.html http://www.latimes.com/science/sciencenow/la-sci-sn-stap-ste... [2] "Water bears’ genetic borrowing questioned" https://www.sciencenews.org/article/water-bears%E2%80%99-genetic-borrowing-questioned https://www.sciencenews.org/article/water-bears%E2%80%99-gen...
ealloc 11y ago
It's rare for bad results to get called out. The usual fate of bad research is a fade into obscurity. Publishing questionable results quickly has many benefits (if you happen to be right) and few consequences. Just don't draw too much attention to yourself by over-hyping.
- danieltillett 11y ago
  Yep if I had not put my foot down it probably would have slipped by and never been noticed until some poor unfortunate PhD student tried to build their research off it. Another anecdote - a certain famous professor (since dead) at my alma mater managed to destroy at least three PhD students careers via fraud. He had faked up the initial data and then put them to work on projects based on the fake data. The whole thing only came to light (internal only as it was all hushed up) when he died and his students moved to a new supervisor.

danieltillett 11y ago

The reason is pressure to get the papers out as fast as possible or else you won’t get any more grants. It should be embarrassing to publish garbage, but for lots of people who have succeeded in the current system it appears not to be. Everyone involved was rather sheepish afterwards except the post doc. He had felt much the same as me, but didn’t feel he could raise the issue with his boss. Privately he was very grateful I had stood up to his boss, as he didn’t feel he could do the same and was under enormous pressure to pump out the data. All round not good.

88e282102ae2e5b 11y ago
Sometimes I wonder if I'm in a bubble. My PI thanked me when I told him I had discovered that out our data was systematically biased and it would take weeks to correct the issue.
- danieltillett 11y ago
  Yes most PI’s are happy to avoid the embarrassment and only want to publish good data. The problem is not that most scientists are not trying to do the right thing, but that they are under enormous pressure to pump out the results. It only takes a few falling to this pressure to destroy the public’s faith in science - the consequence of this are catastrophic. We must solve this problem or we will not have science.
  - netcan 11y ago
    I know very little about academia. But, in the context of today's world of information and collaboration, it always seems to me that it should be a better time than any to (A) have a better than publication and (B) put a lot more into reproducing results independently. I mean what is the whole publication, review, reproduction system if not an early system of crowdsourcing. To take a simplistic example: a publication norm which includes instructions for reproduction, ideally requiring as little resources as possible.
    - 88e282102ae2e5b 11y ago
      Instructions for reproduction are mandatory already. That said, you probably can't pick up any paper and reproduce the experiments exactly without asking the authors a question or two, at least in biology. Systems are getting increasingly complex as well. In my lab, it literally takes several weeks for people to learn how to do our assay, and that's with constant feedback from an experienced user. Every step is published but that still doesn't help when things don't go according to plan. Also, not every lab has the same equipment. Even if we provided free training, a lab would still need to drop $300k to get all the machines required, which are not common. Not every result is worth reproducing either. If someone publishes a paper that shows several lines of evidence for the same thing and they've done all reasonable controls, and it doesn't disagree with any existing models, why would you reproduce that? Outside of deliberate fraud, it's a pretty solid bet that it's true, and you can save enormous amounts of time and money by proceeding to build off of it instead of reproducing it first. And that's really what it comes down to: no one's paying you to do this kind of work. Governments would have to fund it, because you're fundamentally asking for twice as much work to be done, which will cost twice as much money (and time). But take heart: when someone DOES make an outrageous claim, it's very common for labs to try to reproduce it (or really, disprove it). See [1] and [2] for recent examples. [1] "STAP stem cell controversy ends in suicide for Japanese scientist" http://www.latimes.com/science/sciencenow/la-sci-sn-stap-stem-cell-suicide-20140805-story.html http://www.latimes.com/science/sciencenow/la-sci-sn-stap-ste... [2] "Water bears’ genetic borrowing questioned" https://www.sciencenews.org/article/water-bears%E2%80%99-genetic-borrowing-questioned https://www.sciencenews.org/article/water-bears%E2%80%99-gen...

ealloc 11y ago

It's rare for bad results to get called out. The usual fate of bad research is a fade into obscurity. Publishing questionable results quickly has many benefits (if you happen to be right) and few consequences. Just don't draw too much attention to yourself by over-hyping.

danieltillett 11y ago
Yep if I had not put my foot down it probably would have slipped by and never been noticed until some poor unfortunate PhD student tried to build their research off it. Another anecdote - a certain famous professor (since dead) at my alma mater managed to destroy at least three PhD students careers via fraud. He had faked up the initial data and then put them to work on projects based on the fake data. The whole thing only came to light (internal only as it was all hushed up) when he died and his students moved to a new supervisor.