5 ms·

As a complete outsider reading the attitudes behind the original scientists, it seems to me that they resent the oversight and hate to do extra work. In defend

As a complete outsider reading the attitudes behind the original scientists, it seems to me that they resent the oversight and hate to do extra work. In defending their practices they fall back on "expert work" and essentially are arguing that what they are doing is too complex for anyone else to do and they should be left alone to continue to do it. And from their point of view, it seems all very reasonable. But from the rest of humanity who is being asked to materially support them, and waits for their conclusions to make the world a better place, it seems ... frankly... lazy and selfish. 30 emails, wow! 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even? The demands on their time seem so low, yet the complaints are so high, that one can't help but wonder if the concern really is that their results are too 'magical' and irreproducible and they just fear other people learning about it. I've seen this behavior in professional settings, and ultimately it comes down to a lack of confidence in oneself, the tools and technology and the quality of work being done. Careers are at stake, but is the alternative to just give people a free pass?

nmrm2 11y ago
Sorry, but how is this not just anti-intellectualism? > 30 emails, wow! It can take up to half a day to reply to an email containing even one fairly technical question. Obviously the time invested depends on the content of those 30 emails, but it's not at all difficult to imagine this eating a week worth of a PI's time. > 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even? The typical attitude is that this is a justification for not making them spend their time managing these sorts of tasks. You don't get to say "we get to pay you crap because your job is exciting and rewarding" and then load that same person down with a bunch of grunt work (on top of all the normal grunt work). That's how you bleed students and kill the lab's productivity. Also, grad students are not well paid but they are typically no cheaper than post docs; PIs pay tuition. > The demands on their time seem so low, yet the complaints are so high So let's imagine a world where people calling for massive improvements to reproducability get their way. Let's say it is a month worth of time for the lab. For each paper. Multiple papers a year. That's a pretty massive time investment. If you're a top lab, that could become a full-time position. And believe me, you're not going to be able to fill that position with a grad student. That person will have to be well-paid, because their job is going to suck. So it's reasonable that scientists are peeved when they invest all this time and don't perceive their collaborators as acting in good faith, or feel like their collaborators are trying to cut corners to pinch pennies.
- drcode 11y ago
  If an engineer builds the worlds greatest new engine but says "unfortunately it'll only run in my lab, no one else is competent enough to run it or build a copy" then what good is it to society? If the researchers in a lab are such geniuses that they are doing experiments almost nobody else can duplicate and it is therefore impossible to determine the veracity of their claims, how is that helping society and why should society fund them? Isn't the onus on the researchers to focus experiments that are also reproducible by non-supergeniuses?
  - epistasis 11y ago
    Science is not engineering. In engineering, we understand the problem domain enough to make everything reproducible, the governing principles are well enough understood that reasoning from first principles can usually come up with answers. In contrast, in science, we are trying to discover those first principles. So if something doesn't replicate in another lab, we have a responsibility to check it out, but that doesn't mean that the scientist has to discover all the first principles before making a publication! The non-replication could be due to new, solid, first principles, and the assumption that it should be "reproducible" exactly from the English words on a sheet of paper are a faulty assumption. If you want to be able to pick up the scientific literature and read it like a science text book, you're doing it wrong. That is not the purpose of publishing papers.
  - nmrm2 11y ago
    > If an engineer builds the worlds greatest new engine There is more to science than advanced product development. Conflating the two is wrong-headed. > but says "unfortunately it'll only run in my lab, no one else is competent enough to run it or build a copy" then what good is it to society? It's fantastically good to society. A company interested in monetizing the research could provide that researcher with a multi-year sabbatical to come to their company and turn is Research into a Product. Incidentally, that happens. It's also not unheard of for phd students to carry an adviser's idea forward toward application in the context of a permanent position at a relevant company. > almost nobody else can duplicate There's a difference between not being able to duplicate, and duplication being expensive. > and it is therefore impossible to determine the veracity of their claims Again, reproducibility should focus on the veracity of the claims, not the economics of reproducing them. Nothing is wrong with calling for better reproducibility. The problem is in expecting to get it for free, and assuming that it's always appropriate at every stage of research. Investment in reproducibility should be in proportion to the degree of trust the scientific community puts in the claim, and it is absolutely reasonable for that investment to grow over time. But, don't kid yourself, it's an investment. And society would have to be stupid to invest enormous amounts of money into ensuring every single scientific paper ever published is held to an extremely strong standard for reproducibility. > Isn't the onus on the researchers to focus experiments that are also reproducible by non-supergeniuses? If by "super genius" you mean "someone else who does research in the same or a closely related field", then Hell. No. The onus on the scientists is to focus on experiments that push science forward in service of mankind. Sometimes this means helping mega-corps figure out how to reliably reproduce your research without expert help and thereby increase profit by decreasing required investment. Sometimes this means focusing on discovery.
coliveira 11y ago
The real answer to this problem is to invest more money on research. If there are more people looking at a particular problem, there is less and less chance that a big mistake will be hiding. Many have the wishful thinking that if we were just more careful with procedures or software or technology or something else, then the results would be more robust. Unfortunately this doesn't work when you have fixed resources to perform extremely complex research. The power of science is not on individual results (papers), but on how these results confirm or refute each other over time.

nmrm2 11y ago

Sorry, but how is this not just anti-intellectualism? > 30 emails, wow! It can take up to half a day to reply to an email containing even one fairly technical question. Obviously the time invested depends on the content of those 30 emails, but it's not at all difficult to imagine this eating a week worth of a PI's time. > 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even? The typical attitude is that this is a justification for not making them spend their time managing these sorts of tasks. You don't get to say "we get to pay you crap because your job is exciting and rewarding" and then load that same person down with a bunch of grunt work (on top of all the normal grunt work). That's how you bleed students and kill the lab's productivity. Also, grad students are not well paid but they are typically no cheaper than post docs; PIs pay tuition. > The demands on their time seem so low, yet the complaints are so high So let's imagine a world where people calling for massive improvements to reproducability get their way. Let's say it is a month worth of time for the lab. For each paper. Multiple papers a year. That's a pretty massive time investment. If you're a top lab, that could become a full-time position. And believe me, you're not going to be able to fill that position with a grad student. That person will have to be well-paid, because their job is going to suck. So it's reasonable that scientists are peeved when they invest all this time and don't perceive their collaborators as acting in good faith, or feel like their collaborators are trying to cut corners to pinch pennies.

drcode 11y ago
If an engineer builds the worlds greatest new engine but says "unfortunately it'll only run in my lab, no one else is competent enough to run it or build a copy" then what good is it to society? If the researchers in a lab are such geniuses that they are doing experiments almost nobody else can duplicate and it is therefore impossible to determine the veracity of their claims, how is that helping society and why should society fund them? Isn't the onus on the researchers to focus experiments that are also reproducible by non-supergeniuses?
- epistasis 11y ago
  Science is not engineering. In engineering, we understand the problem domain enough to make everything reproducible, the governing principles are well enough understood that reasoning from first principles can usually come up with answers. In contrast, in science, we are trying to discover those first principles. So if something doesn't replicate in another lab, we have a responsibility to check it out, but that doesn't mean that the scientist has to discover all the first principles before making a publication! The non-replication could be due to new, solid, first principles, and the assumption that it should be "reproducible" exactly from the English words on a sheet of paper are a faulty assumption. If you want to be able to pick up the scientific literature and read it like a science text book, you're doing it wrong. That is not the purpose of publishing papers.
- nmrm2 11y ago
  > If an engineer builds the worlds greatest new engine There is more to science than advanced product development. Conflating the two is wrong-headed. > but says "unfortunately it'll only run in my lab, no one else is competent enough to run it or build a copy" then what good is it to society? It's fantastically good to society. A company interested in monetizing the research could provide that researcher with a multi-year sabbatical to come to their company and turn is Research into a Product. Incidentally, that happens. It's also not unheard of for phd students to carry an adviser's idea forward toward application in the context of a permanent position at a relevant company. > almost nobody else can duplicate There's a difference between not being able to duplicate, and duplication being expensive. > and it is therefore impossible to determine the veracity of their claims Again, reproducibility should focus on the veracity of the claims, not the economics of reproducing them. Nothing is wrong with calling for better reproducibility. The problem is in expecting to get it for free, and assuming that it's always appropriate at every stage of research. Investment in reproducibility should be in proportion to the degree of trust the scientific community puts in the claim, and it is absolutely reasonable for that investment to grow over time. But, don't kid yourself, it's an investment. And society would have to be stupid to invest enormous amounts of money into ensuring every single scientific paper ever published is held to an extremely strong standard for reproducibility. > Isn't the onus on the researchers to focus experiments that are also reproducible by non-supergeniuses? If by "super genius" you mean "someone else who does research in the same or a closely related field", then Hell. No. The onus on the scientists is to focus on experiments that push science forward in service of mankind. Sometimes this means helping mega-corps figure out how to reliably reproduce your research without expert help and thereby increase profit by decreasing required investment. Sometimes this means focusing on discovery.

coliveira 11y ago

The real answer to this problem is to invest more money on research. If there are more people looking at a particular problem, there is less and less chance that a big mistake will be hiding. Many have the wishful thinking that if we were just more careful with procedures or software or technology or something else, then the results would be more robust. Unfortunately this doesn't work when you have fixed resources to perform extremely complex research. The power of science is not on individual results (papers), but on how these results confirm or refute each other over time.