3 ms·

I agree with this - but so long as it's not used as an excuse when a problem arises.

by sgt101 3y ago

matthewdgreen 3y ago
The challenge with malicious team-members in science is the same one you face in software. As another person in the thread said, how do you safely develop programs when one contributor is adding “Underhanded C contest”-level backdoors to the code? You can architect your development processes to defend against this threat, but in practice very few software development teams do this because it devours limited resources. Resources in science are even more limited than those in for-profit software development firms, and often involve taxpayer funds. The instinct on HN is to approach this as an angry blame game. But successful industries don’t do this: for example, the aviation industry performs root-cause analysis after each air disaster and then makes intelligent cost/benefit decisions about how to update processes to avoid future occurrences. In science it’s not clear what the right approach is on a cost/benefit level. Should we reduce scientific output by 50% to catch fraud in the first instance, rather than later on when important studies fail to replicate? Is that a good use of society’s limited resources? I don’t know the answer to this, I just want people to behave effectively rather than grinding an axe. We’re not getting out of the messes we’ve made on this planet without a whole lot of science.
- whatyesaid 3y ago
  This is orthogonal to the reproducibility crisis, but not really part of it. When you claim a breakthrough even just within a sub-field, if you simply invented an elaborate lie, it's always gonna be caught it's just a waste of everyone's time in the end. This person is lucky she and her lab caught it themselves before other people started to work further on it. And I guess the other lab members were already trying to reproduce it, as you would. That guy who faked all this can't really be thought of as a sophisticated bad actor, neither was he really gaming the system because it never would have worked. He was just leaping into his own pit. Research needs to first move to being fully open access and end of tens of millions of taxpayer dollars going to journals. This is something tangible you can legislate. Then, you need to incentivize high-quality research; that's not easy to do at all, but you can do things if you have hiring power within your field I guess...
  - sgt101 3y ago
    We also need to architect the processes to cope with the preprint culture. There's a lot being claimed without peer-review, and a lot of it is bullshit and wastes my time...

I agree with this - but so long as it's not used as an excuse when a problem arises.

by sgt101 3y ago

I agree with this - but so long as it's not used as an excuse when a problem arises.

The challenge with malicious team-members in science is the same one you face in software. As another person in the thread said, how do you safely develop programs when one contributor is adding “Underhanded C contest”-level backdoors to the code? You can architect your development processes to defend against this threat, but in practice very few software development teams do this because it devours limited resources. Resources in science are even more limited than those in for-profit software development firms, and often involve taxpayer funds. The instinct on HN is to approach this as an angry blame game. But successful industries don’t do this: for example, the aviation industry performs root-cause analysis after each air disaster and then makes intelligent cost/benefit decisions about how to update processes to avoid future occurrences. In science it’s not clear what the right approach is on a cost/benefit level. Should we reduce scientific output by 50% to catch fraud in the first instance, rather than later on when important studies fail to replicate? Is that a good use of society’s limited resources? I don’t know the answer to this, I just want people to behave effectively rather than grinding an axe. We’re not getting out of the messes we’ve made on this planet without a whole lot of science.

whatyesaid 3y ago
This is orthogonal to the reproducibility crisis, but not really part of it. When you claim a breakthrough even just within a sub-field, if you simply invented an elaborate lie, it's always gonna be caught it's just a waste of everyone's time in the end. This person is lucky she and her lab caught it themselves before other people started to work further on it. And I guess the other lab members were already trying to reproduce it, as you would. That guy who faked all this can't really be thought of as a sophisticated bad actor, neither was he really gaming the system because it never would have worked. He was just leaping into his own pit. Research needs to first move to being fully open access and end of tens of millions of taxpayer dollars going to journals. This is something tangible you can legislate. Then, you need to incentivize high-quality research; that's not easy to do at all, but you can do things if you have hiring power within your field I guess...
- sgt101 3y ago
  We also need to architect the processes to cope with the preprint culture. There's a lot being claimed without peer-review, and a lot of it is bullshit and wastes my time...