5 ms·

> an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud

by davemp 8y ago

> an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.

BurritoAlPastor 8y ago
Social sciences have pretty deep-seated logistical obstacles to reproducibility. If you want to reproduce an archaeology paper, well, how? You can’t exactly go dig the same site back up. If you go out as a cultural anthropologist to hang out with some indigenous tribe somewhere, and your experiences aren’t the same as the last guy, does that mean anything? How much is because you’re different people? How much is because the tribe has already gotten used to having a researcher around? This is part of why shit like the Piltdown Man happens.
mooman219 8y ago
Exactly this. If my banking API relies on user's not trying to commit fraud instead of proper validation, then I have a very insecure API. Making a statement that your system works as long as no one tries to break it sounds almost satirical.
- amw 8y ago
  The result of successful banking API fraud is "I get a lot of money". The result of successful science publication fraud is, "I get some papers published, and as long as I don't get too famous and people don't look too hard at replicating my results later, maybe I get to include them in a bibliography or resume somewhere".
TheCoelacanth 8y ago
Reproduction of results happens after publication, though. Simply getting published does not demonstrate that their ideas were accepted as truthful, you should look at things like whether their work was cited by other work.
amw 8y ago
It's supposed to rely on not-fraud because its incentive structure is not supposed to reward gaming the system (the way that "you get money" incentivizes fraud in a market system). A certain amount of good faith is presumed because the people who are eventually found retracted en mass will have a rough time getting work in the future, and because they won't be any closer to getting work published that will make them well-known scientists in the future.
tptacek 8y ago
Sussing out invalid research is the purpose of replication. A replication is itself a research project. It's not something that peer reviewers do.
- davemp 8y ago
  Yes. My comment implies that you can get invalid research published. At least it was intended to. The problem lies in areas where replication and formalization is difficult. There is certainly knowledge to be gained in these areas. Though calling the fields sciences, when the scientific method cannot be effectively applied, seems dubious. The solution is not obvious, but it is clear that steps should be taken to improve rigor in the less formal (straight forward) bodies of scholarship.
  - foldr 8y ago
    It isn't possible to attempt replication of every scientific paper that's published. A lot of people on this thread seem to have difficulty accepting this fact, but the time and money to do that simply isn't available. If a large percentage of scientists were to engage in deliberate fraud, there wouldn't be any feasible way to regulate that problem away. All available resources would end up being spent on scientists investigating each other for fraud, and no actual science would get done. Bear in mind that in many cases, establishing that an experiment is not fraudulent could require more time and effort than the original experiment! As with many other human institutions, we have no choice but to rely on most people being honest most of the time.

by davemp 8y ago

BurritoAlPastor 8y ago

Social sciences have pretty deep-seated logistical obstacles to reproducibility. If you want to reproduce an archaeology paper, well, how? You can’t exactly go dig the same site back up. If you go out as a cultural anthropologist to hang out with some indigenous tribe somewhere, and your experiences aren’t the same as the last guy, does that mean anything? How much is because you’re different people? How much is because the tribe has already gotten used to having a researcher around? This is part of why shit like the Piltdown Man happens.

mooman219 8y ago

Exactly this. If my banking API relies on user's not trying to commit fraud instead of proper validation, then I have a very insecure API. Making a statement that your system works as long as no one tries to break it sounds almost satirical.

amw 8y ago
The result of successful banking API fraud is "I get a lot of money". The result of successful science publication fraud is, "I get some papers published, and as long as I don't get too famous and people don't look too hard at replicating my results later, maybe I get to include them in a bibliography or resume somewhere".

TheCoelacanth 8y ago

Reproduction of results happens after publication, though. Simply getting published does not demonstrate that their ideas were accepted as truthful, you should look at things like whether their work was cited by other work.

amw 8y ago

It's supposed to rely on not-fraud because its incentive structure is not supposed to reward gaming the system (the way that "you get money" incentivizes fraud in a market system). A certain amount of good faith is presumed because the people who are eventually found retracted en mass will have a rough time getting work in the future, and because they won't be any closer to getting work published that will make them well-known scientists in the future.

tptacek 8y ago

Sussing out invalid research is the purpose of replication. A replication is itself a research project. It's not something that peer reviewers do.

davemp 8y ago
Yes. My comment implies that you can get invalid research published. At least it was intended to. The problem lies in areas where replication and formalization is difficult. There is certainly knowledge to be gained in these areas. Though calling the fields sciences, when the scientific method cannot be effectively applied, seems dubious. The solution is not obvious, but it is clear that steps should be taken to improve rigor in the less formal (straight forward) bodies of scholarship.
- foldr 8y ago
  It isn't possible to attempt replication of every scientific paper that's published. A lot of people on this thread seem to have difficulty accepting this fact, but the time and money to do that simply isn't available. If a large percentage of scientists were to engage in deliberate fraud, there wouldn't be any feasible way to regulate that problem away. All available resources would end up being spent on scientists investigating each other for fraud, and no actual science would get done. Bear in mind that in many cases, establishing that an experiment is not fraudulent could require more time and effort than the original experiment! As with many other human institutions, we have no choice but to rely on most people being honest most of the time.