25 ms·

Depends on the topic. If you're trying to prove (or, more accurately, fail to falsify) a certain set of results or a discovery, open data plus reproducibility s

Depends on the topic. If you're trying to prove (or, more accurately, fail to falsify) a certain set of results or a discovery, open data plus reproducibility should be the gold standard. If you're trying to make decisions about personal health, it pays to take a cautious approach (IMO) and let people decide for themselves. It is not inconceivable that commercial entities finance "peer-reviewed research" in order to support public health policies.

xpe 3y ago
A small but important reframing: I agree that (open data) + (reproducibility) + (review) is a better bar than any one in isolation. (This applies to empirical sciences only. In contrast, for a mathematical paper, rigorous reviews of the formal logic are the gold-standard.) I choose the phrase 'review' intentionally; it doesn't need to follow one of the typical [1] 'peer review' processes from a journal. There are likely alternative methods that are (at least as) or (more) effective. Some fields have seen amazing contributions from 'lay people' all across the world. > If you're trying to make decisions about personal health, it pays to take a cautious approach (IMO) and let people decide for themselves. For a person equipped with the time, skill, rationality, and motivation, I agree. But it is an empirical question to assess how many people have these skills. I also want to push back against what might be an individualistic bias [2] in the statement above. Even if one is an individualist, it isn't optimally efficient nor practical for an individual to 'take on' all this responsibility. It is rational and efficient to place some degree of trust in others. [1] These processes vary and should be treated with a fair bit of skepticism. I want to to dig in more; e.g. to what degree have these processes been designed to account for human realities so as to maximize quality? [2]: See also https://so2020.isosonline.org/conference/the-individualist-bias/ https://so2020.isosonline.org/conference/the-individualist-b... "Individualism claims that social entities (e.g. groups, institutions) are essentially nothing but suitably arranged aggregates of individuals. This is usually tried to be demonstrated through developing accounts of reduction or supervenience. Holism, on the other hand, claims that social entities are sui generis as something over and above individuals."
olyjohn 3y ago
I'm confused. I thought that reproducing a study was pretty much doing a peer review?
- rgrieselhuber 3y ago
  That is the way it was sold but peer review fraud is a major problem.
  - manonthewall 3y ago
    Do you have a peer reviewed source on that? Major usually implies 50%+ . Are you saying that 50%+ of the peer reviewed scientific papers out there bogus and fraudulent?
    - rgrieselhuber 3y ago
      I don’t think 50% or greater is a good standard for defining what we would consider to be a major problem with scientific fraud. I’m sure you’re well aware that “major” can refer to relative impact depending on the context of the issue being discussed. In the case of scientific publishing, I would argue that even 5% fraud rate would be grounds for labeling this as a major problem. That said, this issue has received a lot of attention in recent years. Some articles try to put a positive spin on it but the stark truth of where we are today is that there are no standards enforced requiring data publishing (“available on request” doesn’t cut it) and reproducibility. This, combined with the amount of money in this industry and the impact that it has on public policy is an acceptable situation. Some good coverage: https://www.experimental-history.com/p/the-rise-and-fall-of-peer-review https://www.experimental-history.com/p/the-rise-and-fall-of-... (Lots of links to follow on this above post) https://retractionwatch.com/ https://retractionwatch.com/ https://www.science.org/content/article/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty https://www.science.org/content/article/what-massive-databas... (This one puts a positive spin on the editorial process but the fact remains that there is a massive flood of fraudulent papers being published. Going back to standards of open data plus verified reproducibility would go a long way to mitigate this problem.
- 3y ago

xpe 3y ago

A small but important reframing: I agree that (open data) + (reproducibility) + (review) is a better bar than any one in isolation. (This applies to empirical sciences only. In contrast, for a mathematical paper, rigorous reviews of the formal logic are the gold-standard.) I choose the phrase 'review' intentionally; it doesn't need to follow one of the typical [1] 'peer review' processes from a journal. There are likely alternative methods that are (at least as) or (more) effective. Some fields have seen amazing contributions from 'lay people' all across the world. > If you're trying to make decisions about personal health, it pays to take a cautious approach (IMO) and let people decide for themselves. For a person equipped with the time, skill, rationality, and motivation, I agree. But it is an empirical question to assess how many people have these skills. I also want to push back against what might be an individualistic bias [2] in the statement above. Even if one is an individualist, it isn't optimally efficient nor practical for an individual to 'take on' all this responsibility. It is rational and efficient to place some degree of trust in others. [1] These processes vary and should be treated with a fair bit of skepticism. I want to to dig in more; e.g. to what degree have these processes been designed to account for human realities so as to maximize quality? [2]: See also https://so2020.isosonline.org/conference/the-individualist-bias/ https://so2020.isosonline.org/conference/the-individualist-b... "Individualism claims that social entities (e.g. groups, institutions) are essentially nothing but suitably arranged aggregates of individuals. This is usually tried to be demonstrated through developing accounts of reduction or supervenience. Holism, on the other hand, claims that social entities are sui generis as something over and above individuals."

olyjohn 3y ago

I'm confused. I thought that reproducing a study was pretty much doing a peer review?

rgrieselhuber 3y ago
That is the way it was sold but peer review fraud is a major problem.
- manonthewall 3y ago
  Do you have a peer reviewed source on that? Major usually implies 50%+ . Are you saying that 50%+ of the peer reviewed scientific papers out there bogus and fraudulent?
  - rgrieselhuber 3y ago
    I don’t think 50% or greater is a good standard for defining what we would consider to be a major problem with scientific fraud. I’m sure you’re well aware that “major” can refer to relative impact depending on the context of the issue being discussed. In the case of scientific publishing, I would argue that even 5% fraud rate would be grounds for labeling this as a major problem. That said, this issue has received a lot of attention in recent years. Some articles try to put a positive spin on it but the stark truth of where we are today is that there are no standards enforced requiring data publishing (“available on request” doesn’t cut it) and reproducibility. This, combined with the amount of money in this industry and the impact that it has on public policy is an acceptable situation. Some good coverage: https://www.experimental-history.com/p/the-rise-and-fall-of-peer-review https://www.experimental-history.com/p/the-rise-and-fall-of-... (Lots of links to follow on this above post) https://retractionwatch.com/ https://retractionwatch.com/ https://www.science.org/content/article/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty https://www.science.org/content/article/what-massive-databas... (This one puts a positive spin on the editorial process but the fact remains that there is a massive flood of fraudulent papers being published. Going back to standards of open data plus verified reproducibility would go a long way to mitigate this problem.
3y ago