4 ms·

Does anybody still believe in the peer reviewed journal system? After all these years and the countless stories of passing off fake information? We know people

by smeej 3y ago

Does anybody still believe in the peer reviewed journal system? After all these years and the countless stories of passing off fake information? We know people make up their data. We know peer reviewers sign off on it without actually looking it over. And we know the journals publish it anyway, and make a lot of money off of it. Why does anybody even care if we've added another layer of fabrication to the mix of something that's already so full of lies?

ulrashida 3y ago
This feels very dismissive of the efforts put in by genuine participants into the system. Exceptions and examples of malfeasance are not sufficient to indict all aspects.
- coolhand2120 3y ago
  What happens when fake/fraud/whatever is > 50% as it is in some domains of science? https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis I believe something that is not falsifiable is not science. So what would failure look like to you if it isn’t “> 50% off science is fake”? What do you call failure of this system?
JohnFen 3y ago
> Does anybody still believe in the peer reviewed journal system? Sure, why not? Yes, it's imperfect and BS can get through. But on the whole, it works reasonably well.
- breakwaterlabs 3y ago
  A system that generates plausible, seemingly authoritative information, but often makes hard to detect errors ranging from minor to outright lies is dangeorous. This goes double when the information is either difficult or impossible to verify. This shouldn't be surprising, since the most effective and dangerous liars tell the truth most of the time.
  - JohnFen 3y ago
    > but often makes hard to detect errors ranging from minor to outright lies is dangeorous. Oh, I agree. However, the peer-review system does mitigate this problem to a useful degree. Without such a system, it would be impossible to have any degree of trust in any paper at all. > What we're really finding out is that journals are often little better than LLMs as information sources. It depends on the journal. There are absolutely crap ones out there that need to be ignored. But there are also good ones out there that have earned their reputation. Even there, BS can get through of course -- but to say that such journals are mostly unreliable is seriously overstating the issue.
    - smeej 3y ago
      I've yet to see studies saying this isn't pervasive across journals. The vast majority are apparently affected. The idea of peer-reviewed journals is great. The reality is a circle jerk.
  - xboxnolifes 3y ago
    Reading un-replicated papers as authoritative accounts is definitely a problem. Solution: don't do that.
    - breakwaterlabs 3y ago
      What percentage of papers in your average, reputable journal have been replicated? And how can one easily determine, while looking at a particular paper, whether it has been replicated? And whether those doing the replication have any undisclosed ties to the original? At an epistemological level, the idea of a knowledge source like a journal where the information is only deemed reliable if personally verified seems problematic. Why even have it if all of its uncountable claims are indistinguishable from very clever lies, and attempts to quantify the extent of those lies indicate that they are pervasive?
- coolhand2120 3y ago
  With this system, I’m many domains, a -majority- of published research is not reproducible. Possibly worse than a coin flip. Do you really think this is “reasonably well”. https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis
booleandilemma 3y ago
I still get sick even though I wash my hands. Let me stop washing my hands.
- smeej 3y ago
  Not to take this too far, but, unless you're a surgeon or a childcare worker, it probably isn't making much difference.

by smeej 3y ago

ulrashida 3y ago

This feels very dismissive of the efforts put in by genuine participants into the system. Exceptions and examples of malfeasance are not sufficient to indict all aspects.

coolhand2120 3y ago
What happens when fake/fraud/whatever is > 50% as it is in some domains of science? https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis I believe something that is not falsifiable is not science. So what would failure look like to you if it isn’t “> 50% off science is fake”? What do you call failure of this system?

JohnFen 3y ago

> Does anybody still believe in the peer reviewed journal system? Sure, why not? Yes, it's imperfect and BS can get through. But on the whole, it works reasonably well.

breakwaterlabs 3y ago
A system that generates plausible, seemingly authoritative information, but often makes hard to detect errors ranging from minor to outright lies is dangeorous. This goes double when the information is either difficult or impossible to verify. This shouldn't be surprising, since the most effective and dangerous liars tell the truth most of the time.
- JohnFen 3y ago
  > but often makes hard to detect errors ranging from minor to outright lies is dangeorous. Oh, I agree. However, the peer-review system does mitigate this problem to a useful degree. Without such a system, it would be impossible to have any degree of trust in any paper at all. > What we're really finding out is that journals are often little better than LLMs as information sources. It depends on the journal. There are absolutely crap ones out there that need to be ignored. But there are also good ones out there that have earned their reputation. Even there, BS can get through of course -- but to say that such journals are mostly unreliable is seriously overstating the issue.
  - smeej 3y ago
    I've yet to see studies saying this isn't pervasive across journals. The vast majority are apparently affected. The idea of peer-reviewed journals is great. The reality is a circle jerk.
- xboxnolifes 3y ago
  Reading un-replicated papers as authoritative accounts is definitely a problem. Solution: don't do that.
  - breakwaterlabs 3y ago
    What percentage of papers in your average, reputable journal have been replicated? And how can one easily determine, while looking at a particular paper, whether it has been replicated? And whether those doing the replication have any undisclosed ties to the original? At an epistemological level, the idea of a knowledge source like a journal where the information is only deemed reliable if personally verified seems problematic. Why even have it if all of its uncountable claims are indistinguishable from very clever lies, and attempts to quantify the extent of those lies indicate that they are pervasive?
coolhand2120 3y ago
With this system, I’m many domains, a -majority- of published research is not reproducible. Possibly worse than a coin flip. Do you really think this is “reasonably well”. https://en.m.wikipedia.org/wiki/Replication_crisis https://en.m.wikipedia.org/wiki/Replication_crisis

booleandilemma 3y ago

I still get sick even though I wash my hands. Let me stop washing my hands.

smeej 3y ago
Not to take this too far, but, unless you're a surgeon or a childcare worker, it probably isn't making much difference.