4 ms·

I wonder how this could be done ethically but in a way that doesn't allow peer reviewers to cheat. One thing that comes to mind is ABX testing. Show peer review

I wonder how this could be done ethically but in a way that doesn't allow peer reviewers to cheat. One thing that comes to mind is ABX testing. Show peer reviewers both a paper which they should accept, and a paper which they should reject, and then give them questionable papers to classify. At what point does it become unethical simply to be a peer-reviewer for a broken journal? That's the implicit question that should be asked in this sort of ethics review. It does no good to punish just a submitter of papers, if the entire journal is unhealthy.

tptacek 6y ago
Low-impact-factor journals that accept papers with 1000 of ethnographic field studies aren't "broken" just because the data was fake, because it isn't the premise of peer review to reliably spot fake data. You can make the argument that there are too many low-impact journals and schools shouldn't be allocating resources towards keeping them going; I don't think it's a super strong argument but it's at least coherent. But more importantly: it won't have anything to do with this hoax; if you're willing to fake data, you can get accepted in higher-impact journals outside the social sciences as well.
- Kednicma 6y ago
  I'm not sure how fake data is substantially different from any other part of a fake paper. Just like how randomly-generated sentences might sound reasonable, faked data might look like real data. In both cases, we expect peer review to engage and to consider how data was sourced. This happened in the "Who's Afraid of Peer Review?" incident [0], where the offending papers were intentionally designed to have many red flags that would have helped reviewers understand that the data was bogus. More worryingly, data cannot model itself, so the quality of data should be largely irrelevant to any models which purport to explain it; as a result, a paper's contribution to science needs to be structured to work with other results, and cannot simply proclaim that its modelling is correct because some given equation fits the observed data. To pick a situation where your expertise shines, imagine that somebody submits a paper about a 3-SAT solver, and they include data that shows their solver doing not just extremely well on standard problem sets, but asymptotically scaling better than exponentially. The reason that you might doubt the honest presentation of the data is because you know, from having studied the field, that such behavior is unlikely. [0] https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%3F https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%...
  - tptacek 6y ago
    No, we don't expect peer reviewers to investigate the sourcing of data. A journal reviewer might be assigned a workload of dozens of papers to be completed, in snippets of spare time, over the course of a month or two. Peer reviewers aren't equipped to do that kind of vetting; that's what replication is for, and replications are their own research projects. It's really surprising what kinds of expectations people seem to have about peer review and what it can reasonably accomplish. "Who's Afraid Of Peer Review" targeted fee-charging open-access journals, which are financially incentivized to accept random papers. Journal reviewers are generally unpaid postgraduate academics.
    - jxramos 6y ago
      Spot on about unreasonable expectations; omne ignotum pro magnifico est always pokes its head when guessing on unfamiliar processes and procedures and what not. So funny how human nature when it assumes, assumes big grand intensive things! I'm frequently guilty of it myself. This is news to me about the reasonable costs absorbed from peer review, makes so much sense the level of vetting that's employed.
    - Kednicma 6y ago
      I don't want to quibble with your experiences of peer review; your experiences are valid. But what does a journal provide, then, exactly? Are they relics of the era before cheap self-publication?
      - tptacek 6y ago
        "A panel of experts from this field read this paper and felt it likely to make significant contributions". For high-impact journals, you can add "so much so that it was selected to be among the 25% of submissions this cycle to be published". What peer review does not say is "a panel of experts carefully vetted this paper to ensure that its conclusions are accurate". In fact, part of the premise of replication projects is that peer review doesn't say that. I've only been a reviewer a couple times, and only in computer science. Other people on HN have experience reviewing for other hard science venues. Maybe their experiences are different. But I think it's notable that you don't hear that in comments about what this hoax exposed; has there been any HN comment from a reviewer saying that they were expected to rigorously vet submissions, the way a PhD board does with a thesis? Daniel Engber makes many of these points better than I do, here: https://slate.com/technology/2018/10/grievance-studies-hoax-not-academic-scandal.html https://slate.com/technology/2018/10/grievance-studies-hoax-...

tptacek 6y ago

Low-impact-factor journals that accept papers with 1000 of ethnographic field studies aren't "broken" just because the data was fake, because it isn't the premise of peer review to reliably spot fake data. You can make the argument that there are too many low-impact journals and schools shouldn't be allocating resources towards keeping them going; I don't think it's a super strong argument but it's at least coherent. But more importantly: it won't have anything to do with this hoax; if you're willing to fake data, you can get accepted in higher-impact journals outside the social sciences as well.

Kednicma 6y ago
I'm not sure how fake data is substantially different from any other part of a fake paper. Just like how randomly-generated sentences might sound reasonable, faked data might look like real data. In both cases, we expect peer review to engage and to consider how data was sourced. This happened in the "Who's Afraid of Peer Review?" incident [0], where the offending papers were intentionally designed to have many red flags that would have helped reviewers understand that the data was bogus. More worryingly, data cannot model itself, so the quality of data should be largely irrelevant to any models which purport to explain it; as a result, a paper's contribution to science needs to be structured to work with other results, and cannot simply proclaim that its modelling is correct because some given equation fits the observed data. To pick a situation where your expertise shines, imagine that somebody submits a paper about a 3-SAT solver, and they include data that shows their solver doing not just extremely well on standard problem sets, but asymptotically scaling better than exponentially. The reason that you might doubt the honest presentation of the data is because you know, from having studied the field, that such behavior is unlikely. [0] https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%3F https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%...
- tptacek 6y ago
  No, we don't expect peer reviewers to investigate the sourcing of data. A journal reviewer might be assigned a workload of dozens of papers to be completed, in snippets of spare time, over the course of a month or two. Peer reviewers aren't equipped to do that kind of vetting; that's what replication is for, and replications are their own research projects. It's really surprising what kinds of expectations people seem to have about peer review and what it can reasonably accomplish. "Who's Afraid Of Peer Review" targeted fee-charging open-access journals, which are financially incentivized to accept random papers. Journal reviewers are generally unpaid postgraduate academics.
  - jxramos 6y ago
    Spot on about unreasonable expectations; omne ignotum pro magnifico est always pokes its head when guessing on unfamiliar processes and procedures and what not. So funny how human nature when it assumes, assumes big grand intensive things! I'm frequently guilty of it myself. This is news to me about the reasonable costs absorbed from peer review, makes so much sense the level of vetting that's employed.
  - Kednicma 6y ago
    I don't want to quibble with your experiences of peer review; your experiences are valid. But what does a journal provide, then, exactly? Are they relics of the era before cheap self-publication?
    - tptacek 6y ago
      "A panel of experts from this field read this paper and felt it likely to make significant contributions". For high-impact journals, you can add "so much so that it was selected to be among the 25% of submissions this cycle to be published". What peer review does not say is "a panel of experts carefully vetted this paper to ensure that its conclusions are accurate". In fact, part of the premise of replication projects is that peer review doesn't say that. I've only been a reviewer a couple times, and only in computer science. Other people on HN have experience reviewing for other hard science venues. Maybe their experiences are different. But I think it's notable that you don't hear that in comments about what this hoax exposed; has there been any HN comment from a reviewer saying that they were expected to rigorously vet submissions, the way a PhD board does with a thesis? Daniel Engber makes many of these points better than I do, here: https://slate.com/technology/2018/10/grievance-studies-hoax-not-academic-scandal.html https://slate.com/technology/2018/10/grievance-studies-hoax-...