3 ms·

The whole point of peer review and publications like these is to ensure a quality of published research. If someone conducts a study and determines that a signi

The whole point of peer review and publications like these is to ensure a quality of published research. If someone conducts a study and determines that a significant portion of fraudulent research is published, then this demonstrates that the publication is not doing a good job of detecting fraudulent research. Some may respond that these authors were deliberately attempting to get their fraudulent papers published, and that these pieces were only published because they were bad faith actors. And? Who's to say that other authors aren't doing the same? The whole point of the system is to detect bad faith actors, so pointing out that the authors of the grievance studies papers were not publishing in good faith is no excuse. This is like saying pen testing or red team exercises are fatally flawed. They're not. But they can definitely be embarrassing when they reveal deficiencies, so it's understanding why many would want to reject the results of these exercises. But if the organizations wants to improve they need to react constructively to the issues that were revealed, not dismiss the study as flawed. Unfortunately, in this situation the latter seems to be happening.

tptacek 6y ago
No, it's not. Just the idea that you'd think it would be reasonable to "red team" a program committee shows how far off you are. The red team will succeed every time; program committees generally operate on a presumption of good faith. Reviewers are busy people with their own research commitments; they might spend an hour, tops, if they're generous, on any paper. Of course, the purpose of the "Sokal Squared" "experiment" was to demonstrate how un-rigorous social science research is, but this is common in hard science fields as well, which just amplifies the dishonesty of the whole enterprise.
- manfredo 6y ago
  Good program committees don't operate on the presumption of good faith. In fact, I addressed this in my comment so I'm unsure why you're continuing to make this faulty argument. This is like a website saying that a vulnerability shouldn't be criticized because only bad faith actors would take advantage of it. Is that a reassuring response? Of course not. The whole point of being secure is to be secure from bad faith actors. The whole point of Boghossian and the other researchers was to demonstrate how easily these publications can be taken advantage of by bad faith actors. Pointing out that these publications published these papers because the authors were bad faith actors is no excuse. How many other bad faith actors got fraudulent research published in these papers? We don't know, and we can't know. But we do know that these publications are extremely vulnerable to them. It seems to me you aren't actually disagreeing with the conclusions made by these hoax papers: that these publications operate on a system of blind trust and can be easily exploited. It seems like you're saying that social science publications can't operate on any system other than blind trust, so we shouldn't think of this hoax as revealing anything significant. Personally, though, if these publications operate on a system of blind trust then that makes them inherently untrustworthy - hoax papers or not. This thread is being rate limited, reply in edit: At this point you're not even contending the claim that these publications fail to block bad research, and are instead claiming that other publications are just as ineffective. At this point you're accepting the thesis of these authors: these publications operate on blind faith and are incapable of identifying false papers. If you submitted a paper solving NP hard problems in polynomial time it wouldn't be published - at least not without extensive scrutiny and checking if the most famous computer science question had indeed been answered. Also, regarding the claim that papers don't receive feedback I suggest you read through the responses these authors of the hoax papers received. They did indeed receive feedback from their submissions, contrary to your comment, and some of it was truly astounding. Several praised the content but explicitly rejected the paper on the grounds of the race and gender of the author. In fact I'd say that the feedback was more important than the count of papers published.
  - tptacek 6y ago
    My belief is that this is just false. My basis for that belief is experience with ACM and Usenix program committees, which I've participated in directly, and have talked to peers who have as well; I also chaired one cycle of a Usenix proceeding, and so got to see how a fairly large set of reviewers actually worked. I am not including my experience as a reviewer for non-academic conferences. To be specific: my experience is that reviewers allocate a very small amount of time to reviewing any particular paper, certainly not enough to reliably spot faked research results. Most of the work of reviewing is simply ranking papers, reducing an intractable set of submissions down to a tractable short list. Submitters are lucky to get any substantive feedback at all (hence the "reviewer #2" phenomenon). If you have countervailing experience, describe it. Where have you reviewed, where you believe that program committee could withstand "red teaming"?
    - tptacek 6y ago
      You responded in an edit to your post. But your post doesn't address anything I said. My argument, at this point, boils down to: you don't seem to understand how journal peer review works, or what its purpose is. When confronted with the reality of what peer review is meant to accomplish, you reject the system entirely. That's a coherent (though: bad) argument, but it is not the argument the Areo Hoax authors made. If you can't find support for the Areo Hoax's own arguments, you're not actually making a point relevant to this thread.
      - manfredo 6y ago
        What did I fail to address? Your argument is that peer review doesn't actually involve any sort of review capable of spotting bad research. That publications operate on blind faith, and so failure to spot bad fraudulent research is a non-issue. And I pointed out that this isn't a rebuttal to what these hoaxsters sought to reveal. In fact, it's an admission that their claims are true: These publication do operate on blind faith and are highly vulnerable to false claims. You seem to be under the impression that peer review isn't meant to spot ineffective or false claims, and so this hoax reveals nothing. You're entitled to your own opinions, but many others disagree. Most understand that the purpose of peer review to ensure academic rigor and catch bad research, rather than operating under blind faith. So the revelation that many publications operate on blind faith is indeed a significant result. In short, these hoaxsters are saying, "Look! These publications are incapable of spotting blatantly wrong research." And you're responding, "But most publications can't spot blatantly wrong research.". The second statement does not disprove the former. You're just claiming that the conclusions these hoaxsters made about social science publications can also be made against other publications. And that may be your perspective, but others do not have the same pessimistic attitude towards peer review.

tptacek 6y ago

No, it's not. Just the idea that you'd think it would be reasonable to "red team" a program committee shows how far off you are. The red team will succeed every time; program committees generally operate on a presumption of good faith. Reviewers are busy people with their own research commitments; they might spend an hour, tops, if they're generous, on any paper. Of course, the purpose of the "Sokal Squared" "experiment" was to demonstrate how un-rigorous social science research is, but this is common in hard science fields as well, which just amplifies the dishonesty of the whole enterprise.

manfredo 6y ago
Good program committees don't operate on the presumption of good faith. In fact, I addressed this in my comment so I'm unsure why you're continuing to make this faulty argument. This is like a website saying that a vulnerability shouldn't be criticized because only bad faith actors would take advantage of it. Is that a reassuring response? Of course not. The whole point of being secure is to be secure from bad faith actors. The whole point of Boghossian and the other researchers was to demonstrate how easily these publications can be taken advantage of by bad faith actors. Pointing out that these publications published these papers because the authors were bad faith actors is no excuse. How many other bad faith actors got fraudulent research published in these papers? We don't know, and we can't know. But we do know that these publications are extremely vulnerable to them. It seems to me you aren't actually disagreeing with the conclusions made by these hoax papers: that these publications operate on a system of blind trust and can be easily exploited. It seems like you're saying that social science publications can't operate on any system other than blind trust, so we shouldn't think of this hoax as revealing anything significant. Personally, though, if these publications operate on a system of blind trust then that makes them inherently untrustworthy - hoax papers or not. This thread is being rate limited, reply in edit: At this point you're not even contending the claim that these publications fail to block bad research, and are instead claiming that other publications are just as ineffective. At this point you're accepting the thesis of these authors: these publications operate on blind faith and are incapable of identifying false papers. If you submitted a paper solving NP hard problems in polynomial time it wouldn't be published - at least not without extensive scrutiny and checking if the most famous computer science question had indeed been answered. Also, regarding the claim that papers don't receive feedback I suggest you read through the responses these authors of the hoax papers received. They did indeed receive feedback from their submissions, contrary to your comment, and some of it was truly astounding. Several praised the content but explicitly rejected the paper on the grounds of the race and gender of the author. In fact I'd say that the feedback was more important than the count of papers published.
- tptacek 6y ago
  My belief is that this is just false. My basis for that belief is experience with ACM and Usenix program committees, which I've participated in directly, and have talked to peers who have as well; I also chaired one cycle of a Usenix proceeding, and so got to see how a fairly large set of reviewers actually worked. I am not including my experience as a reviewer for non-academic conferences. To be specific: my experience is that reviewers allocate a very small amount of time to reviewing any particular paper, certainly not enough to reliably spot faked research results. Most of the work of reviewing is simply ranking papers, reducing an intractable set of submissions down to a tractable short list. Submitters are lucky to get any substantive feedback at all (hence the "reviewer #2" phenomenon). If you have countervailing experience, describe it. Where have you reviewed, where you believe that program committee could withstand "red teaming"?
  - tptacek 6y ago
    You responded in an edit to your post. But your post doesn't address anything I said. My argument, at this point, boils down to: you don't seem to understand how journal peer review works, or what its purpose is. When confronted with the reality of what peer review is meant to accomplish, you reject the system entirely. That's a coherent (though: bad) argument, but it is not the argument the Areo Hoax authors made. If you can't find support for the Areo Hoax's own arguments, you're not actually making a point relevant to this thread.
    - manfredo 6y ago
      What did I fail to address? Your argument is that peer review doesn't actually involve any sort of review capable of spotting bad research. That publications operate on blind faith, and so failure to spot bad fraudulent research is a non-issue. And I pointed out that this isn't a rebuttal to what these hoaxsters sought to reveal. In fact, it's an admission that their claims are true: These publication do operate on blind faith and are highly vulnerable to false claims. You seem to be under the impression that peer review isn't meant to spot ineffective or false claims, and so this hoax reveals nothing. You're entitled to your own opinions, but many others disagree. Most understand that the purpose of peer review to ensure academic rigor and catch bad research, rather than operating under blind faith. So the revelation that many publications operate on blind faith is indeed a significant result. In short, these hoaxsters are saying, "Look! These publications are incapable of spotting blatantly wrong research." And you're responding, "But most publications can't spot blatantly wrong research.". The second statement does not disprove the former. You're just claiming that the conclusions these hoaxsters made about social science publications can also be made against other publications. And that may be your perspective, but others do not have the same pessimistic attitude towards peer review.