3 ms·
Good program committees don't operate on the presumption of good faith. In fact, I addressed this in my comment so I'm unsure why you're continuing to make this
by manfredo 6y ago
Good program committees don't operate on the presumption of good faith. In fact, I addressed this in my comment so I'm unsure why you're continuing to make this faulty argument.
This is like a website saying that a vulnerability shouldn't be criticized because only bad faith actors would take advantage of it. Is that a reassuring response? Of course not. The whole point of being secure is to be secure from bad faith actors.
The whole point of Boghossian and the other researchers was to demonstrate how easily these publications can be taken advantage of by bad faith actors. Pointing out that these publications published these papers because the authors were bad faith actors is no excuse. How many other bad faith actors got fraudulent research published in these papers? We don't know, and we can't know. But we do know that these publications are extremely vulnerable to them.
It seems to me you aren't actually disagreeing with the conclusions made by these hoax papers: that these publications operate on a system of blind trust and can be easily exploited. It seems like you're saying that social science publications can't operate on any system other than blind trust, so we shouldn't think of this hoax as revealing anything significant. Personally, though, if these publications operate on a system of blind trust then that makes them inherently untrustworthy - hoax papers or not.
This thread is being rate limited, reply in edit:
At this point you're not even contending the claim that these publications fail to block bad research, and are instead claiming that other publications are just as ineffective. At this point you're accepting the thesis of these authors: these publications operate on blind faith and are incapable of identifying false papers.
If you submitted a paper solving NP hard problems in polynomial time it wouldn't be published - at least not without extensive scrutiny and checking if the most famous computer science question had indeed been answered.
Also, regarding the claim that papers don't receive feedback I suggest you read through the responses these authors of the hoax papers received. They did indeed receive feedback from their submissions, contrary to your comment, and some of it was truly astounding. Several praised the content but explicitly rejected the paper on the grounds of the race and gender of the author. In fact I'd say that the feedback was more important than the count of papers published.
- tptacek 6y agoMy belief is that this is just false. My basis for that belief is experience with ACM and Usenix program committees, which I've participated in directly, and have talked to peers who have as well; I also chaired one cycle of a Usenix proceeding, and so got to see how a fairly large set of reviewers actually worked. I am not including my experience as a reviewer for non-academic conferences. To be specific: my experience is that reviewers allocate a very small amount of time to reviewing any particular paper, certainly not enough to reliably spot faked research results. Most of the work of reviewing is simply ranking papers, reducing an intractable set of submissions down to a tractable short list. Submitters are lucky to get any substantive feedback at all (hence the "reviewer #2" phenomenon). If you have countervailing experience, describe it. Where have you reviewed, where you believe that program committee could withstand "red teaming"?
- tptacek 6y agoYou responded in an edit to your post. But your post doesn't address anything I said. My argument, at this point, boils down to: you don't seem to understand how journal peer review works, or what its purpose is. When confronted with the reality of what peer review is meant to accomplish, you reject the system entirely. That's a coherent (though: bad) argument, but it is not the argument the Areo Hoax authors made. If you can't find support for the Areo Hoax's own arguments, you're not actually making a point relevant to this thread.
- manfredo 6y agoWhat did I fail to address? Your argument is that peer review doesn't actually involve any sort of review capable of spotting bad research. That publications operate on blind faith, and so failure to spot bad fraudulent research is a non-issue. And I pointed out that this isn't a rebuttal to what these hoaxsters sought to reveal. In fact, it's an admission that their claims are true: These publication do operate on blind faith and are highly vulnerable to false claims. You seem to be under the impression that peer review isn't meant to spot ineffective or false claims, and so this hoax reveals nothing. You're entitled to your own opinions, but many others disagree. Most understand that the purpose of peer review to ensure academic rigor and catch bad research, rather than operating under blind faith. So the revelation that many publications operate on blind faith is indeed a significant result. In short, these hoaxsters are saying, "Look! These publications are incapable of spotting blatantly wrong research." And you're responding, "But most publications can't spot blatantly wrong research.". The second statement does not disprove the former. You're just claiming that the conclusions these hoaxsters made about social science publications can also be made against other publications. And that may be your perspective, but others do not have the same pessimistic attitude towards peer review.
- tptacek 6y agoAs I said: the good (not "blind") faith social science venues presume is also presumed by hard science journals, an important fact you're decided to pretend I didn't point out, and one that refutes the Areo Hoax authors arguments. I think we can be done now.