5 ms·

Looks like it boiled down to the IRB's take on the matter. The school asserted that Boghossian had unethically conducted research on human subjects wit

Looks like it boiled down to the IRB's take on the matter. The school asserted that Boghossian had unethically conducted research on human subjects with his experiment. According to the school’s Institutional Review Board, Boghossian would have needed to obtain “informed consent” from the individuals reviewing his hoax articles in order for his actions to have been considered ethical. Does this catch-22 mean no more investigative meta-research would be tolerated by universities? Even if those human subjects are kept anonymous, they're still afforded IRB protections? Is there anything we can know apriori that those subjects aren't actually subjected to anything high risk to them? Maybe that's a bad direction to take to even allow that?

Kednicma 6y ago
I wonder how this could be done ethically but in a way that doesn't allow peer reviewers to cheat. One thing that comes to mind is ABX testing. Show peer reviewers both a paper which they should accept, and a paper which they should reject, and then give them questionable papers to classify. At what point does it become unethical simply to be a peer-reviewer for a broken journal? That's the implicit question that should be asked in this sort of ethics review. It does no good to punish just a submitter of papers, if the entire journal is unhealthy.
- tptacek 6y ago
  Low-impact-factor journals that accept papers with 1000 of ethnographic field studies aren't "broken" just because the data was fake, because it isn't the premise of peer review to reliably spot fake data. You can make the argument that there are too many low-impact journals and schools shouldn't be allocating resources towards keeping them going; I don't think it's a super strong argument but it's at least coherent. But more importantly: it won't have anything to do with this hoax; if you're willing to fake data, you can get accepted in higher-impact journals outside the social sciences as well.
  - Kednicma 6y ago
    I'm not sure how fake data is substantially different from any other part of a fake paper. Just like how randomly-generated sentences might sound reasonable, faked data might look like real data. In both cases, we expect peer review to engage and to consider how data was sourced. This happened in the "Who's Afraid of Peer Review?" incident [0], where the offending papers were intentionally designed to have many red flags that would have helped reviewers understand that the data was bogus. More worryingly, data cannot model itself, so the quality of data should be largely irrelevant to any models which purport to explain it; as a result, a paper's contribution to science needs to be structured to work with other results, and cannot simply proclaim that its modelling is correct because some given equation fits the observed data. To pick a situation where your expertise shines, imagine that somebody submits a paper about a 3-SAT solver, and they include data that shows their solver doing not just extremely well on standard problem sets, but asymptotically scaling better than exponentially. The reason that you might doubt the honest presentation of the data is because you know, from having studied the field, that such behavior is unlikely. [0] https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%3F https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%...
    - tptacek 6y ago
      No, we don't expect peer reviewers to investigate the sourcing of data. A journal reviewer might be assigned a workload of dozens of papers to be completed, in snippets of spare time, over the course of a month or two. Peer reviewers aren't equipped to do that kind of vetting; that's what replication is for, and replications are their own research projects. It's really surprising what kinds of expectations people seem to have about peer review and what it can reasonably accomplish. "Who's Afraid Of Peer Review" targeted fee-charging open-access journals, which are financially incentivized to accept random papers. Journal reviewers are generally unpaid postgraduate academics.
      - jxramos 6y ago
        Spot on about unreasonable expectations; omne ignotum pro magnifico est always pokes its head when guessing on unfamiliar processes and procedures and what not. So funny how human nature when it assumes, assumes big grand intensive things! I'm frequently guilty of it myself. This is news to me about the reasonable costs absorbed from peer review, makes so much sense the level of vetting that's employed.
tomp 6y ago
> Does this catch-22 mean no more investigative meta-research would be tolerated by universities? It's not a Cache-22, it's plain old ultra-wokeness & wrongthink. Similar studies on "human subjects" (e.g. sending fake CVs to employers to spot biases in hiring) are lauded as exemplary and important just because they present socially-favourable results. Social studies are one big hoax, most research is p-hacked & fake, and the rest is heavily politicized.

Kednicma 6y ago

I wonder how this could be done ethically but in a way that doesn't allow peer reviewers to cheat. One thing that comes to mind is ABX testing. Show peer reviewers both a paper which they should accept, and a paper which they should reject, and then give them questionable papers to classify. At what point does it become unethical simply to be a peer-reviewer for a broken journal? That's the implicit question that should be asked in this sort of ethics review. It does no good to punish just a submitter of papers, if the entire journal is unhealthy.

tptacek 6y ago
Low-impact-factor journals that accept papers with 1000 of ethnographic field studies aren't "broken" just because the data was fake, because it isn't the premise of peer review to reliably spot fake data. You can make the argument that there are too many low-impact journals and schools shouldn't be allocating resources towards keeping them going; I don't think it's a super strong argument but it's at least coherent. But more importantly: it won't have anything to do with this hoax; if you're willing to fake data, you can get accepted in higher-impact journals outside the social sciences as well.
- Kednicma 6y ago
  I'm not sure how fake data is substantially different from any other part of a fake paper. Just like how randomly-generated sentences might sound reasonable, faked data might look like real data. In both cases, we expect peer review to engage and to consider how data was sourced. This happened in the "Who's Afraid of Peer Review?" incident [0], where the offending papers were intentionally designed to have many red flags that would have helped reviewers understand that the data was bogus. More worryingly, data cannot model itself, so the quality of data should be largely irrelevant to any models which purport to explain it; as a result, a paper's contribution to science needs to be structured to work with other results, and cannot simply proclaim that its modelling is correct because some given equation fits the observed data. To pick a situation where your expertise shines, imagine that somebody submits a paper about a 3-SAT solver, and they include data that shows their solver doing not just extremely well on standard problem sets, but asymptotically scaling better than exponentially. The reason that you might doubt the honest presentation of the data is because you know, from having studied the field, that such behavior is unlikely. [0] https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%3F https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%...
  - tptacek 6y ago
    No, we don't expect peer reviewers to investigate the sourcing of data. A journal reviewer might be assigned a workload of dozens of papers to be completed, in snippets of spare time, over the course of a month or two. Peer reviewers aren't equipped to do that kind of vetting; that's what replication is for, and replications are their own research projects. It's really surprising what kinds of expectations people seem to have about peer review and what it can reasonably accomplish. "Who's Afraid Of Peer Review" targeted fee-charging open-access journals, which are financially incentivized to accept random papers. Journal reviewers are generally unpaid postgraduate academics.
    - jxramos 6y ago
      Spot on about unreasonable expectations; omne ignotum pro magnifico est always pokes its head when guessing on unfamiliar processes and procedures and what not. So funny how human nature when it assumes, assumes big grand intensive things! I'm frequently guilty of it myself. This is news to me about the reasonable costs absorbed from peer review, makes so much sense the level of vetting that's employed.

tomp 6y ago

> Does this catch-22 mean no more investigative meta-research would be tolerated by universities? It's not a Cache-22, it's plain old ultra-wokeness & wrongthink. Similar studies on "human subjects" (e.g. sending fake CVs to employers to spot biases in hiring) are lauded as exemplary and important just because they present socially-favourable results. Social studies are one big hoax, most research is p-hacked & fake, and the rest is heavily politicized.