7 ms·

That wasn't the actual experiment, though. The actual experiment was to test whether or not these publications were capable of detecting fake papers.

jrumbut 6y ago
Understood, but in the process he claimed to do a large experiment that he didn't do. He faked data, he broke the trust that the entire research enterprise is built on.
- tptacek 6y ago
  I think you're going to find that most people commenting on this have never reviewed a journal article and don't understand the process. You see it in replication crisis threads as well: the belief that academic research is premised on reviewers replicating results before things get published. The idea that academic journals are based on a presumption of good faith is totally alien to a lot of HN commenters.
  - naasking 6y ago
    Reasonable critics aren't expecting reviewers to replicate the results, but the dog park paper illustrates that reviewers can't even recognized clearly bogus data. I mean, have you even read the methods the paper said they employed?
    - tptacek 6y ago
      I read several of the hoax papers, and skimmed one that included dog park data. Which paper are you referring to, where was it submitted, and where was it accepted? The hoax authors were exceptionally dishonest in their presentation of the results: virtually all of their papers were rejected, the accepts were generally in very low-impact venues, and the papers that got accepted were generally not the lurid ones they highlighted in their summary. I ask for specifics because it is not an interesting research result if they created a paper with clearly bogus data that was then rejected. The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
      - manfredo 6y ago
        It was accepted by Gender, Place & Culture. In fact it was not only accepted, it received an award for exemplary scholarship. Out of the 21 papers submitted [1] 9 were rejected, 7 were published, 3 were asked to revised and resubmit for publication, and 1 was currently under review when the hoax was revealed. The fact that totally bogus papers have about a coin flip chance of being accepted is astounding. 1. https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair
        tptacek 6y ago
        You're astounded that a piece that faked 1000 hours of observational studies was accepted in the #15th-ranked gender studies journal? Respectfully: I think what's shocking you is how many journals there are. (Revise-and-resubmit, by the way, is a nice way of saying "reject").
        manfredo 6y ago
        I'm astounded that a research didn't bat an eye about someone claiming to have observed a thousand instances of dog-humping, managed to identify the genders of the dogs, as well as the sexual orientation of the owners - and to have published the paper anyway when the authors claimed to have accidentally thrown away the original data. And to put the cherry on top, to have decided to give this paper an award for exemplary scholarship.
        tptacek 6y ago
        They didn't claim "a thousand instances". They claim 1000 hours of field study. That number shocks people, but over the course of a year, it's a half-time job for a researcher, and so your argument is left at "that seems like a dumb thing for a researcher to allocate half their time to". I mean, sure, I agree, but so what? That's not a very interesting argument, or a damning one.
        manfredo 6y ago
- manfredo 6y ago
  The whole point of the experiment was to test whether the research enterprise is trustworthy. Submitting fake experiments to the paper and seeing if they get published is the experiment. It seems to me that you're rejecting the entire premise of testing publications' abilities to detect fraudulent research.
  - tptacek 6y ago
    Yes, he is, because that premise is fatally flawed.
    - manfredo 6y ago
      The whole point of peer review and publications like these is to ensure a quality of published research. If someone conducts a study and determines that a significant portion of fraudulent research is published, then this demonstrates that the publication is not doing a good job of detecting fraudulent research. Some may respond that these authors were deliberately attempting to get their fraudulent papers published, and that these pieces were only published because they were bad faith actors. And? Who's to say that other authors aren't doing the same? The whole point of the system is to detect bad faith actors, so pointing out that the authors of the grievance studies papers were not publishing in good faith is no excuse. This is like saying pen testing or red team exercises are fatally flawed. They're not. But they can definitely be embarrassing when they reveal deficiencies, so it's understanding why many would want to reject the results of these exercises. But if the organizations wants to improve they need to react constructively to the issues that were revealed, not dismiss the study as flawed. Unfortunately, in this situation the latter seems to be happening.
      - tptacek 6y ago
        No, it's not. Just the idea that you'd think it would be reasonable to "red team" a program committee shows how far off you are. The red team will succeed every time; program committees generally operate on a presumption of good faith. Reviewers are busy people with their own research commitments; they might spend an hour, tops, if they're generous, on any paper. Of course, the purpose of the "Sokal Squared" "experiment" was to demonstrate how un-rigorous social science research is, but this is common in hard science fields as well, which just amplifies the dishonesty of the whole enterprise.
- naasking 6y ago
  He "faked data" that was literally impossible to acquire in the manner described in the paper. Whether reviewers would actually catch the obviously faked data was entirely the point. Reviewers are not supposed to check work, not trust blindly.
avs733 6y ago
and there are processes for doing that...which he did not follow.
- jrumbut 6y ago
  Oh yeah! There are any number of other issues here. But I think there are mitigating factors for many of them. It's the issue of saying he went to a dog park and observed and recorded dog behavior when he never did anything of the kind that really bothers me.
  - avs733 6y ago
    "haha I pulled a hoax" "no, you lied...we have processes for how to lie ethically, we told you about them and you ignored them"
    - jrumbut 6y ago
      "What are you doing here?" "Showing you burglary is possible" "I already knew that, and also no longer trust you as a neighbor"
      - avs733 6y ago
        It's a shame we can't re-title this article 'after getting caught black hat claims to actually be white hat'

jrumbut 6y ago

Understood, but in the process he claimed to do a large experiment that he didn't do. He faked data, he broke the trust that the entire research enterprise is built on.

tptacek 6y ago
I think you're going to find that most people commenting on this have never reviewed a journal article and don't understand the process. You see it in replication crisis threads as well: the belief that academic research is premised on reviewers replicating results before things get published. The idea that academic journals are based on a presumption of good faith is totally alien to a lot of HN commenters.
- naasking 6y ago
  Reasonable critics aren't expecting reviewers to replicate the results, but the dog park paper illustrates that reviewers can't even recognized clearly bogus data. I mean, have you even read the methods the paper said they employed?
  - tptacek 6y ago
    I read several of the hoax papers, and skimmed one that included dog park data. Which paper are you referring to, where was it submitted, and where was it accepted? The hoax authors were exceptionally dishonest in their presentation of the results: virtually all of their papers were rejected, the accepts were generally in very low-impact venues, and the papers that got accepted were generally not the lurid ones they highlighted in their summary. I ask for specifics because it is not an interesting research result if they created a paper with clearly bogus data that was then rejected. The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
    - manfredo 6y ago
      It was accepted by Gender, Place & Culture. In fact it was not only accepted, it received an award for exemplary scholarship. Out of the 21 papers submitted [1] 9 were rejected, 7 were published, 3 were asked to revised and resubmit for publication, and 1 was currently under review when the hoax was revealed. The fact that totally bogus papers have about a coin flip chance of being accepted is astounding. 1. https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair
      - tptacek 6y ago
        You're astounded that a piece that faked 1000 hours of observational studies was accepted in the #15th-ranked gender studies journal? Respectfully: I think what's shocking you is how many journals there are. (Revise-and-resubmit, by the way, is a nice way of saying "reject").
        manfredo 6y ago
        I'm astounded that a research didn't bat an eye about someone claiming to have observed a thousand instances of dog-humping, managed to identify the genders of the dogs, as well as the sexual orientation of the owners - and to have published the paper anyway when the authors claimed to have accidentally thrown away the original data. And to put the cherry on top, to have decided to give this paper an award for exemplary scholarship.
        tptacek 6y ago
        They didn't claim "a thousand instances". They claim 1000 hours of field study. That number shocks people, but over the course of a year, it's a half-time job for a researcher, and so your argument is left at "that seems like a dumb thing for a researcher to allocate half their time to". I mean, sure, I agree, but so what? That's not a very interesting argument, or a damning one.
        manfredo 6y ago
manfredo 6y ago
The whole point of the experiment was to test whether the research enterprise is trustworthy. Submitting fake experiments to the paper and seeing if they get published is the experiment. It seems to me that you're rejecting the entire premise of testing publications' abilities to detect fraudulent research.
- tptacek 6y ago
  Yes, he is, because that premise is fatally flawed.
  - manfredo 6y ago
    The whole point of peer review and publications like these is to ensure a quality of published research. If someone conducts a study and determines that a significant portion of fraudulent research is published, then this demonstrates that the publication is not doing a good job of detecting fraudulent research. Some may respond that these authors were deliberately attempting to get their fraudulent papers published, and that these pieces were only published because they were bad faith actors. And? Who's to say that other authors aren't doing the same? The whole point of the system is to detect bad faith actors, so pointing out that the authors of the grievance studies papers were not publishing in good faith is no excuse. This is like saying pen testing or red team exercises are fatally flawed. They're not. But they can definitely be embarrassing when they reveal deficiencies, so it's understanding why many would want to reject the results of these exercises. But if the organizations wants to improve they need to react constructively to the issues that were revealed, not dismiss the study as flawed. Unfortunately, in this situation the latter seems to be happening.
    - tptacek 6y ago
      No, it's not. Just the idea that you'd think it would be reasonable to "red team" a program committee shows how far off you are. The red team will succeed every time; program committees generally operate on a presumption of good faith. Reviewers are busy people with their own research commitments; they might spend an hour, tops, if they're generous, on any paper. Of course, the purpose of the "Sokal Squared" "experiment" was to demonstrate how un-rigorous social science research is, but this is common in hard science fields as well, which just amplifies the dishonesty of the whole enterprise.
naasking 6y ago
He "faked data" that was literally impossible to acquire in the manner described in the paper. Whether reviewers would actually catch the obviously faked data was entirely the point. Reviewers are not supposed to check work, not trust blindly.

avs733 6y ago

and there are processes for doing that...which he did not follow.

jrumbut 6y ago
Oh yeah! There are any number of other issues here. But I think there are mitigating factors for many of them. It's the issue of saying he went to a dog park and observed and recorded dog behavior when he never did anything of the kind that really bothers me.
- avs733 6y ago
  "haha I pulled a hoax" "no, you lied...we have processes for how to lie ethically, we told you about them and you ignored them"
  - jrumbut 6y ago
    "What are you doing here?" "Showing you burglary is possible" "I already knew that, and also no longer trust you as a neighbor"
    - avs733 6y ago
      It's a shame we can't re-title this article 'after getting caught black hat claims to actually be white hat'