3 ms·

As far as I can tell the hoax did fail. As I recall from the authors' writeup, the initial round of hoax papers they sent out all got rejected. By any reasonabl

As far as I can tell the hoax did fail. As I recall from the authors' writeup, the initial round of hoax papers they sent out all got rejected. By any reasonable standard they could/should have stopped right there and published a paper called "social science journals aren't as gullible as we thought". Instead they iterated on the papers, and kept iterating until they got something published. What's bizarre is that the authors don't seem to realize that this changed the whole nature of their study, and they were now testing a completely different thesis than the one they set out to test.

jdminhbg 8y ago
I’m sorry, but if the barrier to entry for publishing is a few weeks of revising made-up papers, that’s still a gigantic problem. The hoax succeeded.
- fenomas 8y ago
  Look past the false dichotomy - there can still be a gigantic problem even though this particular hoax didn't do a good job of exposing it. I mean, the social science journals targeted by this hoax could be awful, I have no idea. But the hoax started with the thesis "if we submit an awful paper it will get accepted" and that thesis turned out to be false. This implies that the journals are not as bad as the hoax authors originally expected, right? They could still be bad, but the hoax doesn't establish that. The fact that the authors later got other papers accepted, I don't think we can say much about. If an experimenter's methodology is "keep iterating until X happens", then obviously the fact that X eventually happened can't prove anything about their thesis.
  - jdminhbg 8y ago
    There is a LOT of motivated reasoning in here. What method of experimentation are you proposing where if the first attempt at validating a hypothesis fails, you simply quit instead of tweaking the hypothesis and trying again? The hoax proved that you do not need more than a couple weeks of training to become a published author in these journals. I do not expect that these authors could, without any training or expertise, revise-and-submit a half dozen papers to be published in physics or chemistry. That's significant, whether you like it or not.
    - fenomas 8y ago
      > motivated reasoning You have things backwards. I agree with the hoaxers' motivation, and I assume that what they were trying to prove is presumably true. But their experiment didn't actually support those conclusions. > What method of experimentation are you proposing .. Lots of methodologies are valid, but if the whole thing is presented as science then the conclusions need to be consistent with the methodology used! In the hoaxers' writeup, they said they submitted a bunch of absurd papers, but they all got rejected, so they gradually toned things down bit by bit until some of them got accepted. That's fine, insofar as it goes, but you cannot then turn around and claim to have proved that the journals will will publish absurd papers, which is what the authors did. You can't call a result significant if your methodology is to gradually lower the bar for "significant" until you get the result you wanted! What the hoax actually proved was that the authors could publish a paper that they considered absurd, but which the journal considered fine. Which is neat - but isn't that already true of the regular papers these journals normally publish? Then what in the world did the hoax accomplish? > The hoax proved that you do not need more than a couple weeks of training to become a published author in these journals I don't think the authors claimed anything like that, where are you getting that? Their thesis was just that the content of the papers was absurd, not the times or credentials involved.
      - jdminhbg 8y ago
        > You can't call a result significant if your methodology is to gradually lower the bar for "significant" until you get the result you wanted! This didn't happen. You're the one who has post hoc decided that there is some abstract bar of "significance" that they're moving up or down. The point is to publish absurd papers in ostensibly serious journals. There is not some absurdity equivalent to p < 0.05 that they manipulated. They just published absurd papers in ostensibly serious journals. It's not like the absurdity of Dog Rape Culture would be lessened if the first papers they submitted had been the output of /dev/random. > I don't think the authors claimed anything like that, where are you getting that? Their thesis was just that the content of the papers was absurd, not the times or credentials involved. I'm getting it from the literal facts of what happened. They are not trained in these fields, and papers they wrote were published in these journals. Therefore, the hoax shows that you do not need to be trained in these fields to publish in these journals.

jdminhbg 8y ago

I’m sorry, but if the barrier to entry for publishing is a few weeks of revising made-up papers, that’s still a gigantic problem. The hoax succeeded.

fenomas 8y ago
Look past the false dichotomy - there can still be a gigantic problem even though this particular hoax didn't do a good job of exposing it. I mean, the social science journals targeted by this hoax could be awful, I have no idea. But the hoax started with the thesis "if we submit an awful paper it will get accepted" and that thesis turned out to be false. This implies that the journals are not as bad as the hoax authors originally expected, right? They could still be bad, but the hoax doesn't establish that. The fact that the authors later got other papers accepted, I don't think we can say much about. If an experimenter's methodology is "keep iterating until X happens", then obviously the fact that X eventually happened can't prove anything about their thesis.
- jdminhbg 8y ago
  There is a LOT of motivated reasoning in here. What method of experimentation are you proposing where if the first attempt at validating a hypothesis fails, you simply quit instead of tweaking the hypothesis and trying again? The hoax proved that you do not need more than a couple weeks of training to become a published author in these journals. I do not expect that these authors could, without any training or expertise, revise-and-submit a half dozen papers to be published in physics or chemistry. That's significant, whether you like it or not.
  - fenomas 8y ago
    > motivated reasoning You have things backwards. I agree with the hoaxers' motivation, and I assume that what they were trying to prove is presumably true. But their experiment didn't actually support those conclusions. > What method of experimentation are you proposing .. Lots of methodologies are valid, but if the whole thing is presented as science then the conclusions need to be consistent with the methodology used! In the hoaxers' writeup, they said they submitted a bunch of absurd papers, but they all got rejected, so they gradually toned things down bit by bit until some of them got accepted. That's fine, insofar as it goes, but you cannot then turn around and claim to have proved that the journals will will publish absurd papers, which is what the authors did. You can't call a result significant if your methodology is to gradually lower the bar for "significant" until you get the result you wanted! What the hoax actually proved was that the authors could publish a paper that they considered absurd, but which the journal considered fine. Which is neat - but isn't that already true of the regular papers these journals normally publish? Then what in the world did the hoax accomplish? > The hoax proved that you do not need more than a couple weeks of training to become a published author in these journals I don't think the authors claimed anything like that, where are you getting that? Their thesis was just that the content of the papers was absurd, not the times or credentials involved.
    - jdminhbg 8y ago
      > You can't call a result significant if your methodology is to gradually lower the bar for "significant" until you get the result you wanted! This didn't happen. You're the one who has post hoc decided that there is some abstract bar of "significance" that they're moving up or down. The point is to publish absurd papers in ostensibly serious journals. There is not some absurdity equivalent to p < 0.05 that they manipulated. They just published absurd papers in ostensibly serious journals. It's not like the absurdity of Dog Rape Culture would be lessened if the first papers they submitted had been the output of /dev/random. > I don't think the authors claimed anything like that, where are you getting that? Their thesis was just that the content of the papers was absurd, not the times or credentials involved. I'm getting it from the literal facts of what happened. They are not trained in these fields, and papers they wrote were published in these journals. Therefore, the hoax shows that you do not need to be trained in these fields to publish in these journals.