8 ms·

PSU punishes prof who duped academic journal with hoax 'dog rape' article (2019)

37 points by nec4b 6y ago

Kednicma 6y ago
As a Portland State donor, I'm disappointed at PSU's refusal to back legitimate scientific inquiry. Sending bogus garbage to journals occasionally is a reasonable way to test that peer-review systems are working. The events I normally donate to support are canceled due to the pandemic, though, so I probably wouldn't donate anyway. Guess I can't really take a moral stance here.
- valuearb 6y ago
  As a PSU alum, I never donate to this halfassed institution, a glorified community college.
zimpenfish 6y ago
[2019]
- dang 6y ago
  Good catch. Added.
jrumbut 6y ago
I don't know how you can get clearer cut research misconduct than faking an experiment.
- manfredo 6y ago
  That wasn't the actual experiment, though. The actual experiment was to test whether or not these publications were capable of detecting fake papers.
  - jrumbut 6y ago
    Understood, but in the process he claimed to do a large experiment that he didn't do. He faked data, he broke the trust that the entire research enterprise is built on.
    - tptacek 6y ago
      I think you're going to find that most people commenting on this have never reviewed a journal article and don't understand the process. You see it in replication crisis threads as well: the belief that academic research is premised on reviewers replicating results before things get published. The idea that academic journals are based on a presumption of good faith is totally alien to a lot of HN commenters.
      - naasking 6y ago
        Reasonable critics aren't expecting reviewers to replicate the results, but the dog park paper illustrates that reviewers can't even recognized clearly bogus data. I mean, have you even read the methods the paper said they employed?
        tptacek 6y ago
        I read several of the hoax papers, and skimmed one that included dog park data. Which paper are you referring to, where was it submitted, and where was it accepted? The hoax authors were exceptionally dishonest in their presentation of the results: virtually all of their papers were rejected, the accepts were generally in very low-impact venues, and the papers that got accepted were generally not the lurid ones they highlighted in their summary. I ask for specifics because it is not an interesting research result if they created a paper with clearly bogus data that was then rejected. The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
- deleted 6y ago
  [deleted]
kaiju0 6y ago
This looks like a valid test case. How else could you verify the legitimacy of previously submitted articles. The emperor has no clothes.
free_rms 6y ago
So, when Sokal did something similar in the 90s, we all had a laugh and life went on. Now that it's today, we're more vicious and this kind of punking will not go unpunished. Good checkpoint on where we're at.
pjc50 6y ago
This website seems to be an extremely partisan culture war site basically focused on issues academics discuss that conservatives don't like? And then "cancelling" them?
RcouF1uZ4gsC 6y ago
As an aside for everyone coming to the comments first before reading the article, PSU in the title is Portland State University not Penn State University which I immediately thought of when I saw PSU. Basically, Portland State University is retaliating against a professor who showed that the peer review and acceptance process for various journals was woefully broken.
- avs733 6y ago
  another perspective is that they are disciplining a professor who practiced research misconduct
jxramos 6y ago
Looks like it boiled down to the IRB's take on the matter. The school asserted that Boghossian had unethically conducted research on human subjects with his experiment. According to the school’s Institutional Review Board, Boghossian would have needed to obtain “informed consent” from the individuals reviewing his hoax articles in order for his actions to have been considered ethical. Does this catch-22 mean no more investigative meta-research would be tolerated by universities? Even if those human subjects are kept anonymous, they're still afforded IRB protections? Is there anything we can know apriori that those subjects aren't actually subjected to anything high risk to them? Maybe that's a bad direction to take to even allow that?
- Kednicma 6y ago
  I wonder how this could be done ethically but in a way that doesn't allow peer reviewers to cheat. One thing that comes to mind is ABX testing. Show peer reviewers both a paper which they should accept, and a paper which they should reject, and then give them questionable papers to classify. At what point does it become unethical simply to be a peer-reviewer for a broken journal? That's the implicit question that should be asked in this sort of ethics review. It does no good to punish just a submitter of papers, if the entire journal is unhealthy.
  - tptacek 6y ago
    Low-impact-factor journals that accept papers with 1000 of ethnographic field studies aren't "broken" just because the data was fake, because it isn't the premise of peer review to reliably spot fake data. You can make the argument that there are too many low-impact journals and schools shouldn't be allocating resources towards keeping them going; I don't think it's a super strong argument but it's at least coherent. But more importantly: it won't have anything to do with this hoax; if you're willing to fake data, you can get accepted in higher-impact journals outside the social sciences as well.
    - Kednicma 6y ago
      I'm not sure how fake data is substantially different from any other part of a fake paper. Just like how randomly-generated sentences might sound reasonable, faked data might look like real data. In both cases, we expect peer review to engage and to consider how data was sourced. This happened in the "Who's Afraid of Peer Review?" incident [0], where the offending papers were intentionally designed to have many red flags that would have helped reviewers understand that the data was bogus. More worryingly, data cannot model itself, so the quality of data should be largely irrelevant to any models which purport to explain it; as a result, a paper's contribution to science needs to be structured to work with other results, and cannot simply proclaim that its modelling is correct because some given equation fits the observed data. To pick a situation where your expertise shines, imagine that somebody submits a paper about a 3-SAT solver, and they include data that shows their solver doing not just extremely well on standard problem sets, but asymptotically scaling better than exponentially. The reason that you might doubt the honest presentation of the data is because you know, from having studied the field, that such behavior is unlikely. [0] https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%3F https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%...
easterncalculus 6y ago
This happened a bit ago, you can read more about it here: https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair
avs733 6y ago
Just a note that Campusreform.org is...a mess of a site that publishes opinions as factual articles and is primarily writen by undergraduates. Here is a more fair handed presentation of this case: https://www.insidehighered.com/news/2019/01/08/author-recent-academic-hoax-faces-disciplinary-action-portland-state https://www.insidehighered.com/news/2019/01/08/author-recent...
avs733 6y ago
An ongoing misconception here is that academic journals only publish 'facts' Not really. They publish research articles. Depending on the field, and the field's philosophical perspective on the construction of knowledge, that can take differing forms. Those differing forms are not just valid but important. And understanding the differences is also, important. An article about medical treatment necessarily does different things and looks different than a journal in the humanities (for example). They make meaning from information in different ways - and that is what all forms of research do, make meaning. Research at its core is not about the discovery of facts...this is basic philosophy of science, basic Thomas Kuhn. Standards like replicability may be of use for making meaning in some fields, but they are not necessarily equally useful in others, often because the level of contextual situating that needs to occur reaches towards the impossible. Serious members of the field would not have taken the Grievance Studies articles as 'capital-F-FACT' they instead would have interpreted it as a perspective, an argument, an interpretation. Specifically, the entire field of critical studies exists to, in varying forms, critique the ways in which 'discovery of facts' is a reductivist way of looking at meaning making that privileges certain perspectives over others - by treating some perspectives as reality. So the critique here, and looking back at Sokal, isn't that 'hahaha I pulled one over on you' its that 'we assumed that you were giving us a new perspective in good faith that we could collectively learn from' and ...'now you are standing here laughing at us because you acted like a jerk.' It's two different world views with one willing to be open (by choice) to outside critique and perspectives, and the other supremely self-confident that critique and perspective is unnecessary. In effect, it's like a legal opinion that separates matters of interpreting law and matters of finding fact.
morninglight 6y ago
If Mark McLellan remains VP for Research and Graduate Studies, it casts a cloud over academic excellence at PSU. What is his background? How long has he been in this position? What would cause him to make such a politically driven bad decision? It is inconceivable that someone with so little understanding of the toxic academic publishing environment could be influencing the education of 26,000 students.
lwneal 6y ago
The PSU Grievance Studies Affair [1] had the same objective as the 1996 Sokal Affair: to test peer review by submitting obviously bad papers that should be rejected, and seeing how many of them are accepted. The motivation behind this is entirely reasonable. Bad "true negative" as well as known-good "true positive" articles ought to be regularly submitted to every peer review process as a test of accuracy. A great example of a similar idea was the 2014 NIPS consistency experiment [2]. However, there is a problem with the "dog rape" hoax article [3] as a test of the peer review system. Specifically, the author in that paper claims to have spent over 1000 hours carefully cataloguing the behavior of ten thousand people and dogs over the course of one year. The paper then proceeds to produce silly but surprising statistics: female dogs are 70% more likely to be leashed than male dogs, 100% of dogs with shock collars are male, 847 instances of dogs fighting were observed, and so on. The problem here is that the data was falsified: the observations and experiments claimed never happened, the numbers were all made up. The article was likely accepted because of its data, but the data was a lie. If this data had been real, it might have been a minor but useful study relevant to economists, sociologists, or urban planners, regardless of how silly or made-up the conclusions at the end were. Peer review works based on the assumption that the author is telling the truth about what experiments they conducted and what numbers they measured. The grievance studies authors could have submitted fake and silly hoax papers without falsifying data: that's what Sokal did, and that would have been a valid experiment testing the peer review system. But that is not what the authors chose to do. [1] https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair [2] http://blog.mrtz.org/2014/12/15/the-nips-experiment.html http://blog.mrtz.org/2014/12/15/the-nips-experiment.html [3] http://norskk.is/bytta/menn/dog_park.pdf http://norskk.is/bytta/menn/dog_park.pdf
goldcd 6y ago
Would it be equally unethical to submit a paper that 'simply wasn't any good?' i.e. any academic who doesn't think a paper should be published, but submits it on the off-chance they think it might be?
- tptacek 6y ago
  No. Submitting a flawed paper in good faith to a journal isn't unethical; journals exist to evaluate papers. Deliberately submitting bad papers --- in fact, going out of your way to shade what's bad about your papers to try to get them further into the process --- is a waste of volunteer time. Reviewers have offered to evaluate good faith papers; they have not offered to have their time spent as guinea pigs. Hence IRBs, to tell you not to do stuff like this.

37 points by nec4b 6y ago

As a Portland State donor, I'm disappointed at PSU's refusal to back legitimate scientific inquiry. Sending bogus garbage to journals occasionally is a reasonable way to test that peer-review systems are working. The events I normally donate to support are canceled due to the pandemic, though, so I probably wouldn't donate anyway. Guess I can't really take a moral stance here.

valuearb 6y ago
As a PSU alum, I never donate to this halfassed institution, a glorified community college.

zimpenfish 6y ago

[2019]

dang 6y ago
Good catch. Added.

jrumbut 6y ago

I don't know how you can get clearer cut research misconduct than faking an experiment.

manfredo 6y ago
That wasn't the actual experiment, though. The actual experiment was to test whether or not these publications were capable of detecting fake papers.
- jrumbut 6y ago
  Understood, but in the process he claimed to do a large experiment that he didn't do. He faked data, he broke the trust that the entire research enterprise is built on.
  - tptacek 6y ago
    I think you're going to find that most people commenting on this have never reviewed a journal article and don't understand the process. You see it in replication crisis threads as well: the belief that academic research is premised on reviewers replicating results before things get published. The idea that academic journals are based on a presumption of good faith is totally alien to a lot of HN commenters.
    - naasking 6y ago
      Reasonable critics aren't expecting reviewers to replicate the results, but the dog park paper illustrates that reviewers can't even recognized clearly bogus data. I mean, have you even read the methods the paper said they employed?
      - tptacek 6y ago
        I read several of the hoax papers, and skimmed one that included dog park data. Which paper are you referring to, where was it submitted, and where was it accepted? The hoax authors were exceptionally dishonest in their presentation of the results: virtually all of their papers were rejected, the accepts were generally in very low-impact venues, and the papers that got accepted were generally not the lurid ones they highlighted in their summary. I ask for specifics because it is not an interesting research result if they created a paper with clearly bogus data that was then rejected. The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
deleted 6y ago
[deleted]

kaiju0 6y ago

This looks like a valid test case. How else could you verify the legitimacy of previously submitted articles. The emperor has no clothes.

free_rms 6y ago

So, when Sokal did something similar in the 90s, we all had a laugh and life went on. Now that it's today, we're more vicious and this kind of punking will not go unpunished. Good checkpoint on where we're at.

pjc50 6y ago

This website seems to be an extremely partisan culture war site basically focused on issues academics discuss that conservatives don't like? And then "cancelling" them?

RcouF1uZ4gsC 6y ago

As an aside for everyone coming to the comments first before reading the article, PSU in the title is Portland State University not Penn State University which I immediately thought of when I saw PSU. Basically, Portland State University is retaliating against a professor who showed that the peer review and acceptance process for various journals was woefully broken.

avs733 6y ago
another perspective is that they are disciplining a professor who practiced research misconduct

jxramos 6y ago

Looks like it boiled down to the IRB's take on the matter. The school asserted that Boghossian had unethically conducted research on human subjects with his experiment. According to the school’s Institutional Review Board, Boghossian would have needed to obtain “informed consent” from the individuals reviewing his hoax articles in order for his actions to have been considered ethical. Does this catch-22 mean no more investigative meta-research would be tolerated by universities? Even if those human subjects are kept anonymous, they're still afforded IRB protections? Is there anything we can know apriori that those subjects aren't actually subjected to anything high risk to them? Maybe that's a bad direction to take to even allow that?

Kednicma 6y ago
I wonder how this could be done ethically but in a way that doesn't allow peer reviewers to cheat. One thing that comes to mind is ABX testing. Show peer reviewers both a paper which they should accept, and a paper which they should reject, and then give them questionable papers to classify. At what point does it become unethical simply to be a peer-reviewer for a broken journal? That's the implicit question that should be asked in this sort of ethics review. It does no good to punish just a submitter of papers, if the entire journal is unhealthy.
- tptacek 6y ago
  Low-impact-factor journals that accept papers with 1000 of ethnographic field studies aren't "broken" just because the data was fake, because it isn't the premise of peer review to reliably spot fake data. You can make the argument that there are too many low-impact journals and schools shouldn't be allocating resources towards keeping them going; I don't think it's a super strong argument but it's at least coherent. But more importantly: it won't have anything to do with this hoax; if you're willing to fake data, you can get accepted in higher-impact journals outside the social sciences as well.
  - Kednicma 6y ago
    I'm not sure how fake data is substantially different from any other part of a fake paper. Just like how randomly-generated sentences might sound reasonable, faked data might look like real data. In both cases, we expect peer review to engage and to consider how data was sourced. This happened in the "Who's Afraid of Peer Review?" incident [0], where the offending papers were intentionally designed to have many red flags that would have helped reviewers understand that the data was bogus. More worryingly, data cannot model itself, so the quality of data should be largely irrelevant to any models which purport to explain it; as a result, a paper's contribution to science needs to be structured to work with other results, and cannot simply proclaim that its modelling is correct because some given equation fits the observed data. To pick a situation where your expertise shines, imagine that somebody submits a paper about a 3-SAT solver, and they include data that shows their solver doing not just extremely well on standard problem sets, but asymptotically scaling better than exponentially. The reason that you might doubt the honest presentation of the data is because you know, from having studied the field, that such behavior is unlikely. [0] https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%3F https://en.wikipedia.org/wiki/Who%27s_Afraid_of_Peer_Review%...

easterncalculus 6y ago

This happened a bit ago, you can read more about it here: https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair

avs733 6y ago

Just a note that Campusreform.org is...a mess of a site that publishes opinions as factual articles and is primarily writen by undergraduates. Here is a more fair handed presentation of this case: https://www.insidehighered.com/news/2019/01/08/author-recent-academic-hoax-faces-disciplinary-action-portland-state https://www.insidehighered.com/news/2019/01/08/author-recent...

avs733 6y ago

An ongoing misconception here is that academic journals only publish 'facts' Not really. They publish research articles. Depending on the field, and the field's philosophical perspective on the construction of knowledge, that can take differing forms. Those differing forms are not just valid but important. And understanding the differences is also, important. An article about medical treatment necessarily does different things and looks different than a journal in the humanities (for example). They make meaning from information in different ways - and that is what all forms of research do, make meaning. Research at its core is not about the discovery of facts...this is basic philosophy of science, basic Thomas Kuhn. Standards like replicability may be of use for making meaning in some fields, but they are not necessarily equally useful in others, often because the level of contextual situating that needs to occur reaches towards the impossible. Serious members of the field would not have taken the Grievance Studies articles as 'capital-F-FACT' they instead would have interpreted it as a perspective, an argument, an interpretation. Specifically, the entire field of critical studies exists to, in varying forms, critique the ways in which 'discovery of facts' is a reductivist way of looking at meaning making that privileges certain perspectives over others - by treating some perspectives as reality. So the critique here, and looking back at Sokal, isn't that 'hahaha I pulled one over on you' its that 'we assumed that you were giving us a new perspective in good faith that we could collectively learn from' and ...'now you are standing here laughing at us because you acted like a jerk.' It's two different world views with one willing to be open (by choice) to outside critique and perspectives, and the other supremely self-confident that critique and perspective is unnecessary. In effect, it's like a legal opinion that separates matters of interpreting law and matters of finding fact.

morninglight 6y ago

If Mark McLellan remains VP for Research and Graduate Studies, it casts a cloud over academic excellence at PSU. What is his background? How long has he been in this position? What would cause him to make such a politically driven bad decision? It is inconceivable that someone with so little understanding of the toxic academic publishing environment could be influencing the education of 26,000 students.

lwneal 6y ago

The PSU Grievance Studies Affair [1] had the same objective as the 1996 Sokal Affair: to test peer review by submitting obviously bad papers that should be rejected, and seeing how many of them are accepted. The motivation behind this is entirely reasonable. Bad "true negative" as well as known-good "true positive" articles ought to be regularly submitted to every peer review process as a test of accuracy. A great example of a similar idea was the 2014 NIPS consistency experiment [2]. However, there is a problem with the "dog rape" hoax article [3] as a test of the peer review system. Specifically, the author in that paper claims to have spent over 1000 hours carefully cataloguing the behavior of ten thousand people and dogs over the course of one year. The paper then proceeds to produce silly but surprising statistics: female dogs are 70% more likely to be leashed than male dogs, 100% of dogs with shock collars are male, 847 instances of dogs fighting were observed, and so on. The problem here is that the data was falsified: the observations and experiments claimed never happened, the numbers were all made up. The article was likely accepted because of its data, but the data was a lie. If this data had been real, it might have been a minor but useful study relevant to economists, sociologists, or urban planners, regardless of how silly or made-up the conclusions at the end were. Peer review works based on the assumption that the author is telling the truth about what experiments they conducted and what numbers they measured. The grievance studies authors could have submitted fake and silly hoax papers without falsifying data: that's what Sokal did, and that would have been a valid experiment testing the peer review system. But that is not what the authors chose to do. [1] https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair [2] http://blog.mrtz.org/2014/12/15/the-nips-experiment.html http://blog.mrtz.org/2014/12/15/the-nips-experiment.html [3] http://norskk.is/bytta/menn/dog_park.pdf http://norskk.is/bytta/menn/dog_park.pdf

goldcd 6y ago

Would it be equally unethical to submit a paper that 'simply wasn't any good?' i.e. any academic who doesn't think a paper should be published, but submits it on the off-chance they think it might be?

tptacek 6y ago
No. Submitting a flawed paper in good faith to a journal isn't unethical; journals exist to evaluate papers. Deliberately submitting bad papers --- in fact, going out of your way to shade what's bad about your papers to try to get them further into the process --- is a waste of volunteer time. Reviewers have offered to evaluate good faith papers; they have not offered to have their time spent as guinea pigs. Hence IRBs, to tell you not to do stuff like this.