4 ms·
Reasonable critics aren't expecting reviewers to replicate the results, but the dog park paper illustrates that reviewers can't even recognized clearly bogus da
by naasking 6y ago
Reasonable critics aren't expecting reviewers to replicate the results, but the dog park paper illustrates that reviewers can't even recognized clearly bogus data. I mean, have you even read the methods the paper said they employed?
- tptacek 6y agoI read several of the hoax papers, and skimmed one that included dog park data. Which paper are you referring to, where was it submitted, and where was it accepted? The hoax authors were exceptionally dishonest in their presentation of the results: virtually all of their papers were rejected, the accepts were generally in very low-impact venues, and the papers that got accepted were generally not the lurid ones they highlighted in their summary. I ask for specifics because it is not an interesting research result if they created a paper with clearly bogus data that was then rejected. The problem with the experiment, of course, is that reviewing takes a fuckload of time and effort, and most fields barely keep up with the legitimate workload they have. They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that.
- manfredo 6y agoIt was accepted by Gender, Place & Culture. In fact it was not only accepted, it received an award for exemplary scholarship. Out of the 21 papers submitted [1] 9 were rejected, 7 were published, 3 were asked to revised and resubmit for publication, and 1 was currently under review when the hoax was revealed. The fact that totally bogus papers have about a coin flip chance of being accepted is astounding. 1. https://en.wikipedia.org/wiki/Grievance_studies_affair https://en.wikipedia.org/wiki/Grievance_studies_affair
- tptacek 6y agoYou're astounded that a piece that faked 1000 hours of observational studies was accepted in the #15th-ranked gender studies journal? Respectfully: I think what's shocking you is how many journals there are. (Revise-and-resubmit, by the way, is a nice way of saying "reject").
- manfredo 6y agoI'm astounded that a research didn't bat an eye about someone claiming to have observed a thousand instances of dog-humping, managed to identify the genders of the dogs, as well as the sexual orientation of the owners - and to have published the paper anyway when the authors claimed to have accidentally thrown away the original data. And to put the cherry on top, to have decided to give this paper an award for exemplary scholarship.
- tptacek 6y agoThey didn't claim "a thousand instances". They claim 1000 hours of field study. That number shocks people, but over the course of a year, it's a half-time job for a researcher, and so your argument is left at "that seems like a dumb thing for a researcher to allocate half their time to". I mean, sure, I agree, but so what? That's not a very interesting argument, or a damning one.
- manfredo 6y agoI think you're missing the series of events. The authors never submitted any false data. They claimed that this was what their research found, and when the journal asked for the actual research data the authors claimed that they had written the finding on pen and paper and lost the only physical copy. The journal published the paper anyway, without ever even seeing the data.
- tptacek 6y agoWhat you're saying now is that you think it's standard practice for peer reviewers to request the raw data for papers they're reviewing? The paper authors signed an actual contract affirming that they hadn't fabricated the data.
- manfredo 6y agoIt's normal practice for publications to publish papers from authors who explicitly say that they don't even retain the data used to produce the paper? If that's the case what's even the point of peer review? At this point it seems like you're saying that peer review doesn't actually involve any sort of review. Fortunately, though, other academics don't share your experience that peer review is incapable of identifying faulty research. Because if it did, then there'd be little to no reason to put trust in academia.
- naasking 6y ago> The hoax authors were exceptionally dishonest in their presentation of the results: virtually all of their papers were rejected, the accepts were generally in very low-impact venues, and the papers that got accepted were generally not the lurid ones they highlighted in their summary. I think the other poster summarizes why quite well why this charge of dishonesty is ironic. I'll just add a link to the paper itself if you'd like to read it [1], and review one part: > From 10 June 2016, to 10 June 2017, I stationed myself on benches that were in central observational locations at three dog parks in Southeast Portland, Oregon. Observation sessions varied widely according to the day of the week and time of day. These, however, lasted a minimum of two and no more than 7 h and concluded by 7:30 pm (due to visibility). I did not conduct any observations in heavy rain. [...] The usual caveats of observational research also apply here. While I closely and respectfully examined the genitals of slightly fewer than ten thousand dogs [...] So in the span of one year, this lone "researcher" claims to have "closely" inspected the genitals of ~10,000 dogs. That's 1,000 hours to inspect 10,000 dogs, which amounts to 10 dogs per hour, during which they took detailed notes on the dogs and owner's names, gender, and other associated information, while documenting the dogs' behaviour (6 minutes per dog+owner!). That stretches credulity to say the least. Also, for the data to be meaningful, there must be at least 10,000 unique dogs visiting these three dog parks during the given time span. This also beggars belief even for Portland which features a high percentage of dog ownership. Portland has ~264,000 households, ~70% of households own a dog, that's ~185,000 dogs across ~32 dog parks, which is only 5,000 unique dogs per park on average. The basic math just doesn't add up, and then the researcher disclaims their abilities to determine canine breeds, but makes claims like, "NB: the phrase ‘dog rape/humping incident’ documents only those incidents in which the activity appeared unwanted from my perspective – the humped dog having given no encouragement and apparently not enjoying the activity." So apparently they have quite a bit of insight into canine behavioural psychology. There is a lot about the methods and the data that make no sense, and this paper received accolades. > They are literally taking time and resources from program committees, and they do have IRB obligations in order to do that. That's a legitimate concern. Unfortunately, the hoax itself reveals that these program committees may not be doing much meaningful work with those resources anyway, which seems like a far more important matter. Edit: I would add that some way to verify that peer review is doing its job should be part of the publishing process. Periodic random hoaxes seem like a good way of doing it. It will make everyone, particularly reviewers, more skeptical and cautious. > (Revise-and-resubmit, by the way, is a nice way of saying "reject"). No, it's a nice way of saying, "this is good work, you just need to massage your presentation". [1] http://norskk.is/bytta/menn/dog_park.pdf http://norskk.is/bytta/menn/dog_park.pdf