4 ms·
"One evening at the SIPS conference, after sessions had been concluded and beers consumed, a researcher asked Heathers for his opinion of a study that seemed su
by mcguire 8y ago
"One evening at the SIPS conference, after sessions had been concluded and beers consumed, a researcher asked Heathers for his opinion of a study that seemed suspicious. When Heathers decided he’d heard enough to render a verdict, he took a few steps back and began to shout.
""Give me a B!"
""B!" the assembled scientists replied.
""Give me a U!""
And here we see how more bullshit gets started. Deciding that something sounds bad means it is wrong is exactly the same error as deciding that something sounds good means it is right.
- jcranmer 8y ago> Deciding that something sounds bad means it is wrong is exactly the same error as deciding that something sounds good means it is right. False equivalence. A correct paper inherently has no flaws, while a wrong paper inherently has at least one flaw. It is very possible to be able to immediately detect some classes of flaws, and that in no way means that one can immediately find all of the flaws. When I was a TA, I found several people who cheated. One person I discovered because there's no way a struggling student in an introductory C course is going to pull out code with gotos and register keyword usage plastered everywhere. Do I claim to have caught everyone who cheated? Hell no.
- eslaught 8y agoI don't know if you've ever been subject to scientific peer review or not. One of the most frustrating things is reviewers who misunderstand the paper, but are convinced they've found some sort of problem with it. Admittedly, these sorts of issues usually point to places where the paper could be more clear. However, the point still stands that it is far easier to misunderstand something, than it is to understand it and find a flaw. I would be very, very suspicious of any scientist claiming to have found a flaw in a work they've only been given a superficial description of. It may be accurate to say they have a hunch of where a flaw might be. But without really spending time with a paper it would foolish to claim any degree of certainty. Only very low quality papers can be rejected so quickly.
- TangoTrotFox 8y agoThere was an interesting study recently that showed prediction markets actually did a better job of determining 'fake' papers than journals in social sciences. [1][2] A group of researchers took some 21 papers published in Nature and Science from 2010 to 2015. 13 were able to be replicated. The prediction market accurately determined all 13 predictable papers, and 5 of the 'fake' papers. They gave the remaining 3 about 50/50 odds. Though one thing to be said there is that even in studies that they managed to replicate, the effect size was found to be, on average, 50% of the stated effect size. I think there's long been a perception that social sciences are heavily influenced by people who take whatever their biases are, create some experiment specifically designed to show them, and then play with the numbers or experiment's parameters until they manage to do so. This goes all the way back to (and certainly before) Zimbardo's Stanford Prison Experiment. There was nothing organic there. Participants, both prisoner and guard, were heavily coached on how to act and, in their own words, saw the experiment more as an acting role than emergent normal behavior. It seems to be that this perception is accurate. In a society where people are increasingly radicalizing on social views, we ought expect social sciences to become even more dysfunctional in the years to come. This sort of stuff is, in turn, casting a very dangerous cloud over the rest of science since people tend to extrapolate these actions and behaviors in the social sciences, to science as a whole. In my opinion we need to start creating a strong distinction between science driven by science that yields falsifiability, predictability, and is driven exclusively by direct experimental result -- as compared to not-quite-science that is based on models, abstract experimentation, is not falsifiable, and does not provide meaningful predictions. What I mean by meaningful is that the whole point of predictability is not to have something to encourage political action on as is often the case in social science, but to use as a litmus test for the accuracy of a hypothesis. If it's true then that provides strength to the hypothesis, if it's false then the hypothesis is false. Without falsifiability, predictions are worthless. [1] - https://www.bloomberg.com/view/articles/2018-08-30/prediction-markets-seem-to-assess-studies-better-than-peer-review https://www.bloomberg.com/view/articles/2018-08-30/predictio... [2] - https://www.nature.com/articles/s41562-018-0399-z https://www.nature.com/articles/s41562-018-0399-z
- sievebrain 8y ago