4 ms·

There is a video from a debate with Daryl Bem about those results. It is worth a look, because he admits to using various methods that increasing the degrees of

by wuch 9y ago

There is a video from a debate with Daryl Bem about those results. It is worth a look, because he admits to using various methods that increasing the degrees of freedom, and not taking those into account during statistical analysis. Essentially, he honestly admits to p-hacking, but doesn't seem to recognize anything wrong in that. On those grounds, I think, it is little misleading to suggest that those studies surpasses the statistical standards of many fields. The statistical tests are completely misapplied, but you can't tell just by looking at publications.

nkurz 9y ago
Do you have a link to the debate you mean? I'd be interested to see it, but when I searched I found several. One theory is that while the initial experiments were p-hacked to some extent, the replications (by definition) were not. Some trustworthy sources argue that the problem is that the replications actually weren't, but I haven't looked into the specifics closely enough to know whether that's the case. That said, I thought Bem came across quite well in this interview: http://skeptiko.com/daryl-bem-responds-to-parapsychology-debunkers http://skeptiko.com/daryl-bem-responds-to-parapsychology-deb.... I think, it is little misleading to suggest that those studies surpasses the statistical standards of many fields I didn't mean to imply that the statistical standards were good ones, rather that the standards in some fields are abysmally low. For example, there's a high profile case going on with the Cornell Food Lab, where the extremely prolific lead researcher seemed not just blissfully unaware but proud of the unseen pitfalls around him: https://web.archive.org/web/20170312041524/http:/www.brianwansink.com/phd-advice/the-grad-student-who-never-said-no https://web.archive.org/web/20170312041524/http:/www.brianwa.... The statistical tests are completely misapplied, but you can't tell just by looking at publications. I think that's true, but (in my opinion) the problem is finding papers outside the very hard sciences where they _aren't_ completely misapplied. For example, I feel there is an insurmountable issue with applying formal statistics to any sort of meta-studies, where the data is at best a convenience sample, and (almost?) all conclusions are conditional on the unknown biases of the sample. This of course doesn't mean that all (or even most) meta-analyses produce the wrong answer, but I think it does mean that they should be treated as rhetorical rather than logical arguments.
- heymijo 9y ago
  This caught my attention: "...insurmountable issue with applying formal statistics to any sort of meta-studies..." If I wanted to get a better understanding of what formed your opinion on this, where might I look? P.S. I enjoyed the way you formatted and responded to your parent comment.
  - nkurz 9y ago
    My belief is mostly intuitive and likely more extreme than most, and I don't know of a good single source to point to. John Ioannidis' writings are probably a good starting point: http://retractionwatch.com/2016/09/13/we-have-an-epidemic-of-deeply-flawed-meta-analyses-says-john-ioannidis/ http://retractionwatch.com/2016/09/13/we-have-an-epidemic-of.... Searching for "convenience sampling" on Andrew Gelman's blog yields lots of good discussion in the comments: http://andrewgelman.com/?s=%22convenience+sample%22 http://andrewgelman.com/?s=%22convenience+sample%22. Miguel Hernan's book on Causal Inference gives a good sense of the pitfalls of biased sampling: https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/ https://www.hsph.harvard.edu/miguel-hernan/causal-inference-.... Sorry I can't do better, and maybe others can add better sources. P.S. I enjoyed the way you formatted and responded to your parent comment. Thanks, the style is sometimes referred to as "interleaved" or "inline", as opposed to "top posting" and "bottom posting": https://brooksreview.net/2011/01/interleaved-email/ https://brooksreview.net/2011/01/interleaved-email/. It was the norm for early online communications, but has mostly fallen out of favor. I think it works very well for some situations, although it too is surprisingly controversial: https://news.ycombinator.com/item?id=5233428 https://news.ycombinator.com/item?id=5233428.
amelius 9y ago
There's also a Google Tech Talk on the subject of ESP, by Rupert Sheldrake. https://www.youtube.com/watch?v=JnA8GUtXpXY https://www.youtube.com/watch?v=JnA8GUtXpXY

by wuch 9y ago

nkurz 9y ago

Do you have a link to the debate you mean? I'd be interested to see it, but when I searched I found several. One theory is that while the initial experiments were p-hacked to some extent, the replications (by definition) were not. Some trustworthy sources argue that the problem is that the replications actually weren't, but I haven't looked into the specifics closely enough to know whether that's the case. That said, I thought Bem came across quite well in this interview: http://skeptiko.com/daryl-bem-responds-to-parapsychology-debunkers http://skeptiko.com/daryl-bem-responds-to-parapsychology-deb.... I think, it is little misleading to suggest that those studies surpasses the statistical standards of many fields I didn't mean to imply that the statistical standards were good ones, rather that the standards in some fields are abysmally low. For example, there's a high profile case going on with the Cornell Food Lab, where the extremely prolific lead researcher seemed not just blissfully unaware but proud of the unseen pitfalls around him: https://web.archive.org/web/20170312041524/http:/www.brianwansink.com/phd-advice/the-grad-student-who-never-said-no https://web.archive.org/web/20170312041524/http:/www.brianwa.... The statistical tests are completely misapplied, but you can't tell just by looking at publications. I think that's true, but (in my opinion) the problem is finding papers outside the very hard sciences where they _aren't_ completely misapplied. For example, I feel there is an insurmountable issue with applying formal statistics to any sort of meta-studies, where the data is at best a convenience sample, and (almost?) all conclusions are conditional on the unknown biases of the sample. This of course doesn't mean that all (or even most) meta-analyses produce the wrong answer, but I think it does mean that they should be treated as rhetorical rather than logical arguments.

heymijo 9y ago
This caught my attention: "...insurmountable issue with applying formal statistics to any sort of meta-studies..." If I wanted to get a better understanding of what formed your opinion on this, where might I look? P.S. I enjoyed the way you formatted and responded to your parent comment.
- nkurz 9y ago
  My belief is mostly intuitive and likely more extreme than most, and I don't know of a good single source to point to. John Ioannidis' writings are probably a good starting point: http://retractionwatch.com/2016/09/13/we-have-an-epidemic-of-deeply-flawed-meta-analyses-says-john-ioannidis/ http://retractionwatch.com/2016/09/13/we-have-an-epidemic-of.... Searching for "convenience sampling" on Andrew Gelman's blog yields lots of good discussion in the comments: http://andrewgelman.com/?s=%22convenience+sample%22 http://andrewgelman.com/?s=%22convenience+sample%22. Miguel Hernan's book on Causal Inference gives a good sense of the pitfalls of biased sampling: https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/ https://www.hsph.harvard.edu/miguel-hernan/causal-inference-.... Sorry I can't do better, and maybe others can add better sources. P.S. I enjoyed the way you formatted and responded to your parent comment. Thanks, the style is sometimes referred to as "interleaved" or "inline", as opposed to "top posting" and "bottom posting": https://brooksreview.net/2011/01/interleaved-email/ https://brooksreview.net/2011/01/interleaved-email/. It was the norm for early online communications, but has mostly fallen out of favor. I think it works very well for some situations, although it too is surprisingly controversial: https://news.ycombinator.com/item?id=5233428 https://news.ycombinator.com/item?id=5233428.

amelius 9y ago

There's also a Google Tech Talk on the subject of ESP, by Rupert Sheldrake. https://www.youtube.com/watch?v=JnA8GUtXpXY https://www.youtube.com/watch?v=JnA8GUtXpXY