7 ms·

Journal rejects studies contradicting precognition

24 points by wallflower 16y ago

sigzero 16y ago
Did they reject the studies BEFORE they were submitted or AFTER?
schrototo 16y ago
The TL;DR version: "This journal does not publish replication studies, whether successful or unsuccessful"
- faboo 16y ago
  Which, to me, seems to be an admission that the Journal of Personality and Social Psychology isn't actually a scientific journal. An initial experiment/study is only a part of the process. If you get extraordinary results out of an experiment, but no one else sees the same results with the same experiment, well, something was up with your run of it. It is vital to the body of scientific knowledge that re-runs of published experiments are performed, recorded, and published. Otherwise, there's no way to know whether the original result was real or a fluke (or fraudulent).
  - mcburton 16y ago
    When dealing with failed replicability it is important not to conflate a poorly designed experiment with a poorly written methods section. Often there is tacit or socially negotiated knowledge required to properly replicate an experiment. This doesn't mean the science is bad, it means science is more social than some would like to admit. For some interesting reading on this check out Harry Collins's paper "The Seven Sexes: A Study in the Sociology of a Phenomenon, or the Replication of Experiments in Physics"[1] [1] http://soc.sagepub.com/content/9/2/205.short http://soc.sagepub.com/content/9/2/205.short
- pessimizer 16y ago
  The truth of that depends on the accuracy of the title of: Violence is a curvilinear function of temperature in Dallas: A replication. Rotton, James; Cohn, Ellen G. Journal of Personality and Social Psychology, Vol 78(6), Jun 2000, 1074-1081. doi: 10.1037/0022-3514.78.6.1074 http://psycnet.apa.org/psycinfo/2000-07798-005 http://psycnet.apa.org/psycinfo/2000-07798-005
- orangecat 16y ago
  I spluttered incoherently after reading that line. Obligatory xkcd: http://xkcd.com/882/ http://xkcd.com/882/
tokenadult 16y ago
Richard Wiseman, interviewed for the submitted article, is a psychologist who studies human cognitive biases. His concern regarding Bem's purported studies on precognition is that if journals reject studies showing no effect, while accepting flawed studies appearing to show precognition, this results in a file drawer problem http://en.wikipedia.org/wiki/Publication_bias#The_file_drawer_effect http://en.wikipedia.org/wiki/Publication_bias#The_file_drawe... in which publication bias among the published studies will then skew any subsequent attempt at meta-analysis of the published studies. The file drawer effect is of particular concern in meta-analysis of purported extrasensory perception. http://www.skepdic.com/filedrawer.html http://www.skepdic.com/filedrawer.html By the way, Wiseman is perhaps best known online as the author of the Colour-changing Card Trick website, featuring videos of an experiment that demonstrates selective attention in human perception. http://www.quirkology.com/USA/Video_ColourChangingTrick.shtml http://www.quirkology.com/USA/Video_ColourChangingTrick.shtm... If you haven't seen the video before, it is well worth watching. I hope the more responsible journals of psychology will step up their efforts to publish commentaries on the study design http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html of ESP studies and publish in general more studies showing failures of replication of earlier study findings.
Semiapies 16y ago
I'm curious how this will play out in the scientific world - if there's a slew of failed attempts to reproduce the findings, this will be the new cold fusion, and such an embarrassment may actually affect the way papers get published. In the layman world, of course, the original paper will be used as justification for whole shelves of pseudoscience books, and the public won't actually hear about any failed attempts to reproduce the claims.
scott_s 16y ago
As a response to the original study, please refer to "Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi": http://commonsenseatheism.com/wp-content/uploads/2010/11/Wagenmakers-Why-Psychologists-Must-Change-the-Way-They-Analyze-Their-Data.pdf http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... The main point of it is that the parapsychologists have done nothing wrong: they're following the same methods and using the same techniques as the actual psychologists. The real problem is that those methods and techniques are not good enough. Also related, "Parapsychology: The Control Group for Science": http://lesswrong.com/lw/1ib/parapsychology_the_control_group_for_science/ http://lesswrong.com/lw/1ib/parapsychology_the_control_group...
- naveensundar 16y ago
  I don't know how the lesswrong people came to that conclusion about parapsychology a-priori. This is a problem in Science in general. People use parametric-value-based reasoning (e.g. significance values) to get binary conclusions (existence or not of physical processes etc). http://dilbert.com/fast/1998-01-19/ http://dilbert.com/fast/1998-01-19/
nickff 16y ago
This is exactly what Richard Feynman described in his 1974 commencement address at Caltech. http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm

24 points by wallflower 16y ago

sigzero 16y ago

Did they reject the studies BEFORE they were submitted or AFTER?

schrototo 16y ago

The TL;DR version: "This journal does not publish replication studies, whether successful or unsuccessful"

faboo 16y ago
Which, to me, seems to be an admission that the Journal of Personality and Social Psychology isn't actually a scientific journal. An initial experiment/study is only a part of the process. If you get extraordinary results out of an experiment, but no one else sees the same results with the same experiment, well, something was up with your run of it. It is vital to the body of scientific knowledge that re-runs of published experiments are performed, recorded, and published. Otherwise, there's no way to know whether the original result was real or a fluke (or fraudulent).
- mcburton 16y ago
  When dealing with failed replicability it is important not to conflate a poorly designed experiment with a poorly written methods section. Often there is tacit or socially negotiated knowledge required to properly replicate an experiment. This doesn't mean the science is bad, it means science is more social than some would like to admit. For some interesting reading on this check out Harry Collins's paper "The Seven Sexes: A Study in the Sociology of a Phenomenon, or the Replication of Experiments in Physics"[1] [1] http://soc.sagepub.com/content/9/2/205.short http://soc.sagepub.com/content/9/2/205.short
pessimizer 16y ago
The truth of that depends on the accuracy of the title of: Violence is a curvilinear function of temperature in Dallas: A replication. Rotton, James; Cohn, Ellen G. Journal of Personality and Social Psychology, Vol 78(6), Jun 2000, 1074-1081. doi: 10.1037/0022-3514.78.6.1074 http://psycnet.apa.org/psycinfo/2000-07798-005 http://psycnet.apa.org/psycinfo/2000-07798-005
orangecat 16y ago
I spluttered incoherently after reading that line. Obligatory xkcd: http://xkcd.com/882/ http://xkcd.com/882/

tokenadult 16y ago

Richard Wiseman, interviewed for the submitted article, is a psychologist who studies human cognitive biases. His concern regarding Bem's purported studies on precognition is that if journals reject studies showing no effect, while accepting flawed studies appearing to show precognition, this results in a file drawer problem http://en.wikipedia.org/wiki/Publication_bias#The_file_drawer_effect http://en.wikipedia.org/wiki/Publication_bias#The_file_drawe... in which publication bias among the published studies will then skew any subsequent attempt at meta-analysis of the published studies. The file drawer effect is of particular concern in meta-analysis of purported extrasensory perception. http://www.skepdic.com/filedrawer.html http://www.skepdic.com/filedrawer.html By the way, Wiseman is perhaps best known online as the author of the Colour-changing Card Trick website, featuring videos of an experiment that demonstrates selective attention in human perception. http://www.quirkology.com/USA/Video_ColourChangingTrick.shtml http://www.quirkology.com/USA/Video_ColourChangingTrick.shtm... If you haven't seen the video before, it is well worth watching. I hope the more responsible journals of psychology will step up their efforts to publish commentaries on the study design http://norvig.com/experiment-design.html http://norvig.com/experiment-design.html of ESP studies and publish in general more studies showing failures of replication of earlier study findings.

Semiapies 16y ago

I'm curious how this will play out in the scientific world - if there's a slew of failed attempts to reproduce the findings, this will be the new cold fusion, and such an embarrassment may actually affect the way papers get published. In the layman world, of course, the original paper will be used as justification for whole shelves of pseudoscience books, and the public won't actually hear about any failed attempts to reproduce the claims.

scott_s 16y ago

As a response to the original study, please refer to "Why Psychologists Must Change the Way They Analyze Their Data: The Case of Psi": http://commonsenseatheism.com/wp-content/uploads/2010/11/Wagenmakers-Why-Psychologists-Must-Change-the-Way-They-Analyze-Their-Data.pdf http://commonsenseatheism.com/wp-content/uploads/2010/11/Wag... The main point of it is that the parapsychologists have done nothing wrong: they're following the same methods and using the same techniques as the actual psychologists. The real problem is that those methods and techniques are not good enough. Also related, "Parapsychology: The Control Group for Science": http://lesswrong.com/lw/1ib/parapsychology_the_control_group_for_science/ http://lesswrong.com/lw/1ib/parapsychology_the_control_group...

naveensundar 16y ago
I don't know how the lesswrong people came to that conclusion about parapsychology a-priori. This is a problem in Science in general. People use parametric-value-based reasoning (e.g. significance values) to get binary conclusions (existence or not of physical processes etc). http://dilbert.com/fast/1998-01-19/ http://dilbert.com/fast/1998-01-19/

nickff 16y ago

This is exactly what Richard Feynman described in his 1974 commencement address at Caltech. http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm