10 ms·

The Control Group Is Out of Control

179 points by nbouscal 12y ago

tomp 12y ago
From A Meta-Analysis of 90 Experiments on the Anomalous Anticipation of Random Future Events [1]: > [...] a meta-analysis of 90 experiments from 33 laboratories in 14 different countries which yielded an overall positive effect in excess of 6 sigma with an effect size (Hedges’ g) of 0.09, combined z = 6.33, p = 1.2 × 10-10. A Bayesian analysis yielded a Bayes Factor of 7.4 × 10-9, greatly exceeding the criterion value of 100 for "decisive evidence" in favor of the experimental hypothesis (Jeffries, 1961). [...] The number of potentially unretrieved experiments averaging a null effect that would be required to reduce the overall effect size to a trivial value was conservatively calculated to be 520. An analysis of p values across experiments implies that the results were not a product of "p-hacking," [...]. [1] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2423692 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2423692
- robinh 12y ago
  So... I know next to nothing about parapsychology itself, but have seen a lot of the drama around it, and I just have to ask: is it not intellectually dishonest to call something the 'control group of science' when their results overwhelmingly support the hypothesis? I have a hard time seeing how this is different from any other form of science denialism. "We don't like the results because they clash with our preconceived notions of how the universe works so we made up this thing to ignore your evidence"? That's hardly a valid complaint. Basically, on what grounds can people claim one field to be nonsense (e.g. calling parapsychology the control group of science) but not others? Can someone explain this to me?
  - tomp 12y ago
    Have you read the article? It's basically all along this problem... tl;dr: Yes, that is intellectually dishonest, which is a huge problem. Scientists have two options: (1) accept parapsychology as real, or (2) accept that the "scientific method" (in social "sciences", at least) is insufficient. The problem seems to be that Bem (the author of the study) did almost everything "right", and if we increase the bar of scientific proof so that his study is excluded, so are many others... The reason why parapsychology is excluded: the real world doesn't support it (noone is earning huge amounts of money on the stock market using psi). However, the real world often doesn't support very small effects (e.g. Einstein's relativity). Fortunately for physics, they can afford to make their scientific methods much more rigorous, so they can study large effects as well as exceptionally tiny effects. Social studies can't, for the time being.
    - robinh 12y ago
      Okay, but that's not actually an answer to my question. My question basically comes down to this: > Yes, that is intellectually dishonest, which is a huge problem. Scientists have two options: (1) accept parapsychology as real, or (2) accept that the "scientific method" (in social "sciences", at least) is insufficient. I don't get why the whole thing is such a huge problem. The entire problem rests on needing parapsychology effects to not be real. If that need did not exist, we could just go "Okay, interesting, seems likely that there's something to it then. Let's do more research!" because, you know, we take that approach everywhere else. So my question remains: what is it about parapsychology that makes option two even valid to consider? All I can see is people just not liking that that may be how things work.
      - lmm 12y ago
        Parapsychology is physically impossible, and the evidentiary standards in physics are much higher, so we have much more confidence in our physics results than in these experiments - enough that we can reasonably say that physics is true and parapsychology is false. (But these experiments are as good as many in psychology / social science - suggesting that many "proven" results in psychology / social science could be false)
        dragonwriter 12y ago
        > Parapsychology is physically impossible, and the evidentiary standards in physics are much higher, so we have much more confidence in our physics results than in these experiments Conflicting results don't mean one set is impossible (cf., the long-standing apparent conflict between QM and relativity within physics). Apparently conflicting results without a methodological error in either imply that the explanatory model that appears to be supported by at least one of the results (if not both) is, while useful within its own domain, in some way incorrect. The whole idea that the models validated by scientific experimentation are binarily true of false is, well, missing the point badly. While over time we hope they approach truth with them, what they are is useful (that is, they have predictive power) to a greater or lesser extent. And quite often the models with the greatest predictive power in two different domains conflict when either or both are extend outside of their own domain. EDIT: The real problem with parapsychology is that there's little in the way of explanatory models being tested anywhere in the field. There's a lot of hypotheses without models and some experiments testing them, which (concerns about methodology aside, for the moment) might raise interesting questions and serve as inspiration for developing and then testing theoretical models to explain the effects, but very little has been done there -- which makes "parapsychology" more a collection of potentially unexplained phenomena more than a branch of science that provides an explanatory model for some set of phenomena. Which is very different from most of the social sciences.
  - Retric 12y ago
    Individual experiments give false positives and false negatives all the time. Think about how you might film this using a fair coin: http://www.youtube.com/watch?v=X1uJD1O3L08 http://www.youtube.com/watch?v=X1uJD1O3L08 Hint 10 heads is only a 1/1024 chance but 1024 is not that rare. Now realise that 5 coin flips is a 1/32 chance and 1/20 is considered acceptable to publish... Science is not based on a single experiment it's based around replication of experiments by different people at different times using slightly different methods. Parapsychology repeatedly fails this test to the point where there is a million dollar prize for anyone that can demonstrate anything in a controlled setting.
    - jmcmichael 12y ago
      The process to win Randi's prize is not science, it is public relations for the ideology of elimination materialism. Science is the testing of a hypothesis by applying an instrumental injunction to reality, apprehending and analyzing the results, and sharing/verifying the results in with a community of experts who have also performed the same injunction (reproducibility). Randi's process shares none of these characteristics with the scientific process. It is entertainment, a publicity stunt that has little to do with science other than to muddle the waters when discussing these topics.
  - rainmaking 12y ago
    Jack Parsons, who was quite into parapsychology, phrased it quite neatly: It's when science becomes closed minded and degenerates into ancestor worship.
  - jarrett 12y ago
    It's a valid question. The article takes the following premise: Parapsychology gets positive results following the scientific method as defined by prevailing norms in the scientific community. So why can't we draw the conclusion that parapsychology is as legitimate as other branches of science? What is to separate it? I would offer a pseudo-Bayesian[1] answer to that question. Parapsychology aims to prove hypotheses that lacks theoretical foundations. Our current understanding of physics and biology weigh strongly against the existence of psychic phenomena. Even before any experiment is conducted, we must admit that psychic phenomena are unlikely to exist. Our experimental results must be evaluated in light of that prior probability. Thus, parapsychology is and should be held to a higher burden of proof than other branches of science. We should demand more rigorous experimental designs, stronger effects, and smaller p values. This XKCD presents a similar idea, if you substitute "psychic phenomena exist" for "the sun has gone nova": http://www.explainxkcd.com/wiki/index.php?title=1132:_Frequentists_vs._Bayesians http://www.explainxkcd.com/wiki/index.php?title=1132:_Freque... [1] I say "pseudo" because I'm not a statistician by trade. I'm basing my argument on my rather superficial understanding of Bayesian statistics. I still think it's a valid argument in its own right, but I don't claim that it's an accurate representation of Bayesian statistics.
    - ChuckMcM 12y ago
      Parapsychology aims to prove hypotheses that lacks theoretical foundations. This is the lynch pin for most scientists. Before you can have a hypothesis, you must have a theory, and you can work to prove or disprove that theory by experimenting to create or observe results that the theory predicts. Parapsychology doesn't have good, testable, theories. Rather it has some interesting unexplained correlations.
      - pjreddie 12y ago
        Doctors rejected hand washing for a long time because there was no theory behind why it would improve patient care, even when there was overwhelming evidence showing it decreased mortality rates. Do you discount all evidence that doesn't fit into your world view? Maybe theory hasn't caught up with evidence yet.
  - gwern 12y ago
    One man's modus ponens is another man's modus tollens (http://www.gwern.net/Prediction%20markets#modus-tollens-vs-modus-ponens http://www.gwern.net/Prediction%20markets#modus-tollens-vs-m...).
  - onetwofiveten 12y ago
    There are two differences. The first is that you have to replace "preconceived notions of how the universe works" with "vast body of experimental data and scientific understanding of how the universe works". Our evidence for how physics works vastly outweighs current parapsychology research results. The second difference is that the evidence is not being ignored (at least by the author of the article), but is being taken to indicate a real effect, it just isn't the effect of psychic powers. Our knowledge of physics means that the existence of psychic powers is given a much lower prior probability than the possibility of widespread experimental error and bias. Since both explanations are likely to produce slight positive results, the existence of psychic powers is still unlikely once you take the evidence into account. However, the important point in the article is that in order to make this inference in an intellectually honest way, you need to significantly increase your estimation for the probabilities of widespread experimental error in ALL scientific studies that use similar methods. Since the methods in parapsychology are pretty good, this has quite a far reaching effect. In a sense, your question about "on what grounds can people claim one field to be nonsense but not others?" is exactly the same question asked by the author. Except the author isn't implying that parapsychology isn't nonsense, they are implying that many other fields are nonsense too. This makes sense because physics is almost certainly not nonsense.
yummyfajitas 12y ago
If anyone here hasn't read the comments on this as well, go read Eliezer Yudkowsky's comment on meta-analysis: http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/#comment-66077 http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
- SoftwareMaven 12y ago
  I've spent a lot of time in the past year studying nutrition as I've tried to optimize my health. I don't believe there is a field that gets science more wrong and, as a result, whose meta-analyses are, to put it in Yudkowsky's words, more bullshit. Unfortunately, I don't think there is another field whose bullshit gets more media play, either (except maybe politicians, but that's a tautology).
deleted 12y ago
[deleted]
tokenadult 12y ago
This is a very interesting review of some of the most challenging published literature on the reliability of psychological science. The author's point is correct that parapsychology (ESP and the like) basically has no prior plausibility. Yet a researcher who has the chops to do reliable research, Daryl J. Bem, has conducted "experiments" that appear to show that some human experimental subjects can see into the future. As the article says, "Bem definitely picked up a signal. The only question is whether it’s a signal of psi, or a signal of poor experimental technique." Please read this submission from beginning to end and follow the links to see more of the background. I had the opportunity to discuss the original Bem paper on psi in the journal club I regularly participate in. The conclusion is plainly wrong, but the error in procedure is subtle and it took a while for response papers to come out about his initial finding. (And everyone agrees that Bem is a smart man, so the subtle errors are all the more dismaying.) If Bem can't design experiments carefully, who can? AFTER EDIT TO RESPOND TO QUESTIONS IN ANOTHER SUBTHREAD HERE: Another participant is asking why parapsychology has poor prior plausibility. Parapsychology is a rather recent discipline, always with sloppy experimental practices,[1] and meanwhile the disciplines of physics and medicine are much older. The study of how information is transmitted by electromagnetic radiation and how the human body perceives sensations is incompatible with supposed abilities like ESP, and completely incompatible with seeing into the future in the manner described by Bem. Extraordinary claims require extraordinary evidence. A million dollars says that psychic claims are bogus,[2] and so far no one has won a million dollars by proving that proposition wrong. [1] http://www.skepdic.com/projectalpha.html http://www.skepdic.com/projectalpha.html [2] http://www.randi.org/site/index.php/1m-challenge.html http://www.randi.org/site/index.php/1m-challenge.html
- wpietri 12y ago
  To your point on poor plausibility, I'd add that people have been trying to prove and develop their psychic powers for millennia. If you look at other common human traits, you can see that we're pretty good at exploiting our natural capacities. E.g., how the various martial arts traditions have taken modest raw capacity and learned how to develop it to an amazing degree. Ditto science, art, engineering, music, poetry, and literature. The failure of all of the various paranormal traditions to develop anything demonstrable despite a lot of effort suggests that there is little or nothing there.
rainmaking 12y ago
Wait a minute. The author calls for "stricter scientific standards", and than has the audacity to define "strict" as "something that excludes parapsychology". Say it was 1600 A.D. and we would like to improve the standards of science. Everybody knows the cart stops when the ox stops pulling, so why not use this reasonable observation as a litmus test for all future scientific endeavor? Bye, Newton. The whole article is just a particular set of beliefs trying to institutionalize itself, the very thing science is supposed to put an end to.
- Strilanc 12y ago
  That is not what I took away from the article at all. What I took away is: if you think parapsychology is wrong, then you should be really worried about the rest of science because their studies are not better designed. It's most clearly expressed in this paragraph: > This is not a criticism of Bem or a criticism of parapsychology. It’s something that is inherent to the practice of meta-analysis, and even more, inherent to the practice of science. Other than a few very exceptional large medical trials, there is not a study in the world that would survive the level of criticism I am throwing at Bem right now.
- Kutta 12y ago
  It is always right to institutionalize correct ideas, like Newtonian mechanics (while having in mind its later refinements, of course), or the notion that parapsychology doesn't exist.
  - rainmaking 12y ago
    My you have audacious work ethic, examining every human on the planet just to rule that out. Let it be law, then.
  - ksmiley 12y ago
    Ok, but how do you determine which ideas are "correct"?
- 6d0debc071 12y ago
  Alternatively, you can read it that the author is defining strict as, 'Something that would make me believe parapsychology.' At least if it met those standards.
Zigurd 12y ago
Starting out with a flawed analogy to placebos makes the whole article hard to swallow. Placebos are complicated, especially for drugs with receptors in the brain. Stating that we know placebos "don't work" is outright incorrect. We know the opposite: placebos can produce real results, and there are physical mechanisms that explain those results. Nothing paranormal about it. If the conclusion is that there is no magic wand in methodology that makes an experiment "good" - that's true. But the approach gets off to a rough start.
- pdonis 12y ago
  I don't think the analogy with placebos is flawed; it's just not quite the analogy you are describing here. The point is not that placebos "don't work"; it's that, if a placebo works as well as some drug, the drug itself doesn't work. That is, it adds nothing to what you can already get using the placebo effect. Which may indeed be a "real" result; but why pay extra for a drug to get it when you don't need to? Note, btw, that while placebos can indeed produce real results, and there are indeed physical mechanisms that explain those results, those mechanisms have nothing to do with what's in the placebo itself; they are entirely to do with what goes on in the mind and body of the person taking the placebo. In that sense, it's true that placebos "don't work": it doesn't matter what specific substance you put in the placebo pill--sugar, starch, gelatin, whatever--as long as it looks and tastes the same to the subject.
btilly 12y ago
It was an interesting read, but the author showed absolutely no clue why pre-registration was important. The point is to avoid selection bias. If the same exact study is done 100 times, but only the best 50 results get included in your study, a meta analysis of those results is going to show a strong statistical bias. No amount of waving around the word "Bayesian" can fix this - the only way to fix it is pre-registration of honest results. Why am I so certain of this? Because of the following passage: By my count, Bem follows all of the commandments except [6] and [10]. He apologizes for not using pre-registration, but says it’s okay because the studies were exact replications of a previous study that makes it impossible for an unsavory researcher to change the parameters halfway through and does pretty much the same thing. But that misses the whole point. Whether the studies had the same or different parameters is irrelevant - the fact that you see the ones that worked out and don't see the ones that didn't is the only thing that matters. And the fact that the meta analysis didn't get this right makes the output statistical garbage. No further discussion needed.
- Natsu 12y ago
  There is this rather amusing section, though: "(Richard Wiseman – the same guy who provided the negative aura for the Wiseman and Schiltz experiment – has started a pre-register site for Bem replications. He says he has received five of them. This is very promising. There is also a separate pre-register for parapsychology trials in general. I am both extremely pleased at this victory for good science, and ashamed that my own field is apparently behind parapsychology in the “scientific rigor” department)"
  - kumbasha 12y ago
    After reading that article, I actually came away with a very positive view of the field of parapsychology in general. Of course psychics aren't real, but the Flying Spaghetti Monster can be used to make legitimate points about the study of religion, to use an analogy.
- SatvikBeri 12y ago
  To give a concrete example, let's say I want to show that I can influence a coin flip with my mind in order to produce a heads result every time. I do this by publishing trials where each instance involves me flipping a coin four times. I conduct 16,000 trials. On average, 1,000 of them will show that I can flip a coin and get heads every time. If all I publish are these trials, that yields the amazing result that I can, in fact, influence the coin! Furthermore, the p-value is phenomenally small. And it's very easy to design this experiment to meet all the criteria mentioned in the post other than preregistration.
return0 12y ago
I think it's a larger issue with life science that everyone is looking for correlations. In physics, correlation is the hint that leads to the development if a theory, which leads to predictions, which are invalidated by experiments. In life science it seems correlations is the end goal. I think the issue is with the kind of questions one poses, not only with the interpretations.
- epistasis 12y ago
  That is ignorant bullshit.
- epistasis 12y ago
  Just to be clear about how this is incredibly wrong, as clearly this poster has never read life science research, and it's very strange for such a blatantly misinformed comment to stick around on HN. There are no biologists out there who are just looking for correlations, mistaking them for the real finding. Look through the life science papers in Nature, Science, Cell, PLoS Biology, etc. You will find people starting with a hypothesis, and running controlled, causal experiments, often with two or three different methodologies to establish non-correlational evidence. These are often based on initial observations that are correlational, but there is no publication where people are like "whoa, we found a correlation, our work is done!" In contrast, there are some subfields where most of the work is correlational, and those are the fields where researchers are limited to analyzing data more than running experiements: epidemiology and bioinformatics. Now epidemiologists collect tons of data, but there's very little that they can do in noncorrelational analysis, because that's the nature of ethics and human life. But that doesn't mean that they don't try to invalidate their correlation-based hypotheses with experiments, it's just that they get to run very few of them. Bioinformatics is somewhat different, and in some ways is much more like a theoretical physics in that developing new methodology can be a publication-worth, even if there was no new data collected to validate the computational methodology. In short, there's no truth at all to what return0 posted.
DanielBMarkham 12y ago
This is one of the better HN posts in a while. Science is our light in the darkness. It is not a new form of religion. It does not give us the answers to everything. It may not even be able ever to give us the answers to everything. It does not even give us a way to think. It does not advance in a linear way. Instead, many false starts are made before progress takes hold. Better ideas/models do not win out over older ones automatically. Instead, many times an older generation has to die before new ideas can firmly take root. Funding does not equal knowledge. You can throw as much money as you want at some problems and get nowhere. Many times this is because the questions are being asked the wrong way. But you'll never know -- until progress is finally made. It's just Bayesian. Science is that study which tells us if we do something or if some state exists --- then there's a very high likelihood that something else will happen or some new state will occur. We try to create models to explain this tenuous causality, but over the generations, models have gotten us into more trouble than they have gotten us out of. If we understand science as just being the study of what most likely follows what, instead of how things work, it's easier on everybody. After all, at the end of the day, even with a model, for you to manipulate the universe around you? You're going to need to know if you do one thing, what other things will happen. I fucking love science. But it's not what people think it is. We teach science in much too prideful a manner. If anything, the history of how science has been done should give us a deep and overpowering sense of humility.
- kijin 12y ago
  I agree. Excessive romanticization of science in the mass media is one of the greatest problems facing science today. Perhaps people are so used to religion that they think science works just like religion, expecting it to offer answers to questions like "what is the meaning of life?" and whatnot. The ouroboros diagram at the end of the article is an accurate description of not only science but pretty much every human endeavor. We cannot reach "pure unadulterated reality", only a reflection of our beliefs and biases, on the basis of which we form other beliefs and biases. Fortunately, reflections depend not only on what is reflected but also on the reflector's own properties. Nature is our reflector, so we can indeed make progress over long time scales. To think that science will help us overcome our very human weaknesses, however, is not only arrogant but dangerous. After all, the only way to realize this flawed ideal of pure science would be to eliminate all traces of our humanity. What good is a "light in the darkness", as you said, if the light is so bright it blinds the person who wanted to see?
lauradhamilton 12y ago
The fact that a lot of journals refuse to publish negative results is damaging to science and to our understanding. Relevant XKCD: http://xkcd.com/882/ http://xkcd.com/882/
nateabele 12y ago
"You’d have to find a phenomenon that definitely doesn’t exist [...]" (emphasis added) Welp.
zackmorris 12y ago
The Ganzfeld experiment suffers from bad statistics but it would sure be cool if it worked: https://en.wikipedia.org/wiki/Ganzfeld_experiment https://en.wikipedia.org/wiki/Ganzfeld_experiment It required so many lengthy trials to get statistically meaningful results that only large institutions could pull it off. I think it would make a cool crowdsourced/crowdfunded project today though. If there's something to it, I'd like to see the relationship to distance and especially time, but I'm not holding my breath.

179 points by nbouscal 12y ago

tomp 12y ago

From A Meta-Analysis of 90 Experiments on the Anomalous Anticipation of Random Future Events [1]: > [...] a meta-analysis of 90 experiments from 33 laboratories in 14 different countries which yielded an overall positive effect in excess of 6 sigma with an effect size (Hedges’ g) of 0.09, combined z = 6.33, p = 1.2 × 10-10. A Bayesian analysis yielded a Bayes Factor of 7.4 × 10-9, greatly exceeding the criterion value of 100 for "decisive evidence" in favor of the experimental hypothesis (Jeffries, 1961). [...] The number of potentially unretrieved experiments averaging a null effect that would be required to reduce the overall effect size to a trivial value was conservatively calculated to be 520. An analysis of p values across experiments implies that the results were not a product of "p-hacking," [...]. [1] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2423692 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2423692

robinh 12y ago
So... I know next to nothing about parapsychology itself, but have seen a lot of the drama around it, and I just have to ask: is it not intellectually dishonest to call something the 'control group of science' when their results overwhelmingly support the hypothesis? I have a hard time seeing how this is different from any other form of science denialism. "We don't like the results because they clash with our preconceived notions of how the universe works so we made up this thing to ignore your evidence"? That's hardly a valid complaint. Basically, on what grounds can people claim one field to be nonsense (e.g. calling parapsychology the control group of science) but not others? Can someone explain this to me?
- tomp 12y ago
  Have you read the article? It's basically all along this problem... tl;dr: Yes, that is intellectually dishonest, which is a huge problem. Scientists have two options: (1) accept parapsychology as real, or (2) accept that the "scientific method" (in social "sciences", at least) is insufficient. The problem seems to be that Bem (the author of the study) did almost everything "right", and if we increase the bar of scientific proof so that his study is excluded, so are many others... The reason why parapsychology is excluded: the real world doesn't support it (noone is earning huge amounts of money on the stock market using psi). However, the real world often doesn't support very small effects (e.g. Einstein's relativity). Fortunately for physics, they can afford to make their scientific methods much more rigorous, so they can study large effects as well as exceptionally tiny effects. Social studies can't, for the time being.
  - robinh 12y ago
    Okay, but that's not actually an answer to my question. My question basically comes down to this: > Yes, that is intellectually dishonest, which is a huge problem. Scientists have two options: (1) accept parapsychology as real, or (2) accept that the "scientific method" (in social "sciences", at least) is insufficient. I don't get why the whole thing is such a huge problem. The entire problem rests on needing parapsychology effects to not be real. If that need did not exist, we could just go "Okay, interesting, seems likely that there's something to it then. Let's do more research!" because, you know, we take that approach everywhere else. So my question remains: what is it about parapsychology that makes option two even valid to consider? All I can see is people just not liking that that may be how things work.
    - lmm 12y ago
      Parapsychology is physically impossible, and the evidentiary standards in physics are much higher, so we have much more confidence in our physics results than in these experiments - enough that we can reasonably say that physics is true and parapsychology is false. (But these experiments are as good as many in psychology / social science - suggesting that many "proven" results in psychology / social science could be false)
      - dragonwriter 12y ago
        > Parapsychology is physically impossible, and the evidentiary standards in physics are much higher, so we have much more confidence in our physics results than in these experiments Conflicting results don't mean one set is impossible (cf., the long-standing apparent conflict between QM and relativity within physics). Apparently conflicting results without a methodological error in either imply that the explanatory model that appears to be supported by at least one of the results (if not both) is, while useful within its own domain, in some way incorrect. The whole idea that the models validated by scientific experimentation are binarily true of false is, well, missing the point badly. While over time we hope they approach truth with them, what they are is useful (that is, they have predictive power) to a greater or lesser extent. And quite often the models with the greatest predictive power in two different domains conflict when either or both are extend outside of their own domain. EDIT: The real problem with parapsychology is that there's little in the way of explanatory models being tested anywhere in the field. There's a lot of hypotheses without models and some experiments testing them, which (concerns about methodology aside, for the moment) might raise interesting questions and serve as inspiration for developing and then testing theoretical models to explain the effects, but very little has been done there -- which makes "parapsychology" more a collection of potentially unexplained phenomena more than a branch of science that provides an explanatory model for some set of phenomena. Which is very different from most of the social sciences.
- Retric 12y ago
  Individual experiments give false positives and false negatives all the time. Think about how you might film this using a fair coin: http://www.youtube.com/watch?v=X1uJD1O3L08 http://www.youtube.com/watch?v=X1uJD1O3L08 Hint 10 heads is only a 1/1024 chance but 1024 is not that rare. Now realise that 5 coin flips is a 1/32 chance and 1/20 is considered acceptable to publish... Science is not based on a single experiment it's based around replication of experiments by different people at different times using slightly different methods. Parapsychology repeatedly fails this test to the point where there is a million dollar prize for anyone that can demonstrate anything in a controlled setting.
  - jmcmichael 12y ago
    The process to win Randi's prize is not science, it is public relations for the ideology of elimination materialism. Science is the testing of a hypothesis by applying an instrumental injunction to reality, apprehending and analyzing the results, and sharing/verifying the results in with a community of experts who have also performed the same injunction (reproducibility). Randi's process shares none of these characteristics with the scientific process. It is entertainment, a publicity stunt that has little to do with science other than to muddle the waters when discussing these topics.
- rainmaking 12y ago
  Jack Parsons, who was quite into parapsychology, phrased it quite neatly: It's when science becomes closed minded and degenerates into ancestor worship.
- jarrett 12y ago
  It's a valid question. The article takes the following premise: Parapsychology gets positive results following the scientific method as defined by prevailing norms in the scientific community. So why can't we draw the conclusion that parapsychology is as legitimate as other branches of science? What is to separate it? I would offer a pseudo-Bayesian[1] answer to that question. Parapsychology aims to prove hypotheses that lacks theoretical foundations. Our current understanding of physics and biology weigh strongly against the existence of psychic phenomena. Even before any experiment is conducted, we must admit that psychic phenomena are unlikely to exist. Our experimental results must be evaluated in light of that prior probability. Thus, parapsychology is and should be held to a higher burden of proof than other branches of science. We should demand more rigorous experimental designs, stronger effects, and smaller p values. This XKCD presents a similar idea, if you substitute "psychic phenomena exist" for "the sun has gone nova": http://www.explainxkcd.com/wiki/index.php?title=1132:_Frequentists_vs._Bayesians http://www.explainxkcd.com/wiki/index.php?title=1132:_Freque... [1] I say "pseudo" because I'm not a statistician by trade. I'm basing my argument on my rather superficial understanding of Bayesian statistics. I still think it's a valid argument in its own right, but I don't claim that it's an accurate representation of Bayesian statistics.
  - ChuckMcM 12y ago
    Parapsychology aims to prove hypotheses that lacks theoretical foundations. This is the lynch pin for most scientists. Before you can have a hypothesis, you must have a theory, and you can work to prove or disprove that theory by experimenting to create or observe results that the theory predicts. Parapsychology doesn't have good, testable, theories. Rather it has some interesting unexplained correlations.
    - pjreddie 12y ago
      Doctors rejected hand washing for a long time because there was no theory behind why it would improve patient care, even when there was overwhelming evidence showing it decreased mortality rates. Do you discount all evidence that doesn't fit into your world view? Maybe theory hasn't caught up with evidence yet.
- gwern 12y ago
  One man's modus ponens is another man's modus tollens (http://www.gwern.net/Prediction%20markets#modus-tollens-vs-modus-ponens http://www.gwern.net/Prediction%20markets#modus-tollens-vs-m...).
- onetwofiveten 12y ago
  There are two differences. The first is that you have to replace "preconceived notions of how the universe works" with "vast body of experimental data and scientific understanding of how the universe works". Our evidence for how physics works vastly outweighs current parapsychology research results. The second difference is that the evidence is not being ignored (at least by the author of the article), but is being taken to indicate a real effect, it just isn't the effect of psychic powers. Our knowledge of physics means that the existence of psychic powers is given a much lower prior probability than the possibility of widespread experimental error and bias. Since both explanations are likely to produce slight positive results, the existence of psychic powers is still unlikely once you take the evidence into account. However, the important point in the article is that in order to make this inference in an intellectually honest way, you need to significantly increase your estimation for the probabilities of widespread experimental error in ALL scientific studies that use similar methods. Since the methods in parapsychology are pretty good, this has quite a far reaching effect. In a sense, your question about "on what grounds can people claim one field to be nonsense but not others?" is exactly the same question asked by the author. Except the author isn't implying that parapsychology isn't nonsense, they are implying that many other fields are nonsense too. This makes sense because physics is almost certainly not nonsense.

yummyfajitas 12y ago

If anyone here hasn't read the comments on this as well, go read Eliezer Yudkowsky's comment on meta-analysis: http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/#comment-66077 http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...

SoftwareMaven 12y ago
I've spent a lot of time in the past year studying nutrition as I've tried to optimize my health. I don't believe there is a field that gets science more wrong and, as a result, whose meta-analyses are, to put it in Yudkowsky's words, more bullshit. Unfortunately, I don't think there is another field whose bullshit gets more media play, either (except maybe politicians, but that's a tautology).

deleted 12y ago

[deleted]

tokenadult 12y ago

This is a very interesting review of some of the most challenging published literature on the reliability of psychological science. The author's point is correct that parapsychology (ESP and the like) basically has no prior plausibility. Yet a researcher who has the chops to do reliable research, Daryl J. Bem, has conducted "experiments" that appear to show that some human experimental subjects can see into the future. As the article says, "Bem definitely picked up a signal. The only question is whether it’s a signal of psi, or a signal of poor experimental technique." Please read this submission from beginning to end and follow the links to see more of the background. I had the opportunity to discuss the original Bem paper on psi in the journal club I regularly participate in. The conclusion is plainly wrong, but the error in procedure is subtle and it took a while for response papers to come out about his initial finding. (And everyone agrees that Bem is a smart man, so the subtle errors are all the more dismaying.) If Bem can't design experiments carefully, who can? AFTER EDIT TO RESPOND TO QUESTIONS IN ANOTHER SUBTHREAD HERE: Another participant is asking why parapsychology has poor prior plausibility. Parapsychology is a rather recent discipline, always with sloppy experimental practices,[1] and meanwhile the disciplines of physics and medicine are much older. The study of how information is transmitted by electromagnetic radiation and how the human body perceives sensations is incompatible with supposed abilities like ESP, and completely incompatible with seeing into the future in the manner described by Bem. Extraordinary claims require extraordinary evidence. A million dollars says that psychic claims are bogus,[2] and so far no one has won a million dollars by proving that proposition wrong. [1] http://www.skepdic.com/projectalpha.html http://www.skepdic.com/projectalpha.html [2] http://www.randi.org/site/index.php/1m-challenge.html http://www.randi.org/site/index.php/1m-challenge.html

wpietri 12y ago
To your point on poor plausibility, I'd add that people have been trying to prove and develop their psychic powers for millennia. If you look at other common human traits, you can see that we're pretty good at exploiting our natural capacities. E.g., how the various martial arts traditions have taken modest raw capacity and learned how to develop it to an amazing degree. Ditto science, art, engineering, music, poetry, and literature. The failure of all of the various paranormal traditions to develop anything demonstrable despite a lot of effort suggests that there is little or nothing there.

rainmaking 12y ago

Wait a minute. The author calls for "stricter scientific standards", and than has the audacity to define "strict" as "something that excludes parapsychology". Say it was 1600 A.D. and we would like to improve the standards of science. Everybody knows the cart stops when the ox stops pulling, so why not use this reasonable observation as a litmus test for all future scientific endeavor? Bye, Newton. The whole article is just a particular set of beliefs trying to institutionalize itself, the very thing science is supposed to put an end to.

Strilanc 12y ago
That is not what I took away from the article at all. What I took away is: if you think parapsychology is wrong, then you should be really worried about the rest of science because their studies are not better designed. It's most clearly expressed in this paragraph: > This is not a criticism of Bem or a criticism of parapsychology. It’s something that is inherent to the practice of meta-analysis, and even more, inherent to the practice of science. Other than a few very exceptional large medical trials, there is not a study in the world that would survive the level of criticism I am throwing at Bem right now.
Kutta 12y ago
It is always right to institutionalize correct ideas, like Newtonian mechanics (while having in mind its later refinements, of course), or the notion that parapsychology doesn't exist.
- rainmaking 12y ago
  My you have audacious work ethic, examining every human on the planet just to rule that out. Let it be law, then.
- ksmiley 12y ago
  Ok, but how do you determine which ideas are "correct"?
6d0debc071 12y ago
Alternatively, you can read it that the author is defining strict as, 'Something that would make me believe parapsychology.' At least if it met those standards.

Zigurd 12y ago

Starting out with a flawed analogy to placebos makes the whole article hard to swallow. Placebos are complicated, especially for drugs with receptors in the brain. Stating that we know placebos "don't work" is outright incorrect. We know the opposite: placebos can produce real results, and there are physical mechanisms that explain those results. Nothing paranormal about it. If the conclusion is that there is no magic wand in methodology that makes an experiment "good" - that's true. But the approach gets off to a rough start.

pdonis 12y ago
I don't think the analogy with placebos is flawed; it's just not quite the analogy you are describing here. The point is not that placebos "don't work"; it's that, if a placebo works as well as some drug, the drug itself doesn't work. That is, it adds nothing to what you can already get using the placebo effect. Which may indeed be a "real" result; but why pay extra for a drug to get it when you don't need to? Note, btw, that while placebos can indeed produce real results, and there are indeed physical mechanisms that explain those results, those mechanisms have nothing to do with what's in the placebo itself; they are entirely to do with what goes on in the mind and body of the person taking the placebo. In that sense, it's true that placebos "don't work": it doesn't matter what specific substance you put in the placebo pill--sugar, starch, gelatin, whatever--as long as it looks and tastes the same to the subject.

btilly 12y ago

It was an interesting read, but the author showed absolutely no clue why pre-registration was important. The point is to avoid selection bias. If the same exact study is done 100 times, but only the best 50 results get included in your study, a meta analysis of those results is going to show a strong statistical bias. No amount of waving around the word "Bayesian" can fix this - the only way to fix it is pre-registration of honest results. Why am I so certain of this? Because of the following passage: By my count, Bem follows all of the commandments except [6] and [10]. He apologizes for not using pre-registration, but says it’s okay because the studies were exact replications of a previous study that makes it impossible for an unsavory researcher to change the parameters halfway through and does pretty much the same thing. But that misses the whole point. Whether the studies had the same or different parameters is irrelevant - the fact that you see the ones that worked out and don't see the ones that didn't is the only thing that matters. And the fact that the meta analysis didn't get this right makes the output statistical garbage. No further discussion needed.

Natsu 12y ago
There is this rather amusing section, though: "(Richard Wiseman – the same guy who provided the negative aura for the Wiseman and Schiltz experiment – has started a pre-register site for Bem replications. He says he has received five of them. This is very promising. There is also a separate pre-register for parapsychology trials in general. I am both extremely pleased at this victory for good science, and ashamed that my own field is apparently behind parapsychology in the “scientific rigor” department)"
- kumbasha 12y ago
  After reading that article, I actually came away with a very positive view of the field of parapsychology in general. Of course psychics aren't real, but the Flying Spaghetti Monster can be used to make legitimate points about the study of religion, to use an analogy.
SatvikBeri 12y ago
To give a concrete example, let's say I want to show that I can influence a coin flip with my mind in order to produce a heads result every time. I do this by publishing trials where each instance involves me flipping a coin four times. I conduct 16,000 trials. On average, 1,000 of them will show that I can flip a coin and get heads every time. If all I publish are these trials, that yields the amazing result that I can, in fact, influence the coin! Furthermore, the p-value is phenomenally small. And it's very easy to design this experiment to meet all the criteria mentioned in the post other than preregistration.

return0 12y ago

I think it's a larger issue with life science that everyone is looking for correlations. In physics, correlation is the hint that leads to the development if a theory, which leads to predictions, which are invalidated by experiments. In life science it seems correlations is the end goal. I think the issue is with the kind of questions one poses, not only with the interpretations.

epistasis 12y ago
That is ignorant bullshit.
epistasis 12y ago
Just to be clear about how this is incredibly wrong, as clearly this poster has never read life science research, and it's very strange for such a blatantly misinformed comment to stick around on HN. There are no biologists out there who are just looking for correlations, mistaking them for the real finding. Look through the life science papers in Nature, Science, Cell, PLoS Biology, etc. You will find people starting with a hypothesis, and running controlled, causal experiments, often with two or three different methodologies to establish non-correlational evidence. These are often based on initial observations that are correlational, but there is no publication where people are like "whoa, we found a correlation, our work is done!" In contrast, there are some subfields where most of the work is correlational, and those are the fields where researchers are limited to analyzing data more than running experiements: epidemiology and bioinformatics. Now epidemiologists collect tons of data, but there's very little that they can do in noncorrelational analysis, because that's the nature of ethics and human life. But that doesn't mean that they don't try to invalidate their correlation-based hypotheses with experiments, it's just that they get to run very few of them. Bioinformatics is somewhat different, and in some ways is much more like a theoretical physics in that developing new methodology can be a publication-worth, even if there was no new data collected to validate the computational methodology. In short, there's no truth at all to what return0 posted.

DanielBMarkham 12y ago

This is one of the better HN posts in a while. Science is our light in the darkness. It is not a new form of religion. It does not give us the answers to everything. It may not even be able ever to give us the answers to everything. It does not even give us a way to think. It does not advance in a linear way. Instead, many false starts are made before progress takes hold. Better ideas/models do not win out over older ones automatically. Instead, many times an older generation has to die before new ideas can firmly take root. Funding does not equal knowledge. You can throw as much money as you want at some problems and get nowhere. Many times this is because the questions are being asked the wrong way. But you'll never know -- until progress is finally made. It's just Bayesian. Science is that study which tells us if we do something or if some state exists --- then there's a very high likelihood that something else will happen or some new state will occur. We try to create models to explain this tenuous causality, but over the generations, models have gotten us into more trouble than they have gotten us out of. If we understand science as just being the study of what most likely follows what, instead of how things work, it's easier on everybody. After all, at the end of the day, even with a model, for you to manipulate the universe around you? You're going to need to know if you do one thing, what other things will happen. I fucking love science. But it's not what people think it is. We teach science in much too prideful a manner. If anything, the history of how science has been done should give us a deep and overpowering sense of humility.

kijin 12y ago
I agree. Excessive romanticization of science in the mass media is one of the greatest problems facing science today. Perhaps people are so used to religion that they think science works just like religion, expecting it to offer answers to questions like "what is the meaning of life?" and whatnot. The ouroboros diagram at the end of the article is an accurate description of not only science but pretty much every human endeavor. We cannot reach "pure unadulterated reality", only a reflection of our beliefs and biases, on the basis of which we form other beliefs and biases. Fortunately, reflections depend not only on what is reflected but also on the reflector's own properties. Nature is our reflector, so we can indeed make progress over long time scales. To think that science will help us overcome our very human weaknesses, however, is not only arrogant but dangerous. After all, the only way to realize this flawed ideal of pure science would be to eliminate all traces of our humanity. What good is a "light in the darkness", as you said, if the light is so bright it blinds the person who wanted to see?

lauradhamilton 12y ago

The fact that a lot of journals refuse to publish negative results is damaging to science and to our understanding. Relevant XKCD: http://xkcd.com/882/ http://xkcd.com/882/

nateabele 12y ago

"You’d have to find a phenomenon that definitely doesn’t exist [...]" (emphasis added) Welp.

zackmorris 12y ago

The Ganzfeld experiment suffers from bad statistics but it would sure be cool if it worked: https://en.wikipedia.org/wiki/Ganzfeld_experiment https://en.wikipedia.org/wiki/Ganzfeld_experiment It required so many lengthy trials to get statistically meaningful results that only large institutions could pull it off. I think it would make a cool crowdsourced/crowdfunded project today though. If there's something to it, I'd like to see the relationship to distance and especially time, but I'm not holding my breath.