15 ms·

The control group is out of control (2014)

184 points by stakkur 5y ago

mcculley 5y ago
I was intrigued by this wording: "which most reasonable people believe don’t exist" Maybe I am pedantic, but it should be "which most reasonable people don't believe exist"
- intrepidpar 5y ago
  Well, arguably believing that something doesn't exist and not believing that something exists are two different things.
  - mcculley 5y ago
    Exactly. As a skeptic, I have no reason to believe. I don't believe it exists. If I believed that it did not exist, I would have a belief about it.
- xapata 5y ago
  No, it's reasonable to phrase it as a belief that the phenomenon doesn't exist. That's a positive belief of the negative. The alternative of lack of belief in existence is a more neutral stance.
  - mcculley 5y ago
    Lack of belief is the more skeptical stance, which seems more reasonable to me.
    - wtallis 5y ago
      Skepticism doesn't mean resisting any justification for moving the needle of your certainty away from 50%. You are allowed to expect that psychic phenomena being real would lead to the existence of clear evidence that psychic phenomena are real, and to conclude from the lack of such evidence where it ought to be found that it is less likely that psychic phenomena are real.
    - cipheredStones 5y ago
      From another essay by Scott Alexander: > Suppose you have a target that is half red and half blue; you are aiming for red. You would have to be very very confident in your dart skills to say there is only a one in a million chance you will miss it. But if there is a target that is 999,999 millionths red, and 1 millionth blue, then you do not have to be at all good at darts to say confidently that there is only a one in a million chance you will miss the red area. > Suppose a Christian says “Jesus might be God. And he might not be God. 50-50 chance. So you would have to be incredibly overconfident to say you’re sure he isn’t.” The atheist might respond “The target is full of all of these zillions of hypotheses – Jesus is God, Allah is God, Ahura Mazda is God, Vishnu is God, a random guy we’ve never heard of is God. You are taking a tiny tiny submillimeter-sized fraction of a huge blue target, painting it red, and saying that because there are two regions of the target, a blue region and a red region, you have equal chance of hitting either.” Eliezer Yudkowsky calls this “privileging the hypothesis”. [0] > There’s a tougher case. Suppose the Christian says “Okay, I’m not sure about Jesus. But either there is a Hell, or there isn’t. Fifty fifty. Right?” > I think the argument against this is that there are way more ways for there not to be Hell than there are for there to be Hell. If you take a bunch of atoms and shake them up, they usually end up as not-Hell, in much the same way as the creationists’ fabled tornado-going-through-a-junkyard usually ends up as not-a-Boeing-747. For there to be Hell you have to have some kind of mechanism for judging good vs. evil – which is a small part of the space of all mechanisms, let alone the space of all things – some mechanism for diverting the souls of the evil to a specific place, which same, some mechanism for punishing them – again same – et cetera. Most universes won’t have Hell unless you go through a lot of work to put one there. Therefore, Hell existing is only a very tiny part of the target. Making this argument correctly would require an in-depth explanation of formalizations of Occam’s Razor, which is outside the scope of this essay but which you can find on the LW Sequences. [0] https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privileging-the-hypothesis https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privilegin...
- PaulDavisThe1st 5y ago
  this is similar to a kind of subject and object confusion: subject: most reasonable people object: psi phenomena verbs: to believe, to exist The full statement can be stated in at least two ways: "[ these people ] don't believe that [ these phenomena ] exist." "[ these people ] believe that [ these phenomena ] don't exist." Either one is valid, as are the contracted versions, but the first one describes the absence of belief in existence, and the second describes the presence of a belief in non-existence. These are closely related things, but not identical, and I imagine the author likely chose carefully.
teh_infallible 5y ago
“ Parapsychologists are able to produce experimental evidence for psychic phenomena about as easily as normal scientists are able to produce such evidence for normal, non-psychic phenomena.” Therefore psychic phenomena exist? I don’t see the problem.
- selestify 5y ago
  It’s only a problem if you have strong priors telling you that they don’t exist, because that indicates a lot of other scientific fields are in trouble.
darkerside 5y ago
Arthur Conan Doyle, through Sherlock Holmes, said, once you've eliminated the impossible, what remains, no matter how improbable, must be the truth. This tells me that Schlitz actually has psychic powers and Wiseman does not. It actually makes perfect sense. People who are psychic wouldn't be skeptics, and people who are not psychic certainly would be skeptics.
- kibwen 5y ago
  I suppose that, had Conan Doyle been aware of quantum mechanics, Sherlock Holmes would have spent a portion of every case entertaining the notion that quantum phenomena caused the universe to spontaneously arrange the crime scene. "It's elementary particles, my dear Watson."
  - deleted 5y ago
    [deleted]
  - ModernMech 5y ago
    Holmes was famously ignorant of any knowledge that didn’t help him solve crimes. In his own words: “You see,” he explained, “I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things so that he has a difficulty in laying his hands upon it. Now the skillful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.” “But the Solar System!” I protested. “What the deuce is it to me?” he interrupted impatiently; “you say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work. Given this, I’m not sure Holmes would bother with a quantum viewpoint of the world, since the theory wouldn’t help him solve crimes.
jrockway 5y ago
They need to invent control group version 2. It worked for Linux.
gumby 5y ago
> That is, in let’s say a drug testing experiment, you give some people the drug and they recover. That doesn’t tell you much until you give some other people a placebo drug you know doesn’t work – but which they themselves believe in – and see how many of them recover. That number tells you how many people will recover whether the drug works or not. Unless people on your real drug do significantly better than people on the placebo drug, you haven’t found anything. I had a drug program in which we blew away the standard placebo rate … unfortunately our placebo arm did too, though not by as much. But it was just enough that we weren’t enough better than our placebo arm. We changed modality from the current standard (we were an injection rather than a tablet and all the trial participants were super excited.
- civilized 5y ago
  There is no such thing as a standard placebo rate. If there was, there would be no need for every study to have a control group.
  - skrebbel 5y ago
    So maybe that isn't what the GP meant? You're telling the person who did a study that they misunderstand the utter basics of studies, based on a 3-sentence comment they wrote.
    - rictic 5y ago
      Sometimes people misunderstand the basics of their full time work yes We've all seen that person, and almost all of us have at times _been_ that person And unfortunately, given the replication crisis, there is no reason to assume that a published scientist is not making a fundamental error that undermines the validity of their work. It apparently happens quite frequently, and we should be quite grateful for those that question and probe at assumptions and potentially faulty analysis techniques
    - deleted 5y ago
      [deleted]
  - 5y ago
nathanwh 5y ago
Honestly this XKCD arguing that psychic phenomena don't exist because they're not exploited for profit is a comforting difficult to refute argument for me: https://xkcd.com/808/ https://xkcd.com/808/
- mathattack 5y ago
  Yes. The existence of casinos disproves psychic powers.
  - throwaway7033 5y ago
    How so? Casinos routinely ban card counters; if psychic powers existed then psychics would also be banned from casinos.
    - mathattack 5y ago
      Identifying card counters is easier than identifying psychics.
- NateEag 5y ago
  It may be that psychic powers exist but that they are not powerful, general, or controllable enough to be reliably profitable. Also, if every profitable phenomenon in the world is already developed, then no business should ever invest in fundamental research - i.e., the pharmaceutical companies are wasting their research dollars. Why is it comforting to you to believe psychic powers don't exist?
  - gs17 5y ago
    Yeah, e.g. someone who can, for example, consistently predict single bits of output from true random number generators 51% of the time would be evidence of something very strange going on, but not necessarily usable for anything practical. (Maybe if you could gather enough of these people to overcome the "weak" ability?)
    - 5y ago
ALittleLight 5y ago
This post really makes you think and challenged my world view when I first read it. A while back I looked at the CIA "Star Gate" files[1], which, are a collection of documents released under FOIA detailing the CIA's investigation into psychic powers and whether or not those powers serve an intelligence gathering purpose. I just opened up random files and skimmed them - I wish I had taken notes when doing so, but my impression was that the CIA was actually producing some better than chance results with their experiments. I don't really know what to make of parapsychology or psychic powers in general. Sometimes I'll idly wonder if I should dedicate time to trying to uncover my psychic powers. 1 - https://www.cia.gov/readingroom/collection/stargate https://www.cia.gov/readingroom/collection/stargate
- aaaaaaaaaaab 5y ago
  Also, don't forget goat-staring!
amelius 5y ago
Reminds me of a talk of Rupert Sheldrake at Google Talks: https://www.youtube.com/watch?v=hic18Xyk9is https://www.youtube.com/watch?v=hic18Xyk9is and then later he gave a talk called "The Science Delusion" which got banned from TED: https://www.youtube.com/watch?v=JKHUaNAxsTg https://www.youtube.com/watch?v=JKHUaNAxsTg
- 3np 5y ago
  What do you mean by “banned” (as opposed to just taken down from their video channel)? BTW, YouTube title says “TED TALK”; note that it’s TEDx.
  - amelius 5y ago
    Does it matter? Sheldrake makes some wild claims, but eloquently packs them in a talk that is entertaining and at times thought-provoking. In any case, his braveness is certainly admirable. Worth a watch, I would say (perhaps with a pint of beer).
Sniffnoy 5y ago
(2014)
FabHK 5y ago
The "fits of nervous laughter" paper by Wiseman & Schlitz (1998) was replicated by Wiseman & Schlitz (1999) [1], with a similar yet different result: RW (again) found no effect of staring, MS found a significant (p=0.05) effect of staring, but with opposite direction than the 1998 paper. [1] https://richardwiseman.files.wordpress.com/2020/04/expt2.pdf https://richardwiseman.files.wordpress.com/2020/04/expt2.pdf , found on https://richardwiseman.wordpress.com/research/parapsychology/ https://richardwiseman.wordpress.com/research/parapsychology...
- mannykannot 5y ago
  Hence the enthusiasm for preregistration of trials.
Jweb_Guru 5y ago
> I know that standard practice here is to tell the story of Clever Hans and then say That Is Why We Do Double-Blind Studies. Yes, this is the answer to this entire article, and his attempt to dismiss this for some reason is strange to me.
- igorkraw 5y ago
  In the years that I've read him I noticed this is something that happens from time to time with Scott Siskinds (see 3 if you take offense to me using his name) texts, especially when he steps out of his zone of competence to opine on politics, science in general, AI etc. The simple, obvious answer that the establishment believes in is met with some snarky ridicule and on goes the article, with some very subtle rhetoric veiled in the irony ("rigour doesn't matter, all scientists have biases, you can't trust rigorous science if the wrong people do it" is the message behind [quote] Studies are going to be confounded by the allegiance of the researcher. When researchers who don’t believe something discover it, that’s when it’s worth looking into. [/quote] ). I often wondered why this is, because he seems like a very smart guy with interesting ideas sometimes, so it seemed kind of weird and conflicting. Then I learned he's trying to hide his beliefs about scientific racism and political views (1) while often writing in the tone of "pained centrist just wanting to understand things scientifically" and it made much more sense as (in the worst case) an old school propaganda technique of FUD in science and rigorous science to make the HBD go down better and (more charitably) as Mr. Siskind just being another bright-but-human person who when writing on things he has actual expertise on can be very well worth reading, but simply shouldn't be trusted on anything aside from this. Rationalwiki has a decent writeup on him (2) 1: https://web.archive.org/web/20210217195335/https://twitter.com/TopherTBrennan/status/1362108632070905857 https://web.archive.org/web/20210217195335/https://twitter.c... 2: https://rationalwiki.org/wiki/Scott_Alexander https://rationalwiki.org/wiki/Scott_Alexander 3: https://news.ycombinator.com/item?id=25874025 https://news.ycombinator.com/item?id=25874025
  - wpietri 5y ago
    Exactly. He's definitely smart, and so it was a mystery to me why his articles were so incredibly long and rambling. (I get that some people enjoy his prose regardless of content, but I'm not one of them.) Why would a smart person who in theory wants to convey important points be so obscurantist about them? I threw up my hands and put it down to a personality quirk. It was the Sandifer article "The Beigeness" that cleared it up for me: https://www.eruditorumpress.com/blog/the-beigeness-or-how-to-kill-people-with-bad-writing-the-scott-alexander-method https://www.eruditorumpress.com/blog/the-beigeness-or-how-to...
chalst 5y ago
Andrew Gelman had an interesting 9-years-too-late idea for what to do in cases like Bem's article: https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-bem-paper-back-2010-click-find-surprisingly-simple-answer/ https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-... Rather than publish all of the dubious speculations and the analysis whose problems took time to emerge, simply publish the data. The advantage, in his words: > In the example of the ESP study, if anything’s valuable it’s the data. Publishing the data would get the journal off the hook regarding fairness, open-mindedness, and not missing a scoop, while enabling others to move on reanalyses right away, and without saddling the journal with an embarrassing endorsement of a weak theory that, it turns out, was not really supported by data at all.
- MaxBarraclough 5y ago
  Obvious question then: what are the arguments/forces opposing this? Just inertia in the publishing norms?
  - Misdicorl 5y ago
    The approach fails miserably when taken in aggregate. I would read five to ten papers from my own field a week while in grad school. If I had to do the analysis for myself as well there wouldn't be any time to do my own research. Add on twenty more papers a week from adjacent fields where I don't even know all the techniques and the magnitude of the problem hopefully becomes clearer.
    - MaxBarraclough 5y ago
      The raw figures would be available in addition to the paper itself, not instead of it.
      - chalst 5y ago
        Gelman's suggestion is how to handle articles with unlikely conclusions: don't publish the submitted article, though that presumably would be circulated as a preprint, but just the data. Then the journal is bringing attention to interesting data without endorsing the dubious analysis or conclusions.
monocasa 5y ago
[2014]
olliej 5y ago
Well this certainly makes me glad that the only time I was having to do statistical studies for research purposes, it was in the context of generated code performance.
deleted 5y ago
[deleted]
adamgordonbell 5y ago
So if you assume ESP doesn't exist then parapsychology research has something important to say about the replication crisis in social sciences. This is a surprising thing I wouldn't have thought of on my own.
egypturnash 5y ago
Finally, they would conduct the experiment in a series of different batches. Half the batches (randomly assigned) would be conducted by Dr. Schlitz, the other half by Dr. Wiseman. Because the two authors had very carefully standardized the setting, apparatus and procedure beforehand, “conducted by” pretty much just meant greeting the participants, giving the experimental instructions, and doing the staring. The results? Schlitz’s trials found strong evidence of psychic powers, Wiseman’s trials found no evidence whatsoever. Take a second to reflect on how this makes no sense. Two experimenters in the same laboratory, using the same apparatus, having no contact with the subjects except to introduce themselves and flip a few switches – and whether one or the other was there that day completely altered the result. For a good time, watch the gymnastics they have to do to in the paper to make this sound sufficiently sensical to even get published. This is the only journal article I’ve ever read where, in the part of the Discussion section where you’re supposed to propose possible reasons for your findings, both authors suggest maybe their co-author hacked into the computer and altered the results. Well, if you would take the "psi can never exist" blinders off, Mr. Codex, then this seems to be a pointer towards "this works better if you believe it will work", and it makes perfect sense if you accept this. Schlitz's belief that it would work resulted in it working. Wiseman's belief that it wouldn't resulted in it not working. The whole idea behind psi is that the human mind can sense and/or effect the world in currently-inexplicable ways, and here it is doing just that. With at least ten other experiments on this idea getting similar results. But parapsychology is completely and utterly a scam in your eyes so that is not a result you will consider.
- einpoklum 5y ago
  One can get quite a bit of popular traction from arranging for one's theory to be effectively-impossible to disprove (e.g. "it only works if you believe in it during the experiment"). Not much explanatory power though.
  - egypturnash 5y ago
    The experiment I excerpted his description of seems to go along with exactly that. Experimenter who believes? Things happen. Experimenter who does not believe? Nothing happens.
dang 5y ago
Discussed at the time: The Control Group Is Out of Control - https://news.ycombinator.com/item?id=7666575 https://news.ycombinator.com/item?id=7666575 - April 2014 (74 comments)
ed-209 5y ago
"...things we haven’t discovered yet which are at least as weird as subconscious emotional cues" - sounds like psi to me.
Karellen 5y ago
> Pre-registration only assures that your study will not get any worse than it was the first time you thought of it, which may be very bad indeed. > > Searching for publication bias only means you will get all of the confounded studies, instead of just some of them. No, pre-registration assures that when you do a study and get a negative result, you don't simply forget to publish it - helping to reduce publication bias. But also, it prevents you from changing what you were looking for (because that's part of the pre-registration data) and "data dredging" for something that looks like a positive result, helping to reduce p-hacking - specifically of the "Texas sharpshooter fallacy" variety.
- jessriedel 5y ago
  What you wrote and what the author wrote do not conflict. Indeed, the author already noted the goal of preregistration earlier in the essay. He is explaining here why it doesn't save you from different issues.
- BeFlatXIII 5y ago
  How does pre-reg handle cases where unexpected findings are made along the way? Are those data to be scrapped so everyone has to start their pre-registered "real" study from scratch?
  - gjvnq 5y ago
    I imagine you would report both that you failed to find what you expected but that you also found something else.
EGreg 5y ago
What happened with Bem? Did his studies get debunked or he is still able to prove things? And the other guy runs same studies and gets different result ? WhT ?
coldtea 5y ago
Note the implicit (and, to most, justified) a priori convinction that psychic phenomena cannot exist, even if they have evidence to the contrary with similar confidence as other ("coventional") study subjects. Isn't this an "isolated demand for rigor"? [1] Doesn't this prove that science as practiced is not just the empirical process and some raw axioms that guide it (which these "psychic research" both follows), which scientists pretend to ascribe to, but also part blind faith/prejudice? I'm not saying physic phenomena are real. I'm saying if tests are conducted with the same principles as those about physical phenomena, and results are equally carefully collected, and they have the same condidence, science in principle shouldn't treat them any differently than results about "regular" subjects. [1] https://slatestarcodex.com/2014/08/14/beware-isolated-demands-for-rigor/ https://slatestarcodex.com/2014/08/14/beware-isolated-demand...
- amatic 5y ago
  >science in principle shouldn't treat them any differently than results about "regular" subjects. That's the whole problem, as I understand it. If we are going to treat ESP research the same as research in other fields, we either accept ESP as real, or reject both ESP and other fields as deeply flawed. Personally, I'm leaning heavily toward the second option. The methods in ESP are identical to methods in psychology, social sciences, etc, but not physics or chemistry. I'm just wondering where to draw the line. Maybe the whole method of statistical inference needs to be flushed? Maybe it cannot be saved by preregistration, publishing all the data, transparency and other bells and whistles.
  - coldtea 5y ago
    >That's the whole problem, as I understand it. If we are going to treat ESP research the same as research in other fields, we either accept ESP as real, or reject both ESP and other fields as deeply flawed. That's my point them. "If we are going to treat ESP research the same as research in other fields", according to the principles science pays lip service to this shouldn't be a question to begin with, because science shouldn't have a priori ideas about what can be treated "the same as other fields" and what shouldn't. Research (results) should determine that (which means: if there are results in its favor, and we can't see a problem with the research that other - non ESP - accepted research doesn't have, we should accept it, as much as we accept in other such - non ESP - results).

184 points by stakkur 5y ago

I was intrigued by this wording: "which most reasonable people believe don’t exist" Maybe I am pedantic, but it should be "which most reasonable people don't believe exist"

intrepidpar 5y ago
Well, arguably believing that something doesn't exist and not believing that something exists are two different things.
- mcculley 5y ago
  Exactly. As a skeptic, I have no reason to believe. I don't believe it exists. If I believed that it did not exist, I would have a belief about it.
xapata 5y ago
No, it's reasonable to phrase it as a belief that the phenomenon doesn't exist. That's a positive belief of the negative. The alternative of lack of belief in existence is a more neutral stance.
- mcculley 5y ago
  Lack of belief is the more skeptical stance, which seems more reasonable to me.
  - wtallis 5y ago
    Skepticism doesn't mean resisting any justification for moving the needle of your certainty away from 50%. You are allowed to expect that psychic phenomena being real would lead to the existence of clear evidence that psychic phenomena are real, and to conclude from the lack of such evidence where it ought to be found that it is less likely that psychic phenomena are real.
  - cipheredStones 5y ago
    From another essay by Scott Alexander: > Suppose you have a target that is half red and half blue; you are aiming for red. You would have to be very very confident in your dart skills to say there is only a one in a million chance you will miss it. But if there is a target that is 999,999 millionths red, and 1 millionth blue, then you do not have to be at all good at darts to say confidently that there is only a one in a million chance you will miss the red area. > Suppose a Christian says “Jesus might be God. And he might not be God. 50-50 chance. So you would have to be incredibly overconfident to say you’re sure he isn’t.” The atheist might respond “The target is full of all of these zillions of hypotheses – Jesus is God, Allah is God, Ahura Mazda is God, Vishnu is God, a random guy we’ve never heard of is God. You are taking a tiny tiny submillimeter-sized fraction of a huge blue target, painting it red, and saying that because there are two regions of the target, a blue region and a red region, you have equal chance of hitting either.” Eliezer Yudkowsky calls this “privileging the hypothesis”. [0] > There’s a tougher case. Suppose the Christian says “Okay, I’m not sure about Jesus. But either there is a Hell, or there isn’t. Fifty fifty. Right?” > I think the argument against this is that there are way more ways for there not to be Hell than there are for there to be Hell. If you take a bunch of atoms and shake them up, they usually end up as not-Hell, in much the same way as the creationists’ fabled tornado-going-through-a-junkyard usually ends up as not-a-Boeing-747. For there to be Hell you have to have some kind of mechanism for judging good vs. evil – which is a small part of the space of all mechanisms, let alone the space of all things – some mechanism for diverting the souls of the evil to a specific place, which same, some mechanism for punishing them – again same – et cetera. Most universes won’t have Hell unless you go through a lot of work to put one there. Therefore, Hell existing is only a very tiny part of the target. Making this argument correctly would require an in-depth explanation of formalizations of Occam’s Razor, which is outside the scope of this essay but which you can find on the LW Sequences. [0] https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privileging-the-hypothesis https://www.lesswrong.com/posts/X2AD2LgtKgkRNPj2a/privilegin...
PaulDavisThe1st 5y ago
this is similar to a kind of subject and object confusion: subject: most reasonable people object: psi phenomena verbs: to believe, to exist The full statement can be stated in at least two ways: "[ these people ] don't believe that [ these phenomena ] exist." "[ these people ] believe that [ these phenomena ] don't exist." Either one is valid, as are the contracted versions, but the first one describes the absence of belief in existence, and the second describes the presence of a belief in non-existence. These are closely related things, but not identical, and I imagine the author likely chose carefully.

teh_infallible 5y ago

“ Parapsychologists are able to produce experimental evidence for psychic phenomena about as easily as normal scientists are able to produce such evidence for normal, non-psychic phenomena.” Therefore psychic phenomena exist? I don’t see the problem.

selestify 5y ago
It’s only a problem if you have strong priors telling you that they don’t exist, because that indicates a lot of other scientific fields are in trouble.

darkerside 5y ago

Arthur Conan Doyle, through Sherlock Holmes, said, once you've eliminated the impossible, what remains, no matter how improbable, must be the truth. This tells me that Schlitz actually has psychic powers and Wiseman does not. It actually makes perfect sense. People who are psychic wouldn't be skeptics, and people who are not psychic certainly would be skeptics.

kibwen 5y ago
I suppose that, had Conan Doyle been aware of quantum mechanics, Sherlock Holmes would have spent a portion of every case entertaining the notion that quantum phenomena caused the universe to spontaneously arrange the crime scene. "It's elementary particles, my dear Watson."
- deleted 5y ago
  [deleted]
- ModernMech 5y ago
  Holmes was famously ignorant of any knowledge that didn’t help him solve crimes. In his own words: “You see,” he explained, “I consider that a man’s brain originally is like a little empty attic, and you have to stock it with such furniture as you choose. A fool takes in all the lumber of every sort that he comes across, so that the knowledge which might be useful to him gets crowded out, or at best is jumbled up with a lot of other things so that he has a difficulty in laying his hands upon it. Now the skillful workman is very careful indeed as to what he takes into his brain-attic. He will have nothing but the tools which may help him in doing his work, but of these he has a large assortment, and all in the most perfect order. It is a mistake to think that that little room has elastic walls and can distend to any extent. Depend upon it there comes a time when for every addition of knowledge you forget something that you knew before. It is of the highest importance, therefore, not to have useless facts elbowing out the useful ones.” “But the Solar System!” I protested. “What the deuce is it to me?” he interrupted impatiently; “you say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work. Given this, I’m not sure Holmes would bother with a quantum viewpoint of the world, since the theory wouldn’t help him solve crimes.

jrockway 5y ago

They need to invent control group version 2. It worked for Linux.

gumby 5y ago

> That is, in let’s say a drug testing experiment, you give some people the drug and they recover. That doesn’t tell you much until you give some other people a placebo drug you know doesn’t work – but which they themselves believe in – and see how many of them recover. That number tells you how many people will recover whether the drug works or not. Unless people on your real drug do significantly better than people on the placebo drug, you haven’t found anything. I had a drug program in which we blew away the standard placebo rate … unfortunately our placebo arm did too, though not by as much. But it was just enough that we weren’t enough better than our placebo arm. We changed modality from the current standard (we were an injection rather than a tablet and all the trial participants were super excited.

civilized 5y ago
There is no such thing as a standard placebo rate. If there was, there would be no need for every study to have a control group.
- skrebbel 5y ago
  So maybe that isn't what the GP meant? You're telling the person who did a study that they misunderstand the utter basics of studies, based on a 3-sentence comment they wrote.
  - rictic 5y ago
    Sometimes people misunderstand the basics of their full time work yes We've all seen that person, and almost all of us have at times _been_ that person And unfortunately, given the replication crisis, there is no reason to assume that a published scientist is not making a fundamental error that undermines the validity of their work. It apparently happens quite frequently, and we should be quite grateful for those that question and probe at assumptions and potentially faulty analysis techniques
  - deleted 5y ago
    [deleted]
- 5y ago

nathanwh 5y ago

Honestly this XKCD arguing that psychic phenomena don't exist because they're not exploited for profit is a comforting difficult to refute argument for me: https://xkcd.com/808/ https://xkcd.com/808/

mathattack 5y ago
Yes. The existence of casinos disproves psychic powers.
- throwaway7033 5y ago
  How so? Casinos routinely ban card counters; if psychic powers existed then psychics would also be banned from casinos.
  - mathattack 5y ago
    Identifying card counters is easier than identifying psychics.
NateEag 5y ago
It may be that psychic powers exist but that they are not powerful, general, or controllable enough to be reliably profitable. Also, if every profitable phenomenon in the world is already developed, then no business should ever invest in fundamental research - i.e., the pharmaceutical companies are wasting their research dollars. Why is it comforting to you to believe psychic powers don't exist?
- gs17 5y ago
  Yeah, e.g. someone who can, for example, consistently predict single bits of output from true random number generators 51% of the time would be evidence of something very strange going on, but not necessarily usable for anything practical. (Maybe if you could gather enough of these people to overcome the "weak" ability?)
  - 5y ago

ALittleLight 5y ago

This post really makes you think and challenged my world view when I first read it. A while back I looked at the CIA "Star Gate" files[1], which, are a collection of documents released under FOIA detailing the CIA's investigation into psychic powers and whether or not those powers serve an intelligence gathering purpose. I just opened up random files and skimmed them - I wish I had taken notes when doing so, but my impression was that the CIA was actually producing some better than chance results with their experiments. I don't really know what to make of parapsychology or psychic powers in general. Sometimes I'll idly wonder if I should dedicate time to trying to uncover my psychic powers. 1 - https://www.cia.gov/readingroom/collection/stargate https://www.cia.gov/readingroom/collection/stargate

aaaaaaaaaaab 5y ago
Also, don't forget goat-staring!

amelius 5y ago

Reminds me of a talk of Rupert Sheldrake at Google Talks: https://www.youtube.com/watch?v=hic18Xyk9is https://www.youtube.com/watch?v=hic18Xyk9is and then later he gave a talk called "The Science Delusion" which got banned from TED: https://www.youtube.com/watch?v=JKHUaNAxsTg https://www.youtube.com/watch?v=JKHUaNAxsTg

3np 5y ago
What do you mean by “banned” (as opposed to just taken down from their video channel)? BTW, YouTube title says “TED TALK”; note that it’s TEDx.
- amelius 5y ago
  Does it matter? Sheldrake makes some wild claims, but eloquently packs them in a talk that is entertaining and at times thought-provoking. In any case, his braveness is certainly admirable. Worth a watch, I would say (perhaps with a pint of beer).

Sniffnoy 5y ago

(2014)

FabHK 5y ago

The "fits of nervous laughter" paper by Wiseman & Schlitz (1998) was replicated by Wiseman & Schlitz (1999) [1], with a similar yet different result: RW (again) found no effect of staring, MS found a significant (p=0.05) effect of staring, but with opposite direction than the 1998 paper. [1] https://richardwiseman.files.wordpress.com/2020/04/expt2.pdf https://richardwiseman.files.wordpress.com/2020/04/expt2.pdf , found on https://richardwiseman.wordpress.com/research/parapsychology/ https://richardwiseman.wordpress.com/research/parapsychology...

mannykannot 5y ago
Hence the enthusiasm for preregistration of trials.

Jweb_Guru 5y ago

> I know that standard practice here is to tell the story of Clever Hans and then say That Is Why We Do Double-Blind Studies. Yes, this is the answer to this entire article, and his attempt to dismiss this for some reason is strange to me.

igorkraw 5y ago
In the years that I've read him I noticed this is something that happens from time to time with Scott Siskinds (see 3 if you take offense to me using his name) texts, especially when he steps out of his zone of competence to opine on politics, science in general, AI etc. The simple, obvious answer that the establishment believes in is met with some snarky ridicule and on goes the article, with some very subtle rhetoric veiled in the irony ("rigour doesn't matter, all scientists have biases, you can't trust rigorous science if the wrong people do it" is the message behind [quote] Studies are going to be confounded by the allegiance of the researcher. When researchers who don’t believe something discover it, that’s when it’s worth looking into. [/quote] ). I often wondered why this is, because he seems like a very smart guy with interesting ideas sometimes, so it seemed kind of weird and conflicting. Then I learned he's trying to hide his beliefs about scientific racism and political views (1) while often writing in the tone of "pained centrist just wanting to understand things scientifically" and it made much more sense as (in the worst case) an old school propaganda technique of FUD in science and rigorous science to make the HBD go down better and (more charitably) as Mr. Siskind just being another bright-but-human person who when writing on things he has actual expertise on can be very well worth reading, but simply shouldn't be trusted on anything aside from this. Rationalwiki has a decent writeup on him (2) 1: https://web.archive.org/web/20210217195335/https://twitter.com/TopherTBrennan/status/1362108632070905857 https://web.archive.org/web/20210217195335/https://twitter.c... 2: https://rationalwiki.org/wiki/Scott_Alexander https://rationalwiki.org/wiki/Scott_Alexander 3: https://news.ycombinator.com/item?id=25874025 https://news.ycombinator.com/item?id=25874025
- wpietri 5y ago
  Exactly. He's definitely smart, and so it was a mystery to me why his articles were so incredibly long and rambling. (I get that some people enjoy his prose regardless of content, but I'm not one of them.) Why would a smart person who in theory wants to convey important points be so obscurantist about them? I threw up my hands and put it down to a personality quirk. It was the Sandifer article "The Beigeness" that cleared it up for me: https://www.eruditorumpress.com/blog/the-beigeness-or-how-to-kill-people-with-bad-writing-the-scott-alexander-method https://www.eruditorumpress.com/blog/the-beigeness-or-how-to...

chalst 5y ago

Andrew Gelman had an interesting 9-years-too-late idea for what to do in cases like Bem's article: https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-bem-paper-back-2010-click-find-surprisingly-simple-answer/ https://statmodeling.stat.columbia.edu/2019/01/27/jpsp-done-... Rather than publish all of the dubious speculations and the analysis whose problems took time to emerge, simply publish the data. The advantage, in his words: > In the example of the ESP study, if anything’s valuable it’s the data. Publishing the data would get the journal off the hook regarding fairness, open-mindedness, and not missing a scoop, while enabling others to move on reanalyses right away, and without saddling the journal with an embarrassing endorsement of a weak theory that, it turns out, was not really supported by data at all.

MaxBarraclough 5y ago
Obvious question then: what are the arguments/forces opposing this? Just inertia in the publishing norms?
- Misdicorl 5y ago
  The approach fails miserably when taken in aggregate. I would read five to ten papers from my own field a week while in grad school. If I had to do the analysis for myself as well there wouldn't be any time to do my own research. Add on twenty more papers a week from adjacent fields where I don't even know all the techniques and the magnitude of the problem hopefully becomes clearer.
  - MaxBarraclough 5y ago
    The raw figures would be available in addition to the paper itself, not instead of it.
    - chalst 5y ago
      Gelman's suggestion is how to handle articles with unlikely conclusions: don't publish the submitted article, though that presumably would be circulated as a preprint, but just the data. Then the journal is bringing attention to interesting data without endorsing the dubious analysis or conclusions.

monocasa 5y ago

[2014]

olliej 5y ago

Well this certainly makes me glad that the only time I was having to do statistical studies for research purposes, it was in the context of generated code performance.

deleted 5y ago

[deleted]

adamgordonbell 5y ago

So if you assume ESP doesn't exist then parapsychology research has something important to say about the replication crisis in social sciences. This is a surprising thing I wouldn't have thought of on my own.

egypturnash 5y ago

Finally, they would conduct the experiment in a series of different batches. Half the batches (randomly assigned) would be conducted by Dr. Schlitz, the other half by Dr. Wiseman. Because the two authors had very carefully standardized the setting, apparatus and procedure beforehand, “conducted by” pretty much just meant greeting the participants, giving the experimental instructions, and doing the staring. The results? Schlitz’s trials found strong evidence of psychic powers, Wiseman’s trials found no evidence whatsoever. Take a second to reflect on how this makes no sense. Two experimenters in the same laboratory, using the same apparatus, having no contact with the subjects except to introduce themselves and flip a few switches – and whether one or the other was there that day completely altered the result. For a good time, watch the gymnastics they have to do to in the paper to make this sound sufficiently sensical to even get published. This is the only journal article I’ve ever read where, in the part of the Discussion section where you’re supposed to propose possible reasons for your findings, both authors suggest maybe their co-author hacked into the computer and altered the results. Well, if you would take the "psi can never exist" blinders off, Mr. Codex, then this seems to be a pointer towards "this works better if you believe it will work", and it makes perfect sense if you accept this. Schlitz's belief that it would work resulted in it working. Wiseman's belief that it wouldn't resulted in it not working. The whole idea behind psi is that the human mind can sense and/or effect the world in currently-inexplicable ways, and here it is doing just that. With at least ten other experiments on this idea getting similar results. But parapsychology is completely and utterly a scam in your eyes so that is not a result you will consider.

einpoklum 5y ago
One can get quite a bit of popular traction from arranging for one's theory to be effectively-impossible to disprove (e.g. "it only works if you believe in it during the experiment"). Not much explanatory power though.
- egypturnash 5y ago
  The experiment I excerpted his description of seems to go along with exactly that. Experimenter who believes? Things happen. Experimenter who does not believe? Nothing happens.

dang 5y ago

Discussed at the time: The Control Group Is Out of Control - https://news.ycombinator.com/item?id=7666575 https://news.ycombinator.com/item?id=7666575 - April 2014 (74 comments)

ed-209 5y ago

"...things we haven’t discovered yet which are at least as weird as subconscious emotional cues" - sounds like psi to me.

Karellen 5y ago

> Pre-registration only assures that your study will not get any worse than it was the first time you thought of it, which may be very bad indeed. > > Searching for publication bias only means you will get all of the confounded studies, instead of just some of them. No, pre-registration assures that when you do a study and get a negative result, you don't simply forget to publish it - helping to reduce publication bias. But also, it prevents you from changing what you were looking for (because that's part of the pre-registration data) and "data dredging" for something that looks like a positive result, helping to reduce p-hacking - specifically of the "Texas sharpshooter fallacy" variety.

jessriedel 5y ago
What you wrote and what the author wrote do not conflict. Indeed, the author already noted the goal of preregistration earlier in the essay. He is explaining here why it doesn't save you from different issues.
BeFlatXIII 5y ago
How does pre-reg handle cases where unexpected findings are made along the way? Are those data to be scrapped so everyone has to start their pre-registered "real" study from scratch?
- gjvnq 5y ago
  I imagine you would report both that you failed to find what you expected but that you also found something else.

EGreg 5y ago

What happened with Bem? Did his studies get debunked or he is still able to prove things? And the other guy runs same studies and gets different result ? WhT ?

coldtea 5y ago

Note the implicit (and, to most, justified) a priori convinction that psychic phenomena cannot exist, even if they have evidence to the contrary with similar confidence as other ("coventional") study subjects. Isn't this an "isolated demand for rigor"? [1] Doesn't this prove that science as practiced is not just the empirical process and some raw axioms that guide it (which these "psychic research" both follows), which scientists pretend to ascribe to, but also part blind faith/prejudice? I'm not saying physic phenomena are real. I'm saying if tests are conducted with the same principles as those about physical phenomena, and results are equally carefully collected, and they have the same condidence, science in principle shouldn't treat them any differently than results about "regular" subjects. [1] https://slatestarcodex.com/2014/08/14/beware-isolated-demands-for-rigor/ https://slatestarcodex.com/2014/08/14/beware-isolated-demand...

amatic 5y ago
>science in principle shouldn't treat them any differently than results about "regular" subjects. That's the whole problem, as I understand it. If we are going to treat ESP research the same as research in other fields, we either accept ESP as real, or reject both ESP and other fields as deeply flawed. Personally, I'm leaning heavily toward the second option. The methods in ESP are identical to methods in psychology, social sciences, etc, but not physics or chemistry. I'm just wondering where to draw the line. Maybe the whole method of statistical inference needs to be flushed? Maybe it cannot be saved by preregistration, publishing all the data, transparency and other bells and whistles.
- coldtea 5y ago
  >That's the whole problem, as I understand it. If we are going to treat ESP research the same as research in other fields, we either accept ESP as real, or reject both ESP and other fields as deeply flawed. That's my point them. "If we are going to treat ESP research the same as research in other fields", according to the principles science pays lip service to this shouldn't be a question to begin with, because science shouldn't have a priori ideas about what can be treated "the same as other fields" and what shouldn't. Research (results) should determine that (which means: if there are results in its favor, and we can't see a problem with the research that other - non ESP - accepted research doesn't have, we should accept it, as much as we accept in other such - non ESP - results).