10 ms·

I disagree with Turing and Kahneman regarding statistical evidence (2014)

129 points by acqbu 4y ago

woliveirajr 4y ago
@dang: the original title should be "I disagree with Alan Turing and Daniel Kahneman regarding the strength of statistical evidence", the way it was edited it became... clickbait ?
- function_seven 4y ago
  Probably done to get it under 80 characters. Maybe I Disagree with Turing & Kahneman Regarding the Strength of Statistical Evidence will work? It's exactly 80.
- pvg 4y ago
  Email this stuff in if you want it to reach a dang.
kqr 4y ago
The conclusion is a bit at odds with the rest of the piece: > And that’s interesting. When stupid people make a mistake, that’s no big deal. But when brilliant people make a mistake, it’s worth noting. Maybe not. In the words of the author, > maybe these are real effects being discovered, but you should at least consider the possibility that you’re chasing noise. When a striking result appears in the dataset, it’s possible that this result does not represent an enduring truth or even a pattern in the general population but rather is just an artifact of a particular small and noisy dataset.
- pessimizer 4y ago
  Clearly brilliant people saying that something that later turns out to be noise is statistically undeniable signal is worth noting. I'm not sure what you're seeing here.
  - kqr 4y ago
    I don't think it is worth noting. Brilliant people make mistakes too, just not at the same rate as the rest of us. That one or even a few make the a specific mistake means nothing -- it's statistical fluke, even if it seems noteworthy in isolation.
deleted 4y ago
[deleted]
ev7 4y ago
Kahneman admitted that some of the studies his book cited were underpowered here: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/comment-page-1/#comment-1454 https://replicationindex.com/2017/02/02/reconstruction-of-a-...
arolihas 4y ago
Reminds me a lot of Beware the Man of One Study by Scott Alexander https://www.lesswrong.com/s/BQBqPowfxjvoee8jw/p/ythFNoiAotjvuEGkg https://www.lesswrong.com/s/BQBqPowfxjvoee8jw/p/ythFNoiAotjv...
- mannykannot 4y ago
  Perhaps one should also beware of the study by one man. I have a vague recollection of hearing that, at the time Turing was expressing his opinion of telepathy, there seemed to be experimental evidence for it, but it turned out that all the evidence came from one person, using unsound methods (including, IIRC, counting "one before" or "one after" guesses as successes if the results had higher than the expected frequency of one of those outcomes.) Tentatively, that person may be Joseph Banks Rhine.
  - googlryas 4y ago
    Yes - I had to read into that part of Turing, since I had never previously come across his views on ESP, but it seems like his error was not that he believed the statistics to be more powerful than they were - but rather he was too credulous in believing there weren't charlatans intentionally fudging the stats in order to help their point. Basically - the stats were powerful, but they were just fake.
- giraffe_lady 4y ago
  - Marinus 4y ago
    Thats a liber and a smear. I expect better of HN.
  - Veen 4y ago
    > isn't he a eugenicist? No. Although the term is widely abused and misapplied, so he might fall within whatever definition of eugenicist you're working with.
ChrisArchitect 4y ago
Discussion from 8 years ago: https://news.ycombinator.com/item?id=8263243 https://news.ycombinator.com/item?id=8263243
randcraw 4y ago
> It’s good to have an open mind. Is it? That's the very basis for error that begat this article. In fact, Turing's and Kahneman's minds were too open. They didn't express sufficient reservation by demanding more rigorous tests of those claims. Perhaps a better maxim would be, "It's good to have a mind that's open just enough to entertain the impossible."
- clint 4y ago
  Its funny, I considered Turing's state of mind to be not open enough to the possibility that he might be wrong.
  - n4r9 4y ago
    My thoughts exactly. "This claim is true and you mustn't disagree" is quite closed-minded.
- deleted 4y ago
  [deleted]
- arto 4y ago
  "By all means let's be open-minded, but not so open-minded that our brains drop out." -- Richard Dawkins
  - eyelidlessness 4y ago
    An appropriately simplistic quote from someone who habitually and sometimes famously demonstrates being stubbornly narrow minded. But then, how else would he not know that brains don’t fall out of—nor are placed in—minds.
  - bloak 4y ago
    That saying, or ones like it, are also attributed to Carl Sagan, G. K. Chesterton, James Oberg, Arthur Hays Sulzberger, Virginia Gildersleeve, Bertrand Russell, Max Radin, ... This article mentions Walter M. Kotschnig as the author of the earliest known instance of the saying, but notes that even Kotschnig seemed to be quoting something that was already known at the time: https://www.skeptic.com/insight/open-mind-brains-fall-out-maxim-adage-aphorism/ https://www.skeptic.com/insight/open-mind-brains-fall-out-ma...
glacials 4y ago
Hard not to chuckle at: > People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psvchological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions. (emphasis mine) Not only does the statement say that “people” make this mistake then go on to cite a questionnaire only of professional psychologists, but the questionnaire is the prototypical non-random random sample the very psychologists taking it are allegedly proving misinformation about. Questionnaires select for people with the time, inclination, and attention to start and then finish a questionnaire. Infamously, experiments held at universities bias towards undergrads. Experiments that reward participation bias towards people motivated by the reward. Experiments that don’t reward participation bias towards people good-natured enough to contribute just for science. Randomness is hard.
- pixl97 4y ago
  Using undergrads is how you get W.E.I.R.D. science
- autosharp 4y ago
  Certainly very ironic. But is this line of argument incorrect? E.g. if I said "I don't believe in the Holy Book because in verse 7 it says that one cannot trust anything written in any books". Isn't that an analogous reasoning? I think these examples/arguments are ultimately about exposing a liar-paradox statement, and when you can show such a statement you have proven that something isn't right.
- raverbashing 4y ago
  > Randomness is hard. Yes. And might I say everything has a statistical bias. Picking people for an experiment is not like throwing dice Your sample will always be biased, and it would be crazy to try and get a random sample from "the world". (and even then you have temporal bias - you can't run your experiments on people from the XIX century)
jstx1 4y ago
Kahneman's full quote is just embarrassing: > The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. More important, you must accept that they are true about you. So so arrogant, and he ended up being wrong about it too. It's very hard to take him seriously after reading this.
- klysm 4y ago
  How do you end up with such a fundamentally flawed epistemology?
  - FabHK 4y ago
    I'd say that it wasn't the epistemology that was flawed, but the studies. (Though, ok, arguably it was a flaw in the epistemology that it didn't allow for flaws in the studies.) But the more interesting question is how the initial studies could have been so flawed. At any rate, those results were not replicated, and science marches on.
    - Jensson 4y ago
      How is "More important, you must accept that they are true about you." anything but flawed? If something is true for 60% of people must you accept that it is true for you as well, even though evidently it wasn't true for 40%?
- neuronexmachina 4y ago
  It's interesting that his "proposal to deal with questions about priming effects" was just a year later (2012): https://www.nature.com/news/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf https://www.nature.com/news/polopoly_fs/7.6716.1349271308!/s... > My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young people on the job market. Being associated with a controversial and suspicious field will put them at a severe disadvantage in the competition for positions. Because of the high visibility of the issue, you may already expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the replicability of findings: subliminal perception and dissonance reduction. > I believe that you should collectively do something about this mess. To deal effectively with the doubts you should acknowledge their existence and confront them straight on, because a posture of defiant denial is selfdefeating. Specifically, I believe that you should have an association, with a board that might include prominent social psychologists from other field. The first mission of the board would be to organize an effort to examine the replicability of priming results, following a protocol that avoids the questions that have been raised and guarantees credibility among colleagues outside the field. His more recent (2022) retrospective on the episode is also pretty interesting: https://www.edge.org/adversarial-collaboration-daniel-kahneman https://www.edge.org/adversarial-collaboration-daniel-kahnem...
pfisherman 4y ago
> How could Turing have thought this? I don’t know much about Turing but it does seem, when reading old-time literature, that belief in the supernatural was pretty common back then, lots of mention of ghosts etc. I think that today we overestimate how much we know, and discount the existence of “supernatural” phenomena that we don’t fit into our model of the universe. Karma (or something like it) is a thing that probably exists; but we do not have the necessary concepts or tools to qualitatively or quantitatively describe how it works.
- bobthechef 4y ago
  I'm not sure what you mean by karma in this case, but I wanted to note that the word "supernatural" is very often misused. Once you start prodding people about what they mean by "supernatural", you quickly discover that they don't know what "natural" or "physical/material" mean, much less what "supernatural" means, and the whole conversation descends into grumbling. Here's one definition of "supernatural order" [0]. [0] https://www.newadvent.org/cathen/14336b.htm https://www.newadvent.org/cathen/14336b.htm
- CyberDildonics 4y ago
  The only reason to believe that karma exists is that we have an emotional desire for it to exist.
kingludite 4y ago
I doubt we can get places if the reader of this comment ws to produce thoughts about what I think about what the author thinks psychologists think. I'm not comfortable with the idea the unmatched creative engineering powers of mother nature did not discover something as simple as radio. We could power down the electrical grid and do the experiment? But I can just call everyone, what use is telepathy? The cat is in front of the house when someone arrives (which is never at the same time sometimes skipping a day) but the rest of the day he is nowhere near on the camera footage. For him there is a point, it means food! Sheldrake would love it.
amai 4y ago
Kahneman and his book „Thinking fast and slow“ are totaly overrated. In fact „despite Kahneman's previous contributions to the field of decision making, most of the book's ideas are based on 'scientific literature with shaky foundations'. A general lack of replication in the empirical studies cited in the book was given as a justification.“ Unfortunately you will find that statement only at the end of https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow. It should be stated at the beginning. So many wrong assumptions especially in economics, but also management theory are based on this book. But nothing of is book can be replicated: https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow/ https://replicationindex.com/2020/12/30/a-meta-scientific-pe... Unfortunately far to few people know about this.
tpoacher 4y ago
"Whenever there is a simple error that most laymen fall for, there is always a slightly more sophisticated version of the same problem that experts fall for." --Amos Tversky

129 points by acqbu 4y ago

@dang: the original title should be "I disagree with Alan Turing and Daniel Kahneman regarding the strength of statistical evidence", the way it was edited it became... clickbait ?

function_seven 4y ago
Probably done to get it under 80 characters. Maybe I Disagree with Turing & Kahneman Regarding the Strength of Statistical Evidence will work? It's exactly 80.
pvg 4y ago
Email this stuff in if you want it to reach a dang.

kqr 4y ago

The conclusion is a bit at odds with the rest of the piece: > And that’s interesting. When stupid people make a mistake, that’s no big deal. But when brilliant people make a mistake, it’s worth noting. Maybe not. In the words of the author, > maybe these are real effects being discovered, but you should at least consider the possibility that you’re chasing noise. When a striking result appears in the dataset, it’s possible that this result does not represent an enduring truth or even a pattern in the general population but rather is just an artifact of a particular small and noisy dataset.

pessimizer 4y ago
Clearly brilliant people saying that something that later turns out to be noise is statistically undeniable signal is worth noting. I'm not sure what you're seeing here.
- kqr 4y ago
  I don't think it is worth noting. Brilliant people make mistakes too, just not at the same rate as the rest of us. That one or even a few make the a specific mistake means nothing -- it's statistical fluke, even if it seems noteworthy in isolation.

deleted 4y ago

[deleted]

ev7 4y ago

Kahneman admitted that some of the studies his book cited were underpowered here: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/comment-page-1/#comment-1454 https://replicationindex.com/2017/02/02/reconstruction-of-a-...

arolihas 4y ago

Reminds me a lot of Beware the Man of One Study by Scott Alexander https://www.lesswrong.com/s/BQBqPowfxjvoee8jw/p/ythFNoiAotjvuEGkg https://www.lesswrong.com/s/BQBqPowfxjvoee8jw/p/ythFNoiAotjv...

mannykannot 4y ago
Perhaps one should also beware of the study by one man. I have a vague recollection of hearing that, at the time Turing was expressing his opinion of telepathy, there seemed to be experimental evidence for it, but it turned out that all the evidence came from one person, using unsound methods (including, IIRC, counting "one before" or "one after" guesses as successes if the results had higher than the expected frequency of one of those outcomes.) Tentatively, that person may be Joseph Banks Rhine.
- googlryas 4y ago
  Yes - I had to read into that part of Turing, since I had never previously come across his views on ESP, but it seems like his error was not that he believed the statistics to be more powerful than they were - but rather he was too credulous in believing there weren't charlatans intentionally fudging the stats in order to help their point. Basically - the stats were powerful, but they were just fake.
giraffe_lady 4y ago
- Marinus 4y ago
  Thats a liber and a smear. I expect better of HN.
- Veen 4y ago
  > isn't he a eugenicist? No. Although the term is widely abused and misapplied, so he might fall within whatever definition of eugenicist you're working with.

ChrisArchitect 4y ago

Discussion from 8 years ago: https://news.ycombinator.com/item?id=8263243 https://news.ycombinator.com/item?id=8263243

randcraw 4y ago

> It’s good to have an open mind. Is it? That's the very basis for error that begat this article. In fact, Turing's and Kahneman's minds were too open. They didn't express sufficient reservation by demanding more rigorous tests of those claims. Perhaps a better maxim would be, "It's good to have a mind that's open just enough to entertain the impossible."

clint 4y ago
Its funny, I considered Turing's state of mind to be not open enough to the possibility that he might be wrong.
- n4r9 4y ago
  My thoughts exactly. "This claim is true and you mustn't disagree" is quite closed-minded.
deleted 4y ago
[deleted]
arto 4y ago
"By all means let's be open-minded, but not so open-minded that our brains drop out." -- Richard Dawkins
- eyelidlessness 4y ago
  An appropriately simplistic quote from someone who habitually and sometimes famously demonstrates being stubbornly narrow minded. But then, how else would he not know that brains don’t fall out of—nor are placed in—minds.
- bloak 4y ago
  That saying, or ones like it, are also attributed to Carl Sagan, G. K. Chesterton, James Oberg, Arthur Hays Sulzberger, Virginia Gildersleeve, Bertrand Russell, Max Radin, ... This article mentions Walter M. Kotschnig as the author of the earliest known instance of the saying, but notes that even Kotschnig seemed to be quoting something that was already known at the time: https://www.skeptic.com/insight/open-mind-brains-fall-out-maxim-adage-aphorism/ https://www.skeptic.com/insight/open-mind-brains-fall-out-ma...

glacials 4y ago

Hard not to chuckle at: > People have erroneous intuitions about the laws of chance. In particular, they regard a sample randomly drawn from a population as highly representative, that is, similar to the population in all essential characteristics. The prevalence of the belief and its unfortunate consequences for psvchological research are illustrated by the responses of professional psychologists to a questionnaire concerning research decisions. (emphasis mine) Not only does the statement say that “people” make this mistake then go on to cite a questionnaire only of professional psychologists, but the questionnaire is the prototypical non-random random sample the very psychologists taking it are allegedly proving misinformation about. Questionnaires select for people with the time, inclination, and attention to start and then finish a questionnaire. Infamously, experiments held at universities bias towards undergrads. Experiments that reward participation bias towards people motivated by the reward. Experiments that don’t reward participation bias towards people good-natured enough to contribute just for science. Randomness is hard.

pixl97 4y ago
Using undergrads is how you get W.E.I.R.D. science
autosharp 4y ago
Certainly very ironic. But is this line of argument incorrect? E.g. if I said "I don't believe in the Holy Book because in verse 7 it says that one cannot trust anything written in any books". Isn't that an analogous reasoning? I think these examples/arguments are ultimately about exposing a liar-paradox statement, and when you can show such a statement you have proven that something isn't right.
raverbashing 4y ago
> Randomness is hard. Yes. And might I say everything has a statistical bias. Picking people for an experiment is not like throwing dice Your sample will always be biased, and it would be crazy to try and get a random sample from "the world". (and even then you have temporal bias - you can't run your experiments on people from the XIX century)

jstx1 4y ago

Kahneman's full quote is just embarrassing: > The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. More important, you must accept that they are true about you. So so arrogant, and he ended up being wrong about it too. It's very hard to take him seriously after reading this.

klysm 4y ago
How do you end up with such a fundamentally flawed epistemology?
- FabHK 4y ago
  I'd say that it wasn't the epistemology that was flawed, but the studies. (Though, ok, arguably it was a flaw in the epistemology that it didn't allow for flaws in the studies.) But the more interesting question is how the initial studies could have been so flawed. At any rate, those results were not replicated, and science marches on.
  - Jensson 4y ago
    How is "More important, you must accept that they are true about you." anything but flawed? If something is true for 60% of people must you accept that it is true for you as well, even though evidently it wasn't true for 40%?
neuronexmachina 4y ago
It's interesting that his "proposal to deal with questions about priming effects" was just a year later (2012): https://www.nature.com/news/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf https://www.nature.com/news/polopoly_fs/7.6716.1349271308!/s... > My reason for writing this letter is that I see a train wreck looming. I expect the first victims to be young people on the job market. Being associated with a controversial and suspicious field will put them at a severe disadvantage in the competition for positions. Because of the high visibility of the issue, you may already expect the coming crop of graduates to encounter problems. Another reason for writing is that I am old enough to remember two fields that went into a prolonged eclipse after similar outsider attacks on the replicability of findings: subliminal perception and dissonance reduction. > I believe that you should collectively do something about this mess. To deal effectively with the doubts you should acknowledge their existence and confront them straight on, because a posture of defiant denial is selfdefeating. Specifically, I believe that you should have an association, with a board that might include prominent social psychologists from other field. The first mission of the board would be to organize an effort to examine the replicability of priming results, following a protocol that avoids the questions that have been raised and guarantees credibility among colleagues outside the field. His more recent (2022) retrospective on the episode is also pretty interesting: https://www.edge.org/adversarial-collaboration-daniel-kahneman https://www.edge.org/adversarial-collaboration-daniel-kahnem...

pfisherman 4y ago

> How could Turing have thought this? I don’t know much about Turing but it does seem, when reading old-time literature, that belief in the supernatural was pretty common back then, lots of mention of ghosts etc. I think that today we overestimate how much we know, and discount the existence of “supernatural” phenomena that we don’t fit into our model of the universe. Karma (or something like it) is a thing that probably exists; but we do not have the necessary concepts or tools to qualitatively or quantitatively describe how it works.

bobthechef 4y ago
I'm not sure what you mean by karma in this case, but I wanted to note that the word "supernatural" is very often misused. Once you start prodding people about what they mean by "supernatural", you quickly discover that they don't know what "natural" or "physical/material" mean, much less what "supernatural" means, and the whole conversation descends into grumbling. Here's one definition of "supernatural order" [0]. [0] https://www.newadvent.org/cathen/14336b.htm https://www.newadvent.org/cathen/14336b.htm
CyberDildonics 4y ago
The only reason to believe that karma exists is that we have an emotional desire for it to exist.

kingludite 4y ago

I doubt we can get places if the reader of this comment ws to produce thoughts about what I think about what the author thinks psychologists think. I'm not comfortable with the idea the unmatched creative engineering powers of mother nature did not discover something as simple as radio. We could power down the electrical grid and do the experiment? But I can just call everyone, what use is telepathy? The cat is in front of the house when someone arrives (which is never at the same time sometimes skipping a day) but the rest of the day he is nowhere near on the camera footage. For him there is a point, it means food! Sheldrake would love it.

amai 4y ago

Kahneman and his book „Thinking fast and slow“ are totaly overrated. In fact „despite Kahneman's previous contributions to the field of decision making, most of the book's ideas are based on 'scientific literature with shaky foundations'. A general lack of replication in the empirical studies cited in the book was given as a justification.“ Unfortunately you will find that statement only at the end of https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow. It should be stated at the beginning. So many wrong assumptions especially in economics, but also management theory are based on this book. But nothing of is book can be replicated: https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow/ https://replicationindex.com/2020/12/30/a-meta-scientific-pe... Unfortunately far to few people know about this.

tpoacher 4y ago

"Whenever there is a simple error that most laymen fall for, there is always a slightly more sophisticated version of the same problem that experts fall for." --Amos Tversky