I really wanted to start this comment with the HN standard zero-content dick remark of "Is anyone really surprised ...", because I'm really not surprised. Excep

I really wanted to start this comment with the HN standard zero-content dick remark of "Is anyone really surprised ...", because I'm really not surprised. Except the fair comment would be, is anyone who actually bothered to look up the references and citations of some of his more bold (but oh so juicy) claims, really surprised, really? It's a typical situation of that telephone-whisper game; researchers do a test they came up with on a group of 21 students. In social psychology this is considered a reasonable sample size. From where I'm standing, this is a non-starter for doing research and calling it science. Just don't even bother. It's better to not know anything than to do it any way and catch a bias (yes like the disease it is). Stopping attributing meaning to what is essentially random data is what got us into this whole modern era of technology in the first place. It seems pretty clear by now which fields of science took that seriously and which ones preferred to instead read (meta meta) meta studies and "sit in their office, bouncing ideas of one another"[0]. But hey, I'm from computational science and we can always generate more data, which starts at sample sizes of about, oh, 10K or so? (at least) I always get the idea that these guys just wanted to sell books, filled with slightly-counterintuitive yet somehow plausible factoids. Add in the veneer of scientific credibility and you've got a very juicy best-selling combo. For people who like to feel they are scientific / rational intelligent, make it a part of their ego--something which I can totally relate to, btw, but I try to be better. A few things that stood out from Kahneman's comment that reflect his ego (even though he uses words to sound humble, he can't quite find the courage, if he could it wouldn't have gotten this far); I had to look up "file-drawer problem", turns out it is a cutesy euphemism for "publication bias". Care to guess why he doesn't use that word? He does it twice, even, so it's definitely not to add variation or flavour to his writing style. Especially when replacing the two usages of the term in context would become "severe publication bias" and "substantial publication bias undermines the two main tools that psychologists use to accumulate evidence", which sounds really pretty damning, much more so than calling it "file-drawer problem". > first paper that Amos Tversky and I published was about the belief in the “law of small numbers,” which allows researchers to trust the results of underpowered studies with unreasonably small samples. He likes to coin terms a lot. I know the "Law of Small Numbers" in a mathematical context where it means something entirely different. So I looked it up and it turns out to be a euphemism for a "hasty generalization fallacy", kind of the exact opposite of what he suggests here. Anyone care to check this citation? What is this magical law that allows researchers to trust the results of underpowered studies with unreasonably small samples? It sounds beyond implausible to me. You can statistics your way around this in circles but really you can also just like, dismiss underpowered studies with unreasonably small samples. > We also cited Overall (1969) for showing “that the prevalence of studies deficient in statistical power is not only wasteful but actually pernicious: it results in a large proportion of invalid rejections of the null hypothesis among published results.” Our article was written in 1969 and published in 1971, but I failed to internalize its message. Yeah right. The real answer is "because I had books to sell, research grants to obtain and an ego to maintain". You don't need someone to write a paper about this to come to this conclusion. Of course it's harmful, how can this not be obvious? It's also, like, a MAJOR part of social psychology and similar studies because sample sizes are always stupid small. And Kahneman is an expert in this field. So apart from that he already KNEW this because he has common sense, he MUST have already internalized it because he's an expert in this field and you come across this particular bit of common sense all the time. Therefore, no, you CHOOSE to ignore that reality (not message because you were perfectly aware of this before that paper you cited). > if a large body of evidence published in reputable journals supports an initially implausible conclusion, then scientific norms require us to believe that conclusion. Implausibility is not sufficient to justify disbelief, and belief in well-supported scientific conclusions is not optional. If you come from the point of view of the exact, hard sciences, like physics, math or computational science, it's kind of hard to see anything wrong with this statement, on the surface. But if you know a thing or two about social sciences and the like, the bullshit that goes on there, you know that almost everything is wrong about the above statement. Reputable journals often aren't. And there is nothing, absolutely nothing, in this world that can make someone believe. You cannot require it. And to say that "belief" is not optional is almost an oxymoron. Unless you use brainwashing. Except I'm not sure if "brainwashing" even really works because, you know, guess what fields conducted the unethical studies into it. > This position still seems reasonable to me – it is why I think people should believe in climate change. Please don't drag climate science through the same mud as your clusterfuck of research. The hard numbers and sample sizes they have access to, are so large you'd soil your pants. > But the argument only holds when all relevant results are published. Which you KNEW is not the case, so that's not really an excuse is it. > I knew, of course, that the results of priming studies were based on small samples, that the effect sizes were perhaps implausibly large, and that no single study was conclusive on its own. But I had books to sell, research grants to obtain etc etc > However, I now understand that my reasoning was flawed and that I should have known better. But I had books to sell, research grants to obtain etc etc > I knew all I needed to know to moderate my enthusiasm for the surprising and elegant findings that I cited, but I did not think it through. But I had books to sell, research grants to obtain etc etc > I still believe that actions can be primed, sometimes even by stimuli of which the person is unaware. If he's so convinced that "belief in well-supported scientific conclusions is not optional", then it also holds when the scientific conclusions say the opposite, and he should DROP this belief right this instance until proper evidence is obtained. This is not belief, it's stubbornness. I mean sure, I would personally say, don't throw out the baby with the bath-water (because I also don't quite agree with the other one, science can be flawed like any human endeavour). But he only just went hard-line science on this idea, in the very same comment, and you should at least be consequent about it. I really don't think he's learned any lesson. He led his scientific beliefs be guided by ego, clouded by the idea that research proved what he wanted to be true, and he's published books filled with untruths that are out there right now. Is he going to issue retractions? Errata? Cause you know, lay people are going to read this years into the future, and believe this crap. [0] In a book by the Kahneman/Tversky/Taleb trio of juicy pop-psych writers, they described their research methodology this way. Proudly so, because what could be better science than such incredibly smart people giving the freedom to "bounce ideas of one another" ... Sorry this post is not proper science, I can't recall and properly cite what book it was :-/ I think it was Taleb talking about his buddies Kahneman and Tversky.