10 ms·

The conclusion in the article: > Why so angry? I know I’ve taken this far too personally. I have no illusions that everything I read online should be correct,

by quanto 4y ago

The conclusion in the article: > Why so angry? I know I’ve taken this far too personally. I have no illusions that everything I read online should be correct, or about people’s susceptibility to a strong rhetoric cleverly bashing conventional science, even in great communities such as HN. But frankly, for the last few years, the world seems to be accelerating the rate at which it’s going crazy, and it feels to me a lot of that is related to people’s distrust in science (and statistics in particular). Something about the way the author conveniently swapped “purely random” with “null hypothesis” (when it’s inappropriate!) and happily went on to call the authors “unskilled and unaware of it”, and about the ease with which people jumped on to the “lies, damned lies, statistics” wagon but were very stubborn about getting off, got to me. Deeply. I couldn’t let this go. I am afraid I actually agree with the author's point. The anti-intellectual, anti-scientific streak in many poor analyses claiming to debunk some scientific research is deeply concerning in our society. If someone is trying to debunk some scientific research, at least he should learn some basic analytic tools. This observation is independent of whether the original DK paper could have been better. That said, I give the benefit of doubt to the author of "The DK Effect is Autocorrelation." It is a human error to be overly zealous in some opinions without thinking it through.

boppo1 4y ago
What about the replication crisis? It's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I can't tell you how many times I've seen headlines or abstracts of studies that seem to contradict ones I've seen previously, and back and forth! Particularly in the social sciences. I recall one study that said all white people are committing environmental racism against all non-white people. I dove in and read the whole thing wondering what method could have yielded scientific confidence in such a broad result. Turns out the model used was a semi-black box that required a request for access and a supercomputer to run. But it was in a Peer Reviewed Scientific Journal and had lots of Graduate Level Statistics so I guess it seemed trustworthy.
- quanto 4y ago
  A replication crisis indeed exists. All the more reason to analyze rigorously. Poor analyses (and borderline name-calling) in the original article do not help with the crisis.
- whatshisface 4y ago
  >it's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I don't think this is true. It is possible to put a lot of work into unsound statistics and to make a lot of "noise and fury" about how mathematical you are while failing some basic principle, but I don't think sound statistics can mislead. The replication crisis was caused by scientists not being rigorous and journals not forcing them to be. You absolutely cannot accept publication as a sign of sound techniques except in journal/field combinations that have a deserved reputation.
  - kortilla 4y ago
    > but I don't think sound statistics can mislead. Of course they can, unless you magically exclude all statistics that made a bad assumption on independence. I plot all the daily high temperatures and the presence of the ice cream cart and it turns out the ice cream cart causes warmer highs! Solid statistics. Turns out the guy that has the ice cream cart has a weather app on his phone though and doesn’t come out on forecasted cold days.
    - mike_hearn 4y ago
      Is that the fault of statistics though, or the non-statistical implication of causation that was tacked on the end of the statistical detection of correlation? Statistics is pretty explicit that it can't tell you about causality, right?
      - omginternets 4y ago
        Yes, the whole point is that you can use sound statistics in bad faith.
legalcorrection 4y ago
Most social science is shoddy, fake, or otherwise misleading (i.e. it proves nothing meaningful despite the claims of the researchers). If you believed every social science study you heard about, you'd be more wrong about the world than if you disbelieved them all.
- macrolocal 4y ago
  Yep, it's the worst approach to this subject matter, except for all the alternatives.
darawk 4y ago
Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand mathematical statistics and the philosophy of statistics fairly deeply. There just isn't any way around it. 1. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...
- specialist 4y ago
  In other words, emphasize the process more than the outcomes. If the scientific process used proves sound, then I have more confidence in the outcomes. Alas, that's a hard sell to laypersons thru the mediums of soundbites and tweets.
- quanto 4y ago
  Completely agreed. While I strongly believe in citizen science and people's right (and perhaps obligation) to critique established science, there is just so much poor analyses done by people to criticize some scientific findings they do not approve, motivated emotionally or otherwise. This phenomenon does not bring us closer to finding scientific facts or resolving the replication crisis. People should learn some basic analytic tools first.
  - j3th9n 4y ago
    Scientific facts do not exist, scientific observations do.
  - bigfudge 4y ago
    We should really just do (and certainly publish) less research.
    - bigfudge 4y ago
      I can see why this has been downvoted, but I didn’t mean to sound like a luddite. I really just think there is a lot of low quality research, especially in the social sciences, done primarily to keep an publishing schedule up for pressured academics. We’d often be better to spend more time thinking about and planning fewer better studies.
heavyset_go 4y ago
I think what contributes to this phenomenon are both second-option bias[1] and motivated reasoning, at least with respect to those who choose to believe in the poor analyses. [1] https://rationalwiki.org/wiki/Essay:Second-option_bias https://rationalwiki.org/wiki/Essay:Second-option_bias
c1ccccc1 4y ago
It still seems to me like "The DK Effect is Autocorrelation" is basically correct. The important thing isn't whether or not independence should be the null hypothesis, because calling something a "null hypothesis" is just an arbitrary label that doesn't affect reality. The important thing is that what we can actually conclude from the Dunning-Kruger paper is a lot less than popular presentations of the concept claim. In particular, "more skilled people are better at predicting their own performance" is really not supported by the paper, since that's not true of random data, which has everyone being equally terrible at predicting their own performance. If the random data can reproduce that graph, then the graph can't be proof that more skilled people are also better predictors. Anyway, "The DK Effect is Autocorrelation" definitely seems to be both statistically literate, and a good faith criticism of the Dunning-Kruger paper. In light of that, calling it "anti-scientific" seems unfair, since criticism and debate are an important part of science.
- molf 4y ago
  > calling something a "null hypothesis" is just an arbitrary label that doesn't affect reality It does affect your conclusions though. The choice of null hypothesis in "The DK Effect is Autocorrelation" determined how the random data was generated. The hypothesis is: "nobody has any clue whatsoever how competent they are". The random data was specifically crafted for that hypothesis. The choice of null hypothesis in this article is: "everyone roughly knows how competent they are". This random data, too, is specifically crafted for the null hypothesis. So what does this mean? If you pick the a particular null hypothesis then you can try to argue that the DK is a statistical artefact. But it's not, it is an artefact of choosing a particular null hypothesis. There are valid criticisms of the DK study, though. See this comment for example: https://news.ycombinator.com/item?id=31119196 https://news.ycombinator.com/item?id=31119196
- qfwfq_ 4y ago
  No, nulls matter a great deal. If you want to test a claim in Null Hypothesis Statistical Testing, the "significance" of the claim is in direct reference to the null. Changing a null will change the significance of the alternative. My favorite statement of this is from Gelman: > the p-value is a strongly nonlinear transformation of data that is interpretable only under the null hypothesis, yet the usual purpose of the p-value in practice is to reject the null. My criticism here is not merely semantic or a clever tongue-twister or a “howler” (as Deborah Mayo would say); it’s real. In settings where the null hypothesis is not a live option, the p-value does not map to anything relevant. https://statmodeling.stat.columbia.edu/2017/01/07/we-fiddle-while-rome-burns-p-value-edition/ https://statmodeling.stat.columbia.edu/2017/01/07/we-fiddle-...
coffeeblack 4y ago
That happens when science is politicized, and any scientists critical of the “official” results is destroyed. From climate to Covid, so many areas where that happens.
- pavlov 4y ago
  If by “destroyed” you mean “billionaires will happily fund their contrarian research for years regardless of its peer reviews”…
  - coffeeblack 4y ago
    No, I mean researchers and professors suddenly not getting any research grants anymore, suddenly getting fired from their tenured jobs, not being invited anymore to conferences, etc. What billionaire funds “contrarian” research?
    - mellavora 4y ago
      There are some funding sources for biologists trying to prove biblical creation. Think also of all the money funding "climate change isn't real" or "climate change is not anthropomorphic" which can be traced back to big oil money.
    - HWR_14 4y ago
      You could build your whole career from the 50's on discrediting the link between smoking and cancer. The Koch brothers were (and the current brother and estate of the other) are happy to write economists checks to prove laissez-faire capitalism under a libertarian government is the best system. Or that climate change isn't real. If you're willing to generate evidence climate change isn't real, Exxon etc have some nice checks for you. If you're willing to show how corn syrup is good for American than the Iowa farmer association has money for you.
      - kortilla 4y ago
        > You could build your whole career from the 50's on discrediting the link between smoking and cancer. > If you're willing to generate evidence climate change isn't real, Exxon etc have some nice checks for you. Do you have any evidence of either of these being true today?
gambler 4y ago
> The anti-intellectual, anti-scientific streak in many poor analyses claiming to debunk some scientific research is deeply concerning in our society. People endlessly reference the Dunning-Kruger effect as a meme, without ever having read the paper, let alone having checked its methods. You don't seem to have a problem with that. On the other hand, after seeing an article that uses essentially statistical arguments to debate a scientific study you conclude that there is some "anti-intellectual, anti-scientific streak" in our society and that it should be of grave concern. This doesn't make any sense except as an extreme case of virtue-signaling.
- notahacker 4y ago
  Seems quite reasonable to argue that superficially plausible "debunkings" by people that apparently misunderstood a paper are more harmful to scientific progress than people casually referencing the scientist's names as a meme or insult. (And I say that as someone who didn't think the DK "debunking" argument was totally without merit) What's more harmful to medicine: a fashionably non-expert contrarian who doesn't understand the appropriate null hypothesis making a superficially plausible statistical argument that actually the trials suggest the drug is harmful to wide acclaim from laymen, or people casually referencing or even being administered the drug without reading the original trial writeups for themselves?
Rastonbury 4y ago
I read a lot of papers on behavioural economics and psychological decision making experiments for university, like dunning-kruger, kahneman, etc and in my opinion the first autocorrelation article reads like a rebuttal paper but more informal, the approach is scientific even if it may be flawed. This is how knowledge advances. I disagree that it is anti-science. Challenging accepted postulations is good. Even famous professors make mistakes, I don't blame the writer for making an honest mistake. That's how we got this new piece of writing Behavioural science is a pretty new field, its pretty easy to get abberant results or manipulate the results to show 'something' statistically. Many findings in earlier papers could not be replicated, or had applied statistics incorrectly, or showed different results when research participants were not white college kids. This is a whole other problem within academia, the pressure to publish something even when there is nothing and perceived legitimacy based on the number of citations a paper has. My professor always said don't look at the number of citations, understand the method and the rebuttal, there were numerous low citation but solid papers showing flaws in famous ones but everyone who isn't deep into the subject holds the original assertion to be legitimate because its "famous"
abirch 4y ago
> That said, I give the benefit of doubt to the author of "The DK Effect is Autocorrelation." It is a human error to be overly zealous in some opinions without thinking it through. If only there were a term for "a cognitive bias whereby people with limited knowledge or competence in a given intellectual or social domain greatly overestimate their own knowledge or competence in that domain relative to objective criteria or to the performance of their peers or of people in general"

by quanto 4y ago

boppo1 4y ago

What about the replication crisis? It's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I can't tell you how many times I've seen headlines or abstracts of studies that seem to contradict ones I've seen previously, and back and forth! Particularly in the social sciences. I recall one study that said all white people are committing environmental racism against all non-white people. I dove in and read the whole thing wondering what method could have yielded scientific confidence in such a broad result. Turns out the model used was a semi-black box that required a request for access and a supercomputer to run. But it was in a Peer Reviewed Scientific Journal and had lots of Graduate Level Statistics so I guess it seemed trustworthy.

quanto 4y ago
A replication crisis indeed exists. All the more reason to analyze rigorously. Poor analyses (and borderline name-calling) in the original article do not help with the crisis.
whatshisface 4y ago
>it's possible to use rigorously sound statistics to lie (or at least unknowingly spread falsehoods). I don't think this is true. It is possible to put a lot of work into unsound statistics and to make a lot of "noise and fury" about how mathematical you are while failing some basic principle, but I don't think sound statistics can mislead. The replication crisis was caused by scientists not being rigorous and journals not forcing them to be. You absolutely cannot accept publication as a sign of sound techniques except in journal/field combinations that have a deserved reputation.
- kortilla 4y ago
  > but I don't think sound statistics can mislead. Of course they can, unless you magically exclude all statistics that made a bad assumption on independence. I plot all the daily high temperatures and the presence of the ice cream cart and it turns out the ice cream cart causes warmer highs! Solid statistics. Turns out the guy that has the ice cream cart has a weather app on his phone though and doesn’t come out on forecasted cold days.
  - mike_hearn 4y ago
    Is that the fault of statistics though, or the non-statistical implication of causation that was tacked on the end of the statistical detection of correlation? Statistics is pretty explicit that it can't tell you about causality, right?
    - omginternets 4y ago
      Yes, the whole point is that you can use sound statistics in bad faith.

legalcorrection 4y ago

Most social science is shoddy, fake, or otherwise misleading (i.e. it proves nothing meaningful despite the claims of the researchers). If you believed every social science study you heard about, you'd be more wrong about the world than if you disbelieved them all.

macrolocal 4y ago
Yep, it's the worst approach to this subject matter, except for all the alternatives.

darawk 4y ago

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand mathematical statistics and the philosophy of statistics fairly deeply. There just isn't any way around it. 1. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

specialist 4y ago
In other words, emphasize the process more than the outcomes. If the scientific process used proves sound, then I have more confidence in the outcomes. Alas, that's a hard sell to laypersons thru the mediums of soundbites and tweets.
quanto 4y ago
Completely agreed. While I strongly believe in citizen science and people's right (and perhaps obligation) to critique established science, there is just so much poor analyses done by people to criticize some scientific findings they do not approve, motivated emotionally or otherwise. This phenomenon does not bring us closer to finding scientific facts or resolving the replication crisis. People should learn some basic analytic tools first.
- j3th9n 4y ago
  Scientific facts do not exist, scientific observations do.
- bigfudge 4y ago
  We should really just do (and certainly publish) less research.
  - bigfudge 4y ago
    I can see why this has been downvoted, but I didn’t mean to sound like a luddite. I really just think there is a lot of low quality research, especially in the social sciences, done primarily to keep an publishing schedule up for pressured academics. We’d often be better to spend more time thinking about and planning fewer better studies.

heavyset_go 4y ago

I think what contributes to this phenomenon are both second-option bias[1] and motivated reasoning, at least with respect to those who choose to believe in the poor analyses. [1] https://rationalwiki.org/wiki/Essay:Second-option_bias https://rationalwiki.org/wiki/Essay:Second-option_bias

c1ccccc1 4y ago

It still seems to me like "The DK Effect is Autocorrelation" is basically correct. The important thing isn't whether or not independence should be the null hypothesis, because calling something a "null hypothesis" is just an arbitrary label that doesn't affect reality. The important thing is that what we can actually conclude from the Dunning-Kruger paper is a lot less than popular presentations of the concept claim. In particular, "more skilled people are better at predicting their own performance" is really not supported by the paper, since that's not true of random data, which has everyone being equally terrible at predicting their own performance. If the random data can reproduce that graph, then the graph can't be proof that more skilled people are also better predictors. Anyway, "The DK Effect is Autocorrelation" definitely seems to be both statistically literate, and a good faith criticism of the Dunning-Kruger paper. In light of that, calling it "anti-scientific" seems unfair, since criticism and debate are an important part of science.

molf 4y ago
> calling something a "null hypothesis" is just an arbitrary label that doesn't affect reality It does affect your conclusions though. The choice of null hypothesis in "The DK Effect is Autocorrelation" determined how the random data was generated. The hypothesis is: "nobody has any clue whatsoever how competent they are". The random data was specifically crafted for that hypothesis. The choice of null hypothesis in this article is: "everyone roughly knows how competent they are". This random data, too, is specifically crafted for the null hypothesis. So what does this mean? If you pick the a particular null hypothesis then you can try to argue that the DK is a statistical artefact. But it's not, it is an artefact of choosing a particular null hypothesis. There are valid criticisms of the DK study, though. See this comment for example: https://news.ycombinator.com/item?id=31119196 https://news.ycombinator.com/item?id=31119196
qfwfq_ 4y ago
No, nulls matter a great deal. If you want to test a claim in Null Hypothesis Statistical Testing, the "significance" of the claim is in direct reference to the null. Changing a null will change the significance of the alternative. My favorite statement of this is from Gelman: > the p-value is a strongly nonlinear transformation of data that is interpretable only under the null hypothesis, yet the usual purpose of the p-value in practice is to reject the null. My criticism here is not merely semantic or a clever tongue-twister or a “howler” (as Deborah Mayo would say); it’s real. In settings where the null hypothesis is not a live option, the p-value does not map to anything relevant. https://statmodeling.stat.columbia.edu/2017/01/07/we-fiddle-while-rome-burns-p-value-edition/ https://statmodeling.stat.columbia.edu/2017/01/07/we-fiddle-...

coffeeblack 4y ago

That happens when science is politicized, and any scientists critical of the “official” results is destroyed. From climate to Covid, so many areas where that happens.

pavlov 4y ago
If by “destroyed” you mean “billionaires will happily fund their contrarian research for years regardless of its peer reviews”…
- coffeeblack 4y ago
  No, I mean researchers and professors suddenly not getting any research grants anymore, suddenly getting fired from their tenured jobs, not being invited anymore to conferences, etc. What billionaire funds “contrarian” research?
  - mellavora 4y ago
    There are some funding sources for biologists trying to prove biblical creation. Think also of all the money funding "climate change isn't real" or "climate change is not anthropomorphic" which can be traced back to big oil money.
  - HWR_14 4y ago
    You could build your whole career from the 50's on discrediting the link between smoking and cancer. The Koch brothers were (and the current brother and estate of the other) are happy to write economists checks to prove laissez-faire capitalism under a libertarian government is the best system. Or that climate change isn't real. If you're willing to generate evidence climate change isn't real, Exxon etc have some nice checks for you. If you're willing to show how corn syrup is good for American than the Iowa farmer association has money for you.
    - kortilla 4y ago
      > You could build your whole career from the 50's on discrediting the link between smoking and cancer. > If you're willing to generate evidence climate change isn't real, Exxon etc have some nice checks for you. Do you have any evidence of either of these being true today?

gambler 4y ago

> The anti-intellectual, anti-scientific streak in many poor analyses claiming to debunk some scientific research is deeply concerning in our society. People endlessly reference the Dunning-Kruger effect as a meme, without ever having read the paper, let alone having checked its methods. You don't seem to have a problem with that. On the other hand, after seeing an article that uses essentially statistical arguments to debate a scientific study you conclude that there is some "anti-intellectual, anti-scientific streak" in our society and that it should be of grave concern. This doesn't make any sense except as an extreme case of virtue-signaling.

notahacker 4y ago
Seems quite reasonable to argue that superficially plausible "debunkings" by people that apparently misunderstood a paper are more harmful to scientific progress than people casually referencing the scientist's names as a meme or insult. (And I say that as someone who didn't think the DK "debunking" argument was totally without merit) What's more harmful to medicine: a fashionably non-expert contrarian who doesn't understand the appropriate null hypothesis making a superficially plausible statistical argument that actually the trials suggest the drug is harmful to wide acclaim from laymen, or people casually referencing or even being administered the drug without reading the original trial writeups for themselves?

Rastonbury 4y ago

I read a lot of papers on behavioural economics and psychological decision making experiments for university, like dunning-kruger, kahneman, etc and in my opinion the first autocorrelation article reads like a rebuttal paper but more informal, the approach is scientific even if it may be flawed. This is how knowledge advances. I disagree that it is anti-science. Challenging accepted postulations is good. Even famous professors make mistakes, I don't blame the writer for making an honest mistake. That's how we got this new piece of writing Behavioural science is a pretty new field, its pretty easy to get abberant results or manipulate the results to show 'something' statistically. Many findings in earlier papers could not be replicated, or had applied statistics incorrectly, or showed different results when research participants were not white college kids. This is a whole other problem within academia, the pressure to publish something even when there is nothing and perceived legitimacy based on the number of citations a paper has. My professor always said don't look at the number of citations, understand the method and the rebuttal, there were numerous low citation but solid papers showing flaws in famous ones but everyone who isn't deep into the subject holds the original assertion to be legitimate because its "famous"

abirch 4y ago

> That said, I give the benefit of doubt to the author of "The DK Effect is Autocorrelation." It is a human error to be overly zealous in some opinions without thinking it through. If only there were a term for "a cognitive bias whereby people with limited knowledge or competence in a given intellectual or social domain greatly overestimate their own knowledge or competence in that domain relative to objective criteria or to the performance of their peers or of people in general"