6 ms·

Reversals in Psychology (2020)

56 points by apsec112 5y ago

newsbinator 5y ago
In summary: every single thing I learned in Psych 101 was "no good evidence"
tus89 5y ago
Lesson - just because people claim they are doing "science" doesn't mean they are doing either good or correct science. Subtle unchallenged assumptions, cause-and-effect confusion, inadequate root-cause analysis, invalid inferential logic and questionable proof-by-statistics. Maybe a better question - should psychology even be considered scientific or a science in the first place?
- undreren 5y ago
  Sloppy science is done in every field, natural sciences included. The article even leads with the statement that psychology experience more reversals due to being exceptionally open in terms of sharing code and data compared to other social sciences. Reversals through replication failure is scientific progress. Your “better” question is nothing of the sort, just lazy contrarianism.
  - RcouF1uZ4gsC 5y ago
    > Reversals through replication failure is scientific progress. Not really. If a flat earther comes and says that the earth is flat, and you take him up in a airplane and show him the curvature of the earth, he can’t say that flat earth study is science because “reversals through replication failure is scientific progress” There is a difference between making real scientific contributions through careful research, trial design, and statistical analysis, and spouting BS. After a bunch of high profile reversals, psychology is looking more like the spouting BS group that then justifies themselves by saying that clearing up BS is scientific advancement. If we want society to be able to actually use science to make decisions, then we have to be careful to differentiate science from BS.
    - derbOac 5y ago
      https://www.physicsforums.com/threads/shinichi-mochizukis-abc-conjecture-and-replication-crisis-in-maths.1002682/ https://www.physicsforums.com/threads/shinichi-mochizukis-ab...
  - mcswell 5y ago
    Two fields which claim some degree of scientific approach which are probably (IMO) worse than psychology: political science and anthropology. I of course exempt my own field, linguistics, which is the most scientific of all. (Yes, that is a joke.)
- BurningFrog 5y ago
  Even if people have impressive titles at institutions and landmark discoveries to their name, doesn't mean they are doing either good or correct science.
- BurningFrog 5y ago
  > should psychology even be considered scientific or a science in the first place? You can do good scientific research in that field. But it's much harder than in other fields.
  - glial 5y ago
    The inside joke is that psychology is not a hard science, but is the hardest science.
    - BurningFrog 5y ago
      Brilliant!
  - exporectomy 5y ago
    I wish psychology researchers would organize bigger experiments more. Instead of 1000 sample sizes of 20, why not have one sample size of 20000? Less personal glory, and more hard management work, no doubt, but the preponderance of small studies seems like a huge waste.
    - glial 5y ago
      Most psychology work is done in universities with limited funding. Hiring the staff to run a study with 20,000 people is prohibitively expensive for any lab. Some alternatives exist - for example, some studies can be run on Amazon Mechanical Turk. Tom Griffiths using this to good effect. (& I did the same for my dissertation)
- throwawaysea 5y ago
  Everything you say is true, probably for all fields. There is undeserved faith in peer review processes (https://www.vox.com/2015/12/7/9865086/peer-review-science-problems https://www.vox.com/2015/12/7/9865086/peer-review-science-pr...), inadequate understanding of statistics (or maybe purposefully poor application of statistics), poor incentive structures, and other issues that corrupt "science". This is why I always cringe a little when someone says "trust the science". Yet at the same time, I think psychology is especially susceptible to these problems, because it is a more fuzzy "social science" (more like sociology, less like physics). It suffers immense bias due to those who select themselves into the field and also how they study it. Psychology researchers typically conduct non-generalizable experiments with immense sampling bias (https://www.psychologytoday.com/us/blog/non-weird-science/202004/psychologys-weird-problem https://www.psychologytoday.com/us/blog/non-weird-science/20...). Then there are replication problems (https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/ https://www.theatlantic.com/science/archive/2018/11/psycholo...). They're also not making any progress in addressing these issues because of meta cultural problems about acknowledging problems in the field (https://www.wired.com/2016/03/psychology-crisis-whether-crisis/ https://www.wired.com/2016/03/psychology-crisis-whether-cris...).
- derbOac 5y ago
  Killing the messenger, really. This stuff has been documented in all sorts of fields, including pharmacology, neuroscience, oncology, you name it. I think there's been posts here on HN lately about replicability problems with AI research? Psychology is fuzzy because of the subject matter, but (appropriately I think) it's also better at turning the microscope on itself (meta-analysis really has its origins in psychology). Pretending this doesn't happen elsewhere is dangerous. Maybe the rates vary from field to field, but psychology isn't alone. If you applied that standard to every field there would be almost nothing left except maybe physics and some other closely related fields.
- deleted 5y ago
  [deleted]
- analog31 5y ago
  >>>> Maybe a better question - should psychology even be considered scientific or a science in the first place? Psychology has embraced scientific methodology. However, it has struggled to produce a reliable scientific knowledge base. There are some successes in psychology, that tempt one to apply the label "scientific," but they are in areas that might be controversial, such as advertising and manipulation.
  - tus89 5y ago
    > Psychology has embraced scientific methodology. I refer you to my statement about "unchallenged assumptions". Like the assumption that the reductive scientific process can be glibly applied to emergent complex phenomenon like psychology. As I said, people can say they are doing science, it neither means what they are doing is good or correct science.
    - coolgeek 5y ago
      Not sure where you're getting your definition of science from, but it doesn't correspond to those from authoritative sources. Here, for example, is Brittanica's: > Science, any system of knowledge that is concerned with the physical world and its phenomena and that entails unbiased observations and systematic experimentation. In general, a science involves a pursuit of knowledge covering general truths or the operations of fundamental laws. > Science can be divided into different branches based on the subject of study. The physical sciences study the inorganic world and comprise the fields of astronomy, physics, chemistry, and the Earth sciences. The biological sciences such as biology and medicine study the organic world of life and its processes. Social sciences like anthropology and economics study the social and cultural aspects of human behaviour. https://www.britannica.com/science/science https://www.britannica.com/science/science This article has been on the web since 1998. The Encyclopedia from which it is sourced has been publishing in print since 1768. If the entry on Science wasn't in the first edition, it probably wasn't too long before it first appeared. I encourage you to take your issue up with the editors and authors there. https://corporate.britannica.com/contact/ https://corporate.britannica.com/contact/ Do let us know how you make out!
ghostbrainalpha 5y ago
That there is not good evidence that screen time negatively effects well being was a surprise to me. https://www.ox.ac.uk/news/2019-01-15-technology-use-explains-most-04-adolescent-wellbeing https://www.ox.ac.uk/news/2019-01-15-technology-use-explains...
- JadeNB 5y ago
  > That there is not good evidence that screen time negatively effects well being was a surprise to me. That shouldn't be taken as evidence of the opposite, though (not that you said that!).
  - exporectomy 5y ago
    This word "screen time" seems to be very poorly defined. I think the harmfulness surely strongly depends on what you're doing with the screen. Reading a book? Writing a book? Scrolling through social media?
    - vorpalhex 5y ago
      There are (at least) three ways to consider "harmful" too. + Looking at screens is actually just hard on your eyes. You can make this better (eye relief screens, less brightness, make sure to look away from screens occasionally, blink...) + Spending a whole bunch of time staring at a screen may be a sign something else is going on - anxiety, depression, etc. The screen isn't the cause, it's just a form of escapism. + "Vegging" where you endlessly scroll memes or tiktoks or whatever the cool kids are doing these days. It's not quite relaxing, but it's not beneficial either. Especially with social media, it can actually be really anxiety causing.
    - mcswell 5y ago
      I'm pretty sure scrolling through social media raises my blood pressure.
stkdump 5y ago
Is there a similar list for results with particularly strong evidence? Studies that have been replicated multiple times?
- derbOac 5y ago
  It's an interesting question. There are many such studies, but the focus has been on the negative.
- exporectomy 5y ago
  Multiple replications are no guarantee of strong evidence. The first example (elderly priming) in the article shows that. It's the old file drawer problem.
vajrabum 5y ago
I notice that the page says that Daryl Bem's experiements on precognition have no good evidence. I poked around a bit and immediately found this paper on a metanalysis from 2016 on the topic. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4706048/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4706048/ Any thoughts on this? This paper says: To encourage replications, all materials needed to conduct them were made available on request. We here report a meta-analysis of 90 experiments from 33 laboratories in 14 countries which yielded an overall effect greater than 6 sigma, z = 6.40, p = 1.2 × 10 -10 with an effect size (Hedges’ g) of 0.09. A Bayesian analysis yielded a Bayes Factor of 5.1 × 10 9, greatly exceeding the criterion value of 100 for “decisive evidence” in support of the experimental hypothesis. When DJB’s original experiments are excluded, the combined effect size for replications by independent investigators is 0.06, z = 4.16, p = 1.1 × 10 -5, and the BF value is 3,853, again exceeding the criterion for “decisive evidence.” The number of potentially unretrieved experiments required to reduce the overall effect size of the complete database to a trivial value of 0.01 is 544, and seven of eight additional statistical tests support the conclusion that the database is not significantly compromised by either selection bias or by intense “ p-hacking”—the selective suppression of findings or analyses that failed to yield statistical significance. P-curve analysis, a recently introduced statistical technique, estimates the true effect size of the experiments to be 0.20 for the complete database and 0.24 for the independent replications, virtually identical to the effect size of DJB’s original experiments (0.22) and the closely related “presentiment” experiments (0.21). We discuss the controversial status of precognition and other anomalous effects collectively known as psi.
arsome 5y ago
Pretty sure I heard about 90% of these on NPR podcasts. I should probably stop listening to those.
- c3600608-467b 5y ago
  It's much worse than that: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0172650 https://journals.plos.org/plosone/article?id=10.1371/journal... Any science you hear about on the news is by selection bias more likely to be wrong than the science you never hear about. In short: news is destroying our trust in science by picking the worst examples of it for clicks.
NoImmatureAdHom 5y ago
The replication crisis in psychology has been on for a solid decade now. We need to stop treating poor statistical practice as negligence and start treating it as fraud.
- amelius 5y ago
  Hanlon's razor disagrees.
  - NoImmatureAdHom 5y ago
    "Never attribute to malice that which is adequately explained by stupidity" https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor I'm not arguing that these people are malicious, or that they aren't stupid. I'm arguing that we should start punishing them as if they were malicious. The default assumption should be: fraud. The fallback should be: incompetence to the point of negligence. But really we start out by treating them like frauds.
    - amelius 5y ago
      Maybe science should have a concept like "negative citations" which could help. But treating them like criminals seems a step too far.
      - NoImmatureAdHom 5y ago
        I like the idea of "negative citations"
- username90 5y ago
  Just add a statistician to review every single paper in every field for statistical error before publication. Every journal that doesn't do this should be deemed unscientific and lose all its status. Of course this wont happen since many "scientists" got to their current position by publishing non-results using bad statistical methods. And no, this wouldn't be too expensive at all. Is it really too expensive to check that our record of human knowledge is correct? It wouldn't cost more than a couple of hundred dollars per paper, basically nothing compared to writing the paper.
kar5pt 5y ago
The article says "No good evidence for multiple intelligences (in the sense of statistically independent components of cognition)." But shouldn't the burden of proof be the other way around? Why is it more reasonable to believe that ability in different cognitive activities can be abstracted into a single component called "intelligence" than it is to believe that performance in unrelated cognitive tasks would be unrelated?
- exporectomy 5y ago
  We already know that various cognitive abilities are correlated. It's been done. Even reaction times are correlated with IQ. Gardner's multiple intelligences are just made up. They didn't appear out of some analysis. He just imagined them and declared them to exist.
  - perl4ever 5y ago
    Assigning disproportionate importance to IQ is inversely correlated with other types of abilities, even if IQ itself is positively correlated. If you interview someone for a job and ask what they like to do in their free time, and their only answer is they belong to Mensa, do you expect they will have musical, mechanical, linguistic, athletic, etc. talent?
    - exporectomy 5y ago
      Ambiguous question. Everyone has those talents to some extent. Do you mean more than average? Exceptional (eg. top 1%)? More likely to have than if they had a lower IQ, or something else?
      - perl4ever 5y ago
        I think that's a bit overly pedantic, but spelled out: ...in the same proportion as a person randomly selected from the general population with the same IQ. My point is that the overall correlation is obviously not going to hold, among people who derive self-worth from IQ in particular. But those are the people who want to talk about the correlation. It's self-defeating.
rickspencer3 5y ago
Interesting. I have a Master's in Cognitive Psychology (from long ago, the 1990s), but it was in the domain Human Factors, so I view "cognitive psychology" in a very different light than these example. See some common examples like: https://en.wikipedia.org/wiki/Stroop_effect https://en.wikipedia.org/wiki/Stroop_effect https://en.wikipedia.org/wiki/Fitts%27s_law https://en.wikipedia.org/wiki/Fitts%27s_law Also things involving haptics, echo-location, alarm response, etc... What these things had in common, I think, is they treated the human mind as pretty much a black box, the dependent measures were not much subject to bias, and the experiments were easy to run on large samples of people. Such findings are pretty useful when designing things like nuclear power plants, airplane cockpits, shock trauma resuscitation bays, etc...
zug_zug 5y ago
This is completely unusable to me. The title says reversals yet the body doesn’t seem to contain reversals… “No strong evidence of X” is a lot different than “Strong evidence against X.” Any study of anything (true or not) will conclude “insufficient evidence” if the sample size is too small, or there is no control group. I think it’s the general consensus that anything in research, especially psychology is undecided until what’s called a “meta-study” is released. Also, Perhaps universities should have unaffiliated professors do each other’s experiments and analyze their data (all videotaped) to avoid the inherent conflict of interest in research.
machinehermitt 5y ago
My unbreakable first love intellectually is psychology. I think this is Civilization and its Discontents vs psych 101 though. It is not a scientific category of inquiry by itself though the way a theory of nodes does make sense outside of graph theory.

56 points by apsec112 5y ago

In summary: every single thing I learned in Psych 101 was "no good evidence"

Lesson - just because people claim they are doing "science" doesn't mean they are doing either good or correct science. Subtle unchallenged assumptions, cause-and-effect confusion, inadequate root-cause analysis, invalid inferential logic and questionable proof-by-statistics. Maybe a better question - should psychology even be considered scientific or a science in the first place?

undreren 5y ago
Sloppy science is done in every field, natural sciences included. The article even leads with the statement that psychology experience more reversals due to being exceptionally open in terms of sharing code and data compared to other social sciences. Reversals through replication failure is scientific progress. Your “better” question is nothing of the sort, just lazy contrarianism.
- RcouF1uZ4gsC 5y ago
  > Reversals through replication failure is scientific progress. Not really. If a flat earther comes and says that the earth is flat, and you take him up in a airplane and show him the curvature of the earth, he can’t say that flat earth study is science because “reversals through replication failure is scientific progress” There is a difference between making real scientific contributions through careful research, trial design, and statistical analysis, and spouting BS. After a bunch of high profile reversals, psychology is looking more like the spouting BS group that then justifies themselves by saying that clearing up BS is scientific advancement. If we want society to be able to actually use science to make decisions, then we have to be careful to differentiate science from BS.
  - derbOac 5y ago
    https://www.physicsforums.com/threads/shinichi-mochizukis-abc-conjecture-and-replication-crisis-in-maths.1002682/ https://www.physicsforums.com/threads/shinichi-mochizukis-ab...
- mcswell 5y ago
  Two fields which claim some degree of scientific approach which are probably (IMO) worse than psychology: political science and anthropology. I of course exempt my own field, linguistics, which is the most scientific of all. (Yes, that is a joke.)
BurningFrog 5y ago
Even if people have impressive titles at institutions and landmark discoveries to their name, doesn't mean they are doing either good or correct science.
BurningFrog 5y ago
> should psychology even be considered scientific or a science in the first place? You can do good scientific research in that field. But it's much harder than in other fields.
- glial 5y ago
  The inside joke is that psychology is not a hard science, but is the hardest science.
  - BurningFrog 5y ago
    Brilliant!
- exporectomy 5y ago
  I wish psychology researchers would organize bigger experiments more. Instead of 1000 sample sizes of 20, why not have one sample size of 20000? Less personal glory, and more hard management work, no doubt, but the preponderance of small studies seems like a huge waste.
  - glial 5y ago
    Most psychology work is done in universities with limited funding. Hiring the staff to run a study with 20,000 people is prohibitively expensive for any lab. Some alternatives exist - for example, some studies can be run on Amazon Mechanical Turk. Tom Griffiths using this to good effect. (& I did the same for my dissertation)
throwawaysea 5y ago
Everything you say is true, probably for all fields. There is undeserved faith in peer review processes (https://www.vox.com/2015/12/7/9865086/peer-review-science-problems https://www.vox.com/2015/12/7/9865086/peer-review-science-pr...), inadequate understanding of statistics (or maybe purposefully poor application of statistics), poor incentive structures, and other issues that corrupt "science". This is why I always cringe a little when someone says "trust the science". Yet at the same time, I think psychology is especially susceptible to these problems, because it is a more fuzzy "social science" (more like sociology, less like physics). It suffers immense bias due to those who select themselves into the field and also how they study it. Psychology researchers typically conduct non-generalizable experiments with immense sampling bias (https://www.psychologytoday.com/us/blog/non-weird-science/202004/psychologys-weird-problem https://www.psychologytoday.com/us/blog/non-weird-science/20...). Then there are replication problems (https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/ https://www.theatlantic.com/science/archive/2018/11/psycholo...). They're also not making any progress in addressing these issues because of meta cultural problems about acknowledging problems in the field (https://www.wired.com/2016/03/psychology-crisis-whether-crisis/ https://www.wired.com/2016/03/psychology-crisis-whether-cris...).
derbOac 5y ago
Killing the messenger, really. This stuff has been documented in all sorts of fields, including pharmacology, neuroscience, oncology, you name it. I think there's been posts here on HN lately about replicability problems with AI research? Psychology is fuzzy because of the subject matter, but (appropriately I think) it's also better at turning the microscope on itself (meta-analysis really has its origins in psychology). Pretending this doesn't happen elsewhere is dangerous. Maybe the rates vary from field to field, but psychology isn't alone. If you applied that standard to every field there would be almost nothing left except maybe physics and some other closely related fields.
deleted 5y ago
[deleted]
analog31 5y ago
>>>> Maybe a better question - should psychology even be considered scientific or a science in the first place? Psychology has embraced scientific methodology. However, it has struggled to produce a reliable scientific knowledge base. There are some successes in psychology, that tempt one to apply the label "scientific," but they are in areas that might be controversial, such as advertising and manipulation.
- tus89 5y ago
  > Psychology has embraced scientific methodology. I refer you to my statement about "unchallenged assumptions". Like the assumption that the reductive scientific process can be glibly applied to emergent complex phenomenon like psychology. As I said, people can say they are doing science, it neither means what they are doing is good or correct science.
  - coolgeek 5y ago
    Not sure where you're getting your definition of science from, but it doesn't correspond to those from authoritative sources. Here, for example, is Brittanica's: > Science, any system of knowledge that is concerned with the physical world and its phenomena and that entails unbiased observations and systematic experimentation. In general, a science involves a pursuit of knowledge covering general truths or the operations of fundamental laws. > Science can be divided into different branches based on the subject of study. The physical sciences study the inorganic world and comprise the fields of astronomy, physics, chemistry, and the Earth sciences. The biological sciences such as biology and medicine study the organic world of life and its processes. Social sciences like anthropology and economics study the social and cultural aspects of human behaviour. https://www.britannica.com/science/science https://www.britannica.com/science/science This article has been on the web since 1998. The Encyclopedia from which it is sourced has been publishing in print since 1768. If the entry on Science wasn't in the first edition, it probably wasn't too long before it first appeared. I encourage you to take your issue up with the editors and authors there. https://corporate.britannica.com/contact/ https://corporate.britannica.com/contact/ Do let us know how you make out!

ghostbrainalpha 5y ago

That there is not good evidence that screen time negatively effects well being was a surprise to me. https://www.ox.ac.uk/news/2019-01-15-technology-use-explains-most-04-adolescent-wellbeing https://www.ox.ac.uk/news/2019-01-15-technology-use-explains...

JadeNB 5y ago
> That there is not good evidence that screen time negatively effects well being was a surprise to me. That shouldn't be taken as evidence of the opposite, though (not that you said that!).
- exporectomy 5y ago
  This word "screen time" seems to be very poorly defined. I think the harmfulness surely strongly depends on what you're doing with the screen. Reading a book? Writing a book? Scrolling through social media?
  - vorpalhex 5y ago
    There are (at least) three ways to consider "harmful" too. + Looking at screens is actually just hard on your eyes. You can make this better (eye relief screens, less brightness, make sure to look away from screens occasionally, blink...) + Spending a whole bunch of time staring at a screen may be a sign something else is going on - anxiety, depression, etc. The screen isn't the cause, it's just a form of escapism. + "Vegging" where you endlessly scroll memes or tiktoks or whatever the cool kids are doing these days. It's not quite relaxing, but it's not beneficial either. Especially with social media, it can actually be really anxiety causing.
  - mcswell 5y ago
    I'm pretty sure scrolling through social media raises my blood pressure.

stkdump 5y ago

Is there a similar list for results with particularly strong evidence? Studies that have been replicated multiple times?

derbOac 5y ago
It's an interesting question. There are many such studies, but the focus has been on the negative.
exporectomy 5y ago
Multiple replications are no guarantee of strong evidence. The first example (elderly priming) in the article shows that. It's the old file drawer problem.

vajrabum 5y ago

I notice that the page says that Daryl Bem's experiements on precognition have no good evidence. I poked around a bit and immediately found this paper on a metanalysis from 2016 on the topic. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4706048/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4706048/ Any thoughts on this? This paper says: To encourage replications, all materials needed to conduct them were made available on request. We here report a meta-analysis of 90 experiments from 33 laboratories in 14 countries which yielded an overall effect greater than 6 sigma, z = 6.40, p = 1.2 × 10 -10 with an effect size (Hedges’ g) of 0.09. A Bayesian analysis yielded a Bayes Factor of 5.1 × 10 9, greatly exceeding the criterion value of 100 for “decisive evidence” in support of the experimental hypothesis. When DJB’s original experiments are excluded, the combined effect size for replications by independent investigators is 0.06, z = 4.16, p = 1.1 × 10 -5, and the BF value is 3,853, again exceeding the criterion for “decisive evidence.” The number of potentially unretrieved experiments required to reduce the overall effect size of the complete database to a trivial value of 0.01 is 544, and seven of eight additional statistical tests support the conclusion that the database is not significantly compromised by either selection bias or by intense “ p-hacking”—the selective suppression of findings or analyses that failed to yield statistical significance. P-curve analysis, a recently introduced statistical technique, estimates the true effect size of the experiments to be 0.20 for the complete database and 0.24 for the independent replications, virtually identical to the effect size of DJB’s original experiments (0.22) and the closely related “presentiment” experiments (0.21). We discuss the controversial status of precognition and other anomalous effects collectively known as psi.

arsome 5y ago

Pretty sure I heard about 90% of these on NPR podcasts. I should probably stop listening to those.

c3600608-467b 5y ago
It's much worse than that: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0172650 https://journals.plos.org/plosone/article?id=10.1371/journal... Any science you hear about on the news is by selection bias more likely to be wrong than the science you never hear about. In short: news is destroying our trust in science by picking the worst examples of it for clicks.

NoImmatureAdHom 5y ago

The replication crisis in psychology has been on for a solid decade now. We need to stop treating poor statistical practice as negligence and start treating it as fraud.

amelius 5y ago
Hanlon's razor disagrees.
- NoImmatureAdHom 5y ago
  "Never attribute to malice that which is adequately explained by stupidity" https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor I'm not arguing that these people are malicious, or that they aren't stupid. I'm arguing that we should start punishing them as if they were malicious. The default assumption should be: fraud. The fallback should be: incompetence to the point of negligence. But really we start out by treating them like frauds.
  - amelius 5y ago
    Maybe science should have a concept like "negative citations" which could help. But treating them like criminals seems a step too far.
    - NoImmatureAdHom 5y ago
      I like the idea of "negative citations"
username90 5y ago
Just add a statistician to review every single paper in every field for statistical error before publication. Every journal that doesn't do this should be deemed unscientific and lose all its status. Of course this wont happen since many "scientists" got to their current position by publishing non-results using bad statistical methods. And no, this wouldn't be too expensive at all. Is it really too expensive to check that our record of human knowledge is correct? It wouldn't cost more than a couple of hundred dollars per paper, basically nothing compared to writing the paper.

kar5pt 5y ago

The article says "No good evidence for multiple intelligences (in the sense of statistically independent components of cognition)." But shouldn't the burden of proof be the other way around? Why is it more reasonable to believe that ability in different cognitive activities can be abstracted into a single component called "intelligence" than it is to believe that performance in unrelated cognitive tasks would be unrelated?

exporectomy 5y ago
We already know that various cognitive abilities are correlated. It's been done. Even reaction times are correlated with IQ. Gardner's multiple intelligences are just made up. They didn't appear out of some analysis. He just imagined them and declared them to exist.
- perl4ever 5y ago
  Assigning disproportionate importance to IQ is inversely correlated with other types of abilities, even if IQ itself is positively correlated. If you interview someone for a job and ask what they like to do in their free time, and their only answer is they belong to Mensa, do you expect they will have musical, mechanical, linguistic, athletic, etc. talent?
  - exporectomy 5y ago
    Ambiguous question. Everyone has those talents to some extent. Do you mean more than average? Exceptional (eg. top 1%)? More likely to have than if they had a lower IQ, or something else?
    - perl4ever 5y ago
      I think that's a bit overly pedantic, but spelled out: ...in the same proportion as a person randomly selected from the general population with the same IQ. My point is that the overall correlation is obviously not going to hold, among people who derive self-worth from IQ in particular. But those are the people who want to talk about the correlation. It's self-defeating.

rickspencer3 5y ago

Interesting. I have a Master's in Cognitive Psychology (from long ago, the 1990s), but it was in the domain Human Factors, so I view "cognitive psychology" in a very different light than these example. See some common examples like: https://en.wikipedia.org/wiki/Stroop_effect https://en.wikipedia.org/wiki/Stroop_effect https://en.wikipedia.org/wiki/Fitts%27s_law https://en.wikipedia.org/wiki/Fitts%27s_law Also things involving haptics, echo-location, alarm response, etc... What these things had in common, I think, is they treated the human mind as pretty much a black box, the dependent measures were not much subject to bias, and the experiments were easy to run on large samples of people. Such findings are pretty useful when designing things like nuclear power plants, airplane cockpits, shock trauma resuscitation bays, etc...

zug_zug 5y ago

This is completely unusable to me. The title says reversals yet the body doesn’t seem to contain reversals… “No strong evidence of X” is a lot different than “Strong evidence against X.” Any study of anything (true or not) will conclude “insufficient evidence” if the sample size is too small, or there is no control group. I think it’s the general consensus that anything in research, especially psychology is undecided until what’s called a “meta-study” is released. Also, Perhaps universities should have unaffiliated professors do each other’s experiments and analyze their data (all videotaped) to avoid the inherent conflict of interest in research.

machinehermitt 5y ago

My unbreakable first love intellectually is psychology. I think this is Civilization and its Discontents vs psych 101 though. It is not a scientific category of inquiry by itself though the way a theory of nodes does make sense outside of graph theory.