5 ms·

So, a slightly narrower scope than the usual psychology study.

jermaustin1 8y ago
This actually seems more comprehensive than most. They tested multiple times! There is a pretty decent episode of Freakonomics Radio around how reproducible (or not) many psychological studies are. I know this is an issue for most studies, but Psychology seems to be in a league of their own with the "reproduction rate".
- ekianjo 8y ago
  > how reproducible (or not) many psychological studies are. Combine small effect size with a complete lack of understanding of statistics and you can be fairly sure that most psychological studies' conclusions are borderline junk level.
  - misja111 8y ago
    Combine that with the usual publication bias and you can be sure that more than 50% of the studies are wrong. https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248 https://www.nature.com/news/over-half-of-psychology-studies-...
  - superhuzza 8y ago
    It has little to do with statistical understanding. All psych students are required to study stats, and review the study design before starting. It has everything to with a more systemic problem - publish or perish. Students and professors alike are basically forced to run new studies. Conducting psych studies takes a lot of money and time because you're dealing with people. And that money and time isn't always available - so your standards drop a lot so you can just graduate or keep researching.
    - IMTDb 8y ago
      This. And so far my experience is that most (but definitely not all) psych student are usually choosing psych over STEM due to their repulsion of math, physics and 'anything number'. They see stats as an 'unfortunate exam they need to get through to pass' so they aim for a barley passing grade in order to focus on other areas they are more interested in. When they are pressured into publishing something the stats part is once again seen as "the impossibly complex part they need help with to be able to publish their interesting findings". They have guidelines such as "p must be <= 0.05" so they try to mess with the data until they reach that point. Usually, most of the paper (including the conclusion) is already written so stats are considered to be the last administrative bruden to get trough to be able to publish.
      - mygo 8y ago
        they already have their conclusion before doing the maths?
      - mattnewton 8y ago
        I think this is an unfair stereotype. Psychology is really hard, the system you are studying is many times more complex than many other fields. I am going to have to ask for the source on that last paragraph too, that’s basically slandering the whole field as not doing actual science.
        smallnamespace 8y ago
        > Psychology is really hard, the system you are studying is many times more complex than many other fields. If that's true though, then the mathematical rigor that should be demanded from its students must be higher than in other fields in order to achieve the same level of reliability. I doubt that is the case today.
        mattnewton 8y ago
        The gp is basically saying psychologists can’t get the math right because most don’t like math and are just playing house to get publications. I am saying many do appreciate good statistics and go to bed worrying about it at night, and the lack of general reproducibility has more to do with the complexity of the field. In most cases you have to find and convince a statistically representative sample to participate, and that’s so expensive and hard enough to often not be done completely, without having to consider that a statistically representative sample of individuals for many psychological phenomena is an impossible task.
Bartweiss 8y ago
At least they ran multiple trials that all have the same problem, instead of just one? More seriously, the results correlating happiness and success with alphabetical order of your name have all failed to replicate. I'm going to make a prediction now that I expect this will also fail replication if anyone ever bothers.
richjdsmith 8y ago
If more than half their sample wasn't just psych students they'd be coming out on top!

This actually seems more comprehensive than most. They tested multiple times! There is a pretty decent episode of Freakonomics Radio around how reproducible (or not) many psychological studies are. I know this is an issue for most studies, but Psychology seems to be in a league of their own with the "reproduction rate".

ekianjo 8y ago
> how reproducible (or not) many psychological studies are. Combine small effect size with a complete lack of understanding of statistics and you can be fairly sure that most psychological studies' conclusions are borderline junk level.
- misja111 8y ago
  Combine that with the usual publication bias and you can be sure that more than 50% of the studies are wrong. https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248 https://www.nature.com/news/over-half-of-psychology-studies-...
- superhuzza 8y ago
  It has little to do with statistical understanding. All psych students are required to study stats, and review the study design before starting. It has everything to with a more systemic problem - publish or perish. Students and professors alike are basically forced to run new studies. Conducting psych studies takes a lot of money and time because you're dealing with people. And that money and time isn't always available - so your standards drop a lot so you can just graduate or keep researching.
  - IMTDb 8y ago
    This. And so far my experience is that most (but definitely not all) psych student are usually choosing psych over STEM due to their repulsion of math, physics and 'anything number'. They see stats as an 'unfortunate exam they need to get through to pass' so they aim for a barley passing grade in order to focus on other areas they are more interested in. When they are pressured into publishing something the stats part is once again seen as "the impossibly complex part they need help with to be able to publish their interesting findings". They have guidelines such as "p must be <= 0.05" so they try to mess with the data until they reach that point. Usually, most of the paper (including the conclusion) is already written so stats are considered to be the last administrative bruden to get trough to be able to publish.
    - mygo 8y ago
      they already have their conclusion before doing the maths?
    - mattnewton 8y ago
      I think this is an unfair stereotype. Psychology is really hard, the system you are studying is many times more complex than many other fields. I am going to have to ask for the source on that last paragraph too, that’s basically slandering the whole field as not doing actual science.
      - smallnamespace 8y ago
        > Psychology is really hard, the system you are studying is many times more complex than many other fields. If that's true though, then the mathematical rigor that should be demanded from its students must be higher than in other fields in order to achieve the same level of reliability. I doubt that is the case today.
        mattnewton 8y ago
        The gp is basically saying psychologists can’t get the math right because most don’t like math and are just playing house to get publications. I am saying many do appreciate good statistics and go to bed worrying about it at night, and the lack of general reproducibility has more to do with the complexity of the field. In most cases you have to find and convince a statistically representative sample to participate, and that’s so expensive and hard enough to often not be done completely, without having to consider that a statistically representative sample of individuals for many psychological phenomena is an impossible task.

Bartweiss 8y ago

At least they ran multiple trials that all have the same problem, instead of just one? More seriously, the results correlating happiness and success with alphabetical order of your name have all failed to replicate. I'm going to make a prediction now that I expect this will also fail replication if anyone ever bothers.

richjdsmith 8y ago

If more than half their sample wasn't just psych students they'd be coming out on top!