3 ms·

The original paper was proven to be more alarmist than true. Dan Gilbert's team put out an excellent analysis of the paper and found many poorly performed repli

The original paper was proven to be more alarmist than true. Dan Gilbert's team put out an excellent analysis of the paper and found many poorly performed replications. That being said, it sheds light on the greater problem of p-value hacking that may exaggerate any kind of statistical result

Lawtonfogle 11y ago
If it is the article I remember, one example was showing that the study could be replicated if done at the same university (which would be years after the first study), but the criticism was that it couldn't be replicated at a different university. So the criticism of the criticism seems to be quite overstated. If the study can only be replicated at a single university (or even all in a general area), it means that we need to be very careful in generalizing the results. So to the extent that people take a study of one very specific group and generalize to an entire culture (or worse, to all humans), the criticism sticks. Back when I was getting my psychology degree, there was a joke that 90% of the findings in psychology can only be generalized to broke American college students who are willing to be lab rats for beer money. While that overstates the problem, there is still a lot of truth in it.
- machinelearning 11y ago
  Obviously it's not possible to pick a sample size large enough to eliminate all bias. However it's important to pick a sample that doesn't inherently bias the effect that is being measured. This has to apply to both the original and the replication. The response to the paper points out few examples where the replication was not judicious in picking a sample that minimized its influence on the measured effect. Even besides that, the original study had sampling and scoring procedures which contradicted its original aims. Lastly, sensationalist headlines of "more than half of all papers are not replicable" are blatantly wrong and not what the original paper even states. As to your point, I agree that only being able to replicate in one sample type is grounds for skepticism, but more replications have to be done to ensure the replication sample is not the one that is flawed. One can't really draw conclusions unless the replication is repeated a large number of times across a wider sample space. Link to response paper in case anyone's interested: http://projects.iq.harvard.edu/files/psychology-replications/files/gilbert_king_pettigrew_wilson_2016_with_appendix.pdf?m=1456973260 http://projects.iq.harvard.edu/files/psychology-replications...
  - Lawtonfogle 11y ago
    >However it's important to pick a sample that doesn't inherently bias the effect that is being measured. This has to apply to both the original and the replication. All this shows is that psychology is less useful that people thing because any data is likely only applicable to the given group. Even if we say the studies are still good, this means that psychology loses the ability to be generalized to others groups beyond the one studied. And given how often psychology has that done to it, we must question every finding all the more.

Lawtonfogle 11y ago

If it is the article I remember, one example was showing that the study could be replicated if done at the same university (which would be years after the first study), but the criticism was that it couldn't be replicated at a different university. So the criticism of the criticism seems to be quite overstated. If the study can only be replicated at a single university (or even all in a general area), it means that we need to be very careful in generalizing the results. So to the extent that people take a study of one very specific group and generalize to an entire culture (or worse, to all humans), the criticism sticks. Back when I was getting my psychology degree, there was a joke that 90% of the findings in psychology can only be generalized to broke American college students who are willing to be lab rats for beer money. While that overstates the problem, there is still a lot of truth in it.

machinelearning 11y ago
Obviously it's not possible to pick a sample size large enough to eliminate all bias. However it's important to pick a sample that doesn't inherently bias the effect that is being measured. This has to apply to both the original and the replication. The response to the paper points out few examples where the replication was not judicious in picking a sample that minimized its influence on the measured effect. Even besides that, the original study had sampling and scoring procedures which contradicted its original aims. Lastly, sensationalist headlines of "more than half of all papers are not replicable" are blatantly wrong and not what the original paper even states. As to your point, I agree that only being able to replicate in one sample type is grounds for skepticism, but more replications have to be done to ensure the replication sample is not the one that is flawed. One can't really draw conclusions unless the replication is repeated a large number of times across a wider sample space. Link to response paper in case anyone's interested: http://projects.iq.harvard.edu/files/psychology-replications/files/gilbert_king_pettigrew_wilson_2016_with_appendix.pdf?m=1456973260 http://projects.iq.harvard.edu/files/psychology-replications...
- Lawtonfogle 11y ago
  >However it's important to pick a sample that doesn't inherently bias the effect that is being measured. This has to apply to both the original and the replication. All this shows is that psychology is less useful that people thing because any data is likely only applicable to the given group. Even if we say the studies are still good, this means that psychology loses the ability to be generalized to others groups beyond the one studied. And given how often psychology has that done to it, we must question every finding all the more.