4 ms·

Or it wasn't, e.g.: http://neuroneurotic.net/2016/03/03/the-non-replicability-of-soft-science/ http://neuroneurotic.net/2016/03/03/the-non-replicability-of...

by Certhas 11y ago

Or it wasn't, e.g.: http://neuroneurotic.net/2016/03/03/the-non-replicability-of-soft-science/ http://neuroneurotic.net/2016/03/03/the-non-replicability-of... From where I’m standing, social and other forms of traditional psychology can’t say the same. Small contextual or methodological differences can quite likely skew the results because the mind is a damn complex thing. For that reason alone, we should expect psychology to have low replicability and the effect sizes should be pretty small (i.e. smaller than what is common in the literature) because they will always be diluted by a multitude of independent factors. Perhaps more than any other field, psychology can benefit from preregistering experimental protocols to delineate the exploratory garden-path from hypothesis-driven confirmatory results. I agree that a direct replication of a contextually dependent effect in a different country and at a different time makes little sense but that is no excuse. If you just say that the effects are so context-specific it is difficult to replicate them, you are bound to end up chasing lots of phantoms. And that isn’t science – not even a “soft” one.

DanBC 11y ago
It's interesting to compare that link talking about how it's difficult to replicate psychology (and other) experiments, with this link that says it doesn't matter. https://web.archive.org/web/*/http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm https://web.archive.org/web/*/http://wjh.harvard.edu/~jmitch... (I agree much more with your link).
nmrm2 11y ago
Interestingly, variations on "the exploratory garden-path" are widely hailed by qualitative social science researchers as sound methodology.
vacri 11y ago
The article explicitly says that the 39% reproducibility isn't that far off what you would probably expect anyway. The article then goes on to add the the meta-study was awful in it's study selection, giving the example: OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam. The article then goes on to say that reproducibility is still a problem in science, but that the meta-study was simply terrible in it's own methodology. Low reproducability is less of an issue given the context of the problem space, the article says; the meta-study was wrong, both methodologically and in conclusion ('low reproducability' != 'unethical' or 'crisis'). I recommend reading the fine article.
- logicrook 11y ago
  >OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam. If they do conclude things on "students" based on this study, then reproducing it on Dutch students is perfectly fine. If they did conclude something on "Stanford University students", then it is a bad reproduction. You see, there's a very good justification there, that you can interpret as subtyping, or the liskov substitution principle. >'low reproducability' != 'unethical' This is akin to saying "ok there's this theorem, but I won't give proof, just a bogus heuristic argument", or even worse, saying "that person is guilty but I won't justify why". Both these behaviors are unacceptable. Why would "social sciences" be above that? > or 'crisis' A field is filled of politically oriented baseless claims, but this is not a crisis, just another day in Oceania. That said, if the meta-study was sloppy, then it still justifies that research standards should have been upped a while ago; the "terrible"" meta-study was accepted precisely because terrible papers are accepted in the first place.
  - vacri 11y ago
    > If they do conclude things on "students" based on this study, then reproducing it on Dutch students is perfectly fine. No. Just fucking 'no'. It was a study on affirmative action policies, repeated in a distant country with different public opinions on welfare and social fairness. If you think that that's fine, then you really have no claim to be declaring what psychology should and shouldn't be. > filled of politically oriented baseless claims Fuck me, but it's funny when detractors claim that psychology is full of unscientific behaviour, while they're way overgeneralising the entire field on a subsection of it.
    - mystikal 11y ago
      His point was about whether the term "students" is qualified.
      - vacri 11y ago
        If that's the reason for including the study, then it was a bad study to choose to replicate, as it would be clearly not replicable for students of other cultures - in which case, the meta-study researchers are stacking the deck; they would be guilty of the very thing they're decrying.
koentj 11y ago
There are many things wrong with the analyses performed by Gilbert et al. Sanjay Srivastava does a decent job at pointing them out: https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-critique-of-the-reproducibility-project/ https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-cr...

by Certhas 11y ago

DanBC 11y ago

It's interesting to compare that link talking about how it's difficult to replicate psychology (and other) experiments, with this link that says it doesn't matter. https://web.archive.org/web/*/http://wjh.harvard.edu/~jmitchel/writing/failed_science.htm https://web.archive.org/web/*/http://wjh.harvard.edu/~jmitch... (I agree much more with your link).

nmrm2 11y ago

Interestingly, variations on "the exploratory garden-path" are widely hailed by qualitative social science researchers as sound methodology.

vacri 11y ago

The article explicitly says that the 39% reproducibility isn't that far off what you would probably expect anyway. The article then goes on to add the the meta-study was awful in it's study selection, giving the example: OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam. The article then goes on to say that reproducibility is still a problem in science, but that the meta-study was simply terrible in it's own methodology. Low reproducability is less of an issue given the context of the problem space, the article says; the meta-study was wrong, both methodologically and in conclusion ('low reproducability' != 'unethical' or 'crisis'). I recommend reading the fine article.

logicrook 11y ago
>OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam. If they do conclude things on "students" based on this study, then reproducing it on Dutch students is perfectly fine. If they did conclude something on "Stanford University students", then it is a bad reproduction. You see, there's a very good justification there, that you can interpret as subtyping, or the liskov substitution principle. >'low reproducability' != 'unethical' This is akin to saying "ok there's this theorem, but I won't give proof, just a bogus heuristic argument", or even worse, saying "that person is guilty but I won't justify why". Both these behaviors are unacceptable. Why would "social sciences" be above that? > or 'crisis' A field is filled of politically oriented baseless claims, but this is not a crisis, just another day in Oceania. That said, if the meta-study was sloppy, then it still justifies that research standards should have been upped a while ago; the "terrible"" meta-study was accepted precisely because terrible papers are accepted in the first place.
- vacri 11y ago
  > If they do conclude things on "students" based on this study, then reproducing it on Dutch students is perfectly fine. No. Just fucking 'no'. It was a study on affirmative action policies, repeated in a distant country with different public opinions on welfare and social fairness. If you think that that's fine, then you really have no claim to be declaring what psychology should and shouldn't be. > filled of politically oriented baseless claims Fuck me, but it's funny when detractors claim that psychology is full of unscientific behaviour, while they're way overgeneralising the entire field on a subsection of it.
  - mystikal 11y ago
    His point was about whether the term "students" is qualified.
    - vacri 11y ago
      If that's the reason for including the study, then it was a bad study to choose to replicate, as it would be clearly not replicable for students of other cultures - in which case, the meta-study researchers are stacking the deck; they would be guilty of the very thing they're decrying.

koentj 11y ago

There are many things wrong with the analyses performed by Gilbert et al. Sanjay Srivastava does a decent job at pointing them out: https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-critique-of-the-reproducibility-project/ https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-cr...