6 ms·

> if you read the reports carefully, as we did, you discover that many of the replication studies differed in truly astounding ways—ways that make it hard to un

> if you read the reports carefully, as we did, you discover that many of the replication studies differed in truly astounding ways—ways that make it hard to understand how they could even be called replications.” > As an example, Gilbert described an original study that involved showing White students at Stanford University a video of four other Stanford students discussing admissions policies at their university. Three of the discussants were White and one was Black. During the discussion, one of the White students made offensive comments about affirmative action, and the researchers found that the observers looked significantly longer at the Black student when they believed he could hear the others’ comments than when he could not. > So how did they do the replication? With students at the University of Amsterdam!” Gilbert said. “They had Dutch students watch a video of Stanford students, speaking in English, about affirmative action policies at a university more than 5000 miles away.” In other words, unlike the participants in the original study, participants in the replication study watched students at a foreign university speaking in a foreign language about an issue of no relevance to them! I’ve long said it’s much more useful to think of the replication crisis as a generalization crisis, and this is a good case in point. The replication (and failure to replicate in Amsterdam) doesn’t tell us the original study was wrong, but tells us that something in the context was actually part of the subject all along. In this case US/Dutch differences in current social hot topics, or in historical slavery, or maybe it only replicates for students who you believe you might run into in class. In any case, per usual, the solution is more science and more thoughtful study, not throwing in the towel and declaring the sciences conquered by “the replication crisis.”

uniqueuid 4y ago
I agree that a nuanced approach is the right way. But many disciplines, in this case psychology, did have the replication crisis coming. Cohen and Gigerenzer have warned about power for literally decades but their warnings were not heeded by a significant part of the field. So a stark message may in fact have been necessary. By the way, Daniel Kahneman basically wrote a "this is a problem, deal with it NOW" letter to some colleagues in 2012 [1]. Very telling and an insightful read. [1] http://www.decisionsciencenews.com/?p=3634 http://www.decisionsciencenews.com/?p=3634
- jvanderbot 4y ago
  Here's what he suggested: * Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants. * Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate. * The board makes a public commitment to these five specific effects * Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc. * Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed). * Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work * Run enough subjects to guarantee power (probably more than in the original study) * Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others. * Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others. I hope that psychology is a field where it is possible to fund such a protocol from grants or from internal school funds. However, that protocol seems to me to be laborious, expensive, and not at all "easy to get".
- wolverine876 4y ago
  > But many disciplines, in this case psychology, did have the replication crisis coming. The point of the OP is that there isn't a replication crisis. Should we be assuming, for purposes of this discussion, that there is one?
  - dash2 4y ago
    I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. The point of the post is that the size of the replication crisis may have been exaggerated in this study.
    - wolverine876 4y ago
radarsat1 4y ago
> So how did they do the replication? With students at the University of Amsterdam!” Gilbert said. “They had Dutch students watch a video of Stanford students, speaking in English, about affirmative action policies at a university more than 5000 miles away.” In other words, unlike the participants in the original study, participants in the replication study watched students at a foreign university speaking in a foreign language about an issue of no relevance to them! I'm sorry what? Does the author think that there are no black people in Amsterdam? Or no affirmative action? Or that Dutch students don't understand English? Race relations aren't the same in the US as in the Netherlands, but the "no relevance to them" comment is going a little far imho.
- bigDinosaur 4y ago
  I wouldn't really say it's relevant to watch a video from a specific university if you're not from that university. I don't think they're saying that 'affirmative action' is not a relevant topic in some way, only that 'affirmative action at Stanford' specifically is not relevant to a Dutch student.
- jandrese 4y ago
  I think the idea is that the Dutch students wouldn’t have the same underlying racism that the American students acquire in their lifetime due to the lingering effects of slavery. The whole point of the original study presumably being to study unconscious bias due to cultural conditioning. A better study might have tried to replicate it using Dutch students in the video and a distinctly Romani student as the object of discrimination.
  - ethanbond 4y ago
    I think the idea is that all of the above hypotheses are variables that indeed might be relevant, and we don’t know which they are.
  - radarsat1 4y ago
    > I think the idea is that the Dutch students wouldn’t have the same underlying racism that the American students acquire in their lifetime due to the lingering effects of slavery. I mean... the Dutch had slaves. I'm sorry, I don't know how else to spell this out. The Dutch were slavers, just like the English. There are generations of black people living in the Netherlands for the same reason they are in the United States. Many of the slaves that came to America were brought on Dutch ships. They did outlaw it earlier, I think. But there are tons of racial issues in NL. I know I'm getting downvoted but I find this attitude that racism is somehow unique to the US to be pretty narrow to be honest.
fartcannon 4y ago
This kind of makes it sound like the original studies can't be repeated, and that they're so hyper specific as to be effectively worthless.
- ethanbond 4y ago
  It makes it sound like the world is full of variables, many of which are not immediately obvious, and this is why science is extremely hard.
  - fartcannon 4y ago
    Yeah, and if you can't make predictions off the results, it's not science, its history.
    - ethanbond 4y ago
      Err, not necessarily. It could be just unfinished science.
      - fartcannon 4y ago
        That's true, but so could literally anything, if the basis is something like what was written in the original post. Obviously it's up to the investors if they want to fund this stuff, but it sure does look like pretty bad science.
        ethanbond 4y ago
        All science starts bad and the vast majority of it stays bad. Nature of the game.
        fartcannon 4y ago
        Yes, like what's mentioned in the original post.
epistasis 4y ago
> I’ve long said it’s much more useful to think of the replication crisis as a generalization crisis, I haven't bothered to follow the psychology discussion of this, but I did follow the cancer research attempt to claim a replication crisis. The article that most people cite is an editorial where pharma researchers try to "replicate" studies by taking "landmark" studies (that were not revealed), and then do drug development off the findings of these initial papers. Now, sure, if cancer were so simple that surprising new cell behaviors could be used to develop drugs that target lots of different cancers, that would be fantastic. But that sort of generalization should never be expected, it's just silly. If somebody finds a new cellular behavior in a handful of cell lines or animal models or tumor biopsies, there's no reason to think it will generalize. The point of the primary literature is to document the primary evidence. It's not a textbook, with highly generalized scientific principles, it's the messy discovery phase of science where scientists are communicating findings that they can build upon to eventually build a body of work that could become a generalized principle. A single paper should never constitute the basis of an entire drug development program. I think that a lot of the "replication" crisis is just a fundamental misunderstanding of how scientific literature works and how it's used by the intended audience.
snapplebobapple 4y ago
Out of curiosity, have you looked at the original paper and the replication paper? I would think whether the study in Amsterdam shows the original isn't replicatable or not would really depend on how generalized the original paper chose to make the claims in its conclusion. If the original paper concluded their result was generally applicable then the Amsterdam study is, in fact, a valid replication. If it claims to be applicable to only some subset of the population then this is a valid criticism (but if the applicability constraint is too narrow, i.e just that university, then another valid criticism is why did the first paper even warrant publishing??) Makes me wonder if this paper will be overturned in itself in a couple years as the academic machinery turns.
- ethanbond 4y ago
  I haven’t, but really even if it does make an excessively bold claim, it’s definitely a replication failure but I’m still of the belief that it’s not useful to frame that as “replication crisis.” Newton’s failure to account for near-light speed movement - and the rest of physics knowledge - doesn’t really warrant “replication crisis.” It’s just “more science” that’s refining our understanding. My issue with replication crisis framing is that in my experience it leads people to believe the softer sciences are irredeemable. Not only is the “crisis” not limited to softer sciences, but it’s really not a crisis. As we go up the entropic chain (physics -> chem -> bio -> socio etc), we can expect that understanding all of the relevant variables will get harder and harder and harder. It doesn’t mean we shouldn’t continue trying, and it doesn’t point to another route to knowledge other than continuing to try.
  - snapplebobapple 4y ago
    The difference would be that with Newton his results applied to pretty much all observable situations and only when our observation ability improved did we discover problems where his work wouldn't replicate and we had to tighten the specification of what his work applied to. In this regard you could also call the problem in the social sciences a specification crisis because their claims don't hold up over the wide group they attempt to apply them to in the results discussion (I would argue far too often they don't apply beyond the x people in the study, but that's open for debate). I agree, we shouldn't stop trying. We do, however, need to be aware of the things we are doing to be utter failures and correct them, otherwise nothing will come from trying.

uniqueuid 4y ago

I agree that a nuanced approach is the right way. But many disciplines, in this case psychology, did have the replication crisis coming. Cohen and Gigerenzer have warned about power for literally decades but their warnings were not heeded by a significant part of the field. So a stark message may in fact have been necessary. By the way, Daniel Kahneman basically wrote a "this is a problem, deal with it NOW" letter to some colleagues in 2012 [1]. Very telling and an insightful read. [1] http://www.decisionsciencenews.com/?p=3634 http://www.decisionsciencenews.com/?p=3634

jvanderbot 4y ago
Here's what he suggested: * Assemble a group of five labs, where the leading investigators have an established reputation (tenure should perhaps be a requirement). Substantial labs with several students are the most desirable participants. * Each lab selects a recent demonstration of a priming effect, which they consider robust and most likely to replicate. * The board makes a public commitment to these five specific effects * Set up a daisy chain of labs A-B-C-D-E-A, where each lab will replicate the study selected by its neighbor: B replicates A, C replicates B etc. * Have the replicating lab send someone to see how subjects are run (hence the emphasis on recency – the experiments should be in the active repertoire of the original lab, so that additional subjects can be run with confidence that the same procedure is followed). * Have the replicated lab send someone to vet the procedure of the replicating lab as it starts its work * Run enough subjects to guarantee power (probably more than in the original study) * Use technology (e.g. video) to ensure that every detail of the method is documented and can be copied by others. * Pre-commit to publish the results, letting the chips fall where they may, and make all data available for analysis by others. I hope that psychology is a field where it is possible to fund such a protocol from grants or from internal school funds. However, that protocol seems to me to be laborious, expensive, and not at all "easy to get".
wolverine876 4y ago
> But many disciplines, in this case psychology, did have the replication crisis coming. The point of the OP is that there isn't a replication crisis. Should we be assuming, for purposes of this discussion, that there is one?
- dash2 4y ago
  I think that's an exaggeration. The replication crisis wasn't just this one paper. There were also (e.g.) Daniel Bem's studies on ESP and a whole bunch of individual failed replications. The point of the post is that the size of the replication crisis may have been exaggerated in this study.
  - wolverine876 4y ago

radarsat1 4y ago

> So how did they do the replication? With students at the University of Amsterdam!” Gilbert said. “They had Dutch students watch a video of Stanford students, speaking in English, about affirmative action policies at a university more than 5000 miles away.” In other words, unlike the participants in the original study, participants in the replication study watched students at a foreign university speaking in a foreign language about an issue of no relevance to them! I'm sorry what? Does the author think that there are no black people in Amsterdam? Or no affirmative action? Or that Dutch students don't understand English? Race relations aren't the same in the US as in the Netherlands, but the "no relevance to them" comment is going a little far imho.

bigDinosaur 4y ago
I wouldn't really say it's relevant to watch a video from a specific university if you're not from that university. I don't think they're saying that 'affirmative action' is not a relevant topic in some way, only that 'affirmative action at Stanford' specifically is not relevant to a Dutch student.
jandrese 4y ago
I think the idea is that the Dutch students wouldn’t have the same underlying racism that the American students acquire in their lifetime due to the lingering effects of slavery. The whole point of the original study presumably being to study unconscious bias due to cultural conditioning. A better study might have tried to replicate it using Dutch students in the video and a distinctly Romani student as the object of discrimination.
- ethanbond 4y ago
  I think the idea is that all of the above hypotheses are variables that indeed might be relevant, and we don’t know which they are.
- radarsat1 4y ago
  > I think the idea is that the Dutch students wouldn’t have the same underlying racism that the American students acquire in their lifetime due to the lingering effects of slavery. I mean... the Dutch had slaves. I'm sorry, I don't know how else to spell this out. The Dutch were slavers, just like the English. There are generations of black people living in the Netherlands for the same reason they are in the United States. Many of the slaves that came to America were brought on Dutch ships. They did outlaw it earlier, I think. But there are tons of racial issues in NL. I know I'm getting downvoted but I find this attitude that racism is somehow unique to the US to be pretty narrow to be honest.

fartcannon 4y ago

This kind of makes it sound like the original studies can't be repeated, and that they're so hyper specific as to be effectively worthless.

ethanbond 4y ago
It makes it sound like the world is full of variables, many of which are not immediately obvious, and this is why science is extremely hard.
- fartcannon 4y ago
  Yeah, and if you can't make predictions off the results, it's not science, its history.
  - ethanbond 4y ago
    Err, not necessarily. It could be just unfinished science.
    - fartcannon 4y ago
      That's true, but so could literally anything, if the basis is something like what was written in the original post. Obviously it's up to the investors if they want to fund this stuff, but it sure does look like pretty bad science.
      - ethanbond 4y ago
        All science starts bad and the vast majority of it stays bad. Nature of the game.
        fartcannon 4y ago
        Yes, like what's mentioned in the original post.

epistasis 4y ago

> I’ve long said it’s much more useful to think of the replication crisis as a generalization crisis, I haven't bothered to follow the psychology discussion of this, but I did follow the cancer research attempt to claim a replication crisis. The article that most people cite is an editorial where pharma researchers try to "replicate" studies by taking "landmark" studies (that were not revealed), and then do drug development off the findings of these initial papers. Now, sure, if cancer were so simple that surprising new cell behaviors could be used to develop drugs that target lots of different cancers, that would be fantastic. But that sort of generalization should never be expected, it's just silly. If somebody finds a new cellular behavior in a handful of cell lines or animal models or tumor biopsies, there's no reason to think it will generalize. The point of the primary literature is to document the primary evidence. It's not a textbook, with highly generalized scientific principles, it's the messy discovery phase of science where scientists are communicating findings that they can build upon to eventually build a body of work that could become a generalized principle. A single paper should never constitute the basis of an entire drug development program. I think that a lot of the "replication" crisis is just a fundamental misunderstanding of how scientific literature works and how it's used by the intended audience.

snapplebobapple 4y ago

Out of curiosity, have you looked at the original paper and the replication paper? I would think whether the study in Amsterdam shows the original isn't replicatable or not would really depend on how generalized the original paper chose to make the claims in its conclusion. If the original paper concluded their result was generally applicable then the Amsterdam study is, in fact, a valid replication. If it claims to be applicable to only some subset of the population then this is a valid criticism (but if the applicability constraint is too narrow, i.e just that university, then another valid criticism is why did the first paper even warrant publishing??) Makes me wonder if this paper will be overturned in itself in a couple years as the academic machinery turns.

ethanbond 4y ago
I haven’t, but really even if it does make an excessively bold claim, it’s definitely a replication failure but I’m still of the belief that it’s not useful to frame that as “replication crisis.” Newton’s failure to account for near-light speed movement - and the rest of physics knowledge - doesn’t really warrant “replication crisis.” It’s just “more science” that’s refining our understanding. My issue with replication crisis framing is that in my experience it leads people to believe the softer sciences are irredeemable. Not only is the “crisis” not limited to softer sciences, but it’s really not a crisis. As we go up the entropic chain (physics -> chem -> bio -> socio etc), we can expect that understanding all of the relevant variables will get harder and harder and harder. It doesn’t mean we shouldn’t continue trying, and it doesn’t point to another route to knowledge other than continuing to try.
- snapplebobapple 4y ago
  The difference would be that with Newton his results applied to pretty much all observable situations and only when our observation ability improved did we discover problems where his work wouldn't replicate and we had to tighten the specification of what his work applied to. In this regard you could also call the problem in the social sciences a specification crisis because their claims don't hold up over the wide group they attempt to apply them to in the results discussion (I would argue far too often they don't apply beyond the x people in the study, but that's open for debate). I agree, we shouldn't stop trying. We do, however, need to be aware of the things we are doing to be utter failures and correct them, otherwise nothing will come from trying.