4 ms·

Unsurprising. Although this is the first time I can recall reading a psychologist accept culpability for the field’s bad “science” over the last twenty years. H

Unsurprising. Although this is the first time I can recall reading a psychologist accept culpability for the field’s bad “science” over the last twenty years. Have any of the “anthropology of science” researchers published an explanation of that yet?

aprilthird2021 2y ago
Everyone says "unsurprising" after the replication fails. Idk, this was a really popular theory that lots of people believed. I doubt you were all so confident back in the heyday of this social science phenomenon. If tomorrow, they say "growth mindset" is also a non-replicable phenomenon, will HN be full of smug people saying "I knew it all along, lol!"
- renewiltord 2y ago
  That’s just a heterogeneous population with selection bias. Everyone is always convinced because when they’re disproven they stay silent. These days prediction markets allow people to convert beliefs to money so it becomes not very convincing when everyone manages to be perfect at predicting things and is always complaining they aren’t making enough money while simultaneously giving up the chance at doubling their bet on a sure shot. Combine that with people upgrading uncertainty to certainty post-hoc when debunking comes out and you have these entertaining things. Overall, I’m glad you called it out. Once I wished I had a profile for people’s past guesses to see how actually good they are and now I have Manifold, Kalshi, and Polymarket.
- Hasu 2y ago
  I'm not a research psychologist, but my understanding is that growth mindset is already non-replicating.
  - Viliam1234 2y ago
    To me it seems like "growth mindset" is a buzzword that everyone defines differently. The weak version is something like: "Even talented people need to work on improving themselves, if they want to be successful." Most people would agree with this. The strong version is something like: "If people believe that there is such a thing as talent, they automatically stop working on improving themselves." This is used to justify things like: "never tell your kids that they are talented or intelligent, you must attribute all their success to their hard work". In my opinion, this is a false dilemma: there can be things like talent or intelligence, and hard work can be necessary to succeed among those who are comparably talented or intelligent.
    - underlipton 2y ago
      >To me it seems like "growth mindset" is a buzzword that everyone defines differently. This seems like something that could (does) also apply to "stereotype threat."
- bluGill 2y ago
  Even if you did suspect something was wrong you were careful because science said otherwise. Calling this wrong in public would feel like saying a big rock falls faster than a small rock - something else that seems obvious but science has disproved. And so the only one loudly saying this was bunk where the right wing crackpots (the reasonable people on the right side were much quieter about it because they don't like to argue with science even though it went against their bias) - the right wing crackpots were on this only because the left wing embraced it as a science that confirmed their bias. Unfortunately in this cast the conspiracy was true and so the crackpots won (even though they have no understanding of the real reasons it is false and it doesn't seem to have been a conspiracy). Let this be a lesson: even if something seems like science and it confirms you bias - that doesn't mean it is true. You should look closer at things you embrace than things you reject lest you embrace a lie.
  - mistermann 2y ago
    I'm a rather reasonable "on the right" person and arguing about science is literally my hobby, there are few funner things in the world imho.
  - philwelch 2y ago
    > Even if you did suspect something was wrong you were careful because science said otherwise.… And so the only one loudly saying this was bunk where the right wing crackpots (the reasonable people on the right side were much quieter about it because they don't like to argue with science even though it went against their bias) - the right wing crackpots were on this only because the left wing embraced it as a science that confirmed their bias. If “crackpots” turn out to be right when “reasonable people” and “science” were wrong—and this is far from the only instance of this happening—maybe we should reevaluate some things. Just because a peer reviewed paper published in a prestigious journal says something doesn’t mean it’s true. Even a survey of multiple peer reviewed papers published over time isn’t necessarily determinative if there are common methodological issues or publishing biases. Yes, a lot of times actual crackpots will make stupid criticisms, but not every criticism is stupid even if it comes from outside the ivory tower. In particular, whether or not a big rock falls faster than a small rock [1] is a fairly basic question in physics, which has one of the most certain answers. Virtually nothing in psychology, let alone human social psychology, is at that level of certainty, and any psychologist worth their salt will agree with that. Basically any finding in psychology should have a level of certainty somewhere between “yeah, that’s probably mostly true” and “hmm, interesting hypothesis that’s not entirely crazy, I wonder if it holds up”. [1] Also, the literal question of whether a big rock or a small rock falls faster is trickier than you might assume if you only know the middle school version of the question. If we’re doing this on the moon, so as to dispose with the tricky aerodynamic questions, and we are answering the question from the frame of reference of an astronaut standing on the surface of the moon, the bigger rock actually does fall faster. Both rocks accelerate towards the center of the moon at the same rate, yes, but gravity works both ways. Both rocks pull on the moon but the bigger rock pulls harder, resulting in an extremely small torque on the lunar surface, drawing the side of the moon closer to it very slightly upwards. Which, from the surface frame of reference, is equivalent to the bigger rock falling faster.
- jdietrich 2y ago
  It might have been surprising at the time, but it's certainly unsurprising in light of the replication crisis. We now know that most findings don't replicate. Anyone who continues to be surprised by the most likely outcome needs to update their priors. https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
  - neuronexmachina 2y ago
    From your link: >Biostatisticians Jager and Leek criticized the model as being based on justifiable but arbitrary assumptions rather than empirical data, and did an investigation of their own which calculated that the false positive rate in biomedical studies was estimated to be around 14%, not over 50% as Ioannidis asserted.
- saurik 2y ago
  FWIW, to me it is "unsurprising" as I can't remember the last time a major psych study actually did replicate, and it seems frankly like the entire field is in shambles. Does that mean that I knew before that this study in specific was false? No. But, that isn't what anyone means when they say "unsurprising"! If you get to the end of a movie and all of the major characters end up alive, and someone else seemed shocked by that fact, you might still explain to them that that's "unsurprising", even if you yourself got to feel the thrill of uncertainty -- or even were concerned for a bit -- while you were watching.
- mistermann 2y ago
  Be careful to not mistake "theory doesn't replicate" with "theory is not true [in whole or in part]".
  - trhway 2y ago
    Would you expect the theory of non replicative theories to be replicative or not?
    - mistermann 2y ago
      Expectations may be part of the problem.
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- philwelch 2y ago
  Stereotype threat was a “popular theory” because it fit the social and political presuppositions that were fashionable at the time. It’s almost always wise to be skeptical of political fashions or new social science findings even on their own; new social science findings that match political fashions are doubly questionable.
- zahlman 2y ago
  My understanding is that "growth mindset" represents a personal philosophy, not a testable hypothesis, so it would be irrelevant here. It was entirely reasonable to be skeptical of stereotype thread when the concept was new, a priori, An "unsurprising" result is not necessarily one that someone confidently believed. If I flip a coin, I'm not "surprised" when the result is heads, nor when the result is tails.
  - eslaught 2y ago
    I could be wrong but I believe the idea is that the growth mindset has measurable consequences. I.e., if you believe you can grow, you will learn/adapt more effectively to new situations and ultimately achieve more. It's an empirically verifiable claim that is either true or false, and comes with a pretty straightforward intervention (i.e., teaching people that they can grow) if true. Of course people may also adopt it as a personal philosophy, but that's separate.
- PittleyDunkin 2y ago
  > If tomorrow, they say "growth mindset" is also a non-replicable phenomenon, will HN be full of smug people saying "I knew it all along, lol!" I will be one of them! Many people I've met in the industry—disturbingly, especially among founders and VCs—are extremely susceptible to reductive theories about the human experience that have little relation to reality. Just in the last year we went through "founder mode" hell even though any person that this "founder mode" describes should probably be in prison for wildly abusive behavior. Of course the flip-side of this is that there are concepts that are situationally useful even if they aren't broadly applicable. Both "stereotype threat" and "founder mode" may fall into this category. (I've never heard of "growth mindset" before and looking it up I'm not sure what the point of it is—I guess to contradict the idea that people and personalities are static?)

aprilthird2021 2y ago

Everyone says "unsurprising" after the replication fails. Idk, this was a really popular theory that lots of people believed. I doubt you were all so confident back in the heyday of this social science phenomenon. If tomorrow, they say "growth mindset" is also a non-replicable phenomenon, will HN be full of smug people saying "I knew it all along, lol!"

renewiltord 2y ago
That’s just a heterogeneous population with selection bias. Everyone is always convinced because when they’re disproven they stay silent. These days prediction markets allow people to convert beliefs to money so it becomes not very convincing when everyone manages to be perfect at predicting things and is always complaining they aren’t making enough money while simultaneously giving up the chance at doubling their bet on a sure shot. Combine that with people upgrading uncertainty to certainty post-hoc when debunking comes out and you have these entertaining things. Overall, I’m glad you called it out. Once I wished I had a profile for people’s past guesses to see how actually good they are and now I have Manifold, Kalshi, and Polymarket.
Hasu 2y ago
I'm not a research psychologist, but my understanding is that growth mindset is already non-replicating.
- Viliam1234 2y ago
  To me it seems like "growth mindset" is a buzzword that everyone defines differently. The weak version is something like: "Even talented people need to work on improving themselves, if they want to be successful." Most people would agree with this. The strong version is something like: "If people believe that there is such a thing as talent, they automatically stop working on improving themselves." This is used to justify things like: "never tell your kids that they are talented or intelligent, you must attribute all their success to their hard work". In my opinion, this is a false dilemma: there can be things like talent or intelligence, and hard work can be necessary to succeed among those who are comparably talented or intelligent.
  - underlipton 2y ago
    >To me it seems like "growth mindset" is a buzzword that everyone defines differently. This seems like something that could (does) also apply to "stereotype threat."
bluGill 2y ago
Even if you did suspect something was wrong you were careful because science said otherwise. Calling this wrong in public would feel like saying a big rock falls faster than a small rock - something else that seems obvious but science has disproved. And so the only one loudly saying this was bunk where the right wing crackpots (the reasonable people on the right side were much quieter about it because they don't like to argue with science even though it went against their bias) - the right wing crackpots were on this only because the left wing embraced it as a science that confirmed their bias. Unfortunately in this cast the conspiracy was true and so the crackpots won (even though they have no understanding of the real reasons it is false and it doesn't seem to have been a conspiracy). Let this be a lesson: even if something seems like science and it confirms you bias - that doesn't mean it is true. You should look closer at things you embrace than things you reject lest you embrace a lie.
- mistermann 2y ago
  I'm a rather reasonable "on the right" person and arguing about science is literally my hobby, there are few funner things in the world imho.
- philwelch 2y ago
  > Even if you did suspect something was wrong you were careful because science said otherwise.… And so the only one loudly saying this was bunk where the right wing crackpots (the reasonable people on the right side were much quieter about it because they don't like to argue with science even though it went against their bias) - the right wing crackpots were on this only because the left wing embraced it as a science that confirmed their bias. If “crackpots” turn out to be right when “reasonable people” and “science” were wrong—and this is far from the only instance of this happening—maybe we should reevaluate some things. Just because a peer reviewed paper published in a prestigious journal says something doesn’t mean it’s true. Even a survey of multiple peer reviewed papers published over time isn’t necessarily determinative if there are common methodological issues or publishing biases. Yes, a lot of times actual crackpots will make stupid criticisms, but not every criticism is stupid even if it comes from outside the ivory tower. In particular, whether or not a big rock falls faster than a small rock [1] is a fairly basic question in physics, which has one of the most certain answers. Virtually nothing in psychology, let alone human social psychology, is at that level of certainty, and any psychologist worth their salt will agree with that. Basically any finding in psychology should have a level of certainty somewhere between “yeah, that’s probably mostly true” and “hmm, interesting hypothesis that’s not entirely crazy, I wonder if it holds up”. [1] Also, the literal question of whether a big rock or a small rock falls faster is trickier than you might assume if you only know the middle school version of the question. If we’re doing this on the moon, so as to dispose with the tricky aerodynamic questions, and we are answering the question from the frame of reference of an astronaut standing on the surface of the moon, the bigger rock actually does fall faster. Both rocks accelerate towards the center of the moon at the same rate, yes, but gravity works both ways. Both rocks pull on the moon but the bigger rock pulls harder, resulting in an extremely small torque on the lunar surface, drawing the side of the moon closer to it very slightly upwards. Which, from the surface frame of reference, is equivalent to the bigger rock falling faster.
jdietrich 2y ago
It might have been surprising at the time, but it's certainly unsurprising in light of the replication crisis. We now know that most findings don't replicate. Anyone who continues to be surprised by the most likely outcome needs to update their priors. https://en.wikipedia.org/wiki/Why_Most_Published_Research_Findings_Are_False https://en.wikipedia.org/wiki/Why_Most_Published_Research_Fi...
- neuronexmachina 2y ago
  From your link: >Biostatisticians Jager and Leek criticized the model as being based on justifiable but arbitrary assumptions rather than empirical data, and did an investigation of their own which calculated that the false positive rate in biomedical studies was estimated to be around 14%, not over 50% as Ioannidis asserted.
saurik 2y ago
FWIW, to me it is "unsurprising" as I can't remember the last time a major psych study actually did replicate, and it seems frankly like the entire field is in shambles. Does that mean that I knew before that this study in specific was false? No. But, that isn't what anyone means when they say "unsurprising"! If you get to the end of a movie and all of the major characters end up alive, and someone else seemed shocked by that fact, you might still explain to them that that's "unsurprising", even if you yourself got to feel the thrill of uncertainty -- or even were concerned for a bit -- while you were watching.
mistermann 2y ago
Be careful to not mistake "theory doesn't replicate" with "theory is not true [in whole or in part]".
- trhway 2y ago
  Would you expect the theory of non replicative theories to be replicative or not?
  - mistermann 2y ago
    Expectations may be part of the problem.
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
philwelch 2y ago
Stereotype threat was a “popular theory” because it fit the social and political presuppositions that were fashionable at the time. It’s almost always wise to be skeptical of political fashions or new social science findings even on their own; new social science findings that match political fashions are doubly questionable.
zahlman 2y ago
My understanding is that "growth mindset" represents a personal philosophy, not a testable hypothesis, so it would be irrelevant here. It was entirely reasonable to be skeptical of stereotype thread when the concept was new, a priori, An "unsurprising" result is not necessarily one that someone confidently believed. If I flip a coin, I'm not "surprised" when the result is heads, nor when the result is tails.
- eslaught 2y ago
  I could be wrong but I believe the idea is that the growth mindset has measurable consequences. I.e., if you believe you can grow, you will learn/adapt more effectively to new situations and ultimately achieve more. It's an empirically verifiable claim that is either true or false, and comes with a pretty straightforward intervention (i.e., teaching people that they can grow) if true. Of course people may also adopt it as a personal philosophy, but that's separate.
PittleyDunkin 2y ago
> If tomorrow, they say "growth mindset" is also a non-replicable phenomenon, will HN be full of smug people saying "I knew it all along, lol!" I will be one of them! Many people I've met in the industry—disturbingly, especially among founders and VCs—are extremely susceptible to reductive theories about the human experience that have little relation to reality. Just in the last year we went through "founder mode" hell even though any person that this "founder mode" describes should probably be in prison for wildly abusive behavior. Of course the flip-side of this is that there are concepts that are situationally useful even if they aren't broadly applicable. Both "stereotype threat" and "founder mode" may fall into this category. (I've never heard of "growth mindset" before and looking it up I'm not sure what the point of it is—I guess to contradict the idea that people and personalities are static?)