5 ms·

I do not understand why a single experiment is considered good enough for research results in the social sciences. A single experiment with unexpected outcomes

I do not understand why a single experiment is considered good enough for research results in the social sciences. A single experiment with unexpected outcomes must be backed up by replication in order to start becoming interesting. Even if you took the SPE at face value when it was first conducted, surely you have to repeat it before deciding if it is valid. What if the outcome only occurs once in every 100 times and they got lucky that first time? Surely that is somewhat important to know. The most disappointing outcome is that nothing has really changed even after all this time. Each researched is only one great result away from making their career.

deyan 8y ago
This is a false assumption about how social sciences work. Like in other fields, academics are repeating experiments and trying to (in)validate results, including for SPE. [1] That being said, social sciences != hard sciences in the sense that there are no hard laws because humans are unpredictable. That doesn't mean social sciences are worthless, but it does require that one approaches them with a different mindset than e.g. math or physics. FWIW, I do agree that your criticism regarding the gap that exists from publication to repetition / validation has made many famous. Bombastic-sounding results -> headlines -> public perception -> increased social and academic status of the researchers. By the time the results are repeated and (in)validated it is often too late to reverse that. I don't think that pattern is unique to social sciences however - I keep seeing article after article contradicting each other - e.g. regarding drugs, food, fitness, etc. [1] https://en.wikipedia.org/wiki/Stanford_prison_experiment#Similar_studies https://en.wikipedia.org/wiki/Stanford_prison_experiment#Sim...
- throwawaymath 8y ago
  What mindset is that, and how exactly is it different?
  - deyan 8y ago
    (Just my 2 cents, your mileage might vary) I think that in the hard sciences, things are very cut and dry - or at least that is the goal of a study. For example, in physics, the goal of experiments is to prove with a very high certainty that something is true or false. Humans on the other hand are unpredictable. So if you run an experiment that says X, it may or may not replicate later, depending on hidden variables and assumptions. Consider the famous marshmallow test. The latest studies suggest that it is not willpower but actually affluence that is the bigger determinant factor. [1] So that means that in future studies, they probably need to consider this variable and design the experiment in a way that they can control for it. What is interesting is that for all the differences and intervening variables, humans can be studied and exhibit very predictable patterns. A good example of that is the study of power. [2] The Prince was written in 1532 and its principle continue to be just as valid today! [1] https://www.theatlantic.com/family/archive/2018/06/marshmallow-test/561779/ https://www.theatlantic.com/family/archive/2018/06/marshmall... [2] https://en.wikipedia.org/wiki/Machiavellianism https://en.wikipedia.org/wiki/Machiavellianism
    - mehwoot 8y ago
      Humans on the other hand are unpredictable. So if you run an experiment that says X, it may or may not replicate later humans can be studied and exhibit very predictable patterns. Seems somewhat contradictory. If humans can be studied and exhibit predictable patterns, why wouldn't we expect experiments to be repeatable, as the parent comment asked? And if the experiments are highly random, then either you should be conducting more of them over and over to get a valid statistical sample, or you shouldn't be conducting them at all. Either way I see no valid argument that studies in social sciences shouldn't be repeatable.
      - everythingswan 8y ago
        This is more of an exploration than an explanation but it seems like you're pointing to the threshold of success in a series of experiments. As deyan mentioned in physics or chemistry, we get a high level of certainty after isolating all the variables. When it doesn't go according to plan, it's a known or unknown variable to blame. It seems to me that behavioral psychology is still in its infancy in terms of identifying those variables and/or the threshold of Truth is much lower than sciences like chemistry. Lots of patterns exist that have lower thresholds. Sports analogies are fairly illustrative. Hitters in baseball are considered great if they succeed 3 times out of 10. Then again, they run 500+ experiments a year (for hopefully many years) to determine their average... I agree with your conclusion that experiments needed to be repeated more over time. I simply wonder what type of success threshold we will look at as the Truth in time.
  - TeMPOraL 8y ago
    IMO - the mindset should be: social sciences are dealing with things orders of magnitude more complex than anything seen in hard sciences (except biology). Therefore, social scientists need to be much more reserved in their claims, and not much should be taken for a fact unless it's replicated in great many studies and has proven predictive power. Whether this is the mindset social scientists have, I can't tell - but social sciences in popular media seem to be the exact opposite of that.
    - carlmr 8y ago
      >Whether this is the mindset social scientists have, I can't tell - but social sciences in popular media seem to be the exact opposite of that. I think it's just not a very human mindset. I've talked to natural scientists and social scientists alike. All of them want their research to succeed, they all want to get their PhDs, get good results for their postdocs, get publications going etc. The incentive is just there to publish somewhat inflated results. If you could get a PhD for systematcally dismantling studies (which would be much more useful than most PhD studies), we'd have a replication crisis in every field.
      - gepi79 8y ago
        Quote: "I think it's just not a very human mindset. I've talked to natural scientists and social scientists alike. All of them want their research to succeed, they all want to get their PhDs, get good results for their postdocs, get publications going etc." And that is one of the major problems. Hubris and arrogance and thus bad requirements and expectations and thus cheating and lies and half-truths as normal behavior. Also in economics: http://bilbo.economicoutlook.net/blog/?p=39198 http://bilbo.economicoutlook.net/blog/?p=39198 Quote: "Further, in my field (economics) one can never really get a publication if the research only produces ‘negative’ results. That is, the researcher fails to find anything. I believe this is a common problem in other disciplines as well."
bakhy 8y ago
The outcome was not entirely unexpected. The article mentions that this is what Zimbardo originally intended. But I think even more important in this story is the great story-like quality of the result. It's a great story, and it "makes you think", as they say. One professor in this text is quoted actually saying that the story is bigger, meaning presumably more important, than the science. I remember similar reactions when some of Gladwell's shenanigans was uncovered. It just doesn't matter if it's true, if it makes a nice story. In fact, it's probably even better if it's not true.
- xamuel 8y ago
  Reminds me of religious parables. No-one objects to Jesus's parables on the basis of their being factually untrue--they're obviously fiction and that's not a mark against them. On the other hand, the SPE is marketed as empirical science, not spiritual revelation!
  - bakhy 8y ago
    Yes. And the fact that, being taken as serious science, it actually affected penal policy, makes it particularly ugly.
- pas 8y ago
  Is there a definitive writeup of what the fuck is wrong with Gladwell('s writing)?
  - bakhy 8y ago
    I stumpled upon a good text here once, but I don't remember where it was published, so I guess you should best google it, sorry.
UncleMeat 8y ago
> I do not understand why a single experiment is considered good enough for research results in the social sciences. It isn't. This is a problem of presentation to nonexperts. Talk to grad students in psych and they will have an accurate understanding of the SPE and its place in the literature.

deyan 8y ago

This is a false assumption about how social sciences work. Like in other fields, academics are repeating experiments and trying to (in)validate results, including for SPE. [1] That being said, social sciences != hard sciences in the sense that there are no hard laws because humans are unpredictable. That doesn't mean social sciences are worthless, but it does require that one approaches them with a different mindset than e.g. math or physics. FWIW, I do agree that your criticism regarding the gap that exists from publication to repetition / validation has made many famous. Bombastic-sounding results -> headlines -> public perception -> increased social and academic status of the researchers. By the time the results are repeated and (in)validated it is often too late to reverse that. I don't think that pattern is unique to social sciences however - I keep seeing article after article contradicting each other - e.g. regarding drugs, food, fitness, etc. [1] https://en.wikipedia.org/wiki/Stanford_prison_experiment#Similar_studies https://en.wikipedia.org/wiki/Stanford_prison_experiment#Sim...

throwawaymath 8y ago
What mindset is that, and how exactly is it different?
- deyan 8y ago
  (Just my 2 cents, your mileage might vary) I think that in the hard sciences, things are very cut and dry - or at least that is the goal of a study. For example, in physics, the goal of experiments is to prove with a very high certainty that something is true or false. Humans on the other hand are unpredictable. So if you run an experiment that says X, it may or may not replicate later, depending on hidden variables and assumptions. Consider the famous marshmallow test. The latest studies suggest that it is not willpower but actually affluence that is the bigger determinant factor. [1] So that means that in future studies, they probably need to consider this variable and design the experiment in a way that they can control for it. What is interesting is that for all the differences and intervening variables, humans can be studied and exhibit very predictable patterns. A good example of that is the study of power. [2] The Prince was written in 1532 and its principle continue to be just as valid today! [1] https://www.theatlantic.com/family/archive/2018/06/marshmallow-test/561779/ https://www.theatlantic.com/family/archive/2018/06/marshmall... [2] https://en.wikipedia.org/wiki/Machiavellianism https://en.wikipedia.org/wiki/Machiavellianism
  - mehwoot 8y ago
    Humans on the other hand are unpredictable. So if you run an experiment that says X, it may or may not replicate later humans can be studied and exhibit very predictable patterns. Seems somewhat contradictory. If humans can be studied and exhibit predictable patterns, why wouldn't we expect experiments to be repeatable, as the parent comment asked? And if the experiments are highly random, then either you should be conducting more of them over and over to get a valid statistical sample, or you shouldn't be conducting them at all. Either way I see no valid argument that studies in social sciences shouldn't be repeatable.
    - everythingswan 8y ago
      This is more of an exploration than an explanation but it seems like you're pointing to the threshold of success in a series of experiments. As deyan mentioned in physics or chemistry, we get a high level of certainty after isolating all the variables. When it doesn't go according to plan, it's a known or unknown variable to blame. It seems to me that behavioral psychology is still in its infancy in terms of identifying those variables and/or the threshold of Truth is much lower than sciences like chemistry. Lots of patterns exist that have lower thresholds. Sports analogies are fairly illustrative. Hitters in baseball are considered great if they succeed 3 times out of 10. Then again, they run 500+ experiments a year (for hopefully many years) to determine their average... I agree with your conclusion that experiments needed to be repeated more over time. I simply wonder what type of success threshold we will look at as the Truth in time.
- TeMPOraL 8y ago
  IMO - the mindset should be: social sciences are dealing with things orders of magnitude more complex than anything seen in hard sciences (except biology). Therefore, social scientists need to be much more reserved in their claims, and not much should be taken for a fact unless it's replicated in great many studies and has proven predictive power. Whether this is the mindset social scientists have, I can't tell - but social sciences in popular media seem to be the exact opposite of that.
  - carlmr 8y ago
    >Whether this is the mindset social scientists have, I can't tell - but social sciences in popular media seem to be the exact opposite of that. I think it's just not a very human mindset. I've talked to natural scientists and social scientists alike. All of them want their research to succeed, they all want to get their PhDs, get good results for their postdocs, get publications going etc. The incentive is just there to publish somewhat inflated results. If you could get a PhD for systematcally dismantling studies (which would be much more useful than most PhD studies), we'd have a replication crisis in every field.
    - gepi79 8y ago
      Quote: "I think it's just not a very human mindset. I've talked to natural scientists and social scientists alike. All of them want their research to succeed, they all want to get their PhDs, get good results for their postdocs, get publications going etc." And that is one of the major problems. Hubris and arrogance and thus bad requirements and expectations and thus cheating and lies and half-truths as normal behavior. Also in economics: http://bilbo.economicoutlook.net/blog/?p=39198 http://bilbo.economicoutlook.net/blog/?p=39198 Quote: "Further, in my field (economics) one can never really get a publication if the research only produces ‘negative’ results. That is, the researcher fails to find anything. I believe this is a common problem in other disciplines as well."

bakhy 8y ago

The outcome was not entirely unexpected. The article mentions that this is what Zimbardo originally intended. But I think even more important in this story is the great story-like quality of the result. It's a great story, and it "makes you think", as they say. One professor in this text is quoted actually saying that the story is bigger, meaning presumably more important, than the science. I remember similar reactions when some of Gladwell's shenanigans was uncovered. It just doesn't matter if it's true, if it makes a nice story. In fact, it's probably even better if it's not true.

xamuel 8y ago
Reminds me of religious parables. No-one objects to Jesus's parables on the basis of their being factually untrue--they're obviously fiction and that's not a mark against them. On the other hand, the SPE is marketed as empirical science, not spiritual revelation!
- bakhy 8y ago
  Yes. And the fact that, being taken as serious science, it actually affected penal policy, makes it particularly ugly.
pas 8y ago
Is there a definitive writeup of what the fuck is wrong with Gladwell('s writing)?
- bakhy 8y ago
  I stumpled upon a good text here once, but I don't remember where it was published, so I guess you should best google it, sorry.

UncleMeat 8y ago

> I do not understand why a single experiment is considered good enough for research results in the social sciences. It isn't. This is a problem of presentation to nonexperts. Talk to grad students in psych and they will have an accurate understanding of the SPE and its place in the literature.