10 ms·

In social sciences, there’s an absolutely massive bias in favor of publishing results that confirm current educated opinion, or that deviate from the consensus

by ced 10y ago

In social sciences, there’s an absolutely massive bias in favor of publishing results that confirm current educated opinion, or that deviate from the consensus in ways that will be seen as quirky or interesting rather than cold or cruel or politically tone-deaf. Isn't that a massive problem for meta-analyses? If there are 15 studies supporting the consensus/politically-correct position and 5 against it, that might be evidence against the consensus, if less than a third of researchers with against-consensus results have dared to publish their results.

elcapitan 10y ago
How does meta-analysis in a highly opinion based field like the social sciences, where there are different, mutually exclusive schools of thought even make sense in the first place?
- pron 10y ago
  And how does a lack of even rudimentary knowledge of a specific field of study qualify one to reject it wholesale? The difficulties in meta-analysis in the social sciences vary from field to field -- psychology is qualitatively different from anthropology or history -- but if we had to generalize, they have more to do with the sheer dimensionality of the problem combined with the difficulty of precise measurement (and sometimes obtaining a large-enough sample), than with the field being "opinion based" or the existence of "mutually exclusive schools of thought" (I don't know how this differs from many other, more tractable research areas, and I don't see how this can be reconciled with saying that there's too much consensus).
  - mafribe 10y ago
    I'd say that scientific fields being "opinion based" or forming around "mutually exclusive schools of thought" is a reaction to "the sheer dimensionality of the problem combined with the difficulty of precise measurement". You see similar social phenomena in other fields with a measurement problem, for example programming language research ...
    - pron 10y ago
      > You see similar social phenomena in other fields with a measurement problem, for example programming language research ... Touché! :) (although there's plenty of social science research that is much more empirically rigorous than PL research, and the certainty/evidence ratio is certainly higher in PL)
pron 10y ago
With all due respect to Aaronson (with whom I find myself agreeing on many things), I wouldn't take his evaluation on the state of social science and his analysis of its core problems as correct. He's not a social scientist, and I don't think he'd appreciate an anthropologist evaluating the state of affairs in quantum computing research.
- privong 10y ago
  > He's not a social scientist, and I don't think he'd appreciate an anthropologist evaluating the state of affairs in quantum computing research. But, from things I have read elsewhere, there does seem to be a reasonably convincing body of evidence that there are systematic biases in social sciences. Some of this perhaps stems from discrimination (e.g., in hiring practices) based on political allegiance. Jonathan Haidt (a social scientist) has spoken out on this topic[0], including specifically on sociology[1]. While that issue likely affects academics in general, political leaning likely has a much smaller influence on one's research in physics and chemistry, compared to one's research in sociology. It's true that Aaronson is not a social scientist, but I don't think that means he is wrong. [0] http://people.stern.nyu.edu/jhaidt/postpartisan.html http://people.stern.nyu.edu/jhaidt/postpartisan.html [1] http://www.mindingthecampus.org/2016/02/a-conversation-with-jonathan-haidt/ http://www.mindingthecampus.org/2016/02/a-conversation-with-...
  - pron 10y ago
    > there does seem to be a reasonably convincing body of evidence that there is systematic biases in social sciences. But Aaronson makes a far more specific assertion than "there is evidence for systematic bias in social sciences". I agree that there is political bias in social science departments. It is painfully clear to anyone who ever sets foot in one, especially as an outsider (much like the sexism in SV). I think that overall it harms research, perhaps significantly, and it must be addressed. But outsiders sometimes view this (rather obvious) criticism as a complete refutation of the field(s), which it is most certainly not. To their credit, social researchers actively study this bias without need for external pressure, and even acknowledge it's a problem, something which cannot be said for other disciplines (one could argue that it is social researchers' job to study precisely such things, but at least you can see that they're not intellectually dishonest enough to avoid the issue or to dismiss it as inconsequential). BTW, in general, I think that outsiders usually tend to overestimate internal criticisms (in any discipline). When (internal) critics say "our estimate of the variable x to be 10 is far too low!" they really mean "x is more likely to be 12", while outsiders interpret it as "x is probably 1000". > political leaning likely has a much smaller influence on one's research in physics and chemistry, compared to one's research in sociology True, but the correlation between political leaning and social research may very well be a reverse causal relationship: that one's research in sociology is far more likely to affect one's politics than research in physics. If there is an over-representation of a particular ideology in the social sciences (although it's hard for me to put all the social sciences in one category, because they are quite different from one another -- more different than, say, physics and chemistry), it may serve as evidence that some common and politically pertinent ideas arise from the research itself[1] as much as evidence of anything else. BTW, I've read the interview with Haidt that you linked to -- thank you for that -- it is truly horrifying. But I get the sense that this is a rather American phenomenon. I certainly didn't experience anything close to it in magnitude at my university. [1]: And I don't mean necessarily from the results, but possibly from the process.
moultano 10y ago
You can assess this using funnel plots. If publication bias were not a factor, then you expect effect size to be uncorrelated from sample size. If they are correlated, then it's likely publication bias is having some effect on the results.
- ced 10y ago
  Clever solution. Is there a more general principle? I thought that most MNAR problems were hopeless, but maybe not.
  - moultano 10y ago
    It helps that we have a rough model of the publication process that we can correct for. It's kinda like assessing the underlying distribution with only access to the tail of it, which isn't ideal, but can be done.

by ced 10y ago

elcapitan 10y ago

How does meta-analysis in a highly opinion based field like the social sciences, where there are different, mutually exclusive schools of thought even make sense in the first place?

pron 10y ago
And how does a lack of even rudimentary knowledge of a specific field of study qualify one to reject it wholesale? The difficulties in meta-analysis in the social sciences vary from field to field -- psychology is qualitatively different from anthropology or history -- but if we had to generalize, they have more to do with the sheer dimensionality of the problem combined with the difficulty of precise measurement (and sometimes obtaining a large-enough sample), than with the field being "opinion based" or the existence of "mutually exclusive schools of thought" (I don't know how this differs from many other, more tractable research areas, and I don't see how this can be reconciled with saying that there's too much consensus).
- mafribe 10y ago
  I'd say that scientific fields being "opinion based" or forming around "mutually exclusive schools of thought" is a reaction to "the sheer dimensionality of the problem combined with the difficulty of precise measurement". You see similar social phenomena in other fields with a measurement problem, for example programming language research ...
  - pron 10y ago
    > You see similar social phenomena in other fields with a measurement problem, for example programming language research ... Touché! :) (although there's plenty of social science research that is much more empirically rigorous than PL research, and the certainty/evidence ratio is certainly higher in PL)

pron 10y ago

With all due respect to Aaronson (with whom I find myself agreeing on many things), I wouldn't take his evaluation on the state of social science and his analysis of its core problems as correct. He's not a social scientist, and I don't think he'd appreciate an anthropologist evaluating the state of affairs in quantum computing research.

privong 10y ago
> He's not a social scientist, and I don't think he'd appreciate an anthropologist evaluating the state of affairs in quantum computing research. But, from things I have read elsewhere, there does seem to be a reasonably convincing body of evidence that there are systematic biases in social sciences. Some of this perhaps stems from discrimination (e.g., in hiring practices) based on political allegiance. Jonathan Haidt (a social scientist) has spoken out on this topic[0], including specifically on sociology[1]. While that issue likely affects academics in general, political leaning likely has a much smaller influence on one's research in physics and chemistry, compared to one's research in sociology. It's true that Aaronson is not a social scientist, but I don't think that means he is wrong. [0] http://people.stern.nyu.edu/jhaidt/postpartisan.html http://people.stern.nyu.edu/jhaidt/postpartisan.html [1] http://www.mindingthecampus.org/2016/02/a-conversation-with-jonathan-haidt/ http://www.mindingthecampus.org/2016/02/a-conversation-with-...
- pron 10y ago
  > there does seem to be a reasonably convincing body of evidence that there is systematic biases in social sciences. But Aaronson makes a far more specific assertion than "there is evidence for systematic bias in social sciences". I agree that there is political bias in social science departments. It is painfully clear to anyone who ever sets foot in one, especially as an outsider (much like the sexism in SV). I think that overall it harms research, perhaps significantly, and it must be addressed. But outsiders sometimes view this (rather obvious) criticism as a complete refutation of the field(s), which it is most certainly not. To their credit, social researchers actively study this bias without need for external pressure, and even acknowledge it's a problem, something which cannot be said for other disciplines (one could argue that it is social researchers' job to study precisely such things, but at least you can see that they're not intellectually dishonest enough to avoid the issue or to dismiss it as inconsequential). BTW, in general, I think that outsiders usually tend to overestimate internal criticisms (in any discipline). When (internal) critics say "our estimate of the variable x to be 10 is far too low!" they really mean "x is more likely to be 12", while outsiders interpret it as "x is probably 1000". > political leaning likely has a much smaller influence on one's research in physics and chemistry, compared to one's research in sociology True, but the correlation between political leaning and social research may very well be a reverse causal relationship: that one's research in sociology is far more likely to affect one's politics than research in physics. If there is an over-representation of a particular ideology in the social sciences (although it's hard for me to put all the social sciences in one category, because they are quite different from one another -- more different than, say, physics and chemistry), it may serve as evidence that some common and politically pertinent ideas arise from the research itself[1] as much as evidence of anything else. BTW, I've read the interview with Haidt that you linked to -- thank you for that -- it is truly horrifying. But I get the sense that this is a rather American phenomenon. I certainly didn't experience anything close to it in magnitude at my university. [1]: And I don't mean necessarily from the results, but possibly from the process.

moultano 10y ago

You can assess this using funnel plots. If publication bias were not a factor, then you expect effect size to be uncorrelated from sample size. If they are correlated, then it's likely publication bias is having some effect on the results.

ced 10y ago
Clever solution. Is there a more general principle? I thought that most MNAR problems were hopeless, but maybe not.
- moultano 10y ago
  It helps that we have a rough model of the publication process that we can correct for. It's kinda like assessing the underlying distribution with only access to the tail of it, which isn't ideal, but can be done.