7 ms·

To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences

by Thrymr 9y ago

To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences (nasonline.org) [1], a nongovernmental organization that consists of scientists elected by their peers to provide independent scientific advice to the US government. There was in fact a study published recently in PNAS, the Proceedings of the National Academy of Sciences, on this topic [2]. [0] https://en.wikipedia.org/wiki/National_Association_of_Scholars https://en.wikipedia.org/wiki/National_Association_of_Schola... [1] https://en.wikipedia.org/wiki/National_Academy_of_Sciences https://en.wikipedia.org/wiki/National_Academy_of_Sciences [2] http://www.pnas.org/content/early/2018/03/08/1802324115 http://www.pnas.org/content/early/2018/03/08/1802324115

jrochkind1 9y ago
Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education" Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one. I wonder what National Association of Scholars' motivation is here exactly.
- DanielBMarkham 9y ago
  Isn't the entire purpose of reproducability is to eliminate such questions from discussions about science?
  - da_chicken 9y ago
    The purpose of reproducibility is to be able to properly analyze the quality and accuracy of a study. The purpose is to support critical thinking and objective facts. However, this paper isn't a study. It's just an essay. And while, yes, the paper in question does highlight numerous issues in reasonable and rational ways, it also uses this as the breakout quote on page 36: > Scientists readily accept results that confirm liberal political arguments, and frequently reject contrary results out of hand. Now, there's a couple problems with this statement. The most obvious is that it implies blame on liberal politics. On the same page they detail that the percentage of self-identified liberal psychologists at a conference (~800 of 1,000) wildly outnumbers the number of self-identified conservative psychologists at the same conference (3 of 1,000). Well, this conference is the Society for Personality and Social Psychology, which is based in southern California. Would that bias attendance? However, even if we assume that distribution is correct, the quoted statement doesn't say anything about the conservative scientists. Would it also be correct that conservative scientists readily accept results that confirm their political arguments? We don't know. Maybe it would be more correct to say this: > Scientists readily accept results that confirm pre-existing personal political arguments, and frequently reject contrary results out of hand. Instead, this paper chooses to imply that scientists who are more liberal are somehow inherently more biased. If we assert that good scientists are those who don't allow political arguments to influence their acceptance of their results, then the natural conclusion to draw from that is that being liberal is incompatible with being a good scientist. Don't get me wrong, this paper makes a lot of legitimately good points with good examples, but the whole section on groupthink comes across a little bit biased itself. However, it's existence means I can't shake the feeling that maybe I'm missing something. So having show that there's a reproducibility problem, the paper concludes. Then very end of the paper, the afterword lists these points introduced seemingly out of the blue: > Symptoms of Pathological Science: > 1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause. > 2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results. > 3. Claims of great accuracy. > 4. Fantastic theories contrary to experience. > 5. Criticisms are met by ad hoc excuses thought up on the spur of the moment. > 6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion. These are from Irving Langmuir (https://en.wikipedia.org/wiki/Irving_Langmuir https://en.wikipedia.org/wiki/Irving_Langmuir) and his criticisms on research conducted with unconscious biases or subjective elements. He meant it as a criticism of research like cold fusion. Now, there's not any real one-for-one discussion of these items or how they relate to what came before. They're just presented basically as-is as the final portion of the essay with few notes that they correlate to what came before. You'll notice, however, they they all call into question the credibility of the scientist as well as the credibility of the scientific community, and not the quality of their research or data. However, let's say you're politically conservative, and you're reading this essay. Maybe you don't believe in climate change. You get to the end of this paper, and it tells you that in spite of the claims of accuracy of the data, in spite of the consensus of the community, in spite of all the evidence, if it's contrary to [your] experience or if the defenses to criticism feel invented on the spot or if you keep hearing claims that support is waning, and apparently especially if it's liberal, then you've found a legitimate case of bad science. I don't think that's necessarily the point they're trying to make here, but I really don't like the way this afterword is presented. It's introducing new concepts and arguments and then handwaves away having to justify them or explain their relevance.
  - jrochkind1 9y ago
    To eliminate what such questions? To eliminate any questions about social context or bias or agenda? Totally not possible, social context is an inherent part of science, as in all human endeavors. I think the purpose of reproducibility is to well, make science _possible_, it is the fundamental bedrock of science -- to determine what and how we can make reliable predictions about the world. It doesn't mean bias and effects of social context no longer exists though. Science exists anyway, in a social context -- as evidenced by the past couple hundred years! In general, I don't think science exists to eliminate _any_ questions, science exists to ask questions. Eliminating questions may be a goal of ideology or religion, but not science. (and in the real world, they certainly influence each other. science is a goal to strive for, not a fait accompli. Scientific understandings of the world are always changing; science is an approach to understanding the world, not a set of settled conclusions).
    - jrochkind1 9y ago
      Also, if we're looking for the social context that leads to the reproducibility crisis, I think it's a lot more likely to be about academic career incentives (you _gotta_ publish things, a lot of things, that all seem like important and socially useful results, none of which are negative results -- when the actual practice of science means negative results are an expected and useful thing, not indication you are a bad scientist), than the political ideologies/biases of researchers. Maybe I'm wrong. There are probably ways to investigate it. This essay isn't one of them.

by Thrymr 9y ago

jrochkind1 9y ago

Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education" Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one. I wonder what National Association of Scholars' motivation is here exactly.

DanielBMarkham 9y ago
Isn't the entire purpose of reproducability is to eliminate such questions from discussions about science?
- da_chicken 9y ago
  The purpose of reproducibility is to be able to properly analyze the quality and accuracy of a study. The purpose is to support critical thinking and objective facts. However, this paper isn't a study. It's just an essay. And while, yes, the paper in question does highlight numerous issues in reasonable and rational ways, it also uses this as the breakout quote on page 36: > Scientists readily accept results that confirm liberal political arguments, and frequently reject contrary results out of hand. Now, there's a couple problems with this statement. The most obvious is that it implies blame on liberal politics. On the same page they detail that the percentage of self-identified liberal psychologists at a conference (~800 of 1,000) wildly outnumbers the number of self-identified conservative psychologists at the same conference (3 of 1,000). Well, this conference is the Society for Personality and Social Psychology, which is based in southern California. Would that bias attendance? However, even if we assume that distribution is correct, the quoted statement doesn't say anything about the conservative scientists. Would it also be correct that conservative scientists readily accept results that confirm their political arguments? We don't know. Maybe it would be more correct to say this: > Scientists readily accept results that confirm pre-existing personal political arguments, and frequently reject contrary results out of hand. Instead, this paper chooses to imply that scientists who are more liberal are somehow inherently more biased. If we assert that good scientists are those who don't allow political arguments to influence their acceptance of their results, then the natural conclusion to draw from that is that being liberal is incompatible with being a good scientist. Don't get me wrong, this paper makes a lot of legitimately good points with good examples, but the whole section on groupthink comes across a little bit biased itself. However, it's existence means I can't shake the feeling that maybe I'm missing something. So having show that there's a reproducibility problem, the paper concludes. Then very end of the paper, the afterword lists these points introduced seemingly out of the blue: > Symptoms of Pathological Science: > 1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause. > 2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results. > 3. Claims of great accuracy. > 4. Fantastic theories contrary to experience. > 5. Criticisms are met by ad hoc excuses thought up on the spur of the moment. > 6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion. These are from Irving Langmuir (https://en.wikipedia.org/wiki/Irving_Langmuir https://en.wikipedia.org/wiki/Irving_Langmuir) and his criticisms on research conducted with unconscious biases or subjective elements. He meant it as a criticism of research like cold fusion. Now, there's not any real one-for-one discussion of these items or how they relate to what came before. They're just presented basically as-is as the final portion of the essay with few notes that they correlate to what came before. You'll notice, however, they they all call into question the credibility of the scientist as well as the credibility of the scientific community, and not the quality of their research or data. However, let's say you're politically conservative, and you're reading this essay. Maybe you don't believe in climate change. You get to the end of this paper, and it tells you that in spite of the claims of accuracy of the data, in spite of the consensus of the community, in spite of all the evidence, if it's contrary to [your] experience or if the defenses to criticism feel invented on the spot or if you keep hearing claims that support is waning, and apparently especially if it's liberal, then you've found a legitimate case of bad science. I don't think that's necessarily the point they're trying to make here, but I really don't like the way this afterword is presented. It's introducing new concepts and arguments and then handwaves away having to justify them or explain their relevance.
- jrochkind1 9y ago
  To eliminate what such questions? To eliminate any questions about social context or bias or agenda? Totally not possible, social context is an inherent part of science, as in all human endeavors. I think the purpose of reproducibility is to well, make science _possible_, it is the fundamental bedrock of science -- to determine what and how we can make reliable predictions about the world. It doesn't mean bias and effects of social context no longer exists though. Science exists anyway, in a social context -- as evidenced by the past couple hundred years! In general, I don't think science exists to eliminate _any_ questions, science exists to ask questions. Eliminating questions may be a goal of ideology or religion, but not science. (and in the real world, they certainly influence each other. science is a goal to strive for, not a fait accompli. Scientific understandings of the world are always changing; science is an approach to understanding the world, not a set of settled conclusions).
  - jrochkind1 9y ago
    Also, if we're looking for the social context that leads to the reproducibility crisis, I think it's a lot more likely to be about academic career incentives (you _gotta_ publish things, a lot of things, that all seem like important and socially useful results, none of which are negative results -- when the actual practice of science means negative results are an expected and useful thing, not indication you are a bad scientist), than the political ideologies/biases of researchers. Maybe I'm wrong. There are probably ways to investigate it. This essay isn't one of them.