7 ms·

> by asking e.g. what the largest organ in the human body is. Which is part of why they had subscores for related fields. Obviously we can't quiz about the ac

by mlyle 4y ago

> by asking e.g. what the largest organ in the human body is. Which is part of why they had subscores for related fields. Obviously we can't quiz about the actual controversy, because we disgaree on the facts. > It doesn't even make any sense. I think it's interesting that people who disagree with these consensus have, on average, lower overall literacy in the sciences-- both adjacent sciences and other science, but believe they have above average literacy and will outperform the average on tests.

somenameforme 4y ago
Ignoring your attempt at predictably headed ad hominem, you again cannot make your 'conclusion' from the study. They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally did everything trying to massage their numbers into the conclusion they wanted to make. Ironically, the numbers don't even support that if you do use the subscales. In all studies the overall effect from opposition was a fraction of a single point difference. In most studies opposition to the consensus in climate change was also predictive of a higher than average level of field-specific knowledge.
- mlyle 4y ago
  > Ignoring your attempt at predictably headed ad hominem ???? ad hominem? Where? I'm saying it's very difficult to quiz about the actual controversy itself, because then we get very close to the point of disagreement and risk confounding. The research explicitly mentions this aspect of design. > They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally did everything trying to massage their numbers into the conclusion they wanted to make. I don't love use of Mechanical Turk for social sciences. It's still an interesting finding. Of course, more and higher quality research should be used to confirm the effect and gain additional nuance. > In all studies the overall effect from opposition was a fraction of a single point difference. The overall effect from opposition was a fraction of a single point difference per unit of opposition.
  - somenameforme 4y ago
    You're definitely correct there on my misreading of the tables. To further clarify, I also decided to see precisely what "points" meant rather than continuing to just skim. They chose to rate true/false answers on a -3 to +3 scale driven by the respondent's certainty. This means one question wrong, out of 34, was able to drive up to a 6 point difference at max certainty. And so, somewhat serendipitously, my point, with some 'modification' remains that opposition in no case was rarely able to explain even a single field-specific question missed, at least not at max certainty. As for the replicability - again, this study arbitrarily removed people fully in line with the consensus. That makes it fairly safe to say that replication is a nonstarter. But I'd also add that another red flag for social science papers is when they collect a large number of variables that end up having no relevance to the published conclusion. Large numbers of variables is a key resource in p-hacking. And this paper was collecting all sorts of data that went completely unused and had nothing whatsoever to do with what they ultimately chose to publish.
    - mlyle 4y ago
      > The effect of opposition on the binarized full set of 34 objective knowledge items variable: > Estimate Std. Error df t value Pr(>|t|) > (Intercept) 25.90592 0.43454 13.11332 59.617 <2e-16 ** > opposition -0.664790.06842 2130.90896 -9.717 <2e-16 ** Looks like each point of opposition was about 2/3rds of one more incorrect true/false question. So the people who were most opposed scored >3 questions worse on a 34 question test, on average. > again, this study arbitrarily removed people fully in line with the consensus. Median filtering and trimming saturated measures is common in research like this-- hopefully designed into the original protocol. I do agree it would be nice to see their preregistration. But, weird things happen at the tails-- truncation effects, etc.
      - somenameforme 4y ago
        Like mentioned, I was more interested in the impact on the subtopic specific results because of what we discussed. Giving a quiz on American history to judge your knowledge of French history is obviously not a reasonable idea, even if the skill-set overlap would probably give at least some weak correlation. In the field-specific binarized case (which removes the -3 to +3 noise), the impact is 0.09 points. So that translates to a fraction of a question difference in results. And while I'm aware trimming extreme outliers, regardless of the side they end up on, I'm unaware of any study entirely removing a segment from their sample, from one side only, which is critical to your entire hypothesis, and doing so without any explanation, let alone justification, whatsoever. One thing I'd observe is that the observed effect is small enough that if this culled group had a knowledge specific score below the mean, then it's likely that their entire conclusion would be invalid. One other issue we have not discussed is the methodology not only resulted in a very non-representative sample, but also was indirectly testing something else. The surveys were done on the internet, and all of the general knowledge questions (besides the covid ones, which were just...) have answers which can be looked up in a matter of seconds.

by mlyle 4y ago

somenameforme 4y ago

Ignoring your attempt at predictably headed ad hominem, you again cannot make your 'conclusion' from the study. They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally did everything trying to massage their numbers into the conclusion they wanted to make. Ironically, the numbers don't even support that if you do use the subscales. In all studies the overall effect from opposition was a fraction of a single point difference. In most studies opposition to the consensus in climate change was also predictive of a higher than average level of field-specific knowledge.

mlyle 4y ago
> Ignoring your attempt at predictably headed ad hominem ???? ad hominem? Where? I'm saying it's very difficult to quiz about the actual controversy itself, because then we get very close to the point of disagreement and risk confounding. The research explicitly mentions this aspect of design. > They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally did everything trying to massage their numbers into the conclusion they wanted to make. I don't love use of Mechanical Turk for social sciences. It's still an interesting finding. Of course, more and higher quality research should be used to confirm the effect and gain additional nuance. > In all studies the overall effect from opposition was a fraction of a single point difference. The overall effect from opposition was a fraction of a single point difference per unit of opposition.
- somenameforme 4y ago
  You're definitely correct there on my misreading of the tables. To further clarify, I also decided to see precisely what "points" meant rather than continuing to just skim. They chose to rate true/false answers on a -3 to +3 scale driven by the respondent's certainty. This means one question wrong, out of 34, was able to drive up to a 6 point difference at max certainty. And so, somewhat serendipitously, my point, with some 'modification' remains that opposition in no case was rarely able to explain even a single field-specific question missed, at least not at max certainty. As for the replicability - again, this study arbitrarily removed people fully in line with the consensus. That makes it fairly safe to say that replication is a nonstarter. But I'd also add that another red flag for social science papers is when they collect a large number of variables that end up having no relevance to the published conclusion. Large numbers of variables is a key resource in p-hacking. And this paper was collecting all sorts of data that went completely unused and had nothing whatsoever to do with what they ultimately chose to publish.
  - mlyle 4y ago
    > The effect of opposition on the binarized full set of 34 objective knowledge items variable: > Estimate Std. Error df t value Pr(>|t|) > (Intercept) 25.90592 0.43454 13.11332 59.617 <2e-16 ** > opposition -0.664790.06842 2130.90896 -9.717 <2e-16 ** Looks like each point of opposition was about 2/3rds of one more incorrect true/false question. So the people who were most opposed scored >3 questions worse on a 34 question test, on average. > again, this study arbitrarily removed people fully in line with the consensus. Median filtering and trimming saturated measures is common in research like this-- hopefully designed into the original protocol. I do agree it would be nice to see their preregistration. But, weird things happen at the tails-- truncation effects, etc.
    - somenameforme 4y ago
      Like mentioned, I was more interested in the impact on the subtopic specific results because of what we discussed. Giving a quiz on American history to judge your knowledge of French history is obviously not a reasonable idea, even if the skill-set overlap would probably give at least some weak correlation. In the field-specific binarized case (which removes the -3 to +3 noise), the impact is 0.09 points. So that translates to a fraction of a question difference in results. And while I'm aware trimming extreme outliers, regardless of the side they end up on, I'm unaware of any study entirely removing a segment from their sample, from one side only, which is critical to your entire hypothesis, and doing so without any explanation, let alone justification, whatsoever. One thing I'd observe is that the observed effect is small enough that if this culled group had a knowledge specific score below the mean, then it's likely that their entire conclusion would be invalid. One other issue we have not discussed is the methodology not only resulted in a very non-representative sample, but also was indirectly testing something else. The surveys were done on the internet, and all of the general knowledge questions (besides the covid ones, which were just...) have answers which can be looked up in a matter of seconds.