3 ms·
No, you're the one that's suffered an incorrect reading here. They created different objective knowledge questions for each of the different studies (there were
No, you're the one that's suffered an incorrect reading here. They created different objective knowledge questions for each of the different studies (there were 5) within this study. I am referencing the COVID "objective knowledge" quiz. "Subjective knowledge" for each study was determined by a single question where participants were asked to rate their knowledge of the various topics on a scale of 1-7.
The questions for this ("objective knowledge" of COVID) begin on page 36 of the supplementary materials page. [1] I predictably offered the most extreme example, but there are various other questions that are also more about liminal consensus than "objective fact".
But, perhaps most importantly, the observed effect from this study was extremely small. And with only 34 questions, even a small number of questions that measure anti-consensus views, rather than scientific knowledge, is enough to create a false effect through what's basically an obfuscated tautology: people who hold anti-consensus views, hold anti-consensus views.
[1] - https://www.science.org/doi/suppl/10.1126/sciadv.abo0038/suppl_file/sciadv.abo0038_sm.pdf https://www.science.org/doi/suppl/10.1126/sciadv.abo0038/sup...
I now see the question you mentioned from study 5. IMO study 5 is not the interesting substudy. As you mention, you cherry-picked the extreme example.
> But, perhaps most importantly, the observed effect from this study was extremely small.
The substudies that looked at those set of 34 questions above had an extremely significant result-- both statistical significance and magnitude of effect. I also don't see anything too controversial in that set of 34 questions. Do you?
Yes. It's a complete non sequitur to ask somebody to self rate their knowledge on e.g. climate change, and then try to objectively measure that knowledge by asking e.g. what the largest organ in the human body is. It doesn't even make any sense. It's like me asking to self estimate your knowledge of javascript, and then giving you a quiz on C# to measure that objective knowledge.
The study is also filled with various issues. For instance they also removed everybody who completely agreed with the scientific consensus on studies 1-4 (see buried in page 44 on the materials and methods supplement) who undoubtedly also rated their objective knowledge highly. They drew participants from outlets like Amazon Mechanical Turk where people are paid peanuts to carry out menial tasks ($0.85 for this study). That's obviously not going to be a representative sample of society, let alone America. The paper claimed to have preregistered their study, yet didn't provide a link to such - which completely negates the point of preregistering, and so on.
Something I find quite troubling is how increasingly common studies of this sort are. This study is mostly junk, but it's enticing click bait junk that will also undoubtedly bump up this journal's impact fact as other's unquestioningly cite it more with the goal of generating interest even if in lieu of science. What exactly were the peer reviewers/editors doing? It seems that so long as one concludes the right thing in exciting enough fashion, what editorial process there is becomes quite deferent, a la Sokal.
> by asking e.g. what the largest organ in the human body is.
Which is part of why they had subscores for related fields. Obviously we can't quiz about the actual controversy, because we disgaree on the facts.
> It doesn't even make any sense.
I think it's interesting that people who disagree with these consensus have, on average, lower overall literacy in the sciences-- both adjacent sciences and other science, but believe they have above average literacy and will outperform the average on tests.
Ignoring your attempt at predictably headed ad hominem, you again cannot make your 'conclusion' from the study. They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally did everything trying to massage their numbers into the conclusion they wanted to make.
Ironically, the numbers don't even support that if you do use the subscales. In all studies the overall effect from opposition was a fraction of a single point difference. In most studies opposition to the consensus in climate change was also predictive of a higher than average level of field-specific knowledge.
> Ignoring your attempt at predictably headed ad hominem
???? ad hominem? Where? I'm saying it's very difficult to quiz about the actual controversy itself, because then we get very close to the point of disagreement and risk confounding. The research explicitly mentions this aspect of design.
> They used a non-representative sample, removed the results of those who fully agreed with the consensus, and just generally did everything trying to massage their numbers into the conclusion they wanted to make.
I don't love use of Mechanical Turk for social sciences. It's still an interesting finding. Of course, more and higher quality research should be used to confirm the effect and gain additional nuance.
> In all studies the overall effect from opposition was a fraction of a single point difference.
The overall effect from opposition was a fraction of a single point difference per unit of opposition.
You're definitely correct there on my misreading of the tables. To further clarify, I also decided to see precisely what "points" meant rather than continuing to just skim. They chose to rate true/false answers on a -3 to +3 scale driven by the respondent's certainty. This means one question wrong, out of 34, was able to drive up to a 6 point difference at max certainty. And so, somewhat serendipitously, my point, with some 'modification' remains that opposition in no case was rarely able to explain even a single field-specific question missed, at least not at max certainty.
As for the replicability - again, this study arbitrarily removed people fully in line with the consensus. That makes it fairly safe to say that replication is a nonstarter. But I'd also add that another red flag for social science papers is when they collect a large number of variables that end up having no relevance to the published conclusion. Large numbers of variables is a key resource in p-hacking. And this paper was collecting all sorts of data that went completely unused and had nothing whatsoever to do with what they ultimately chose to publish.
Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study.
Imagine that there was a secular Islamic expert. And he was quizzed on Islamic ideology in a similar fashion, such that he was obligated to imply belief in such, in his answers. Assuming he is being as accurate as he can in his responses, he would end up scoring as completely ignorant of Islam as he marked all beliefs as false.
I disagree there's a clear trend here, even without the 1s, because there were clear exceptions such as e.g. climate change where, even by their metrics, anti-consensus views were predictive of greater knowledge. So you end up having to, at a minimum, limit the stated claim to specific fields.
> Nobody is forced to accept a fact if they do not want to. There's a large gulf between awareness of knowledge, including facts, and agreement with such. This is an especially important nuance in this quiz, where the precise implications/meanings of knowledge vs ideology are a critical part of the study.
OK, well, here they expressed disagreement with stuff that is not just the scientific consensus, but the overwhelming agreement. If you want to phrase it as disagreeing rather than knowledge being lacking-- I think this is unhelpful. I do not think the nuance you're driving at here is worth complicating the questions or instructions.
> because there were clear exceptions such as e.g. climate change
No.
All 7 areas of controversy did worse on the entire test. It was not statistically significant for climate change and evolution. It was highly statistically significant for 4 of the remaining 5.
6 of the 7 did worse on their on subscales (significantly significant for 4 of these 6), with effect sizes of -.83, -.65, -.28, -.82, -.88, -.55; climate change had a non-statistically significant finding of an effect with slope .03. I do not consider this a counterexample.
You've been really advocating for study power and significance of results... when it favors your argument.
Argumentum ad populum my friend.
It is absolutely worth making the questions more clear, because this is the entire point of the study. Do people who do not agree with the consensus on something lack knowledge, or do simply not agree with said knowledge? Conflating the two in your own study does nothing but set yourself up to observe a tautology (those who disagree with a consensus, disagree with a consensus), which a cynic may interpret as not entirely accidental.
And no, I'm not referencing statistical power. If I haven't made myself clear, I think this study and their numbers, are both going to fall well into the endless black hole that is the replication crisis, which is especially pronounced in social psychology - which has an aggregate replication success rate in the 20s. What I am saying is that you can't make your broad claim based even what I suspect are deeply massaged figures.
> Argumentum ad populum my friend.
Sorry, no. Overwhelming agreement along with readily self-observable characteristics is good enough. You can claim that the kidney is actually the largest organ of the human body, but everyone else agreeing is good enough-- especially when we can look at pictures from other people we trust, arrange to check a cadaver ourselves if we really care, etc.
I mean, I guess it's possible Big Skin (tm) has rigged all the measures and faked everything. /s
> What I am saying is that you can't make your broad claim based even what I suspect are deeply massaged figures.
One submeasure on one subpopulation having an outcome that does not support the claim but is also not inconsistent with that claim doesn't invalidate the claim.
You are talking about:
* A non-statistically significant finding
* Showing an opposite slope, but of tiny magnitude compared to the slopes in the other direction
* On one subpopulation
* On one submeasure.
When Galileo made his case for a heliocentric universe, there was no silver bullet he offered. He simply personally felt it to be more probable than the geocentric model in spite the overwhelming evidence to the contrary given the evidence of the time. Would you thus claim he was lacked knowledge (of the geocentric universe) if he chose to respond false on a question "True/False Everything in the universe revolves around the Earth?" Measuring agreement is different than measuring knowledge.
As for your hypothesis, what would you propose would be a null hypothesis to test your hypothesis? It seems to me that it would be "Somebody who opposes a scientific consensus will score about the same in a test of general knowledge as somebody who supports it." And the climate change example confirms that null hypothesis, which would thus reject your hypothesis.
I'm getting less and less interested in galloping around talking about different minutiae. I basically disagree with your entire argument. But:
> And the climate change example confirms that null hypothesis, which would thus reject your hypothesis.
???! fails to reject the null hypothesis. This seems to be a rather large and fundamental error. You don't ever "confirm[ the] null hypothesis"-- and you certainly don't in a way that lets you exclude other explanations.
e.g. I flip a coin twice. It is heads once and tails once. I perform a statistical test and this outcome is consistent with random chance. In no way does this let me reject the idea that the coin is unfair.
(Indeed, doing a trial of 4 flips and getting 4 heads in a row is consistent with random chance; such a trial would not be likely to increase my belief that the coin is fair, even if it is not strong statistical evidence that the coin is unfair).
Back to our case: we have a whole lot of strong results in one direction, and one ambiguous near-zero result in one submeasure in one subpopulation. Perhaps that subpopulation really is different. It's an interesting thing to measure again. But if you measure enough things, you can expect to find some of these in your result set purely by chance.
You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan. You now cannot say swans are universally white. All it takes is a single exception to break a rule.
This is one of the many issues I have with social science - lack of falsifiability. Social science effectively lacks any degree of falsifiability simply because everything is based on vague probabilistic distributions which vary dramatically based on subtle nuances like asking an identical question two different ways, or testing on different populations in cases where the hypothesis is suggested to generalize - as here.
If there was any notion of falsification, it would all be false. In many ways it's quite analogous to astrology which, if you are unaware, was indeed considered a science, studied in universities for hundreds of years. It's end, somewhat ironically, came not from scientific objection to the fact it suffered the same issues as the social sciences today, but from the Church who deemed it heretical in claims of defacto divination.
> You look at a billion and one swans. For the first billion you see nothing but white swans, yet for the final one you look at you see a black swan.
This analogy doesn't fit our circumstance.
You take two photos each of 7 swans, a close-up and a wide-angle one.
For all of the 7 wide-angle photos, the swan is obviously white.
For the 7 close up-photos: you get 4 good close-up photos of white swan. Then there's 3 bad photos. 2 of them look probably white, and 1 looks like the swan is maybe dark grey.
All the evidence you have is consistent with all the swans being white. Looking to see if you can find those 3 swans and take a better close look at them might be a nice bit of followup.
So you have 7 photos, only 4 of them clearly confirm what you want to say, 1 likely contradicting it, and you're happy to take this as a universal? I also think your analogy doesn't hold for another reason. Look at the last study.
This study is intentionally juking their numbers. They "learned" from their first study and removed every single sample that contradicted what they want to 'prove', magnified those that confirmed it, and even under this form of "science" - they still failed to show statistical significance for multiple categories.
It'd be like noticing there were far "dark" swans in Eastland and so going out of your way to make sure in your next safari you limit yourself to Westland. If you genuinely believed all swans were white, and wanted to test that hypothesis, you'd have done the exact opposite. There is only one conclusion to make.
> So you have 7 photos, only 4 of them clearly confirm what you want to say, 1 likely contradicting it, and you're happy to take this as a universal? I also think your analogy doesn't hold for another reason
I'm sorry; I disagree and I don't think you understand the statistics.
> Look at the last study.
This is another new goalpost. I am not willing to discuss further.
Imagine for a moment that this study was trying to demonstrate that e.g. climate change was primarily caused by solar variation a la Willie Wei-Hock Soon [1]. And he provided the exact same quality of evidence, and the exact same "issues." There is zero chance you would now be claiming this to be a "universal" in spite of the exact same quality, or lack thereof, of evidence.
I was not shifting any goalposts, but simply referencing more evidence on the fundamental issue: that this study was intentionally biased and misleading with the goal of "proving", by any means necessary, a conclusion of which the authors had 0 interest in genuinely testing.
As I suspect we'll be wrapping up, I simply want to thank you for the interesting and lively discussion on this issue. It's rare two people can disagree on the internet for more than a few posts without things devolving into 'You're a poopy face.' 'No, you!'
[1] - https://en.wikipedia.org/wiki/Willie_Soon https://en.wikipedia.org/wiki/Willie_Soon
> Imagine for a moment that this study was trying to demonstrate that e.g. climate change was primarily caused by solar variation a la Willie Wei-Hock Soon [1]. And he provided the exact same quality of evidence, and the exact same "issues."
I suspect we don't agree on what the "exact same quality of evidence" would be.
But the prior probability matters, too. A trial where a die looks slightly-biased-towards-6 (with weak significance) after 3 prior trials where it looked strongly-biased-towards-1 (with high significance) doesn't do much to change our mind about the previous findings.
Base rate fallacy, yada yada yada.