3 ms·

There's a "risk" of bias here? The point seems to be that the professors don't have a good prior -- they have a bias and are confusing it for a prior, and that'

There's a "risk" of bias here? The point seems to be that the professors don't have a good prior -- they have a bias and are confusing it for a prior, and that's what was demonstrated by the experiment. (that, and perhaps CS professors aren't particularly good at statistics)

Bartweiss 10y ago
I disagree with the "bias not prior" claim. The professors were shown constructed data, and primed to apply a prior to it about grades. Prompting someone to use a prior inappropriately isn't the same as revealing a bias. I see what you're getting at (if they wrongly assessed these curves, why would they be right about real data?), but there's a connective step that seems missing. If you show me a histogram and ask if it's bimodal, I can't tell you with certainty, so I have to guess. That's partly by shape, and partly by my knowledge of what the data is. Misjudging an actively-misleading histogram doesn't prove that I'm bad at assessing real data. (Since they just said 'yes' or 'no', I wouldn't accuse them of being bad at statistics - they didn't computer an answer.) So even if real CS grades are bimodal, I would have expected the 'priming' result they found here. The professors were wrong, but this doesn't seem to have much predictive value. More importantly, though, this whole study is bad. Section 3 is based off a single university's final grades, with no discussion of whether assignment grades were being curved before going into the final score (they are in at least some universities). So it's entirely possible that they took normalized data and found that it had normal distributions. Section 4 is unconvincing in several ways. "Assess this constructed data, maybe we lied about the origin" isn't a direct analogue to "are your actual grades bimodal?". "Assess this data for bimodality" threatens a priming effect that's incomparable to unprompted observations about grades. Priming effects are struggling to replicate, and may simply not work the way these researchers expected at all. The study does an admirable job of acknowledging a lot of this, like the researcher-priming risk, but it doesn't actually remedy any of them. So I'm not sure what to say except "any of these methods could be meaningless".

Bartweiss 10y ago

I disagree with the "bias not prior" claim. The professors were shown constructed data, and primed to apply a prior to it about grades. Prompting someone to use a prior inappropriately isn't the same as revealing a bias. I see what you're getting at (if they wrongly assessed these curves, why would they be right about real data?), but there's a connective step that seems missing. If you show me a histogram and ask if it's bimodal, I can't tell you with certainty, so I have to guess. That's partly by shape, and partly by my knowledge of what the data is. Misjudging an actively-misleading histogram doesn't prove that I'm bad at assessing real data. (Since they just said 'yes' or 'no', I wouldn't accuse them of being bad at statistics - they didn't computer an answer.) So even if real CS grades are bimodal, I would have expected the 'priming' result they found here. The professors were wrong, but this doesn't seem to have much predictive value. More importantly, though, this whole study is bad. Section 3 is based off a single university's final grades, with no discussion of whether assignment grades were being curved before going into the final score (they are in at least some universities). So it's entirely possible that they took normalized data and found that it had normal distributions. Section 4 is unconvincing in several ways. "Assess this constructed data, maybe we lied about the origin" isn't a direct analogue to "are your actual grades bimodal?". "Assess this data for bimodality" threatens a priming effect that's incomparable to unprompted observations about grades. Priming effects are struggling to replicate, and may simply not work the way these researchers expected at all. The study does an admirable job of acknowledging a lot of this, like the researcher-priming risk, but it doesn't actually remedy any of them. So I'm not sure what to say except "any of these methods could be meaningless".