3 ms·

I'm less impressed by "wrong about rigged data" results than most people seem to be. Yes, it implies that people bring assumptions to their answer, but is that

I'm less impressed by "wrong about rigged data" results than most people seem to be. Yes, it implies that people bring assumptions to their answer, but is that really shocking? People with a high prior for a thing assume that new data is most likely to conform to that prior. If you handed me a questionably-bimodal data set, my judgement of its distribution would absolutely depend on what you told me the data represented. Hopefully I'd answer right if I sat down and analyzed the thing in depth, but if you simply go "what kind of distribution does this look like?" then I'm going to include my outside expectations. Yeah, there's a risk of bias here, largely from people not conserving expectations (if you already assume Geek Gene is true, unfavorable data should count more than favorable data). But saying "people used experience to interpret new data!" doesn't seem like it's proving much.

elcritch 10y ago
Great point. To go further, I'd argue biases are crucial for efficient decision making as biases are a form human judgement essentially boiled down to priors and used to make complex decisions quickly based on limited data. Without effective biases we'd all probably starve to death arguing. Of course the fatal flaw (the modern age?), is an unwillingness to both critique biases while not outright dismissing them just for being biases.
ubernostrum 10y ago
There's a "risk" of bias here? The point seems to be that the professors don't have a good prior -- they have a bias and are confusing it for a prior, and that's what was demonstrated by the experiment. (that, and perhaps CS professors aren't particularly good at statistics)
- Bartweiss 10y ago
  I disagree with the "bias not prior" claim. The professors were shown constructed data, and primed to apply a prior to it about grades. Prompting someone to use a prior inappropriately isn't the same as revealing a bias. I see what you're getting at (if they wrongly assessed these curves, why would they be right about real data?), but there's a connective step that seems missing. If you show me a histogram and ask if it's bimodal, I can't tell you with certainty, so I have to guess. That's partly by shape, and partly by my knowledge of what the data is. Misjudging an actively-misleading histogram doesn't prove that I'm bad at assessing real data. (Since they just said 'yes' or 'no', I wouldn't accuse them of being bad at statistics - they didn't computer an answer.) So even if real CS grades are bimodal, I would have expected the 'priming' result they found here. The professors were wrong, but this doesn't seem to have much predictive value. More importantly, though, this whole study is bad. Section 3 is based off a single university's final grades, with no discussion of whether assignment grades were being curved before going into the final score (they are in at least some universities). So it's entirely possible that they took normalized data and found that it had normal distributions. Section 4 is unconvincing in several ways. "Assess this constructed data, maybe we lied about the origin" isn't a direct analogue to "are your actual grades bimodal?". "Assess this data for bimodality" threatens a priming effect that's incomparable to unprompted observations about grades. Priming effects are struggling to replicate, and may simply not work the way these researchers expected at all. The study does an admirable job of acknowledging a lot of this, like the researcher-priming risk, but it doesn't actually remedy any of them. So I'm not sure what to say except "any of these methods could be meaningless".

elcritch 10y ago

Great point. To go further, I'd argue biases are crucial for efficient decision making as biases are a form human judgement essentially boiled down to priors and used to make complex decisions quickly based on limited data. Without effective biases we'd all probably starve to death arguing. Of course the fatal flaw (the modern age?), is an unwillingness to both critique biases while not outright dismissing them just for being biases.

ubernostrum 10y ago

There's a "risk" of bias here? The point seems to be that the professors don't have a good prior -- they have a bias and are confusing it for a prior, and that's what was demonstrated by the experiment. (that, and perhaps CS professors aren't particularly good at statistics)

Bartweiss 10y ago
I disagree with the "bias not prior" claim. The professors were shown constructed data, and primed to apply a prior to it about grades. Prompting someone to use a prior inappropriately isn't the same as revealing a bias. I see what you're getting at (if they wrongly assessed these curves, why would they be right about real data?), but there's a connective step that seems missing. If you show me a histogram and ask if it's bimodal, I can't tell you with certainty, so I have to guess. That's partly by shape, and partly by my knowledge of what the data is. Misjudging an actively-misleading histogram doesn't prove that I'm bad at assessing real data. (Since they just said 'yes' or 'no', I wouldn't accuse them of being bad at statistics - they didn't computer an answer.) So even if real CS grades are bimodal, I would have expected the 'priming' result they found here. The professors were wrong, but this doesn't seem to have much predictive value. More importantly, though, this whole study is bad. Section 3 is based off a single university's final grades, with no discussion of whether assignment grades were being curved before going into the final score (they are in at least some universities). So it's entirely possible that they took normalized data and found that it had normal distributions. Section 4 is unconvincing in several ways. "Assess this constructed data, maybe we lied about the origin" isn't a direct analogue to "are your actual grades bimodal?". "Assess this data for bimodality" threatens a priming effect that's incomparable to unprompted observations about grades. Priming effects are struggling to replicate, and may simply not work the way these researchers expected at all. The study does an admirable job of acknowledging a lot of this, like the researcher-priming risk, but it doesn't actually remedy any of them. So I'm not sure what to say except "any of these methods could be meaningless".