3 ms·
So... This is a bad study, top to bottom. Section 3 is based off a single university's final grades. - There's no discussion of whether assignment grades were
by Bartweiss 10y ago
So... This is a bad study, top to bottom.
Section 3 is based off a single university's final grades.
- There's no discussion of whether assignment grades were being curved before going into the final score (they are in at least some universities). So it's entirely possible that they took normalized data and found that it had normal distributions.
- Even if it's raw data, bimodality would appear per-assignment. Averaging problem sets with papers with tests, and assignments of varying difficulty, threatens to obscure any task-level bimodality.
- Finally, the paper acknowledges the risk of university-specific effects, but can't actually adjust for them. So Section 3 may not generalize at all.
Section 4 is unconvincing in several ways.
- The participant selection was open solicitation from multiple forums, with a high dropout rate after providing tests. That screams selection bias, but isn't acknowledged.
- "Assess this constructed data based on an inaccurate origin" threatens to carry over a prior of "all my real grades are actually bimodal" to the sample data. This would produce the observed results regardless of whether professors are misjudging normal data. This risk would be irrelevant if grades aren't bimodal, but Section 3 didn't sell me on that.
- "Assess this data for bimodality" threatens to prime seeing bimodality for any set of data. This renders it incomparable to unprompted observations about grades.
Supporting Literature is a disaster. It's a hit parade of papers and topics which have or may yet fail to replicate. The male/female candidates paper has a sibling which found the inverse result. The weapon bias study hasn't been found predictive outside the original study. The "brilliance-requiring disciplines" literature is such a disaster that there are multi-thousand-word essays breaking down why it's unconvincing.
I'd say that's not the author's fault, but this is dated September 2016. The male/female candidates result at least should have come with an acknowledgement that other studies have found complete different outcomes.
In summary: The data analysis here is fragile and unconvincing. The human studies misuse unproven effects to draw unsupported conclusions. The supporting literature grounds the preceding mess on other studies which were suspect by the time the paper was written.