13 ms·

Evidence That Computer Science Grades Are Not Bimodal [pdf]

146 points by moyix 10y ago

nerdponx 10y ago
Aren't final grades often "curved", i.e. re-graded according to rank? That would explicitly destroy bimodality, making the final grade distribution consist of non-iid order statistics. Also, it's commendable that they point out that they expect 5% false positives, but frustrating that they don't go further and explicitly plan for a multiple testing correction procedure. It seems that they don't need it to fail to reject the null "meta-hypothesis", but still.
- mcguire 10y ago
  The curving process should be linear, I think.
  - tamana 10y ago
    That sounds funny, but it's true under the common modern curving regime that is really just lowering the bar for raw score to letter grade conversion, not forcing a bell curve.
    - turnip1979 10y ago
      Square root of grade times 10 :-p
  - matteotom 10y ago
    I've seen a few different curving processes: 1. Formula, like square root of grade times 10, that's designed to bring up grades. There're a lot of possibilities here 2. Top x% get A's, next y% get B's, etc 3. Look for clusters: top cluster gets A's, next B's, etc These can also be applied to classes as a whole or individual assignments and tests
  - BeetleB 10y ago
    The curving process shouldn't exist. If you make an exam that is difficult enough to need curving, you're getting a poor measure of ability. This is because exams ask only a few questions, and unreasonably difficult exams result in low scores even from high achieving students. Low scores are very susceptible to noise (the delta between 50% and 60% is greater than between 85% and 95%). If that doesn't convince you, take my argument to the obvious extreme. The Putnam Competition in mathematics is tough. Sometimes over half of the people score 0. Getting 1 question correct (out of 12) at times puts you in the top 20%. Imagine I gave this as an exam to a math class of 20 students. One person scores a 1. The rest score 0. Is it meaningful to curve this? I could correct by giving partial credit: Some people get 0.25, others 0.5, and others 0.75, so we now have 5 different grades. Should I just give A, B, C, D and F? The lower the scores, the higher the effect of noise. It's a bad idea.
    - smallnamespace 10y ago
      > Low scores are very susceptible to noise (the delta between 50% and 60% is greater than between 85% and 95%) This seems untrue from information theory. The entropy of a question is maximized when its probability of being answered correctly is exactly 50% [1]. If your only goal is to have the least amount of measurement noise given a fixed number of questions, then you'll want each question to be hard enough to filter 50% of the class out, and to minimize the correlation between questions. For example, 10 independent 50% questions reveals as much information as 16 independent 85% questions. > Imagine I gave this as an exam to a math class of 20 students. One person scores a 1. The rest score 0. Is it meaningful to curve this? You're looking at only 1 tail but ignoring the other. By symmetry, a question that only 1 person gets correct tells you exactly as much as a question that 19 get correct. An exam with a 90% pass rate (before curving) is no better than a 10% exam. [1] https://en.wikipedia.org/wiki/Binary_entropy_function https://en.wikipedia.org/wiki/Binary_entropy_function
      - mcguire 10y ago
        Thanks. I'm now having flashbacks to the summer of 1990, when I worked on a Education professor's research project, creating a math test in a Hypercard stack that presented the question that provided the most information about the testee's abilities, given their previous answers.
  - Bartweiss 10y ago
    But I've seen a lot of courses where it isn't. Among the many techniques I've seen used: - Raw average, then set letter cutoffs for a normal distribution across students. Arbitrary on distribution, normal on letter. - "Skew high" by counting missed points more weakly as you miss more. Inflated letters with a score-accurate relative distribution. - "Fixed and curved", where each cutoff is based on the more (or less!) generous of standard deviations and 'conventional' grade brackets: the floor for A is either 92%+ or +2 SD from mean, B is 83%+ or +1 SD, etc. Bizarre and distorted everything, since the relevant cap is determined by the local distribution. - God knows what else. So honestly, I think post-curve data is a hopelessly course-specific, nonlinear mess.
- Bartweiss 10y ago
  Not only are they curved, different courses are 'curved' according to vastly different rules. Some curve percentages, some curve only letters, and different distributions are enforced for different classes. So if this is post-curve data, I would expect it to be basically worthless as a reflection of skill distribution.
  - JoeAltmaier 10y ago
    Yeah, lots of ways to fudge. One exam, my friend Brian and I were allowed to skip the final and take our grade. So the rest of the class could go from failing, to getting a grade on a curve. There was that much gap between us and the rest.
  - spike021 10y ago
    My Operating Systems course had a really odd grade curve. There was a listed one in the syllabus, but then the exams were so difficult that nobody was scoring above 80-85 out of 100, so there had to be an additional full grade curve after that. So yeah, post-curve isn't helpful. Unless there's some way to normalize the data.
mcguire 10y ago
"Are CS grades bimodal, or unimodal? To test this, we ac- quired the final grades distributions for every undergraduate CS class at the University of British Columbia (UBC), from 1996 to 2013. This represents 778 different lecture sections, containing a total of 30,214 final grades (average class size: 75)." My understanding of the bimodal situation, if it exists, is that it primarily applies to earlier classes---later classes only include those who did well, or at least passed, the previous classes. Ah, yes... "Of the 45 classes which were multimodal, 16 were 100- level classes (35%), 5 were 200-level (11%), 12 were 300-level (27%), and 12 were 400-level (27%). For comparison, in the full set of 778 classes, 171 were 100-level (22%), 165 were 200-level (21%), 243 were 300-level (31%), and 199 were 400-level (26%)." How about we take a closer look at those 100 level classes, hmmm?
- nerdponx 10y ago
  I remember my real analysis professor explicitly telling us that he ran the numbers on one of our exams and found bimodality. Real analysis is not what I would call an earlier class. I'm not sure if we were an outlier or a representative data point, however.
  - trendia 10y ago
    If there are two major topics in a test (say Topic 1 and Topic 2), then bimodality could arise simply because * some people studied both Topic 1 and Topic 2 * some people studied Topic 1 more than Topic 2 * some people studied Topic 2 more than Topic 1 This would cause bimodality even when inherent "genetic" skill is the same.
    - sqeaky 10y ago
      This seems unlikely across multiple students and years (though it is possible so more than one schoool should be studied). You comment also presumes there are no students who can study well enough to pass all exams. Considering the ease of 100 level courses CS courses acing that and another 100 level is not exactly hard.
    - 10y ago
JoeAltmaier 10y ago
A class (30?) students is too small to produce a curve which is unambiguously either normal or bimodal. Its always noisy. So if the CS profs, with their thousands of hours of experience with students, chose the bimodal interpretation I for one tend to believe it.
deleted 10y ago
[deleted]
yodsanklai 10y ago
"We theorized that the perception of bimodal grades in CS is a social defense. It is easier for the CS education community to believe that some students “have it” and others do not than it is for the community to come to terms with the shortfalls of our pedagogical approaches and assessment tools." I didn't read the whole article, but I'm not sure I agree with the conclusion. Even with a standard gaussian distribution, one may still believe that some students have it and some don't. Let say there is consistently 25% of students that are unable to pass their exams (for various reasons: lack of interest, motivation, intelligence, discipline...), why would a "social defense" be needed? While I agree that we always should try to improve our pedagogical tools, I don't believe that anyone can learn anything provided they have a good teacher.
- throwaway729 10y ago
  Read section 4 of the paper. They fix some data that as a matter of mathematical fact is/is not bimodal, and then ask people who do/don't believe in the "geek gene" hypotehsis to interpret the data. Conclusion of the section: "We found a statistically significant relationship between seeing-bimodality and participants’ responses to the questions relating to the Geek Gene hypothesis" So the "social defense mechanism" theory is definitely debatable, but apparently people who believe in the "Geek Gene hypothesis" are more likely to see bimodality where there is none. > I don't believe that anyone can learn anything provided they have a good teacher. The point isn't that everyone can learn anything. The point is just that the distribution isn't as bimodal as people seem to think it is. Also worth noting that the researchers posit normal distrbutions, not uniform distributions, as the alternative to bimodality.
  - Bartweiss 10y ago
    I'm less impressed by "wrong about rigged data" results than most people seem to be. Yes, it implies that people bring assumptions to their answer, but is that really shocking? People with a high prior for a thing assume that new data is most likely to conform to that prior. If you handed me a questionably-bimodal data set, my judgement of its distribution would absolutely depend on what you told me the data represented. Hopefully I'd answer right if I sat down and analyzed the thing in depth, but if you simply go "what kind of distribution does this look like?" then I'm going to include my outside expectations. Yeah, there's a risk of bias here, largely from people not conserving expectations (if you already assume Geek Gene is true, unfavorable data should count more than favorable data). But saying "people used experience to interpret new data!" doesn't seem like it's proving much.
groundhogday1 10y ago
So the assumption being made here is that students either get it or they don't based on genetic predisposition? I, for one, don't particularly subscribe to the notion that every CS student needs to score an INTP on the Meyers-Briggs to be successful.
- deleted 10y ago
  [deleted]
- Practicality 10y ago
  ENTP makes a better developer anyway. :D (In case anybody doesn't get the joke, ENTP is the "debater" type, so I am picking a fight about personality type, since that is what the ENTP is supposed to do, so self-referential humor: https://www.16personalities.com/entp-personality https://www.16personalities.com/entp-personality)
losteverything 10y ago
"Your scores are bimodal!!!!" trying to get the "so what?" Does that say something (e.g ,bad instructor) or does it support something ( idea that 50% of students should not even be taking course, for example) Organic Chem class was bimodal which led to dismissal of instructor. Different instructor meant better learning and a different distribution.
moyix 10y ago
Folks may also be interested in the accompanying blog post by the first author: http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-normal-than-you.html http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-...
Animats 10y ago
Programming tests amplify modality because they require success on multiple items to succeed. Consider teaching N skills. Success is binary, uncorrelated and everyone randomly succeeds some high percentage P on each skill. Scoring is the number of binary successes. You'll get a Gaussian distribution. That's most of education. Now suppose scoring is 1 if all N skills are learned. You'll get a bimodal distribution. That's programming tested on whether the program runs.
- aidenn0 10y ago
  There are two links discussing that in a blog post by the author of this article[1] (second to last paragraph) 1: http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-normal-than-you.html http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-...
  - Animats 10y ago
    Sadly, they're both paywalled.
deleted 10y ago
[deleted]
lifeisstillgood 10y ago
I am not sure we should assume bimodality in programming means we are all special snowflakes. That's a dangerous and self serving interpretation It seems bimodality is correlated with "does it even compile" - the first hump is those who can't get it to compile and the second is those who can but then spread out normally on ability. I would conjecture that the first hump would be seen in any educational environment where, for example we took illiterates and made them write essays. Those who had tried reading and writing in high school would have a better chance of putting ink on paper. We just are seeing an artifact of software not being taught early enough in everyone. Edit - how does my spelling corrector turn bimodality into bumps skirt...
htns 10y ago
They don't consider participation or hours spent studying as explanations for bimodality. This makes their method of determining belief in the "geek gene" questionable, and could invert the moral story of their results: Perhaps lecturers seeing low lecture attendance would like to believe their grade distributions are bimodal, but the data shows the lecturers' impact is at best a tiny factor among many, and grades are determined by factors out of the lecturers' control, such as students' ability.
scott00 10y ago
The whole premise of the survey part of this paper strikes me as very flawed. The idea was "show some professors histograms we know are normal, and see if they see bimodality that isn't there". The problem is, they didn't show them 6 random histograms of the normal distribution. For one, they didn't actually show them normal distributions at all; they showed them normal distributions capped at 100. That's going to give you a point mass at 100... making it a bimodal distribution! Second, they didn't give 6 random examples of a capped normal histogram; they gave them 6 examples chosen from a random set of histograms chosen in such a way that 4 of the 6 look bimodal. The survey participants didn't see bimodality that wasn't there, they saw bimodality that was purposefully generated by the histogram selection methodology!
makmanalp 10y ago
I get especially mad when people suggest that there are some ideas that are just special, that only special people can get. Pointers and recursion are at the top of this list. Take Joel Spolsky, a person whose ideas I read and respect: http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html http://www.joelonsoftware.com/articles/GuerrillaInterviewing... > For some reason most people seem to be born without the part of the brain that understands pointers. Pointers require a complex form of doubly-indirected thinking that some people just can’t do Really? Here's my 20 dollar challenge: give me a person and I can teach them pointers. Pointers, referencing and dereferencing, null pointers, pointers to pointers, the whole deal. The problem is the pedagogy. Learning a new abstraction and a new way of thinking takes time. It's easy to get it but really not /get/ it. You have to do a lot of examples. One on one with a person, with real time feedback. Until they internalize it. Many many many simple examples at first, and then examples of using it in a real world context. Then more. If you already know this abstraction, consciously or unconsciously, it's impossible to un-know it, and the reaction when someone is struggling is "I explained it as simply as I can and I don't know why they still don't get it". It's not like that. It's like basketball: you can explain how to shoot, but really the student needs to do it, and you comment while they're doing it.
- Eliezer 10y ago
  Some people get pointers immediately after one exposure before puberty. Not every student needs to practice. Some people, who maybe never came to your university at all, will never get linear algebra matter how much they practice. To say the distribution is unimodal is one claim. It's another matter to claim the distribution is narrow.
- ap22213 10y ago
  Absolutely - I'm sure that a bit of innate talent can improve one's rate of learning, but 90% of the willing can do it. The best programmers that I've met just loved the art and science of computing, had respect for their craft and very high standards of quality, were undaunted or even motivated by new challenges, but most importantly, put in the necessary time. I doubt things have changed much since I went to college in the 90s, but back then, the ones that did well at CS either had been hacking computers since the Apple II or spending enormous amounts of time at the lab (or both). Also, when at the Microsoft campus in the 90s, it was not a coincidence that the most expensive cars were the ones that were in the parking garage the latest.
k__ 10y ago
I don't know much about this, but I wasn't good at studying. I'm totally living for development and everything to do with it, but most of the formal stuff completely eluded me. I failed every math lecture at least once. Because of my projects and thesis I still finished with a 2,7 which isn't good but also not really bad. Maybe I would have been one of those "college-dropouts who followed their dream" if companies in Germany didn't look so much on your degrees. Also I never had the impression our grades were bimodal. Yes about 40% dopped out and yes there were a few "eager beavers", but most were simply okay enough to finish their degrees.
- groundhogday1 10y ago
  I'm about the opposite of you. Not a good programmer, but I excelled at the 'formal stuff.' Automata theory, regular expressions, compilers, operating systems, discrete math... these are all areas where I did well. Labs and programming assignments held me back. Although I will say that I feel infinitely more comfortable working with memory allocation in C than I do with anything involving JavaScript. I also don't think the overall grades were bimodal. I guess I was in the second mode in programming intense courses and the first mode in theory heavy courses. I also feel that most of my peers were either the same as me or the reverse, like you. Wouldn't that create a single mode bell curve? Of course, this is all purely anecdotal
hyperion2010 10y ago
Having worked with many of my fellow graduate students to try to teach CS concepts to incoming students without any CS background my conclusion here is that we simply don't know how to teach computer science. My suspicion is that many people who learn CS usually do it inspite of attempts to teach them, not because of instruction. Think about how long it took humans to figure out how to teach reading to _everyone_. We had to break the old whole reading method and switch over to phonics, and that was decades after we had the hard scientific evidence that phonics was the missing perspective needed to get that last percentage of students to read. We are probably decades away from understanding how to teach CS to everyone, let alone getting those techniques implemented.
- GregBuchholz 10y ago
  >We had to break the old whole reading method and switch over to phonics, and that was decades after we had the hard scientific evidence that phonics was the missing perspective needed to get that last percentage of students to read. https://en.wikipedia.org/wiki/Phoenician_alphabet#Spread_of_the_alphabet_and_its_social_effects https://en.wikipedia.org/wiki/Phoenician_alphabet#Spread_of_...
  - taeric 10y ago
    I'm not clear on what the link is supposed to be getting me. Phonetics and Phoenician are two different things. Right?
    - jdmichal 10y ago
      I agree that the post is severely underspecified. I think the point is the last paragraph of the linked section: > Phoenician had long-term effects on the social structures of the civilizations which came in contact with it. As mentioned above, the script was the first widespread phonetic script. Its simplicity not only allowed it to be used in multiple languages, but it also allowed the common people to learn how to write. This upset the long-standing status of writing systems only being learned and employed by members of the royal and religious hierarchies of society, who used writing as an instrument of power to control access to information by the larger population.[11] The appearance of Phoenician disintegrated many of these class divisions, although many Middle Eastern kingdoms such as Assyria, Babylonia and Adiabene would continue to use cuneiform for legal and liturgical matters well into the Common Era. That is, Phoenician was the first wide-spread phonetic script. That, in turn, had major effects on civilizations as it was simple enough for commoners to learn. That being the rather tenuous link back to the original idea of teaching reading based on phonetics. Of course, there is a bit of a gap here, in that English phonetics are notoriously unstandardized across the lexicon.
cdevs 10y ago
I hated trying to read about pointers in c when I was about 15, a few years later a different book put it more clear and it all snapped into place for me.
a_puppy 10y ago
This is mostly unrelated to the main topic of the article, but this paragraph caught my eye: > For example, Padavic et al. [20] found that the "work-family" narrative in business is an example of a social defense: people will say that women leave the workplace because of "family", despite the large amount of evidence that women leave their jobs because of inadequate pay or opportunities for advancement [20], particularly when they see male co-workers promoted ahead of them. The "work-family" narrative is a more palatable explanation rather than to confront sexual discrimination in the workplace, and so the narrative continues. > > [20] I. Padavic and R. J. Ely. The work-family narrative as a social defense, 2013. I tried to track down the work by Padavic et al. that Patitsas et al. are citing, and found this: http://www.hbs.edu/faculty/conferences/2013-w50-research-symposium/Documents/Gender_and_work_web_update2015.pdf http://www.hbs.edu/faculty/conferences/2013-w50-research-sym... However, this article by Padavic et al. is actually saying something different from the way that Patitsas et al. summarized it. Padavic et al. appear to actually be arguing that the work-family narrative is a social defense against the uncomfortable truth of excessively long working hours for both genders at the corporation, not against any uncomfortable truth about sexual discrimination. Also, Padavic et al. don't seem to discuss sexual discrimination in wages or promotions at all. Did Patitsas et al. make an error in citing their sources? Am I missing something?
- danieltillett 10y ago
  Mis-citation occurs all the time. When I was a professional scientist I could only find the source for a claim I was interested in less than 25% of the time. I would find some claim and about 50% of the time the reference cited another reference and so on until the chain broke. The other 25% of the time the reference cited said something completely different to what was being claimed. Very frustrating.
  - tjl 10y ago
    During my Ph.D. I tried tracking down the reasoning behind a couple sets of assumptions made in non-uniform torsion of beams (e.g., twisting a beam with one end fixed). It turns out that they just kept the same assumptions from uniform torsion (just twisting the beam but with no ends fixed) without checking if they still held (they don't). All because nobody thought to check if the assumptions might be in conflict. The uniform torsion models go back to the mid 1800s, while the non-uniform torsion models are from the early to mid 20th century. So, it's been over 60 years people have been working with wrong assumptions.
rcheu 10y ago
I've never heard that computer science grades are bimodal, I thought it was that performance in programming is bimodal? It's always been relatively obvious to me that grades were not bimodal since I could just click and see the grade distribution, which didn't appear to be bimodal.
toomanybeersies 10y ago
Oh, useful anecdote time! I was talking to a professor a while back about this, and he reckoned that when the University of Canterbury (in NZ) switched from Java to Python for their 100 level papers, the grades went from bimodal to single-modal. Not sure why that is, and I don't remember him giving me a reason, but I guess it's fair to assume that a lot of students were really struggling with the initial learning curve of Java (boilerplate, static typing, more difficult array and string manipulation) compared to Python, and either just not getting it, or getting frustrated and giving up. Now I didn't see any numbers to back up his claims, as it was a casual conversation, but I'd be inclined to believe him that they were bimodal. Anyway, the point I'm making is that the study here references only one university, different universities have different curriculum and different teaching styles, which could affect whether the distribution is bimodal or not.
Bartweiss 10y ago
So... This is a bad study, top to bottom. Section 3 is based off a single university's final grades. - There's no discussion of whether assignment grades were being curved before going into the final score (they are in at least some universities). So it's entirely possible that they took normalized data and found that it had normal distributions. - Even if it's raw data, bimodality would appear per-assignment. Averaging problem sets with papers with tests, and assignments of varying difficulty, threatens to obscure any task-level bimodality. - Finally, the paper acknowledges the risk of university-specific effects, but can't actually adjust for them. So Section 3 may not generalize at all. Section 4 is unconvincing in several ways. - The participant selection was open solicitation from multiple forums, with a high dropout rate after providing tests. That screams selection bias, but isn't acknowledged. - "Assess this constructed data based on an inaccurate origin" threatens to carry over a prior of "all my real grades are actually bimodal" to the sample data. This would produce the observed results regardless of whether professors are misjudging normal data. This risk would be irrelevant if grades aren't bimodal, but Section 3 didn't sell me on that. - "Assess this data for bimodality" threatens to prime seeing bimodality for any set of data. This renders it incomparable to unprompted observations about grades. Supporting Literature is a disaster. It's a hit parade of papers and topics which have or may yet fail to replicate. The male/female candidates paper has a sibling which found the inverse result. The weapon bias study hasn't been found predictive outside the original study. The "brilliance-requiring disciplines" literature is such a disaster that there are multi-thousand-word essays breaking down why it's unconvincing. I'd say that's not the author's fault, but this is dated September 2016. The male/female candidates result at least should have come with an acknowledgement that other studies have found complete different outcomes. In summary: The data analysis here is fragile and unconvincing. The human studies misuse unproven effects to draw unsupported conclusions. The supporting literature grounds the preceding mess on other studies which were suspect by the time the paper was written.

146 points by moyix 10y ago

Aren't final grades often "curved", i.e. re-graded according to rank? That would explicitly destroy bimodality, making the final grade distribution consist of non-iid order statistics. Also, it's commendable that they point out that they expect 5% false positives, but frustrating that they don't go further and explicitly plan for a multiple testing correction procedure. It seems that they don't need it to fail to reject the null "meta-hypothesis", but still.

mcguire 10y ago
The curving process should be linear, I think.
- tamana 10y ago
  That sounds funny, but it's true under the common modern curving regime that is really just lowering the bar for raw score to letter grade conversion, not forcing a bell curve.
  - turnip1979 10y ago
    Square root of grade times 10 :-p
- matteotom 10y ago
  I've seen a few different curving processes: 1. Formula, like square root of grade times 10, that's designed to bring up grades. There're a lot of possibilities here 2. Top x% get A's, next y% get B's, etc 3. Look for clusters: top cluster gets A's, next B's, etc These can also be applied to classes as a whole or individual assignments and tests
- BeetleB 10y ago
  The curving process shouldn't exist. If you make an exam that is difficult enough to need curving, you're getting a poor measure of ability. This is because exams ask only a few questions, and unreasonably difficult exams result in low scores even from high achieving students. Low scores are very susceptible to noise (the delta between 50% and 60% is greater than between 85% and 95%). If that doesn't convince you, take my argument to the obvious extreme. The Putnam Competition in mathematics is tough. Sometimes over half of the people score 0. Getting 1 question correct (out of 12) at times puts you in the top 20%. Imagine I gave this as an exam to a math class of 20 students. One person scores a 1. The rest score 0. Is it meaningful to curve this? I could correct by giving partial credit: Some people get 0.25, others 0.5, and others 0.75, so we now have 5 different grades. Should I just give A, B, C, D and F? The lower the scores, the higher the effect of noise. It's a bad idea.
  - smallnamespace 10y ago
    > Low scores are very susceptible to noise (the delta between 50% and 60% is greater than between 85% and 95%) This seems untrue from information theory. The entropy of a question is maximized when its probability of being answered correctly is exactly 50% [1]. If your only goal is to have the least amount of measurement noise given a fixed number of questions, then you'll want each question to be hard enough to filter 50% of the class out, and to minimize the correlation between questions. For example, 10 independent 50% questions reveals as much information as 16 independent 85% questions. > Imagine I gave this as an exam to a math class of 20 students. One person scores a 1. The rest score 0. Is it meaningful to curve this? You're looking at only 1 tail but ignoring the other. By symmetry, a question that only 1 person gets correct tells you exactly as much as a question that 19 get correct. An exam with a 90% pass rate (before curving) is no better than a 10% exam. [1] https://en.wikipedia.org/wiki/Binary_entropy_function https://en.wikipedia.org/wiki/Binary_entropy_function
    - mcguire 10y ago
      Thanks. I'm now having flashbacks to the summer of 1990, when I worked on a Education professor's research project, creating a math test in a Hypercard stack that presented the question that provided the most information about the testee's abilities, given their previous answers.
- Bartweiss 10y ago
  But I've seen a lot of courses where it isn't. Among the many techniques I've seen used: - Raw average, then set letter cutoffs for a normal distribution across students. Arbitrary on distribution, normal on letter. - "Skew high" by counting missed points more weakly as you miss more. Inflated letters with a score-accurate relative distribution. - "Fixed and curved", where each cutoff is based on the more (or less!) generous of standard deviations and 'conventional' grade brackets: the floor for A is either 92%+ or +2 SD from mean, B is 83%+ or +1 SD, etc. Bizarre and distorted everything, since the relevant cap is determined by the local distribution. - God knows what else. So honestly, I think post-curve data is a hopelessly course-specific, nonlinear mess.
Bartweiss 10y ago
Not only are they curved, different courses are 'curved' according to vastly different rules. Some curve percentages, some curve only letters, and different distributions are enforced for different classes. So if this is post-curve data, I would expect it to be basically worthless as a reflection of skill distribution.
- JoeAltmaier 10y ago
  Yeah, lots of ways to fudge. One exam, my friend Brian and I were allowed to skip the final and take our grade. So the rest of the class could go from failing, to getting a grade on a curve. There was that much gap between us and the rest.
- spike021 10y ago
  My Operating Systems course had a really odd grade curve. There was a listed one in the syllabus, but then the exams were so difficult that nobody was scoring above 80-85 out of 100, so there had to be an additional full grade curve after that. So yeah, post-curve isn't helpful. Unless there's some way to normalize the data.

mcguire 10y ago

"Are CS grades bimodal, or unimodal? To test this, we ac- quired the final grades distributions for every undergraduate CS class at the University of British Columbia (UBC), from 1996 to 2013. This represents 778 different lecture sections, containing a total of 30,214 final grades (average class size: 75)." My understanding of the bimodal situation, if it exists, is that it primarily applies to earlier classes---later classes only include those who did well, or at least passed, the previous classes. Ah, yes... "Of the 45 classes which were multimodal, 16 were 100- level classes (35%), 5 were 200-level (11%), 12 were 300-level (27%), and 12 were 400-level (27%). For comparison, in the full set of 778 classes, 171 were 100-level (22%), 165 were 200-level (21%), 243 were 300-level (31%), and 199 were 400-level (26%)." How about we take a closer look at those 100 level classes, hmmm?

nerdponx 10y ago
I remember my real analysis professor explicitly telling us that he ran the numbers on one of our exams and found bimodality. Real analysis is not what I would call an earlier class. I'm not sure if we were an outlier or a representative data point, however.
- trendia 10y ago
  If there are two major topics in a test (say Topic 1 and Topic 2), then bimodality could arise simply because * some people studied both Topic 1 and Topic 2 * some people studied Topic 1 more than Topic 2 * some people studied Topic 2 more than Topic 1 This would cause bimodality even when inherent "genetic" skill is the same.
  - sqeaky 10y ago
    This seems unlikely across multiple students and years (though it is possible so more than one schoool should be studied). You comment also presumes there are no students who can study well enough to pass all exams. Considering the ease of 100 level courses CS courses acing that and another 100 level is not exactly hard.
  - 10y ago

JoeAltmaier 10y ago

A class (30?) students is too small to produce a curve which is unambiguously either normal or bimodal. Its always noisy. So if the CS profs, with their thousands of hours of experience with students, chose the bimodal interpretation I for one tend to believe it.

deleted 10y ago

[deleted]

yodsanklai 10y ago

"We theorized that the perception of bimodal grades in CS is a social defense. It is easier for the CS education community to believe that some students “have it” and others do not than it is for the community to come to terms with the shortfalls of our pedagogical approaches and assessment tools." I didn't read the whole article, but I'm not sure I agree with the conclusion. Even with a standard gaussian distribution, one may still believe that some students have it and some don't. Let say there is consistently 25% of students that are unable to pass their exams (for various reasons: lack of interest, motivation, intelligence, discipline...), why would a "social defense" be needed? While I agree that we always should try to improve our pedagogical tools, I don't believe that anyone can learn anything provided they have a good teacher.

throwaway729 10y ago
Read section 4 of the paper. They fix some data that as a matter of mathematical fact is/is not bimodal, and then ask people who do/don't believe in the "geek gene" hypotehsis to interpret the data. Conclusion of the section: "We found a statistically significant relationship between seeing-bimodality and participants’ responses to the questions relating to the Geek Gene hypothesis" So the "social defense mechanism" theory is definitely debatable, but apparently people who believe in the "Geek Gene hypothesis" are more likely to see bimodality where there is none. > I don't believe that anyone can learn anything provided they have a good teacher. The point isn't that everyone can learn anything. The point is just that the distribution isn't as bimodal as people seem to think it is. Also worth noting that the researchers posit normal distrbutions, not uniform distributions, as the alternative to bimodality.
- Bartweiss 10y ago
  I'm less impressed by "wrong about rigged data" results than most people seem to be. Yes, it implies that people bring assumptions to their answer, but is that really shocking? People with a high prior for a thing assume that new data is most likely to conform to that prior. If you handed me a questionably-bimodal data set, my judgement of its distribution would absolutely depend on what you told me the data represented. Hopefully I'd answer right if I sat down and analyzed the thing in depth, but if you simply go "what kind of distribution does this look like?" then I'm going to include my outside expectations. Yeah, there's a risk of bias here, largely from people not conserving expectations (if you already assume Geek Gene is true, unfavorable data should count more than favorable data). But saying "people used experience to interpret new data!" doesn't seem like it's proving much.

groundhogday1 10y ago

So the assumption being made here is that students either get it or they don't based on genetic predisposition? I, for one, don't particularly subscribe to the notion that every CS student needs to score an INTP on the Meyers-Briggs to be successful.

deleted 10y ago
[deleted]
Practicality 10y ago
ENTP makes a better developer anyway. :D (In case anybody doesn't get the joke, ENTP is the "debater" type, so I am picking a fight about personality type, since that is what the ENTP is supposed to do, so self-referential humor: https://www.16personalities.com/entp-personality https://www.16personalities.com/entp-personality)

losteverything 10y ago

"Your scores are bimodal!!!!" trying to get the "so what?" Does that say something (e.g ,bad instructor) or does it support something ( idea that 50% of students should not even be taking course, for example) Organic Chem class was bimodal which led to dismissal of instructor. Different instructor meant better learning and a different distribution.

moyix 10y ago

Folks may also be interested in the accompanying blog post by the first author: http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-normal-than-you.html http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-...

Animats 10y ago

Programming tests amplify modality because they require success on multiple items to succeed. Consider teaching N skills. Success is binary, uncorrelated and everyone randomly succeeds some high percentage P on each skill. Scoring is the number of binary successes. You'll get a Gaussian distribution. That's most of education. Now suppose scoring is 1 if all N skills are learned. You'll get a bimodal distribution. That's programming tested on whether the program runs.

aidenn0 10y ago
There are two links discussing that in a blog post by the author of this article[1] (second to last paragraph) 1: http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-normal-than-you.html http://patitsas.blogspot.ca/2016/01/cs-grades-probably-more-...
- Animats 10y ago
  Sadly, they're both paywalled.

deleted 10y ago

[deleted]

lifeisstillgood 10y ago

I am not sure we should assume bimodality in programming means we are all special snowflakes. That's a dangerous and self serving interpretation It seems bimodality is correlated with "does it even compile" - the first hump is those who can't get it to compile and the second is those who can but then spread out normally on ability. I would conjecture that the first hump would be seen in any educational environment where, for example we took illiterates and made them write essays. Those who had tried reading and writing in high school would have a better chance of putting ink on paper. We just are seeing an artifact of software not being taught early enough in everyone. Edit - how does my spelling corrector turn bimodality into bumps skirt...

htns 10y ago

They don't consider participation or hours spent studying as explanations for bimodality. This makes their method of determining belief in the "geek gene" questionable, and could invert the moral story of their results: Perhaps lecturers seeing low lecture attendance would like to believe their grade distributions are bimodal, but the data shows the lecturers' impact is at best a tiny factor among many, and grades are determined by factors out of the lecturers' control, such as students' ability.

scott00 10y ago

The whole premise of the survey part of this paper strikes me as very flawed. The idea was "show some professors histograms we know are normal, and see if they see bimodality that isn't there". The problem is, they didn't show them 6 random histograms of the normal distribution. For one, they didn't actually show them normal distributions at all; they showed them normal distributions capped at 100. That's going to give you a point mass at 100... making it a bimodal distribution! Second, they didn't give 6 random examples of a capped normal histogram; they gave them 6 examples chosen from a random set of histograms chosen in such a way that 4 of the 6 look bimodal. The survey participants didn't see bimodality that wasn't there, they saw bimodality that was purposefully generated by the histogram selection methodology!

makmanalp 10y ago

I get especially mad when people suggest that there are some ideas that are just special, that only special people can get. Pointers and recursion are at the top of this list. Take Joel Spolsky, a person whose ideas I read and respect: http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html http://www.joelonsoftware.com/articles/GuerrillaInterviewing... > For some reason most people seem to be born without the part of the brain that understands pointers. Pointers require a complex form of doubly-indirected thinking that some people just can’t do Really? Here's my 20 dollar challenge: give me a person and I can teach them pointers. Pointers, referencing and dereferencing, null pointers, pointers to pointers, the whole deal. The problem is the pedagogy. Learning a new abstraction and a new way of thinking takes time. It's easy to get it but really not /get/ it. You have to do a lot of examples. One on one with a person, with real time feedback. Until they internalize it. Many many many simple examples at first, and then examples of using it in a real world context. Then more. If you already know this abstraction, consciously or unconsciously, it's impossible to un-know it, and the reaction when someone is struggling is "I explained it as simply as I can and I don't know why they still don't get it". It's not like that. It's like basketball: you can explain how to shoot, but really the student needs to do it, and you comment while they're doing it.

Eliezer 10y ago
Some people get pointers immediately after one exposure before puberty. Not every student needs to practice. Some people, who maybe never came to your university at all, will never get linear algebra matter how much they practice. To say the distribution is unimodal is one claim. It's another matter to claim the distribution is narrow.
ap22213 10y ago
Absolutely - I'm sure that a bit of innate talent can improve one's rate of learning, but 90% of the willing can do it. The best programmers that I've met just loved the art and science of computing, had respect for their craft and very high standards of quality, were undaunted or even motivated by new challenges, but most importantly, put in the necessary time. I doubt things have changed much since I went to college in the 90s, but back then, the ones that did well at CS either had been hacking computers since the Apple II or spending enormous amounts of time at the lab (or both). Also, when at the Microsoft campus in the 90s, it was not a coincidence that the most expensive cars were the ones that were in the parking garage the latest.

k__ 10y ago

I don't know much about this, but I wasn't good at studying. I'm totally living for development and everything to do with it, but most of the formal stuff completely eluded me. I failed every math lecture at least once. Because of my projects and thesis I still finished with a 2,7 which isn't good but also not really bad. Maybe I would have been one of those "college-dropouts who followed their dream" if companies in Germany didn't look so much on your degrees. Also I never had the impression our grades were bimodal. Yes about 40% dopped out and yes there were a few "eager beavers", but most were simply okay enough to finish their degrees.

groundhogday1 10y ago
I'm about the opposite of you. Not a good programmer, but I excelled at the 'formal stuff.' Automata theory, regular expressions, compilers, operating systems, discrete math... these are all areas where I did well. Labs and programming assignments held me back. Although I will say that I feel infinitely more comfortable working with memory allocation in C than I do with anything involving JavaScript. I also don't think the overall grades were bimodal. I guess I was in the second mode in programming intense courses and the first mode in theory heavy courses. I also feel that most of my peers were either the same as me or the reverse, like you. Wouldn't that create a single mode bell curve? Of course, this is all purely anecdotal

hyperion2010 10y ago

Having worked with many of my fellow graduate students to try to teach CS concepts to incoming students without any CS background my conclusion here is that we simply don't know how to teach computer science. My suspicion is that many people who learn CS usually do it inspite of attempts to teach them, not because of instruction. Think about how long it took humans to figure out how to teach reading to _everyone_. We had to break the old whole reading method and switch over to phonics, and that was decades after we had the hard scientific evidence that phonics was the missing perspective needed to get that last percentage of students to read. We are probably decades away from understanding how to teach CS to everyone, let alone getting those techniques implemented.

GregBuchholz 10y ago
>We had to break the old whole reading method and switch over to phonics, and that was decades after we had the hard scientific evidence that phonics was the missing perspective needed to get that last percentage of students to read. https://en.wikipedia.org/wiki/Phoenician_alphabet#Spread_of_the_alphabet_and_its_social_effects https://en.wikipedia.org/wiki/Phoenician_alphabet#Spread_of_...
- taeric 10y ago
  I'm not clear on what the link is supposed to be getting me. Phonetics and Phoenician are two different things. Right?
  - jdmichal 10y ago
    I agree that the post is severely underspecified. I think the point is the last paragraph of the linked section: > Phoenician had long-term effects on the social structures of the civilizations which came in contact with it. As mentioned above, the script was the first widespread phonetic script. Its simplicity not only allowed it to be used in multiple languages, but it also allowed the common people to learn how to write. This upset the long-standing status of writing systems only being learned and employed by members of the royal and religious hierarchies of society, who used writing as an instrument of power to control access to information by the larger population.[11] The appearance of Phoenician disintegrated many of these class divisions, although many Middle Eastern kingdoms such as Assyria, Babylonia and Adiabene would continue to use cuneiform for legal and liturgical matters well into the Common Era. That is, Phoenician was the first wide-spread phonetic script. That, in turn, had major effects on civilizations as it was simple enough for commoners to learn. That being the rather tenuous link back to the original idea of teaching reading based on phonetics. Of course, there is a bit of a gap here, in that English phonetics are notoriously unstandardized across the lexicon.

cdevs 10y ago

I hated trying to read about pointers in c when I was about 15, a few years later a different book put it more clear and it all snapped into place for me.

a_puppy 10y ago

This is mostly unrelated to the main topic of the article, but this paragraph caught my eye: > For example, Padavic et al. [20] found that the "work-family" narrative in business is an example of a social defense: people will say that women leave the workplace because of "family", despite the large amount of evidence that women leave their jobs because of inadequate pay or opportunities for advancement [20], particularly when they see male co-workers promoted ahead of them. The "work-family" narrative is a more palatable explanation rather than to confront sexual discrimination in the workplace, and so the narrative continues. > > [20] I. Padavic and R. J. Ely. The work-family narrative as a social defense, 2013. I tried to track down the work by Padavic et al. that Patitsas et al. are citing, and found this: http://www.hbs.edu/faculty/conferences/2013-w50-research-symposium/Documents/Gender_and_work_web_update2015.pdf http://www.hbs.edu/faculty/conferences/2013-w50-research-sym... However, this article by Padavic et al. is actually saying something different from the way that Patitsas et al. summarized it. Padavic et al. appear to actually be arguing that the work-family narrative is a social defense against the uncomfortable truth of excessively long working hours for both genders at the corporation, not against any uncomfortable truth about sexual discrimination. Also, Padavic et al. don't seem to discuss sexual discrimination in wages or promotions at all. Did Patitsas et al. make an error in citing their sources? Am I missing something?

danieltillett 10y ago
Mis-citation occurs all the time. When I was a professional scientist I could only find the source for a claim I was interested in less than 25% of the time. I would find some claim and about 50% of the time the reference cited another reference and so on until the chain broke. The other 25% of the time the reference cited said something completely different to what was being claimed. Very frustrating.
- tjl 10y ago
  During my Ph.D. I tried tracking down the reasoning behind a couple sets of assumptions made in non-uniform torsion of beams (e.g., twisting a beam with one end fixed). It turns out that they just kept the same assumptions from uniform torsion (just twisting the beam but with no ends fixed) without checking if they still held (they don't). All because nobody thought to check if the assumptions might be in conflict. The uniform torsion models go back to the mid 1800s, while the non-uniform torsion models are from the early to mid 20th century. So, it's been over 60 years people have been working with wrong assumptions.

rcheu 10y ago

I've never heard that computer science grades are bimodal, I thought it was that performance in programming is bimodal? It's always been relatively obvious to me that grades were not bimodal since I could just click and see the grade distribution, which didn't appear to be bimodal.

toomanybeersies 10y ago

Oh, useful anecdote time! I was talking to a professor a while back about this, and he reckoned that when the University of Canterbury (in NZ) switched from Java to Python for their 100 level papers, the grades went from bimodal to single-modal. Not sure why that is, and I don't remember him giving me a reason, but I guess it's fair to assume that a lot of students were really struggling with the initial learning curve of Java (boilerplate, static typing, more difficult array and string manipulation) compared to Python, and either just not getting it, or getting frustrated and giving up. Now I didn't see any numbers to back up his claims, as it was a casual conversation, but I'd be inclined to believe him that they were bimodal. Anyway, the point I'm making is that the study here references only one university, different universities have different curriculum and different teaching styles, which could affect whether the distribution is bimodal or not.

Bartweiss 10y ago

So... This is a bad study, top to bottom. Section 3 is based off a single university's final grades. - There's no discussion of whether assignment grades were being curved before going into the final score (they are in at least some universities). So it's entirely possible that they took normalized data and found that it had normal distributions. - Even if it's raw data, bimodality would appear per-assignment. Averaging problem sets with papers with tests, and assignments of varying difficulty, threatens to obscure any task-level bimodality. - Finally, the paper acknowledges the risk of university-specific effects, but can't actually adjust for them. So Section 3 may not generalize at all. Section 4 is unconvincing in several ways. - The participant selection was open solicitation from multiple forums, with a high dropout rate after providing tests. That screams selection bias, but isn't acknowledged. - "Assess this constructed data based on an inaccurate origin" threatens to carry over a prior of "all my real grades are actually bimodal" to the sample data. This would produce the observed results regardless of whether professors are misjudging normal data. This risk would be irrelevant if grades aren't bimodal, but Section 3 didn't sell me on that. - "Assess this data for bimodality" threatens to prime seeing bimodality for any set of data. This renders it incomparable to unprompted observations about grades. Supporting Literature is a disaster. It's a hit parade of papers and topics which have or may yet fail to replicate. The male/female candidates paper has a sibling which found the inverse result. The weapon bias study hasn't been found predictive outside the original study. The "brilliance-requiring disciplines" literature is such a disaster that there are multi-thousand-word essays breaking down why it's unconvincing. I'd say that's not the author's fault, but this is dated September 2016. The male/female candidates result at least should have come with an acknowledgement that other studies have found complete different outcomes. In summary: The data analysis here is fragile and unconvincing. The human studies misuse unproven effects to draw unsupported conclusions. The supporting literature grounds the preceding mess on other studies which were suspect by the time the paper was written.