Myself and a colleague entered this competition. We came 9th. We were doing this for fun, and aren't experts in the domain, but I think our score was within 'di

Myself and a colleague entered this competition. We came 9th. We were doing this for fun, and aren't experts in the domain, but I think our score was within 'diminishing returns' of the better teams. There are a couple of things to realize about this challenge: I wouldn't conceptualise the challenge as trying to find features of good essays. It is more about trying to find features that are predictive of the essay being good. This is a subtle but important distinction. One example is that the length of essay was hugely predictive of the score the essay would get - longer meant better. Is a longer essay really a better one? No. But, at the level the students were at, it just so happened that the students who were able to write better, also were able to write longer essays. While you get good accuracy using techniques like this, its debatable how useful or robust this general approach is - because you aren't really measuring the quality of the essay, so much as you are trying to find features that just happen to be predictive of the quality. Certainly, it would seem fairly easy for future students to game, if such a system was deployed. This isn't a general attack on machine learning competitions - but I wonder, if for situations that are in some sense adversarial like this (in that future students would have an incentive to game the system), whether some sort of iterated challenge would be better? After a couple of rounds of grading and attempting to game the grading, we'd probably have a more accurate assessment of how a system would work in practice. There is another important feature of this essay grading challenge, that should be taken into consideration. There were 8 sets of essays, each on a different topic. So, for example, essay set 4 might have had the topic 'Write an essay on what you feel about technology in schools'. To improve accuracy, competitors could (and I would guess most of the better teams did) build separate models for each individual essay-set/topic. This then increased the accuracy of, say, a bag of words approach - if an essay-set 1 essay mentioned the word 'Internet' then maybe that was predictive of a good essay-set-1 essay, even though the inclusion of 'Internet' would not be predictive of essay quality, across all student essays. Its important to remember this when thinking about the success of the algorithms. The essay grading algorithms were not necessarily general purpose, and could be fitted to each individual essay topic. Which is fine, as long as we realize it. The fact that it was so easy to surpass inter annotator agreement (how predictive one of the human graders scoring was of the other human graders scoring) was interesting. Its just important to realize the limits of the machine learning contest setup. I would guess that accuracy would down on essays of older, more advanced students, or in an adversarial situation where there was an incentive to game the system.