3 ms·

I am fascinated with automatic grading. Predicting the grade of an essay isn't fundamentally different than classifying an email as spam. There was a kaggle com

I am fascinated with automatic grading. Predicting the grade of an essay isn't fundamentally different than classifying an email as spam. There was a kaggle competition for automatic grading, and IIRC the best results were pretty close to, or even better than human graders. As NLP improves, this will too. But there is huge resistance to it because of Algorithm Aversion (http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort...). People really distrust statistical algorithms over (even really bad) human judgement. Unless they believe the algorithms are infallible. One interesting thing you can do with automatic grading, is semi-supervised learning. If you know some students are good or bad already (perhaps from multiple choice, or from already graded essays), you can predict the grade of their essays. And then use the predictions to train a more accurate classifier.

SomeStupidPoint 10y ago
The reason for not trusting algorithms is that most of them have unexpected failure modes, and then no way to dispute their result through a more holistic (eg, human based) method. Human grading has relatively well understood failure modes and a (semi-)functional dispute system. The danger is that too often when we switch to algorithms, we "forget" (read: often choose not to implement) dispute mechanisms. Im not against algorithmic grading, per se, but I expect that we're at the phase (and will be for some time) where a human and a computer do a better job together than either does alone.
analog31 10y ago
A potential issue with algorithms is that the successful ones might become a resilient standard, with nobody knowing why, or being able to challenge it. Suppose computer programs were graded by algorithm. That might be the end of the road for developing new languages and coding styles. And I could imagine a consolidation of the grading tools, down to a small handful, or maybe even just one vendor. This is the pattern in textbook publishing. Once the human graders are laid off and their skill vanishes, education will be wholly dependent on the whims of those businesses. Do you want to hire or work with 1000 engineers, who have all been through exactly the same training, and have adapted their styles to exactly the same template, worldwide? What if a new idea or problem comes along, that isn't addressed within the capabilities established by that template? This is not too far fetched. I was in college when word processing and microcomputers were introduced to academia. I was one of the early adopters at my college. Both of these things radically changed how people did academic work, but a word processed essay might have gotten a low grade if the algorithms were tuned to the established preferences of teachers for reading hand-written essays.
jayajay 10y ago
> Predicting the grade of an essay isn't fundamentally different than classifying an email as spam https://media.giphy.com/media/3oz8xLd9DJq2l2VFtu/giphy.gif https://media.giphy.com/media/3oz8xLd9DJq2l2VFtu/giphy.gif It is fundamentally very different, as the former requires domain knowledge (i.e. to a larger extent) and the latter does not (i.e. to a lesser extent). It's just not that simple; even in doing the things you mention, the algorithm will be marginally better than a different algorithm. The magic lies in the 1% that the algorithm misclassifies, not the 99% that it doesn't. The ML community is centered around establishing high classification rates of static data sets. This will soon change, when people realize "99.9% accuracy on a fixed data set" is a meaningless metric for real artificial intelligence, not the monkey-see-monkey-do stuff that is currently considered "state-of-the-art". There's about to be a big reckoning in this field, similarly to what happened in 1905.
inimino 10y ago
It's not algorithm aversion. Predicting the grade is one thing, assigning the grade is another. Remember the Sokal affair? Any writing assignment that can meaningfully be scored by less than human-level AGI should not be assigned in the first place.

SomeStupidPoint 10y ago

The reason for not trusting algorithms is that most of them have unexpected failure modes, and then no way to dispute their result through a more holistic (eg, human based) method. Human grading has relatively well understood failure modes and a (semi-)functional dispute system. The danger is that too often when we switch to algorithms, we "forget" (read: often choose not to implement) dispute mechanisms. Im not against algorithmic grading, per se, but I expect that we're at the phase (and will be for some time) where a human and a computer do a better job together than either does alone.

analog31 10y ago

A potential issue with algorithms is that the successful ones might become a resilient standard, with nobody knowing why, or being able to challenge it. Suppose computer programs were graded by algorithm. That might be the end of the road for developing new languages and coding styles. And I could imagine a consolidation of the grading tools, down to a small handful, or maybe even just one vendor. This is the pattern in textbook publishing. Once the human graders are laid off and their skill vanishes, education will be wholly dependent on the whims of those businesses. Do you want to hire or work with 1000 engineers, who have all been through exactly the same training, and have adapted their styles to exactly the same template, worldwide? What if a new idea or problem comes along, that isn't addressed within the capabilities established by that template? This is not too far fetched. I was in college when word processing and microcomputers were introduced to academia. I was one of the early adopters at my college. Both of these things radically changed how people did academic work, but a word processed essay might have gotten a low grade if the algorithms were tuned to the established preferences of teachers for reading hand-written essays.

jayajay 10y ago

> Predicting the grade of an essay isn't fundamentally different than classifying an email as spam https://media.giphy.com/media/3oz8xLd9DJq2l2VFtu/giphy.gif https://media.giphy.com/media/3oz8xLd9DJq2l2VFtu/giphy.gif It is fundamentally very different, as the former requires domain knowledge (i.e. to a larger extent) and the latter does not (i.e. to a lesser extent). It's just not that simple; even in doing the things you mention, the algorithm will be marginally better than a different algorithm. The magic lies in the 1% that the algorithm misclassifies, not the 99% that it doesn't. The ML community is centered around establishing high classification rates of static data sets. This will soon change, when people realize "99.9% accuracy on a fixed data set" is a meaningless metric for real artificial intelligence, not the monkey-see-monkey-do stuff that is currently considered "state-of-the-art". There's about to be a big reckoning in this field, similarly to what happened in 1905.

inimino 10y ago

It's not algorithm aversion. Predicting the grade is one thing, assigning the grade is another. Remember the Sokal affair? Any writing assignment that can meaningfully be scored by less than human-level AGI should not be assigned in the first place.