7 ms·
The Algorithm Didn’t Like My Essay
- kenrikm 14y agoIf it's ok for them to use an algorithm to grade essays then it would be ok for students to use an algorithm to write the papers no? How is it different from using a calculator? it's a tool right? mathematics used to only be only pen and paper. (I know it says it would still be graded by people, I'm making a point) Writing is an artform and as such should be treated and graded as one. Sometimes you need to use language that is grammatically incorrect to get a point across and software is not nuanced enough to pick up on this. I can see this helping to turn out students that are good grammaticians but terrible writers - optimizing for the wrong outcome.
- imd 14y agoThe evaluations are on "how well algorithms submitted by professional data scientists and amateur statistics wizards could predict the scores assigned by human graders." As long as some humans are always included as a control, this shouldn't be too big of a problem.
- jacktoole1 14y agoI upvoted you for the interesting line of thought, although I disagree with the premise that acceptability of grading algorithmically implies acceptability of writing algorithmically. In CS, we often grade assignments automatically, although we expect students to write the assignments by hand. Although there are many classes where this would be a bad idea, there are some where it works quite well - if the problems are strictly specified, grading an assignment with what are effectively unit tests is reasonable. Grading and doing an assignment are different classes of problems - roughly mapping to decision problems and search problems. Just because the grading is computationally tractable or acceptable does not necessarily imply that the act of writing the assignment is. I agree with your second paragraph by and large. There is a time and place for automatic grading, and most writing should not fall into that category.
- _delirium 14y agoOne difference with CS algorithmic grading is that it's much more specific and causally connected to outcomes. It's not looking for statistical correlates of the input text and a notion of "good program", but is actually finding things like compile errors and failed testcases. And, crucially, it can also report those to the student: your program is wrong because when fed 5 it gives an output of 23 when the output should be 25. It's probably possible to grade student programs like this essay-grading system works: instead of feeding the programs to a compiler and running against test-cases, just turn the source code into a bag-of-words representation and feed it to a statistical classifier. There are almost certainly some statistical correlates between the source code of programs and whether they're good/bad submissions. But the results would be both less reliable and less informative as feedback.
- jacktoole1 14y agoGood point; I agree this makes code much more naturally machine graded than essays, and a statistical classifier for source code sounds disturbing (although it might do better on grading style than the usual autograding does).
- majormajor 14y agoFor the standardized test discussed in the article, it says the human graders gave three minutes, on average, to reading each essay. These tests are not wanting art. Writing one of these essays is no more art than writing a basic linked list implementation. It's not concerned with how good a writer a kid is, but that the kid can write at all. Edit: basically, the sheer scale of the grading required for a standardized test means that it's never going to be a real test of writing skills. It wouldn't surprise me if overly skilled writing is actually a negative even with the human grader—I could easily see a rushed grader misreading something clever and giving it a low score. (The wisdom or effectiveness of these tests in the first place is a whole different discussion.)
- bermanoid 14y ago"Art" is not even a question - these tests typically apply to high school kids, and you're lucky if bare-bones basic communication is possible there. Most of these kids are straight-up idiots, and I say this having spent several years teaching them. I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT Writing scores. Add in a factor for essay length, unique word cound, and grammar errors, and you'd probably get pretty damn close to perfect, within a reasonable statistical tolerance. Children don't produce art, almost all of them produce garbage, right up to the top 1%. Make no mistake, all that standardized tests aim to measure is how stinky the shit that they produce is. I'm fine with missing a bit of brilliance at the top; those students will do just fine anyways. It's ranking the relative crappitude of the middle that's important here, and I think the standard (if crappy) grading procedures do a pretty good job here.
- kenrikm 14y agoJaded much? Maybe I'm overly optimistic of the quality of students the school system produces? (I never went to high school, I took the GED and scored in the 100th percentile across the board and went directly to college at 16)
- unimpressive 14y ago>I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT Writing scores. The best part about that is the fact that people misspell words even with spellcheck! Come on, how hard is it to notice the squiggly lines and correct your error by picking the correct spelling from the drop down list? And I wouldn't say what I produce is shit, but then I would have said that when I was twelve. And I mostly have to double time on projects and essays, so they probably are shit. EDIT: And since losethos (Schizophrenic dude who's hellbanned.) asked, the boss; it's pretty clever to use someones tendency for pedantry against them.
- numlocked 14y ago(disclaimer - I work at Kaggle) I would recommend downloading the dataset[1] and taking a look at some of the essays. There is a tendency for an educated audience to (justifiably) worry about exactly what you've articulated, but when you read through the essays, you get an appreciation of how valuable a tool like this could be. These are elementary and middle school essays rife with rudimentary mistakes (which are at times extremely charming) and generally not filled with subtle arguments prone to misinterpretation. If algorithms can help every seventh grader in the country write a solid 5 paragraph essay, that's progress. [1]: http://www.kaggle.com/c/asap-aes/data http://www.kaggle.com/c/asap-aes/data
- gammarator 14y agoI participated in the Kaggle competition, but I'm less optimistic about the pedagogic value of these systems. A black box algorithm won't explain to a student what their rudimentary mistakes are. I wrote about this more extensively here: http://bellm.org/blog/2012/04/23/arguing-to-the-algorithm-machine-learned-scoring-of-student-essays/ http://bellm.org/blog/2012/04/23/arguing-to-the-algorithm-ma...
- numlocked 14y agoGreat write up! I hadn't seen it, thanks for sharing. Two things: 1) Agree about the explanation bit. In fairness, that was not part of the competition. But I do believe that, given the way the features were constructed, many of the algorithms could be modified to also provide "explanation" of their scoring. The value is highly dependent on the features though (length, for instance, would probably be a prominent "explanation"), and I haven't actually tried it. 2) The "This would grade Dickinson/Hemingway/DFW/etc poorly" argument is true but, again, read the essays. These are serious edge cases - ones that should be addressed (by, for instance, correlating algorithmic scoring with human scoring), but this argument doesn't diminish the applicability of the algorithm to the vast majority of essays and students.
- Jach 14y agoWill this help the nearly-illiterate masses? I think so. Will it hurt those who are one standard deviation or more above the norm? I think that's a problem, too. The solution then is to use the algorithm on the first set and more individual-oriented grading methods for the other set. Effort should be spent finding those in the other set and that isn't restricted to a single course or assignment. This challenges the assumption that students are equal and deserve equal grading, but I don't really see another way forward that doesn't hurt any of the best, mediocre, or poor writers.
- rwallace 14y ago> then it would be ok for students to use an algorithm to write the papers no? If a student used an essay-writing algorithm he designed himself, I would certainly consider that okay.
- alextp 14y agoGiving students close access to a program that evaluates essays many times would be the worst possible thing to do. People are generally very good at adapting to systems, and the algorithm would break down pretty quickly as people find out which factors it favors and which ones it penalises (while human graders are arguably less strict in their criteria and can hence recognize these attempts at writing to the letter).
- biot 14y agoThis inevitably leads to an arms race, with the students and software getting better and better. At some point, sufficiently advanced gaming of essay scores is indistinguishable from students who naturally write well by following the rules which lead to good writing. As long as creativity and originality aren't sacrificed, is it really a problem?
- CapitalistCartr 14y agoIt does not necessarily lead to an arms race. Once the students learn to write to the gradind software, the schools can claim the students are getting better, and point to an 'objective, unbiased' measuring stick. Everyone wins except for society, i.e. sll of us.
- biot 14y agoI'm assuming there will be competition among the companies who develop such software where, periodically, schools will re-evaluate their provider and select the best one. Company A will demonstrate their competitive advantage by showing how Company B's software incorrectly grades as "Excellent" an essay that looks like English but reads as gibberish. Company B will correct their software, but meanwhile Company C has come along and introduced a sophisticated analyzer which is able to grade the presence and quality of the logical argument being made in the essay. Company C will demonstrate how Company A and B's software grades as "Excellent" an essay which is correct English and isn't gibberish but makes no logical sense. After an indeterminate number of iterations with the software getting better and better at finding issues, the only way to game it will be to write a great essay.
- mingpan 14y agoI think this says more about the assignment and the human grading procedures than anything else. Even the supposedly creative sections in standardized tests are surprisingly narrow in what they consider to be good.
- frankydp 14y agoExactly what I was thinking. The ability to reliably predict human scores, says more about the terrible way we "standardize" creative writing, than about the quality of the algorithms. If the grading curve of an essay is a checklist how is this finding a surprise?
- losethos 14y agoIt should be built-into word processors, so all students start with A grammar. The biggest problem I see is, in the real world, concise is king... and marketing is the epitome of persuasiveness. Microsoft marketing is God-awful fluff, but I guess that's where most consumers are -- stupid. God says... 25:47 And if a sojourner or stranger wax rich by thee, and thy brother that dwelleth by him wax poor, and sell himself unto the stranger or sojourner by thee, or to the stock of the stranger's family: 25:48 After that he is sold he may be redeemed again; one of his brethren may redeem him: 25:49 Either his uncle, or his uncle's son, may redeem him, or any that is nigh of kin unto him of his family may redeem him; or if he be able, he may redeem himself. 25:50 And he shall reckon with him that bought him from the year that he was sold to him unto the year of jubile: and the price of his sale shall be according unto the number of years, according to the time of an hired servant shall it be with him. 25:51 If there be yet many years behind, according unto them he shall give again the price of his redemption out of the money that he was bought for. ---- My India-nigger doctor probably thinks a college English professor knows more about persuasive writing than a Coke marketing manager. God says... stoked Mozambique thank_you_very_much you're_no_fun impossible wastoid ghastly fer_sure Honduras let_me_count_the_ways I'm_off_today obviously special_case Israel what_a_mess resume freak fool Malaysia by_the_way Oh_really Poland Wow snap_out_of_it just_lovely fortitude California humility patience why_do_I_put_up_with_this ingrate begs_the_question pet caution anger what_would_Jesus_do figuratively other umm_the_other_answer later how_high Norway yada_yada_yada ---- India-nigger programmers don't know how to use a hash table to collect unique words from a document. "But it's not secure without salt!" ---- For what it's worth, math skill is pretty synanomous with IQ -- the real world idea of IQ. Sure, liberals can claim other things are IQ, but nobody really believes it. Wisdom, by the way, is different from IQ. God says... his house, and from his labour, that performeth not this promise, even thus be he shaken out, and emptied. And all the congregation said, Amen, and praised the LORD. And the people did according to this promise. 5:14 Moreover from the time that I was appointed to be their governor in the land of Judah, from the twentieth year even unto the two and thirtieth year of Artaxerxes the king, that is, twelve years, I and my brethren have not eaten the bread of the governor. 5:15 But the former governors that had been before me were chargeable unto the people, and had taken of them bread and wine, beside forty shekels of silver; yea, even their servants bare rule over the people: but so did not I, because of the fear of God. ----- Stupid shrink! I threw him a bone saying marketeers understood persuasive writing. Hint: that's psychology. God says... weakest uses told accompany save death disdained hogs builded disentangle stated economy familiarly regardest likeness kindled instinctive bathing abide action basically wilfully efforts unthankful Estonia faileth fer_sure duties balneum START dance starting tempt devotion Sri_Lanka forethought forenoon heads sees heightening insensibly encumbered vow readest Image creation kindness awoke turned covered pair fool do_not_disturb no_more_tears X Glory Angels congratulation countenance direction Jupiter proper combine shrunk how_high sheep raging named frame recorded tm dear disregard com rocket_science balneum excelled frenzied gladly grave's firmament figurative resist taunt imparted glorious finds bounces Venus unlearned omened desirous constrain divine lick challenged panting visions infringement holiness this builded file agonised ran justify judgements amazed uncase perturbations Grace honied Exodus East_Timor endureth suffice limitation broke delivers You_da_man altogether astonishment incorruption sensation lengthened what_part_of_God_do_you_not_understand Jesus confidentially notices KIND thread Money perils excel shoe biggot out regulate wert written infantine unworthy waterest Episcopal rich boil observe Foul believed confidently dinner Speak murmured WHAT disputing plainer remission That's_my_favorite unforgotten been their DOMINE cogitated embalmed haphazard Psychology is unlived wisdom, in my understanding. India-nigger is going-off on a particular x86 instruction. Go write an assembler and compiler and get back to me.
- InclinedPlane 14y agoWe keep paying more and more per student for public education. And what do we get in return? Robo-grading and prison-like treatment of students (closed campuses, zero tolerance taken to ridiculous extent). And again we come around to the notion that if only we'd spend more everything would get better, which has worked so well every time it's happened in the past. The increasing reliance on automated tools and standardized tests is just one of many symptoms of the increasing detachment between teachers and students. We have to stop doubling down on the prussian style educational system, it stopped returning dividends decades ago. More so we have to stop acting as though we can just solve our educational problems by throwing more money at it. We need to empower good teachers and parents, we need to encourage and support students. Right now all of the incentives in the system run counter to those things.
- adrianN 14y agoBut this article advocates using automated grading to reduce costs and free up teacher time for better things.
- InclinedPlane 14y agoCost is orthogonal to educational quality. As I said, this isn't an example of reducing costs so much as an example of the continual dehumanization of the education system. Any cost savings will just be gobbled up by the entrenched interests in the system, bureaucrats, administrators, etc. And any efficiencies making it easier to treat students like cattle and push "education" more and more toward useless rote memorization (teaching to the test) and busywork. These are not good things.
- ludflu 14y ago> Cost is orthogonal to educational quality. I've heard this (anecodotal) argument alot. Is there any empirical backing for it, and within what parameters does it hold true? My mother taught at a very expensive private school, which enabled me to switch from a lousy public school . The difference in the quality of teaching was dramatic. Private school teachers don't get paid alot more than public school teachers, and often less, although they are compensated with fringe benefits like I enjoyed, and prestige. In any case, elite private schools certainly get better results, which is why rich people pay alot of money to send their children to elite private schools. (Lots of this is attributable to the selection process, which is part of what parents are paying for.) In contrast, public inner city schools can't choose their student population, and lack basic things like, books, paper, and air conditioning in the middle of the summer. I'm sure that proper and equitable funding of public education wouldn't solve all the problems in US public schools, but starving them of much needed resources certainly exacerbates the problems they do have.
- dhx 14y agoHow well did the algorithms detect insightful analysis, deep understanding beyond the immediate subject matter, factual correctness, salience and an ability to write to a specific audience? When reading articles on the Internet I look out for superficial factors including: ‣ Misuse of U+0022 typewriter double quotation marks in place of U+201C and U+201D double quotation marks ‣ Misuse of U+002D hyphens in place of U+2013 en dashes and U+2014 em dashes ‣ Poor understanding of comma and semicolon usage ‣ Choice of typeface, use of ligatures and micro-typographical features, amount of leading used… ‣ Misuse of their, there, they’re, where, were, we’re, affect, effect… ‣ RAS syndrome (sīc) These factors can assist with analysis of writing. An author can generate credibility in certain contexts by using triangular Unicode bullets instead of ASCII asterisks. Use of lengthy and complex words in place of simple English can also be mistaken for credibility. Analysis of these techniques will show that the writer is often attempting to belittle readers. Documentaries utilise similar tactics that the untrained eye will likely mistake for credibility. Soft filter effects, use of bookcase backdrops for expert interviews and silence have profound manipulative effects on viewers. Grading algorithms will likely reward use of manipulative and belittling writing techniques and penalise honest superficial mistakes. I would rather read the honest opinion from an author that misuses their and there than have to carefully analyse the carefully crafted writings of a dishonest linguist. As an aside, I recommend bookmarking The Browser[1] (Writing Worth Reading). It would be interesting to see the results of algorithmic essay grading against this curated collection of articles from across the Internet. [1] http://thebrowser.com/ http://thebrowser.com/
- DougBTX 14y agoHow well did the algorithms detect insightful analysis, deep understanding beyond the immediate subject matter, factual correctness, salience and an ability to write to a specific audience? I'm far from well informed, but my understanding of standardised tests is that the standard specifies the algorithm, which already ignores your good points above to achieve standardised grading. All that really changes is whether a human or a robot executes the algorithm, the human insight has already been squeezed out of the system.
- SagelyGuru 14y agoAt best, these algorithms can only pick on obvious spelling, grammar, punctuation and style errors. They have no concept of semantics. Therefore the students will have to polish the form and abandon the content. To prove this point, I suggest setting up a testing competition, this time for algorithms to generate meaningless 'essays'. I expect their easy success obtaining full marks. So, what will be the educational outcome of this? In three simple words: more dumbing down. Is this another example of the obsession of officials everywhere with the pedestrian form at the expense of original content? I think it tells us a lot about their own limited mentality.
- ericd 14y agoThis article is coming to the wrong conclusion from the facts given. That the grades given by these algorithms matched the grades given by human essay reviewers on standardized tests almost exactly doesn't mean that our algorithms are great and that they should be rolled out to normal school environments to help students learn to write. It means that our method of grading essays on standardized tests is absolute garbage, because the algorithms can't judge semantic meaning or depth of insight. We're basically grading people on how prepared they are for writing About.com content farm articles.
- PaulHoule 14y agoI was told by the head search engineer for a well-known job board site that they'd trained a bag-of-words classifier to find badly written resumes that would benefit from professional editing. They'd found it was one of the easiest text classification problems that had ever been tried. With a very small training set they'd attained off-the-charts accuracys.
- _delirium 14y agoI wonder how the state-of-the-art in algorithmic essay grading compares to the state-of-the-art in scoring the quality of online content. Does Google have content-quality-estimating algorithms that could improve algorithmic essay grading?
- keithpeter 14y ago"And humans are not necessarily ideal graders: they provide an average of only three minutes of attention per essay, Ms. Chow says. " Well, I'm hardly surprised that an algorithm can duplicate that kind of superficial marking of an essay! Good heavens, what are you doing to your children in the US?
- feral 14y agoMyself and a colleague entered this competition. We came 9th. We were doing this for fun, and aren't experts in the domain, but I think our score was within 'diminishing returns' of the better teams. There are a couple of things to realize about this challenge: I wouldn't conceptualise the challenge as trying to find features of good essays. It is more about trying to find features that are predictive of the essay being good. This is a subtle but important distinction. One example is that the length of essay was hugely predictive of the score the essay would get - longer meant better. Is a longer essay really a better one? No. But, at the level the students were at, it just so happened that the students who were able to write better, also were able to write longer essays. While you get good accuracy using techniques like this, its debatable how useful or robust this general approach is - because you aren't really measuring the quality of the essay, so much as you are trying to find features that just happen to be predictive of the quality. Certainly, it would seem fairly easy for future students to game, if such a system was deployed. This isn't a general attack on machine learning competitions - but I wonder, if for situations that are in some sense adversarial like this (in that future students would have an incentive to game the system), whether some sort of iterated challenge would be better? After a couple of rounds of grading and attempting to game the grading, we'd probably have a more accurate assessment of how a system would work in practice. There is another important feature of this essay grading challenge, that should be taken into consideration. There were 8 sets of essays, each on a different topic. So, for example, essay set 4 might have had the topic 'Write an essay on what you feel about technology in schools'. To improve accuracy, competitors could (and I would guess most of the better teams did) build separate models for each individual essay-set/topic. This then increased the accuracy of, say, a bag of words approach - if an essay-set 1 essay mentioned the word 'Internet' then maybe that was predictive of a good essay-set-1 essay, even though the inclusion of 'Internet' would not be predictive of essay quality, across all student essays. Its important to remember this when thinking about the success of the algorithms. The essay grading algorithms were not necessarily general purpose, and could be fitted to each individual essay topic. Which is fine, as long as we realize it. The fact that it was so easy to surpass inter annotator agreement (how predictive one of the human graders scoring was of the other human graders scoring) was interesting. Its just important to realize the limits of the machine learning contest setup. I would guess that accuracy would down on essays of older, more advanced students, or in an adversarial situation where there was an incentive to game the system.
- fiatmoney 14y agoAs long as we're training algorithms to recognize correlates of "high-quality writing" rather than high-quality writing itself, why not use as many predictive features as possible? I'll bet parental income and education level, average home price in the school district, and the percentage of students at the school receiving free or reduced-price lunches, are incredibly correlated with writing quality.
- deleted 14y ago[deleted]
- reubensutton 14y agoMy instinct is that most algorithms to test this may end up optimizing for essay length and word complexity, rather than being able to assess the content, however I imagine this is also how a lot of teachers grade, given that this article states that they receive 3 minutes of attention each.