4 ms·

If it's ok for them to use an algorithm to grade essays then it would be ok for students to use an algorithm to write the papers no? How is it different from us

by kenrikm 14y ago

If it's ok for them to use an algorithm to grade essays then it would be ok for students to use an algorithm to write the papers no? How is it different from using a calculator? it's a tool right? mathematics used to only be only pen and paper. (I know it says it would still be graded by people, I'm making a point) Writing is an artform and as such should be treated and graded as one. Sometimes you need to use language that is grammatically incorrect to get a point across and software is not nuanced enough to pick up on this. I can see this helping to turn out students that are good grammaticians but terrible writers - optimizing for the wrong outcome.

imd 14y ago
The evaluations are on "how well algorithms submitted by professional data scientists and amateur statistics wizards could predict the scores assigned by human graders." As long as some humans are always included as a control, this shouldn't be too big of a problem.
jacktoole1 14y ago
I upvoted you for the interesting line of thought, although I disagree with the premise that acceptability of grading algorithmically implies acceptability of writing algorithmically. In CS, we often grade assignments automatically, although we expect students to write the assignments by hand. Although there are many classes where this would be a bad idea, there are some where it works quite well - if the problems are strictly specified, grading an assignment with what are effectively unit tests is reasonable. Grading and doing an assignment are different classes of problems - roughly mapping to decision problems and search problems. Just because the grading is computationally tractable or acceptable does not necessarily imply that the act of writing the assignment is. I agree with your second paragraph by and large. There is a time and place for automatic grading, and most writing should not fall into that category.
- _delirium 14y ago
  One difference with CS algorithmic grading is that it's much more specific and causally connected to outcomes. It's not looking for statistical correlates of the input text and a notion of "good program", but is actually finding things like compile errors and failed testcases. And, crucially, it can also report those to the student: your program is wrong because when fed 5 it gives an output of 23 when the output should be 25. It's probably possible to grade student programs like this essay-grading system works: instead of feeding the programs to a compiler and running against test-cases, just turn the source code into a bag-of-words representation and feed it to a statistical classifier. There are almost certainly some statistical correlates between the source code of programs and whether they're good/bad submissions. But the results would be both less reliable and less informative as feedback.
  - jacktoole1 14y ago
    Good point; I agree this makes code much more naturally machine graded than essays, and a statistical classifier for source code sounds disturbing (although it might do better on grading style than the usual autograding does).
majormajor 14y ago
For the standardized test discussed in the article, it says the human graders gave three minutes, on average, to reading each essay. These tests are not wanting art. Writing one of these essays is no more art than writing a basic linked list implementation. It's not concerned with how good a writer a kid is, but that the kid can write at all. Edit: basically, the sheer scale of the grading required for a standardized test means that it's never going to be a real test of writing skills. It wouldn't surprise me if overly skilled writing is actually a negative even with the human grader—I could easily see a rushed grader misreading something clever and giving it a low score. (The wisdom or effectiveness of these tests in the first place is a whole different discussion.)
- bermanoid 14y ago
  "Art" is not even a question - these tests typically apply to high school kids, and you're lucky if bare-bones basic communication is possible there. Most of these kids are straight-up idiots, and I say this having spent several years teaching them. I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT Writing scores. Add in a factor for essay length, unique word cound, and grammar errors, and you'd probably get pretty damn close to perfect, within a reasonable statistical tolerance. Children don't produce art, almost all of them produce garbage, right up to the top 1%. Make no mistake, all that standardized tests aim to measure is how stinky the shit that they produce is. I'm fine with missing a bit of brilliance at the top; those students will do just fine anyways. It's ranking the relative crappitude of the middle that's important here, and I think the standard (if crappy) grading procedures do a pretty good job here.
  - kenrikm 14y ago
    Jaded much? Maybe I'm overly optimistic of the quality of students the school system produces? (I never went to high school, I took the GED and scored in the 100th percentile across the board and went directly to college at 16)
  - unimpressive 14y ago
    >I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT Writing scores. The best part about that is the fact that people misspell words even with spellcheck! Come on, how hard is it to notice the squiggly lines and correct your error by picking the correct spelling from the drop down list? And I wouldn't say what I produce is shit, but then I would have said that when I was twelve. And I mostly have to double time on projects and essays, so they probably are shit. EDIT: And since losethos (Schizophrenic dude who's hellbanned.) asked, the boss; it's pretty clever to use someones tendency for pedantry against them.
numlocked 14y ago
(disclaimer - I work at Kaggle) I would recommend downloading the dataset[1] and taking a look at some of the essays. There is a tendency for an educated audience to (justifiably) worry about exactly what you've articulated, but when you read through the essays, you get an appreciation of how valuable a tool like this could be. These are elementary and middle school essays rife with rudimentary mistakes (which are at times extremely charming) and generally not filled with subtle arguments prone to misinterpretation. If algorithms can help every seventh grader in the country write a solid 5 paragraph essay, that's progress. [1]: http://www.kaggle.com/c/asap-aes/data http://www.kaggle.com/c/asap-aes/data
- gammarator 14y ago
  I participated in the Kaggle competition, but I'm less optimistic about the pedagogic value of these systems. A black box algorithm won't explain to a student what their rudimentary mistakes are. I wrote about this more extensively here: http://bellm.org/blog/2012/04/23/arguing-to-the-algorithm-machine-learned-scoring-of-student-essays/ http://bellm.org/blog/2012/04/23/arguing-to-the-algorithm-ma...
  - numlocked 14y ago
    Great write up! I hadn't seen it, thanks for sharing. Two things: 1) Agree about the explanation bit. In fairness, that was not part of the competition. But I do believe that, given the way the features were constructed, many of the algorithms could be modified to also provide "explanation" of their scoring. The value is highly dependent on the features though (length, for instance, would probably be a prominent "explanation"), and I haven't actually tried it. 2) The "This would grade Dickinson/Hemingway/DFW/etc poorly" argument is true but, again, read the essays. These are serious edge cases - ones that should be addressed (by, for instance, correlating algorithmic scoring with human scoring), but this argument doesn't diminish the applicability of the algorithm to the vast majority of essays and students.
- Jach 14y ago
  Will this help the nearly-illiterate masses? I think so. Will it hurt those who are one standard deviation or more above the norm? I think that's a problem, too. The solution then is to use the algorithm on the first set and more individual-oriented grading methods for the other set. Effort should be spent finding those in the other set and that isn't restricted to a single course or assignment. This challenges the assumption that students are equal and deserve equal grading, but I don't really see another way forward that doesn't hurt any of the best, mediocre, or poor writers.
rwallace 14y ago
> then it would be ok for students to use an algorithm to write the papers no? If a student used an essay-writing algorithm he designed himself, I would certainly consider that okay.

by kenrikm 14y ago

imd 14y ago

The evaluations are on "how well algorithms submitted by professional data scientists and amateur statistics wizards could predict the scores assigned by human graders." As long as some humans are always included as a control, this shouldn't be too big of a problem.

jacktoole1 14y ago

I upvoted you for the interesting line of thought, although I disagree with the premise that acceptability of grading algorithmically implies acceptability of writing algorithmically. In CS, we often grade assignments automatically, although we expect students to write the assignments by hand. Although there are many classes where this would be a bad idea, there are some where it works quite well - if the problems are strictly specified, grading an assignment with what are effectively unit tests is reasonable. Grading and doing an assignment are different classes of problems - roughly mapping to decision problems and search problems. Just because the grading is computationally tractable or acceptable does not necessarily imply that the act of writing the assignment is. I agree with your second paragraph by and large. There is a time and place for automatic grading, and most writing should not fall into that category.

_delirium 14y ago
One difference with CS algorithmic grading is that it's much more specific and causally connected to outcomes. It's not looking for statistical correlates of the input text and a notion of "good program", but is actually finding things like compile errors and failed testcases. And, crucially, it can also report those to the student: your program is wrong because when fed 5 it gives an output of 23 when the output should be 25. It's probably possible to grade student programs like this essay-grading system works: instead of feeding the programs to a compiler and running against test-cases, just turn the source code into a bag-of-words representation and feed it to a statistical classifier. There are almost certainly some statistical correlates between the source code of programs and whether they're good/bad submissions. But the results would be both less reliable and less informative as feedback.
- jacktoole1 14y ago
  Good point; I agree this makes code much more naturally machine graded than essays, and a statistical classifier for source code sounds disturbing (although it might do better on grading style than the usual autograding does).

majormajor 14y ago

For the standardized test discussed in the article, it says the human graders gave three minutes, on average, to reading each essay. These tests are not wanting art. Writing one of these essays is no more art than writing a basic linked list implementation. It's not concerned with how good a writer a kid is, but that the kid can write at all. Edit: basically, the sheer scale of the grading required for a standardized test means that it's never going to be a real test of writing skills. It wouldn't surprise me if overly skilled writing is actually a negative even with the human grader—I could easily see a rushed grader misreading something clever and giving it a low score. (The wisdom or effectiveness of these tests in the first place is a whole different discussion.)

bermanoid 14y ago
"Art" is not even a question - these tests typically apply to high school kids, and you're lucky if bare-bones basic communication is possible there. Most of these kids are straight-up idiots, and I say this having spent several years teaching them. I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT Writing scores. Add in a factor for essay length, unique word cound, and grammar errors, and you'd probably get pretty damn close to perfect, within a reasonable statistical tolerance. Children don't produce art, almost all of them produce garbage, right up to the top 1%. Make no mistake, all that standardized tests aim to measure is how stinky the shit that they produce is. I'm fine with missing a bit of brilliance at the top; those students will do just fine anyways. It's ranking the relative crappitude of the middle that's important here, and I think the standard (if crappy) grading procedures do a pretty good job here.
- kenrikm 14y ago
  Jaded much? Maybe I'm overly optimistic of the quality of students the school system produces? (I never went to high school, I took the GED and scored in the 100th percentile across the board and went directly to college at 16)
- unimpressive 14y ago
  >I'd guess that the algorithm "map number of Microsoft Word reported spelling errors from (0 -> 800) to (20 -> 400)" does a pretty damn good job of guessing students' SAT Writing scores. The best part about that is the fact that people misspell words even with spellcheck! Come on, how hard is it to notice the squiggly lines and correct your error by picking the correct spelling from the drop down list? And I wouldn't say what I produce is shit, but then I would have said that when I was twelve. And I mostly have to double time on projects and essays, so they probably are shit. EDIT: And since losethos (Schizophrenic dude who's hellbanned.) asked, the boss; it's pretty clever to use someones tendency for pedantry against them.

numlocked 14y ago

(disclaimer - I work at Kaggle) I would recommend downloading the dataset[1] and taking a look at some of the essays. There is a tendency for an educated audience to (justifiably) worry about exactly what you've articulated, but when you read through the essays, you get an appreciation of how valuable a tool like this could be. These are elementary and middle school essays rife with rudimentary mistakes (which are at times extremely charming) and generally not filled with subtle arguments prone to misinterpretation. If algorithms can help every seventh grader in the country write a solid 5 paragraph essay, that's progress. [1]: http://www.kaggle.com/c/asap-aes/data http://www.kaggle.com/c/asap-aes/data

gammarator 14y ago
I participated in the Kaggle competition, but I'm less optimistic about the pedagogic value of these systems. A black box algorithm won't explain to a student what their rudimentary mistakes are. I wrote about this more extensively here: http://bellm.org/blog/2012/04/23/arguing-to-the-algorithm-machine-learned-scoring-of-student-essays/ http://bellm.org/blog/2012/04/23/arguing-to-the-algorithm-ma...
- numlocked 14y ago
  Great write up! I hadn't seen it, thanks for sharing. Two things: 1) Agree about the explanation bit. In fairness, that was not part of the competition. But I do believe that, given the way the features were constructed, many of the algorithms could be modified to also provide "explanation" of their scoring. The value is highly dependent on the features though (length, for instance, would probably be a prominent "explanation"), and I haven't actually tried it. 2) The "This would grade Dickinson/Hemingway/DFW/etc poorly" argument is true but, again, read the essays. These are serious edge cases - ones that should be addressed (by, for instance, correlating algorithmic scoring with human scoring), but this argument doesn't diminish the applicability of the algorithm to the vast majority of essays and students.
Jach 14y ago
Will this help the nearly-illiterate masses? I think so. Will it hurt those who are one standard deviation or more above the norm? I think that's a problem, too. The solution then is to use the algorithm on the first set and more individual-oriented grading methods for the other set. Effort should be spent finding those in the other set and that isn't restricted to a single course or assignment. This challenges the assumption that students are equal and deserve equal grading, but I don't really see another way forward that doesn't hurt any of the best, mediocre, or poor writers.

rwallace 14y ago

> then it would be ok for students to use an algorithm to write the papers no? If a student used an essay-writing algorithm he designed himself, I would certainly consider that okay.