4 ms·

> Developers of so-called “robo-graders” say they understand why many students and teachers would be skeptical of the idea. But they insist, with computers alre

> Developers of so-called “robo-graders” say they understand why many students and teachers would be skeptical of the idea. But they insist, with computers already doing jobs as complicated and as fraught as driving cars, detecting cancer, and carrying on conversations, they can certainly handle grading students’ essays. > One year, she says, a student who wrote a whole page of the letter “b” ended up with a good score. Other students have figured out that they could do well writing one really good paragraph and just copying that four times to make a five-paragraph essay that scores well. Others have pulled one over on the computer by padding their essays with long quotes from the text they’re supposed to analyze, or from the question they’re supposed to answer. The science is just not nearly there yet. We can barely determine the sentiment of IMDB reviews. How should we be able to grade an entire essay? Working in NLP, it really bugs me when people overstate what can be done like this. It creates massive expectations that are bound to fail, and then the entire field suffers.

psandersen 8y ago
So much this, even if you can create a decently accurate ML system to score student essays it absolutely doesn't handle the adversarial nature of scoring and is frankly insulting given how much students are paying for tuition these days. The problem is AI-complete. Its a good tool for feedback and its acceptable for a MOOC or low-cost courses as long as there are adequate disclaimers and the student's final score is determined by a properly human graded assignment or exam.
JustSomeNobody 8y ago
> Working in NLP, it really bugs me when people overstate what can be done like this. But they have to. Otherwise funding goes away. > It creates massive expectations that are bound to fail, and then the entire field suffers. Yes. But by then the people overstating now will have made their money and moved on. I agree with you, but honestly companies and investors are so shallow it isn't hard to see what is going on.
- greenleafjacob 8y ago
  That is fraud.
- crispyambulance 8y ago
  > Yes. But by then the people overstating now will have made their money and moved on. Yep, and those people will likely be sending THEIR kids to elite private schools with 5:1 student-teacher ratios and 20-40K/year tuition for elementary and high school. These schools will DEFINITELY NOT use "AI" for grading.
  - nickserv 8y ago
    It's weird that what is luxury today was once the norm: organic produce, a stay at home parent, natural (as opposed to synthetic) clothing, etc. I won't be surprised if having human teachers / caregivers will be reserved for the upper classes.
  - zeth___ 8y ago
    If you think you can get a 5:1 student ratio on 40k a year I have a bridge you can buy. For a 10:1 you'd be looking at 100k.
swebs 8y ago
>Others have pulled one over on the computer by padding their essays with long quotes from the text they’re supposed to analyze, or from the question they’re supposed to answer. This is a long standing tradition for students on pretty much every humanities essay.
Spooky23 8y ago
I used to tutor kids in AP US History and World History when I was a senior and freshman/sophmore in college. The key success factors were getting good scores on two essays and a document-based longer essay where you would use primary source material provided in the test booklet. These tests were all about test strategy. The written material was graded (in the 90s) by tables of people basically working a checklist for a few different things. If you had insight into how they graded (our teacher had been a grader and had our class start grading each others work when I took the class), you could get a great score even in an area where your knowledge was weak on the essay questions. Another trick was to know 3-4 general concepts from a well regarded historian to support whatever conclusion you cook up. I think you could score these with NLP with a reduced number of humans to fine tune it. (In the 90s, I think up to 4-5 people would touch a paper) The exam would get dumbed down over time as a result as the test developers would narrow the scope of the exam to make scoring better, but I'm sure that's happening anyway as the testing vendors shave cost.

psandersen 8y ago

So much this, even if you can create a decently accurate ML system to score student essays it absolutely doesn't handle the adversarial nature of scoring and is frankly insulting given how much students are paying for tuition these days. The problem is AI-complete. Its a good tool for feedback and its acceptable for a MOOC or low-cost courses as long as there are adequate disclaimers and the student's final score is determined by a properly human graded assignment or exam.

JustSomeNobody 8y ago

> Working in NLP, it really bugs me when people overstate what can be done like this. But they have to. Otherwise funding goes away. > It creates massive expectations that are bound to fail, and then the entire field suffers. Yes. But by then the people overstating now will have made their money and moved on. I agree with you, but honestly companies and investors are so shallow it isn't hard to see what is going on.

greenleafjacob 8y ago
That is fraud.
crispyambulance 8y ago
> Yes. But by then the people overstating now will have made their money and moved on. Yep, and those people will likely be sending THEIR kids to elite private schools with 5:1 student-teacher ratios and 20-40K/year tuition for elementary and high school. These schools will DEFINITELY NOT use "AI" for grading.
- nickserv 8y ago
  It's weird that what is luxury today was once the norm: organic produce, a stay at home parent, natural (as opposed to synthetic) clothing, etc. I won't be surprised if having human teachers / caregivers will be reserved for the upper classes.
- zeth___ 8y ago
  If you think you can get a 5:1 student ratio on 40k a year I have a bridge you can buy. For a 10:1 you'd be looking at 100k.

swebs 8y ago

>Others have pulled one over on the computer by padding their essays with long quotes from the text they’re supposed to analyze, or from the question they’re supposed to answer. This is a long standing tradition for students on pretty much every humanities essay.

Spooky23 8y ago

I used to tutor kids in AP US History and World History when I was a senior and freshman/sophmore in college. The key success factors were getting good scores on two essays and a document-based longer essay where you would use primary source material provided in the test booklet. These tests were all about test strategy. The written material was graded (in the 90s) by tables of people basically working a checklist for a few different things. If you had insight into how they graded (our teacher had been a grader and had our class start grading each others work when I took the class), you could get a great score even in an area where your knowledge was weak on the essay questions. Another trick was to know 3-4 general concepts from a well regarded historian to support whatever conclusion you cook up. I think you could score these with NLP with a reduced number of humans to fine tune it. (In the 90s, I think up to 4-5 people would touch a paper) The exam would get dumbed down over time as a result as the test developers would narrow the scope of the exam to make scoring better, but I'm sure that's happening anyway as the testing vendors shave cost.