6 ms·

Show HN: I made an NLP based essay grader

49 points by glex 10y ago

jayajay 10y ago
"Actually, reasonably capricious investors tend to emulate the science of quantum mechanics enhanced to mars gyroscope. Fourteen decided to whimsically trade symphonic trends on the market. Subsequently, eight and over 15.84 percent of astronauts landed on mars with mechanically whirring robotics capable neural networks. Tend to miss the forest for the individual trees. Although over seventeen of our reports turned up airplane maneuver killed over two hundred and fifty police men. After wikipedia, high schools demanded articulation and substantial loss. In conclusion, we have decided that over Harvard nineteen hippopotamus giraffe and lions decimate forests every year. Therefore, carnivorous angiosperm and glycoproteins tend to transpire sodium ions at 456.56 Hz. Thus it can be concluded, therefore, indeed." Got me an A+, according to this algorithm. EDIT: I'm not criticizing you! This is really cool, and you should keep working to improve it. Automating graduate students is definitely gonna happen, and this will bring us one step closer -- if we can work out the kinks. That said, it's always fun to poke and prod the AI nerds :D Makes them better!
- glex 10y ago
  Thanks for the feedback. Yeah I've noticed it's not too hard to trick the grader into giving you a good grade. That's probably a serious issue with AI graders going forward. Almost a bit too reminiscent of the defeat of Deep Blue.
  - homarp 10y ago
    at least detecting the above example is 'easy-ish', as it keeps jumping from one topic to another. So you can use something like 'topic classification' per sentence and detect the jumping around. (e.g. http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/ http://nadbordrozd.github.io/blog/2016/05/20/text-classifica... )
    - jayajay 10y ago
      This is true; language, like other complex systems, exhibits order at various scales. However, even using a "state-of-the-art" algorithm like Wavenet will be insufficient. For example, suppose we implement short range order detection, as you mention. Then, a very well written essay about the history of various metaphors used to describe General Relativity would receive a low score, because we'd be talking about trampolines, marbles, spider webs, and maybe even balloons at the same time. Your algorithm would immediately classify this as nonsense (similar to what I have written above). The problem is not in detecting structural integrity. The problem is in detecting information integrity. A nonsense essay about metaphors cannot be distinguished from a reasonable essay about metaphors unless your network knows what a metaphor is, how isomorphism works, and knows how various ideas relate to that. Long story short, you can't hire an english teacher to grade a paper you're submitting for your GR course. Such a paper would be immune from random insertions of the word "not" or changing random real-valued numbers to other numbers, etc. Any NLP-only algorithm would immediately identify a false positive here. On the other hand, a grammar-only-grader would be reasonable, because grammar is mostly independent of topic, meaning the AI would not need domain knowledge to grade it. To understand language, AI is going to have be a lot more versatile than simply finding order within a paragraph, which is the most naive approach. When a human reads a paragraph, they are playing with much more than simply the words contained inside the paragraph. IIRC, the ML community isn't there yet -- not because they don't know -- but because computing isn't there yet.
- rosstex 10y ago
  I give you an A+ for effort!
mailmrg 10y ago
can you provide some insight on how its built internally, i am kind of new to NLP.
infradig 10y ago
"The cat ate my homework. The cat ate my homework.The cat ate my homework.The cat ate my homework... Etc" Got me an A-
- asafira 10y ago
  Welcome to America!
neerajk 10y ago
Lincoln's second inaugural is an A- as well, and is estimated as "High School" Grade Level. Harsh. https://www.gradeflex.com/graded/5856996dde6a20085a2d2b97 https://www.gradeflex.com/graded/5856996dde6a20085a2d2b97
- tyingq 10y ago
  Edgar Allen Poe gets a B+ and "Junior High" Grade Level. https://www.gradeflex.com/graded/5856a79dde6a2008572d2b84 https://www.gradeflex.com/graded/5856a79dde6a2008572d2b84
Fnoord 10y ago
Interesting project. It only seemed to work with English. I got no problem with that, but it'd be nice to give the user a warning or note about this. Either a warning/note ahead of time, in the FAQ, and/or if non-English is detected.
mtmail 10y ago
I'm surprised I got a 80/100 on vocabulary when I pasted a German news article.
longlost 10y ago
https://www.gradeflex.com/graded/5856a241de6a2008562d2b88 https://www.gradeflex.com/graded/5856a241de6a2008562d2b88 A+ Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like; Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like. Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like; Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like. Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like?
- Retr0spectrum 10y ago
  So apparently that has perfect grammar?
Houshalter 10y ago
I am fascinated with automatic grading. Predicting the grade of an essay isn't fundamentally different than classifying an email as spam. There was a kaggle competition for automatic grading, and IIRC the best results were pretty close to, or even better than human graders. As NLP improves, this will too. But there is huge resistance to it because of Algorithm Aversion (http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort...). People really distrust statistical algorithms over (even really bad) human judgement. Unless they believe the algorithms are infallible. One interesting thing you can do with automatic grading, is semi-supervised learning. If you know some students are good or bad already (perhaps from multiple choice, or from already graded essays), you can predict the grade of their essays. And then use the predictions to train a more accurate classifier.
- SomeStupidPoint 10y ago
  The reason for not trusting algorithms is that most of them have unexpected failure modes, and then no way to dispute their result through a more holistic (eg, human based) method. Human grading has relatively well understood failure modes and a (semi-)functional dispute system. The danger is that too often when we switch to algorithms, we "forget" (read: often choose not to implement) dispute mechanisms. Im not against algorithmic grading, per se, but I expect that we're at the phase (and will be for some time) where a human and a computer do a better job together than either does alone.
- analog31 10y ago
  A potential issue with algorithms is that the successful ones might become a resilient standard, with nobody knowing why, or being able to challenge it. Suppose computer programs were graded by algorithm. That might be the end of the road for developing new languages and coding styles. And I could imagine a consolidation of the grading tools, down to a small handful, or maybe even just one vendor. This is the pattern in textbook publishing. Once the human graders are laid off and their skill vanishes, education will be wholly dependent on the whims of those businesses. Do you want to hire or work with 1000 engineers, who have all been through exactly the same training, and have adapted their styles to exactly the same template, worldwide? What if a new idea or problem comes along, that isn't addressed within the capabilities established by that template? This is not too far fetched. I was in college when word processing and microcomputers were introduced to academia. I was one of the early adopters at my college. Both of these things radically changed how people did academic work, but a word processed essay might have gotten a low grade if the algorithms were tuned to the established preferences of teachers for reading hand-written essays.
OisinMoran 10y ago
Paul Graham's "Writing, Briefly" gets a B+ and a Grade Level Estimate of Junior High. (https://www.gradeflex.com/graded/5856ad12de6a2008562d2b95 https://www.gradeflex.com/graded/5856ad12de6a2008562d2b95) It seems like the criteria used here would favor longer, more complex, and more formal pieces going against most of the writing advice in Graham's piece above. These would generally match up with how writing is graded in schools though so it makes sense in that regard. Incredibly cool project! Well done. I'd love to read more about the process of making this.
3131s 10y ago
Postmodern Essay Generator gets an A-. https://www.gradeflex.com/graded/5856b8c2de6a20085a2d2bd9 https://www.gradeflex.com/graded/5856b8c2de6a20085a2d2bd9 http://www.elsewhere.org/journal/pomo/ http://www.elsewhere.org/journal/pomo/
MatchaPunk 10y ago
I would consider substituting "sentence length" with "sentence variability". It's a bit harder to cheat, basically a super-set of sentence-length, and more accurate to 'quality writing' IMO.
godelski 10y ago
I pasted in the first chapter to A Tale Of Two Cities. Got an F. https://www.gradeflex.com/graded/5856c806de6a2008572d2b9d https://www.gradeflex.com/graded/5856c806de6a2008572d2b9d
bobochan 10y ago
I had a history teacher in college that assigned a 20 page paper. It was one of the biggest classes at the school, 100 students, so he had about 2,000 pages of grading to do. I got a decent grade, an A- if I recall, and two pages of handwritten notes about which arguments I could strengthen and sources that would have bolstered my arguments. Finally, the entire paper was marked in red with style and grammar notes, including the bibliography. Grading is indeed time consuming, but it is one of the best opportunities that we have to give personalized feedback to our students. If we throw away that chance to analyze, consider, and connect, then we might as well substitute YouTube videos for all of education and be done with it.
deepnotderp 10y ago
This is clearly giving out terrible grades. If anyone is interested in something real like this, check out gradescope. It's by pieter abbeel and his phd students, top researchers in Deep reinforcement learning. It uses a similarity based system to assign scores from humans to new essays. Disclaimer: I'm not associated with gradescope.
- rosstex 10y ago
  If I'm not mistaken, the GRE test grades each essay using one human and one automated grader, and if their scores differ by >1, it goes to a second human for arbitration.
Dowwie 10y ago
I put the Declaration of Independence through the engine. It seems the NLP has some strong views about our president elect: http://imgur.com/a/pjsQy http://imgur.com/a/pjsQy source: http://www.ushistory.org/declaration/document/ http://www.ushistory.org/declaration/document/
rm_-rf_slash 10y ago
I did a multiple-choice grader with image recognition that operated at a fraction of the exorbitant Scantron costs and at the time I thought we had a business model in the bag. Turned out that none of the schools we contacted wanted to actually implement our system because 1: they didn't want to switch from all-paper to paper-digital now to all-digital in the future because education moves at geologic timescales, and 2: nobody had tried it before, so if something went wrong, whoever vouched for the program would get the blame. Just something to consider if you plan to commercialize this application. CYA matters a lot on purchasing decisions.

49 points by glex 10y ago

"Actually, reasonably capricious investors tend to emulate the science of quantum mechanics enhanced to mars gyroscope. Fourteen decided to whimsically trade symphonic trends on the market. Subsequently, eight and over 15.84 percent of astronauts landed on mars with mechanically whirring robotics capable neural networks. Tend to miss the forest for the individual trees. Although over seventeen of our reports turned up airplane maneuver killed over two hundred and fifty police men. After wikipedia, high schools demanded articulation and substantial loss. In conclusion, we have decided that over Harvard nineteen hippopotamus giraffe and lions decimate forests every year. Therefore, carnivorous angiosperm and glycoproteins tend to transpire sodium ions at 456.56 Hz. Thus it can be concluded, therefore, indeed." Got me an A+, according to this algorithm. EDIT: I'm not criticizing you! This is really cool, and you should keep working to improve it. Automating graduate students is definitely gonna happen, and this will bring us one step closer -- if we can work out the kinks. That said, it's always fun to poke and prod the AI nerds :D Makes them better!

glex 10y ago
Thanks for the feedback. Yeah I've noticed it's not too hard to trick the grader into giving you a good grade. That's probably a serious issue with AI graders going forward. Almost a bit too reminiscent of the defeat of Deep Blue.
- homarp 10y ago
  at least detecting the above example is 'easy-ish', as it keeps jumping from one topic to another. So you can use something like 'topic classification' per sentence and detect the jumping around. (e.g. http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/ http://nadbordrozd.github.io/blog/2016/05/20/text-classifica... )
  - jayajay 10y ago
    This is true; language, like other complex systems, exhibits order at various scales. However, even using a "state-of-the-art" algorithm like Wavenet will be insufficient. For example, suppose we implement short range order detection, as you mention. Then, a very well written essay about the history of various metaphors used to describe General Relativity would receive a low score, because we'd be talking about trampolines, marbles, spider webs, and maybe even balloons at the same time. Your algorithm would immediately classify this as nonsense (similar to what I have written above). The problem is not in detecting structural integrity. The problem is in detecting information integrity. A nonsense essay about metaphors cannot be distinguished from a reasonable essay about metaphors unless your network knows what a metaphor is, how isomorphism works, and knows how various ideas relate to that. Long story short, you can't hire an english teacher to grade a paper you're submitting for your GR course. Such a paper would be immune from random insertions of the word "not" or changing random real-valued numbers to other numbers, etc. Any NLP-only algorithm would immediately identify a false positive here. On the other hand, a grammar-only-grader would be reasonable, because grammar is mostly independent of topic, meaning the AI would not need domain knowledge to grade it. To understand language, AI is going to have be a lot more versatile than simply finding order within a paragraph, which is the most naive approach. When a human reads a paragraph, they are playing with much more than simply the words contained inside the paragraph. IIRC, the ML community isn't there yet -- not because they don't know -- but because computing isn't there yet.
rosstex 10y ago
I give you an A+ for effort!

mailmrg 10y ago

can you provide some insight on how its built internally, i am kind of new to NLP.

infradig 10y ago

"The cat ate my homework. The cat ate my homework.The cat ate my homework.The cat ate my homework... Etc" Got me an A-

asafira 10y ago
Welcome to America!

neerajk 10y ago

Lincoln's second inaugural is an A- as well, and is estimated as "High School" Grade Level. Harsh. https://www.gradeflex.com/graded/5856996dde6a20085a2d2b97 https://www.gradeflex.com/graded/5856996dde6a20085a2d2b97

tyingq 10y ago
Edgar Allen Poe gets a B+ and "Junior High" Grade Level. https://www.gradeflex.com/graded/5856a79dde6a2008572d2b84 https://www.gradeflex.com/graded/5856a79dde6a2008572d2b84

Fnoord 10y ago

Interesting project. It only seemed to work with English. I got no problem with that, but it'd be nice to give the user a warning or note about this. Either a warning/note ahead of time, in the FAQ, and/or if non-English is detected.

mtmail 10y ago

I'm surprised I got a 80/100 on vocabulary when I pasted a German news article.

longlost 10y ago

https://www.gradeflex.com/graded/5856a241de6a2008562d2b88 https://www.gradeflex.com/graded/5856a241de6a2008562d2b88 A+ Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like; Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like. Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like? Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like; Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like. Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like, Has Anyone Really Been Far Even as Decided to Use Even Go Want to do Look More Like?

Retr0spectrum 10y ago
So apparently that has perfect grammar?

Houshalter 10y ago

I am fascinated with automatic grading. Predicting the grade of an essay isn't fundamentally different than classifying an email as spam. There was a kaggle competition for automatic grading, and IIRC the best results were pretty close to, or even better than human graders. As NLP improves, this will too. But there is huge resistance to it because of Algorithm Aversion (http://opim.wharton.upenn.edu/risk/library/WPAF201410-AlgorthimAversion-Dietvorst-Simmons-Massey.pdf http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort...). People really distrust statistical algorithms over (even really bad) human judgement. Unless they believe the algorithms are infallible. One interesting thing you can do with automatic grading, is semi-supervised learning. If you know some students are good or bad already (perhaps from multiple choice, or from already graded essays), you can predict the grade of their essays. And then use the predictions to train a more accurate classifier.

SomeStupidPoint 10y ago
The reason for not trusting algorithms is that most of them have unexpected failure modes, and then no way to dispute their result through a more holistic (eg, human based) method. Human grading has relatively well understood failure modes and a (semi-)functional dispute system. The danger is that too often when we switch to algorithms, we "forget" (read: often choose not to implement) dispute mechanisms. Im not against algorithmic grading, per se, but I expect that we're at the phase (and will be for some time) where a human and a computer do a better job together than either does alone.
analog31 10y ago
A potential issue with algorithms is that the successful ones might become a resilient standard, with nobody knowing why, or being able to challenge it. Suppose computer programs were graded by algorithm. That might be the end of the road for developing new languages and coding styles. And I could imagine a consolidation of the grading tools, down to a small handful, or maybe even just one vendor. This is the pattern in textbook publishing. Once the human graders are laid off and their skill vanishes, education will be wholly dependent on the whims of those businesses. Do you want to hire or work with 1000 engineers, who have all been through exactly the same training, and have adapted their styles to exactly the same template, worldwide? What if a new idea or problem comes along, that isn't addressed within the capabilities established by that template? This is not too far fetched. I was in college when word processing and microcomputers were introduced to academia. I was one of the early adopters at my college. Both of these things radically changed how people did academic work, but a word processed essay might have gotten a low grade if the algorithms were tuned to the established preferences of teachers for reading hand-written essays.

OisinMoran 10y ago

Paul Graham's "Writing, Briefly" gets a B+ and a Grade Level Estimate of Junior High. (https://www.gradeflex.com/graded/5856ad12de6a2008562d2b95 https://www.gradeflex.com/graded/5856ad12de6a2008562d2b95) It seems like the criteria used here would favor longer, more complex, and more formal pieces going against most of the writing advice in Graham's piece above. These would generally match up with how writing is graded in schools though so it makes sense in that regard. Incredibly cool project! Well done. I'd love to read more about the process of making this.

3131s 10y ago

Postmodern Essay Generator gets an A-. https://www.gradeflex.com/graded/5856b8c2de6a20085a2d2bd9 https://www.gradeflex.com/graded/5856b8c2de6a20085a2d2bd9 http://www.elsewhere.org/journal/pomo/ http://www.elsewhere.org/journal/pomo/

MatchaPunk 10y ago

I would consider substituting "sentence length" with "sentence variability". It's a bit harder to cheat, basically a super-set of sentence-length, and more accurate to 'quality writing' IMO.

godelski 10y ago

I pasted in the first chapter to A Tale Of Two Cities. Got an F. https://www.gradeflex.com/graded/5856c806de6a2008572d2b9d https://www.gradeflex.com/graded/5856c806de6a2008572d2b9d

bobochan 10y ago

I had a history teacher in college that assigned a 20 page paper. It was one of the biggest classes at the school, 100 students, so he had about 2,000 pages of grading to do. I got a decent grade, an A- if I recall, and two pages of handwritten notes about which arguments I could strengthen and sources that would have bolstered my arguments. Finally, the entire paper was marked in red with style and grammar notes, including the bibliography. Grading is indeed time consuming, but it is one of the best opportunities that we have to give personalized feedback to our students. If we throw away that chance to analyze, consider, and connect, then we might as well substitute YouTube videos for all of education and be done with it.

deepnotderp 10y ago

This is clearly giving out terrible grades. If anyone is interested in something real like this, check out gradescope. It's by pieter abbeel and his phd students, top researchers in Deep reinforcement learning. It uses a similarity based system to assign scores from humans to new essays. Disclaimer: I'm not associated with gradescope.

rosstex 10y ago
If I'm not mistaken, the GRE test grades each essay using one human and one automated grader, and if their scores differ by >1, it goes to a second human for arbitration.

Dowwie 10y ago

I put the Declaration of Independence through the engine. It seems the NLP has some strong views about our president elect: http://imgur.com/a/pjsQy http://imgur.com/a/pjsQy source: http://www.ushistory.org/declaration/document/ http://www.ushistory.org/declaration/document/

rm_-rf_slash 10y ago

I did a multiple-choice grader with image recognition that operated at a fraction of the exorbitant Scantron costs and at the time I thought we had a business model in the bag. Turned out that none of the schools we contacted wanted to actually implement our system because 1: they didn't want to switch from all-paper to paper-digital now to all-digital in the future because education moves at geologic timescales, and 2: nobody had tried it before, so if something went wrong, whoever vouched for the program would get the blame. Just something to consider if you plan to commercialize this application. CYA matters a lot on purchasing decisions.