3 ms·

"Actually, reasonably capricious investors tend to emulate the science of quantum mechanics enhanced to mars gyroscope. Fourteen decided to whimsically trade sy

by jayajay 10y ago

"Actually, reasonably capricious investors tend to emulate the science of quantum mechanics enhanced to mars gyroscope. Fourteen decided to whimsically trade symphonic trends on the market. Subsequently, eight and over 15.84 percent of astronauts landed on mars with mechanically whirring robotics capable neural networks. Tend to miss the forest for the individual trees. Although over seventeen of our reports turned up airplane maneuver killed over two hundred and fifty police men. After wikipedia, high schools demanded articulation and substantial loss. In conclusion, we have decided that over Harvard nineteen hippopotamus giraffe and lions decimate forests every year. Therefore, carnivorous angiosperm and glycoproteins tend to transpire sodium ions at 456.56 Hz. Thus it can be concluded, therefore, indeed." Got me an A+, according to this algorithm. EDIT: I'm not criticizing you! This is really cool, and you should keep working to improve it. Automating graduate students is definitely gonna happen, and this will bring us one step closer -- if we can work out the kinks. That said, it's always fun to poke and prod the AI nerds :D Makes them better!

glex 10y ago
Thanks for the feedback. Yeah I've noticed it's not too hard to trick the grader into giving you a good grade. That's probably a serious issue with AI graders going forward. Almost a bit too reminiscent of the defeat of Deep Blue.
- homarp 10y ago
  at least detecting the above example is 'easy-ish', as it keeps jumping from one topic to another. So you can use something like 'topic classification' per sentence and detect the jumping around. (e.g. http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/ http://nadbordrozd.github.io/blog/2016/05/20/text-classifica... )
  - jayajay 10y ago
    This is true; language, like other complex systems, exhibits order at various scales. However, even using a "state-of-the-art" algorithm like Wavenet will be insufficient. For example, suppose we implement short range order detection, as you mention. Then, a very well written essay about the history of various metaphors used to describe General Relativity would receive a low score, because we'd be talking about trampolines, marbles, spider webs, and maybe even balloons at the same time. Your algorithm would immediately classify this as nonsense (similar to what I have written above). The problem is not in detecting structural integrity. The problem is in detecting information integrity. A nonsense essay about metaphors cannot be distinguished from a reasonable essay about metaphors unless your network knows what a metaphor is, how isomorphism works, and knows how various ideas relate to that. Long story short, you can't hire an english teacher to grade a paper you're submitting for your GR course. Such a paper would be immune from random insertions of the word "not" or changing random real-valued numbers to other numbers, etc. Any NLP-only algorithm would immediately identify a false positive here. On the other hand, a grammar-only-grader would be reasonable, because grammar is mostly independent of topic, meaning the AI would not need domain knowledge to grade it. To understand language, AI is going to have be a lot more versatile than simply finding order within a paragraph, which is the most naive approach. When a human reads a paragraph, they are playing with much more than simply the words contained inside the paragraph. IIRC, the ML community isn't there yet -- not because they don't know -- but because computing isn't there yet.
rosstex 10y ago
I give you an A+ for effort!

by jayajay 10y ago

glex 10y ago

Thanks for the feedback. Yeah I've noticed it's not too hard to trick the grader into giving you a good grade. That's probably a serious issue with AI graders going forward. Almost a bit too reminiscent of the defeat of Deep Blue.

homarp 10y ago
at least detecting the above example is 'easy-ish', as it keeps jumping from one topic to another. So you can use something like 'topic classification' per sentence and detect the jumping around. (e.g. http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/ http://nadbordrozd.github.io/blog/2016/05/20/text-classifica... )
- jayajay 10y ago
  This is true; language, like other complex systems, exhibits order at various scales. However, even using a "state-of-the-art" algorithm like Wavenet will be insufficient. For example, suppose we implement short range order detection, as you mention. Then, a very well written essay about the history of various metaphors used to describe General Relativity would receive a low score, because we'd be talking about trampolines, marbles, spider webs, and maybe even balloons at the same time. Your algorithm would immediately classify this as nonsense (similar to what I have written above). The problem is not in detecting structural integrity. The problem is in detecting information integrity. A nonsense essay about metaphors cannot be distinguished from a reasonable essay about metaphors unless your network knows what a metaphor is, how isomorphism works, and knows how various ideas relate to that. Long story short, you can't hire an english teacher to grade a paper you're submitting for your GR course. Such a paper would be immune from random insertions of the word "not" or changing random real-valued numbers to other numbers, etc. Any NLP-only algorithm would immediately identify a false positive here. On the other hand, a grammar-only-grader would be reasonable, because grammar is mostly independent of topic, meaning the AI would not need domain knowledge to grade it. To understand language, AI is going to have be a lot more versatile than simply finding order within a paragraph, which is the most naive approach. When a human reads a paragraph, they are playing with much more than simply the words contained inside the paragraph. IIRC, the ML community isn't there yet -- not because they don't know -- but because computing isn't there yet.

rosstex 10y ago

I give you an A+ for effort!