5 ms·
I would be interested in hearing the results from humans presented with adversarial answer options. You may say that a machine learning correlations between wor
by wrs 6y ago
I would be interested in hearing the results from humans presented with adversarial answer options. You may say that a machine learning correlations between words isn’t really learning science, but I wonder how many human students aren’t either, just pretty much learning correlations between words to pass tests...
- FiberBundle 6y agoThey do give an example of a question, in which the model chose an incorrect answer in the adversarial setting: "The condition of the air outdoors at a certain time ofday is known as (A) friction (B) light (C) force (D)weather[correct](Q) joule (R) gradient[selected](S)trench (T) add heat" I assume this might be characteristic for other questions as well, although I don't know anything about the Regents Science Exam and whether there are multiple questions about closely related topics.
- taneq 6y agoThat’s a terribly worded question anyway. Of the original answers, ‘weather’ is the least worst but it’s still vague.
- mannykannot 6y agoIt is a well-worded question for its purpose. The whole point is that, of all the options given, only one is justifiable (and it does not require a tendentious stretch to justify it, either.) Even “light” (which was not chosen) only applies half the time, on average. This is a valid test of natural language understanding.
- rvense 6y agoRemember when IBM went on Jeopardy? There was a question about which Egyptian pharaoh. A human with some knowledge of history might mix up Ramses and Seti, or whatever, or just not know the answer, but know that they didn't know. Watson answered "What are trousers?" with supreme confidence. Jeopardy is fun and games and it was great for the blooper reel, but they're trying to sell this stuff to diagnose cancer and guide police efforts. Failure modes are kind of important.
- jacobwilliamroy 6y agoMost multiple choice math problems can be completely circumvented by simply finding the digital root of the expression in the problem. I was surprised to find this to be true, even on college entrance exams.
- flir 6y agoSeriously? Can you explain the mechanism? 'cos that sounds like - well, numerology, to be honest.
- jacobwilliamroy 6y agoThe digital root of the problem expression will be the digital root of the answer, because the answer IS the problem expression. They're the same number, just with all the pieces jumbled up. Comparing against digital roots of answer choices will usually eliminate most of the wrong answers. Any remaining wrong answers tend to be obviously wrong. 2 + 2 != 13 obviously.
- plutonorm 6y agoCan you give an example? I am intrigued, although suspect you are making a joke!
- deleted 6y ago[deleted]
- jacobwilliamroy 6y agoYou can check the properties of digital roots on wikipedia: https://en.wikipedia.org/wiki/Digital_root#Properties https://en.wikipedia.org/wiki/Digital_root#Properties Example from ASVAB practice math test: (x+4)(x+4) = A. x^2+16x+8 B. x^2+16x+16 C. x^2+8x+16 D. x^2+8x+8 Since we don't have to solve for X in this problem, we can just assume x is 1, which would make the digital root of the problem expression 7. Assuming x = 1, the digital roots of A, B, C and D are 7, 6, 7 and 8 respectively. C is the answer because its last term is the square of 4.