4 ms·
The trivia game (https://t5-trivia.glitch.me/ https://t5-trivia.glitch.me/) needs a little work. > Q: How did Gus Grissom, Ed White and Roger B. Chaffee die in
by modeless 7y ago
The trivia game (https://t5-trivia.glitch.me/ https://t5-trivia.glitch.me/) needs a little work.
> Q: How did Gus Grissom, Ed White and Roger B. Chaffee die in 1967?
> You: "Apollo 1" WRONG
> T5: "They were killed when their Apollo 1 spacecraft exploded" WRONG
> Correct answer: burned to death
> Q: Which Alpine peak is known in Italy as Monte Cervino?
> You: "Monte Cervino" CORRECT
I wonder how many of the problems with this game could be fixed by applying T5 itself to the answer grading.
- craffel 7y agoYes, unfortunately we have to rely on the very brittle "exact match" method of evaluating whether an answer is correct. FWIW and perhaps surprisingly, this is the primary way question-answering systems are evaluated in common benchmarks. I totally agree that fine-tuning T5 for answer grading would be super interesting!
- modeless 7y agoI think it makes some sense to evaluate models like this, as you want to be conservative with the answers you accept (though my second example shows that it isn't always conservative), and models don't have feelings to hurt if they are docked points for not being precise enough. Humans, of course, are more sensitive.
- dmit 7y agoI'm sorry for being blunt, but is it possible that the `very brittle "exact match" method of evaluating whether an answer is correct` means value equality? Is `==` the secret sauce?
- craffel 7y agoIt's slightly more than that -- it also involves lowercasing and removing articles before testing for string equality.
- lsb 7y agoDoes that mean that answer grading would become like comparing summaries of a given text?
- svnpenn 7y agoWhy are you replying to every single comment?
- schoen 7y agoI think craffel (probably "Colin Raffel, Senior Research Scientist, Google Research") was directly involved in this research!
- kps 7y agoReminds me of the early Unix ‘quiz’ game.
- markdog12 7y agoYeah, it asked who was the director of the CIA from 1976-1981. I answered "George Bush" (lazy, typing on phone), it answered the more exact "George H. W. Bush". It marked me correct, but it was marked wrong. It also asked me what country the island of "Honsh" was in (should be Honshu). I answered Japan and was correct, though. It guessed Iceland.
- kazinator 7y agoTIL: Leonardo Di Caprio, not Robert Redford, played Jay Gatsby in a 1974 adaptation of The Great Gatsby.
- deleted 7y ago[deleted]
- Someone 7y ago”Correct answer: burned to death” More likely: suffocated (as with most fire deaths) https://history.nasa.gov/Apollo204/invest.html https://history.nasa.gov/Apollo204/invest.html: ”d. MEDICAL ANALYSIS Loss of consciousness was due to cerebral hypoxia due to cardiac arrest resulting from myocardial hypoxia. Factors of temperature, pressure and environmental concentrations of carbon monoxide, carbon dioxide, oxygen and pulmonary irritants were changing extremely rapidly. It is impossible to integrate these variables on the basis of available information with the dynamic physiological and metabolic conditions they produced, in order to arrive at a precise statement of time when consciousness was lost and when death supervened. The combined effect of these environmental factors dramatically increased the lethal effect of any factor by itself. It is estimated that consciousness was lost between 15 and 30 seconds after the first suit failed. Chances of resuscitation decreased rapidly thereafter and were irrevocably lost within 4 minutes.”