33 ms·

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

303 points by alexwg 7y ago

alexwg 7y ago
Paper: https://arxiv.org/abs/1910.10683 https://arxiv.org/abs/1910.10683
- throwaway_bad 7y ago
  Twitter summary: https://threadreaderapp.com/thread/1187161460033458177.html https://threadreaderapp.com/thread/1187161460033458177.html
martincmartin 7y ago
"The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems." "We take into account the lessons learnt from original GLUE benchmark and present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard."
pmoriarty 7y ago
There was an article[1] posted to HN recently about these benchmarks, and it was pretty skeptical. Regarding SuperGLUE specifically, it asked: "Indeed, Bowman and his collaborators recently introduced a test called SuperGLUE that's specifically designed to be hard for BERT-based systems. So far, no neural network can beat human performance on it. But even if (or when) it happens, does it mean that machines can really understand language any better than before? Or does just it mean that science has gotten better at teaching machines to the test?" [1] - https://www.quantamagazine.org/machines-beat-humans-on-a-reading-test-but-do-they-understand-20191017 https://www.quantamagazine.org/machines-beat-humans-on-a-rea...
- gradys 7y ago
  This feels hollow. Can't this be said about any benchmark? It seems natural and proper that as one benchmark becomes saturated, we introduce harder benchmarks. I don't think anyone in the field thinks that once we match human performance on benchmark X, we're officially done. It just means it's time for more interesting benchmarks. Over time, if it starts to become difficult to design benchmarks that humans can outperform machines on, then that will prompt interesting conceptual work about what exactly the difference between human and machine language competency is. And then that will lead either to more sophisticated benchmarks or alternatively gradually more sophisticated and persuasive arguments that machines really have surpassed us in language competence. I don't think we're yet at a point where we don't know how to make harder benchmarks, and if and when we do hit such a point, I'd definitely bet the result will be a conceptual advance in benchmark design rather than declaring machine superiority once and for all. At least for the first few rounds of this cycle.
  - not2b 7y ago
    From the Quanta article: "But instead of concluding that BERT could apparently imbue neural networks with near-Aristotelian reasoning skills, they suspected a simpler explanation: that BERT was picking up on superficial patterns in the way the warrants were phrased. Indeed, after re-analyzing their training data, the authors found ample evidence of these so-called spurious cues. For example, simply choosing a warrant with the word “not” in it led to correct answers 61% of the time. After these patterns were scrubbed from the data, BERT’s score dropped from 77 to 53 — equivalent to random guessing."
    - nl 7y ago
      This is true, and absolutely a weakness of these tests. However they don't publish how well a human performs on the dataset without "not" in it. They do initially note that Even human beings don’t do particularly well on this task without practice I've looked at the warrant task. It's pretty tricky! I'd bet real money that untrained humans perform much, much lower than the 80% correct rate they get on the full set on ones without "not". I don't think it would be as low as the 53% BERT gets, but it would drop significantly. I find the HANS analysis[1] much more compelling, but again I'd note that humans suffer on this dataset too (although again - not as badly as models do). [1] https://www.aclweb.org/anthology/P19-1334.pdf https://www.aclweb.org/anthology/P19-1334.pdf
- ehsankia 7y ago
  As long as they're not training on the test data, and they're not submitting hundreds of submissions tweaking parameters trying to improve their score, I don't see what the problem is. If the algorithm can do a great job at classifying hundreds of new test cases it has never seen, and it isn't over-fitted, then that means it is good at that specific task. Of course the task itself may or may not be useful, and you can have some meta discussion about what "understanding language" is, but the computer definitely is doing a super human job at that given task.
  - blazespin 7y ago
    Maybe it's over-fitted on the new data. There has to be a constant infusion of new training data and a system can only prove itself over time. These rankings, if real, should be in constant flux.
- Iv 7y ago
  Even when you will be able to have a 100% coherent and deep discussion with an AI over a niche technical domain, there will be people to pretend that the AI "fakes" it. Systems like GPT-2, incredibly (I used to be a skeptic of a pure statistical approach) manage to extract meaning, keep a theme, and understand the intent behind a sentence. They are amazing. When you have a system that displays all the characteristics of understanding something, it is irrelevant whether or not it "fakes" it. No one ever proved that humans are not "faking" intelligence either.
  - deleted 7y ago
    [deleted]
- nl 7y ago
  (I work in this field, although not specifically on benchmarking) I think that this article makes a good point, and correctly identifies weaknesses. However, I also think that humans often take very similar shortcuts. There are good reasons why "bag of words" approaches work much of the time. Additionally there's lots of evidence showing that very rapid reading by humans does not imply deep understanding. I think it's very important that people are aware of the weaknesses of these types of models. However, I think it's interesting that these weaknesses are becoming harder and harder to find.
- make3 7y ago
  the machines are always trained with the same dataset for each task. the biggest difference right now is small technical modifications on models that are also pre trained on gigantic unlabelled datasets. this doesn't feel like we're teaching them to do the test specifically at all
hn_throwaway_99 7y ago
I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ https://pilehvar.github.io/wic/) One of their examples, though, didn't make any sense to me: 1. The pilot managed to land the airplane safely 2. The enemy landed several of our aircrafts It says that the word "land" does NOT mean the same thing in those sentences. I am a native English speaker, and I honestly don't understand what they are thinking the second sentence means. Shot them down? If so, I have never heard "landed" used in that context, and it appears neither has Merriam-Webster. Also, the plural of aircraft is just "aircraft", without the s.
- mooman219 7y ago
  This is definitely where the 10.2% of human failures are.
  - jdale27 7y ago
    .
    - p1esk 7y ago
      Ever tried taking SAT or GRE tests?
- matthewowen 7y ago
  I think it's landed in the same sense as "landed a deal": got, or achieved, in this case achieving shooting them down. For me, my first read of the sentence would definitely be that it means shot down.
  - otterpop 7y ago
    so this is why the human score is 89.8 :)
throwaway_bad 7y ago
Possibly dumb question: How do you ensure there's no data leakage when benchmarking transfer learning techniques? Is that even a problem anymore when the whole point is to learn "common sense" knowledge? For example their “Colossal Clean Crawled Corpus” (C4), a dataset consisting of hundreds of gigabytes of clean English text scraped from the web, might contain much of the same information as the benchmark datasets, which I presume is also scraped from the web.
- taneq 7y ago
  You don't. Even humans frequently leak information like this. It's just a consequence of having incomplete or incompletely analyzed information.
- calabin 7y ago
  I think that this is a good question that I would also like to know the answer to. Additionally, are there other benchmarks or tests where this issue (possibly) presents itself?
- jcims 7y ago
  Not dumb at all and probably a major challenge when developing benchmarks.
- craffel 7y ago
  Hi, one of the paper authors here. Indeed this is a good question. A couple of comments: - Common Crawl overall is a sparse web dump, it is unlikely that the month we used includes any of the data that are in any of the test sets. - In order for the data to be useful to our model, it would have to be in the correct preprocessed format. ("mnli: hypothesis: ... premise: ...") with the label in a format our model could extract meaning from. We introduced this preprocessing format so I don't believe this would ever happen. - Further, most of these datasets live in .zip files. The Common Crawl dump doesn't unzip zip files. - C4 is so large that our model sees each example (corresponding to a block of text from a website) roughly once ever over the entire course of training. Big neural nets trained with SGD are unlikely to memorize something if they only see it once over the course of one million training steps.
lettergram 7y ago
Although those are some great results, I wish I could try it out locally... https://github.com/google-research/text-to-text-transfer-transformer https://github.com/google-research/text-to-text-transfer-tra... It drives me nuts that most of these papers / publications don't have code where I can just run: > python evaluate_model.py Still exciting, just annoying that I'd have to set up google cloud to try this out.
- ehsankia 7y ago
  They often do setup Python notebooks / Colabs you can simply run, especially with the data hosted on GCloud. Unfortunately not this time.
rrival 7y ago
Where do I take the SuperGLUE test?
riku_iki 7y ago
> T5-11B (11 billion parameters) So, this is largest language model so far?
- lucidrains 7y ago
  Yes. The last one was 8.3B https://arxiv.org/pdf/1909.08053.pdf https://arxiv.org/pdf/1909.08053.pdf
- deleted 7y ago
  [deleted]
vonseel 7y ago
I wonder what I would score on this test. Are these things correlated to standardized test scores at all for humans ?
RcouF1uZ4gsC 7y ago
I think classifying this as human level is misleading. Look at the sub-scores on the page. One score that looks very different from humans is AX-b. The SuperGlue paper provides more context about AX-b https://arxiv.org/pdf/1905.00537.pdf https://arxiv.org/pdf/1905.00537.pdf AX-b "is the broad-coverage diagnostic task, scored using Matthews’ correlation (MCC). " This is how the paper describes this test " Analyzing Linguistic and World Knowledge in Models GLUE includes an expert-constructed, diagnostic dataset that automatically tests models for a broad range of linguistic, commonsense, and world knowledge. Each example in this broad-coverage diagnostic is a sentence pair labeled with a three-way entailment relation (entailment, neutral, or contradiction) and tagged with labels that indicate the phenomena that characterize the relationship between the two sentences. Submissions to the GLUE leaderboard are required to include predictions from the submission’s MultiNLI classifier on the diagnostic dataset, and analyses of the results were shown alongside the main leaderboard. Since this broad-coverage diagnostic task has proved difficult for top models, we retain it in SuperGLUE. However, since MultiNLI is not part of SuperGLUE, we collapse contradiction and neutral into a single not_entailment label, and request that submissions include predictions on the resulting set from the model used for the RTE task. We collect non-expert annotations to estimate human performance, following the same procedure we use for the main benchmark tasks (Section 5.2). We estimate an accuracy of 88% and a Matthew’s correlation coefficient (MCC, the two-class variant of the R3 metric used in GLUE) of 0.77. " If you look at the scores, humans are estimated to score 0.77. Google T5 scores -0.4 on the test. How did T5 get such a high score if it scored so abysmally on the AX-b test? The AX scores are not included in the total score. From the paper: "The Avg column is the overall benchmarkscore on non-AX∗ tasks." If the AX scores were included, the gap between humans and machines would be bigger than the current score indicates.
- foota 7y ago
  I find it strange that they exclude it? Perhaps the reason is related?
- craffel 7y ago
  Hi, one of the paper's authors here. We didn't submit our model's predictions for the AX-b task yet, we just copied over the predictions from the example submission. We will submit predictions for AX-b in the next few days.
6gvONxR4sf7o 7y ago
One thing to always point out in these cases is that the human baseline isn't "how well people do at this task," like it's often hyped to be. It's "how well does a person quickly and repetitively doing this do, on average." The 'quickly and repetitively' part is important because we all make more boneheaded errors in this scenario. The 'on average' part is important because the errors the algo makes aren't just fewer than people, they're different. The algos often still get certain things wrong that humans almost never would. This is really really super great, let's be clear. It's just not up to the hype "omg super human" usually gets.
- jcims 7y ago
  Great point! It makes sense in the context of what these algorithms would generally be tasked with.
- IshKebab 7y ago
  Regarding the type of errors, it seems like the benchmark should be able to take that into account. That is, get a load of humans to do the task on the same specific examples, then for each example you know how hard it is, and what acceptable answers are (I bet a lot of the ground truth is wrong or ambiguous). Then you can benchmark your AI but penalise it more heavily for getting things wrong that are obvious to a human.
  - 6gvONxR4sf7o 7y ago
    That would be ideal, if money weren't a factor. Since money is a factor, I wonder what the tradeoff is between labelling each instance N more times versus just getting N times more instances labeled.
- TheOtherHobbes 7y ago
  It seems to mean "How well does Mechanical Turk do the task?" which is a separate thing again. And yes - error type is at least as revealing as error frequency. I have no idea where the real human baseline is, or how to find it. Also, consider this discussion. GLUE winners may be able to make informed parsing guesses about single text blocks, but they're years away from being able to make a useful contribution to a discussion like this one.
woodgrainz 7y ago
Several of the systems in this leaderboard utilize the BERT model, a clever approach devised by Google for natural language processing. A nice laymen's guide to BERT: https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270 https://towardsdatascience.com/bert-explained-state-of-the-a...
skybrian 7y ago
They came up with the SuperGLUE benchmark because they found that the GLUE benchmark was flawed and too easy to game. There were correlations in the dataset that made it possible to get questions right without real understanding, and so the results didn't generalize. Could the same thing happen again with the better benchmark due to more subtle correlations? These things are tough to judge, so I'd say wait and see if it turns out to be a real result.
enisberk 7y ago
I attended one of the talks(1) of the Sam Bowman. His talk was about "Task-Independent Language Understanding" and he also talked about GLUE and super GLUE; he mentioned that some models are passing an average person in experiments. They did some experiments to understand BERT's performance (2). (similar to article 'NLP's Clever Hans Moment') But they found a different answer to question "what BERT really knows," so he was skeptical about all conclusions. Check these out if you are interested in. (1)[https://www.nyu.edu/projects/bowman/TILU-talk-19-09.pdf https://www.nyu.edu/projects/bowman/TILU-talk-19-09.pdf] (2)[https://arxiv.org/abs/1905.06316 https://arxiv.org/abs/1905.06316]
YeGoblynQueenne 7y ago
Assuming that the baseline human score was set according to the performance of adult humans, then according to these results T5 has a language understanding ability at least as accurate as a human child. In fact it's not just T5 that should be able to understand language as well as a human child, but also BERT++, BERT-mtl and RoBERTa, each of which has a score of 70 or more. There really shouldn't be anything else on the planet that has 70% of human language understanding, other than humans. So if the benchmarks mean what they think they mean, there are currently fully-fledged strongly artificially intelligent systems. That must mean that, in a very short time we should see strong evidence of having created human-like intelligence. Because make no mistake: language understanding is not like image recognition, say, or speech processing. Understanding anything is an AI-complete task, to use a colloquial term. Let's wait and see then. It shouldn't take more than five or six years to figure out what all this means.
- YeGoblynQueenne 7y ago
  To clarify, I meant this comment as an expression of skepticism- I don't believe that the SuperGLUE benchmark really evaluates language understanding, or that BERT and friends are within a few percents of human language understanding. I think SuperGLUE is just another benchmark that is measuring something else than what it's supposed to be measuring (machine learning benchmarks usually do). It seems that the teams behind the attempts to beat such benchmarks are aware of the weaknesses of the benchmarks though, so that's encouraging.
femto113 7y ago
My experience with image classification benchmarks was that they approached human levels only because the scoring only counts how much they get “right” and doesn’t penalize completely whack answers as much as they should (like getting full credit for being pretty sure a picture of a dog was either a dog or an alligator). I suspect there’s something similar going on in these language benchmarks.
nightnight 7y ago
This clearly demonstrates once again that Google is miles ahead of the competition in AI. I mean, they just have the best data. If you want to have an every day example of Google's AI skills: Switch you phone's keyboard to GBoard, especially all iOS users, and you will face a night and day difference to any other keyboard esepcially the stock one. When using multiple languages at the same time the leap to other keyboards gets even bigger. GBoard is my phone's killer app and if Google dropped it for iOS I'd left the same day to Android.
- dingle_thunk 7y ago
  Have you tried the iOS 13 keyboard's built in swipe feature? Are you aware of Swiftkey?
  - occamrazor 7y ago
    I think GP is talking about predictive text, rather than keyboard ergonomics.
- tremon 7y ago
  I have the opposite experience. Yes, some of the suggestions from GBoard are useful, but I feel there's an equal number of times where I've typed a complete word, only to hit space and have the word auto-corrected to what GBoard was expecting. As a typing aid, it's almost unusable because of that.
- htfu 7y ago
  That's how I used to feel, but it's turning into a nuisance. It used to stick to single words or sometimes splitting one if missing a space, but now will sometimes attempt to "correct" the sum of two perfectly valid standalone words after the fact, 97% of the time resulting in nonsense. I cannot for the life of me understand why.
ilaksh 7y ago
The AIs in the benchmark are all trained exclusively on text, correct? My assumption has always been that to get human-level understanding, the AI systems need to be trained on things like visual data in addition to text. This is because there is a fair amount of information that is not encoded at all in text, or at least is not described in enough detail. I mean, humans can't learn to understand language properly without using their other senses. You need something visual or auditory or to associate with the words which are really supposed to represent full systems that are complex and detailed. I think it would be much more obvious if there were questions that involved things like spatial reasoning, or combining image recognition with that and comprehension.
- BiasRegularizer 7y ago
  I think maybe CLEVR[0] dataset is what you are talking about? Keep in mind that a most of the current ML systems have diverged from biology. A majority of the recent breakthroughs come from mathematics, the rational is that just because human brain does it in a certain way does not necessarily mean it is the only way to do it. [0] https://cs.stanford.edu/people/jcjohns/clevr/ https://cs.stanford.edu/people/jcjohns/clevr/
  - ilaksh 7y ago
    In no way do I think that AGI needs to mimic animal/human intelligence. I was just trying to explain why text input alone isn't going to be adequate and that was an example. Thanks for the link, that is one example of the type of thing I was talking about I think.
- visarga 7y ago
  It's not just grounding the language in vision, but the embodiment, first person perspective and ability to interact with the environment. Humans have had the benefit of slowly evolving in a complex environment which is too expensive to recreate for artificial agents. We can only create very limited sims vs the real world.
- deleted 7y ago
  [deleted]
nopinsight 7y ago
As someone working in the field, I congratulate the excellent accomplishment but agree with the authors that we shouldn't get too excited yet (their quote below after the four reasons). Here are some reasons: 1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here: http://www.ericswallace.com/triggers http://www.ericswallace.com/triggers 2) T5 was trained with ~750GB of texts or ~150 billion words, which is > 100 times the number of words native English speakers acquire by the age of 20. 3) Most or all of the tests are multiple-choice. Learning complex correlations from sufficient data should help solve most of them. This is useful but human-level understanding is more than correlations. 4) The performance on datasets that require commonsense knowledge, COPA and WSC, are the weakest relative to humans (who score 100.0 on both). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, p.32 https://arxiv.org/pdf/1910.10683.pdf https://arxiv.org/pdf/1910.10683.pdf "Interestingly, on the reading comprehension tasks (MultiRC and ReCoRD) we exceed human performance by a large margin, suggesting the evaluation metrics used for these tasks may be biased towards machine-made predictions. On the other hand, humans achieve 100% accuracy on both COPA and WSC, which is significantly better than our model’s performance. This suggests that there remain linguistic tasks that are hard for our model to perfect, particularly in the low-resource setting." I’d like to emphasize that the work and the paper are excellent. Still, we are quite far from human-level language understanding. --- We may need more advanced tests to probe the actual language understanding ability of AI systems. Here are some ideas: * Test for conceptual understanding in a non-multiple-choice format. Example: Write a summary for a New Yorker article, rather than standard news pieces (which tend to follow repeated patterns). * Commonsense test with longer chains of inference than those needed for solving Winograd Schema and set in non-standard situations (e.g. fantasy world). This should greatly reduce the chance that an approach can simply detect correlations from huge datasets. * Understanding novel, creative metaphors like those used in some essays by professional writers or some of the Economist's title articles.
- VikingCoder 7y ago
  > 2) T5 was trained with ~750GB of texts or ~150 billion words, which is > 100 times the number of words native English speakers acquire by the age of 20. ...but, humans evolved the ability to use language over hundreds of generations... So... Maybe that's not such a bad thing?
LukeB42 7y ago
http://www.irc.org/history_docs/tao.html http://www.irc.org/history_docs/tao.html
deleted 7y ago
[deleted]
pauljurczak 7y ago
Use of Natural Language Understanding term in context of this benchmark is preposterous. No understanding takes place there. Please stick to NLP (Natural Language Processing) term for the next couple of decades. Thank you.
- deleted 7y ago
  [deleted]
Al-Khwarizmi 7y ago
This surprised me a bit, on the creation of the corpus they use for training: "We removed any page that contained any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”." I don't understand this decision. This list contains words that can be used in a perfectly objective sense, like "anus", "bastard", "erotic", "eunuch", "fecal", etc. I can understand that they want to avoid websites full of expletives and with no useful content, but outright excluding any website with even one occurrence of such words sounds too radical. If we ask this model a text comprehension question about a legitimized bastard that inherited the throne, or about fecal transplants, I suppose it would easily fail. Strange way of limiting such a powerful model.
- Veedrac 7y ago
  They say they removed pages, not websites. Having false positives isn't a problem when you're still left with 750GB of data—quality matters more than slightly higher quantity at that point.
  - Al-Khwarizmi 7y ago
    Sorry, I was thinking about pages even though I said websites. Native language interference (typically, we use the same term for pages and websites in my language). Anyway, my point is not a matter of quantity. The way they're doing it, they have 750 GB of data, but they have exactly zero data that talks about bastards, fecal transplants, etc. So they may have a hard time answering questions about those specific subjects.
vagab0nd 7y ago
This is cool. Since they released a 11B pre-trained model, can we finally reproduce "unicorn-level" text generation now?
- penagwin 7y ago
  My understanding is that a lot of these really high performance models that reach for every percentage-point possible require an absurd amount of hardware - specifically an absurd amount of GPU memory. For example I have what I consider a fairly "high end" rig for being a hobbyist individual, with 32GB of RAM, i7 8700k, 1080ti - there's 0 chance their model would fit on my system. So I mean maybe if you have a ton of money? Usually what happens is a slimmer model with not "quite" as high of a score gets released that actually fits on consumer hardware.
  - vagab0nd 7y ago
    Maybe I'm oversimplifying, but it seems to me that once you have the model trained, it should be possible to partition it somehow when inferencing, to fit smaller machines. At least for a proof of concept it should be possible.
    - penagwin 7y ago
      The problem is that there is so many weights in the model that they don't fit in memory. You can lower the number of weights, which will lower the effectiveness of the model. The thing is that when you're going for leaderboards you're reaching for every last percentage point, so the efficiency of the model size/performance isn't a concern, you want to ramp up the resource usage to as you have access to. TL;DR - Yeah basically most people will run a "slimmed down" version of the model that isn't "as" performant, but is still an improvement over previous models and actually fits on your machine.
    - nmfisher 7y ago
      I'm not aware of any "partioning" strategies per se (at least during inference), but it's now common practice to distill a larger model to a smaller one by either (a) training a smaller "student" network to replicate the larger "teacher" network, or (b) pruning smaller weights from the larger network to reduce the size. Just brainstorming here, but a vanilla network partition strategy might be to load each layer's weight into memory and perform the forward pass sequentially. I think that would be prohibitively slow - some of these models (e.g. BERT) can already take up to 3-4 seconds to perform a single forward pass on a CPU, and that's with all model weights already loaded into main memory. I suspect fetching/loading each layer separately would blow this out by an order of magnitude.
ArtWomb 7y ago
"Attention is all you need", indeed. Of course, our instinct tells us there is more to language inference than word proximity. And so results approaching or exceeding expert-level human baseline raise more questions than providing cause for popping champagne corks. In Question Answering, which is also advancing rapidly with insights from transformers and denoising auto-encoders, but still far from human baseline. The ease with which these models can answer a sample question such as: "Who was the first human in space", demonstrates both their efficacy and limitations. Pre-trained on a large corpus of text, almost every document that contains the the name "Yuri Gagarin" will in its near vicinity describe him in relation to his pioneering accomplishment for which he became a cultural icon. And for even more generalizable scenarios, such as "what might you find on a Mayan monument"? It becomes imperative that an agent explain its reasoning in natural language as well to enable self-correcting backpropagation of error correction. Language may be considered low-dimensional relatively speaking. And sentence prediction across quotidian tasks manageable in current state-of-the-art architectures. But looking at how difficult it is to predict the next N frames of video given a short input example demonstrates the intractability of the problem in higher dimensional spaces. Neural Models for Speech and Language: Successes, Challenges, and the Relationship to COmputational Models of the Brain - Michael Collins https://www.youtube.com/watch?v=HVnFKmPaU8c https://www.youtube.com/watch?v=HVnFKmPaU8c
mrtweetyhack 7y ago
"T" for terminator. Scary.

303 points by alexwg 7y ago

alexwg 7y ago

Paper: https://arxiv.org/abs/1910.10683 https://arxiv.org/abs/1910.10683

throwaway_bad 7y ago
Twitter summary: https://threadreaderapp.com/thread/1187161460033458177.html https://threadreaderapp.com/thread/1187161460033458177.html

martincmartin 7y ago

"The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems." "We take into account the lessons learnt from original GLUE benchmark and present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard."

pmoriarty 7y ago

There was an article[1] posted to HN recently about these benchmarks, and it was pretty skeptical. Regarding SuperGLUE specifically, it asked: "Indeed, Bowman and his collaborators recently introduced a test called SuperGLUE that's specifically designed to be hard for BERT-based systems. So far, no neural network can beat human performance on it. But even if (or when) it happens, does it mean that machines can really understand language any better than before? Or does just it mean that science has gotten better at teaching machines to the test?" [1] - https://www.quantamagazine.org/machines-beat-humans-on-a-reading-test-but-do-they-understand-20191017 https://www.quantamagazine.org/machines-beat-humans-on-a-rea...

gradys 7y ago
This feels hollow. Can't this be said about any benchmark? It seems natural and proper that as one benchmark becomes saturated, we introduce harder benchmarks. I don't think anyone in the field thinks that once we match human performance on benchmark X, we're officially done. It just means it's time for more interesting benchmarks. Over time, if it starts to become difficult to design benchmarks that humans can outperform machines on, then that will prompt interesting conceptual work about what exactly the difference between human and machine language competency is. And then that will lead either to more sophisticated benchmarks or alternatively gradually more sophisticated and persuasive arguments that machines really have surpassed us in language competence. I don't think we're yet at a point where we don't know how to make harder benchmarks, and if and when we do hit such a point, I'd definitely bet the result will be a conceptual advance in benchmark design rather than declaring machine superiority once and for all. At least for the first few rounds of this cycle.
- not2b 7y ago
  From the Quanta article: "But instead of concluding that BERT could apparently imbue neural networks with near-Aristotelian reasoning skills, they suspected a simpler explanation: that BERT was picking up on superficial patterns in the way the warrants were phrased. Indeed, after re-analyzing their training data, the authors found ample evidence of these so-called spurious cues. For example, simply choosing a warrant with the word “not” in it led to correct answers 61% of the time. After these patterns were scrubbed from the data, BERT’s score dropped from 77 to 53 — equivalent to random guessing."
  - nl 7y ago
    This is true, and absolutely a weakness of these tests. However they don't publish how well a human performs on the dataset without "not" in it. They do initially note that Even human beings don’t do particularly well on this task without practice I've looked at the warrant task. It's pretty tricky! I'd bet real money that untrained humans perform much, much lower than the 80% correct rate they get on the full set on ones without "not". I don't think it would be as low as the 53% BERT gets, but it would drop significantly. I find the HANS analysis[1] much more compelling, but again I'd note that humans suffer on this dataset too (although again - not as badly as models do). [1] https://www.aclweb.org/anthology/P19-1334.pdf https://www.aclweb.org/anthology/P19-1334.pdf
ehsankia 7y ago
As long as they're not training on the test data, and they're not submitting hundreds of submissions tweaking parameters trying to improve their score, I don't see what the problem is. If the algorithm can do a great job at classifying hundreds of new test cases it has never seen, and it isn't over-fitted, then that means it is good at that specific task. Of course the task itself may or may not be useful, and you can have some meta discussion about what "understanding language" is, but the computer definitely is doing a super human job at that given task.
- blazespin 7y ago
  Maybe it's over-fitted on the new data. There has to be a constant infusion of new training data and a system can only prove itself over time. These rankings, if real, should be in constant flux.
Iv 7y ago
Even when you will be able to have a 100% coherent and deep discussion with an AI over a niche technical domain, there will be people to pretend that the AI "fakes" it. Systems like GPT-2, incredibly (I used to be a skeptic of a pure statistical approach) manage to extract meaning, keep a theme, and understand the intent behind a sentence. They are amazing. When you have a system that displays all the characteristics of understanding something, it is irrelevant whether or not it "fakes" it. No one ever proved that humans are not "faking" intelligence either.
- deleted 7y ago
  [deleted]
nl 7y ago
(I work in this field, although not specifically on benchmarking) I think that this article makes a good point, and correctly identifies weaknesses. However, I also think that humans often take very similar shortcuts. There are good reasons why "bag of words" approaches work much of the time. Additionally there's lots of evidence showing that very rapid reading by humans does not imply deep understanding. I think it's very important that people are aware of the weaknesses of these types of models. However, I think it's interesting that these weaknesses are becoming harder and harder to find.
make3 7y ago
the machines are always trained with the same dataset for each task. the biggest difference right now is small technical modifications on models that are also pre trained on gigantic unlabelled datasets. this doesn't feel like we're teaching them to do the test specifically at all

hn_throwaway_99 7y ago

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ https://pilehvar.github.io/wic/) One of their examples, though, didn't make any sense to me: 1. The pilot managed to land the airplane safely 2. The enemy landed several of our aircrafts It says that the word "land" does NOT mean the same thing in those sentences. I am a native English speaker, and I honestly don't understand what they are thinking the second sentence means. Shot them down? If so, I have never heard "landed" used in that context, and it appears neither has Merriam-Webster. Also, the plural of aircraft is just "aircraft", without the s.

mooman219 7y ago
This is definitely where the 10.2% of human failures are.
- jdale27 7y ago
  .
  - p1esk 7y ago
    Ever tried taking SAT or GRE tests?
matthewowen 7y ago
I think it's landed in the same sense as "landed a deal": got, or achieved, in this case achieving shooting them down. For me, my first read of the sentence would definitely be that it means shot down.
- otterpop 7y ago
  so this is why the human score is 89.8 :)

throwaway_bad 7y ago

Possibly dumb question: How do you ensure there's no data leakage when benchmarking transfer learning techniques? Is that even a problem anymore when the whole point is to learn "common sense" knowledge? For example their “Colossal Clean Crawled Corpus” (C4), a dataset consisting of hundreds of gigabytes of clean English text scraped from the web, might contain much of the same information as the benchmark datasets, which I presume is also scraped from the web.

taneq 7y ago
You don't. Even humans frequently leak information like this. It's just a consequence of having incomplete or incompletely analyzed information.
calabin 7y ago
I think that this is a good question that I would also like to know the answer to. Additionally, are there other benchmarks or tests where this issue (possibly) presents itself?
jcims 7y ago
Not dumb at all and probably a major challenge when developing benchmarks.
craffel 7y ago
Hi, one of the paper authors here. Indeed this is a good question. A couple of comments: - Common Crawl overall is a sparse web dump, it is unlikely that the month we used includes any of the data that are in any of the test sets. - In order for the data to be useful to our model, it would have to be in the correct preprocessed format. ("mnli: hypothesis: ... premise: ...") with the label in a format our model could extract meaning from. We introduced this preprocessing format so I don't believe this would ever happen. - Further, most of these datasets live in .zip files. The Common Crawl dump doesn't unzip zip files. - C4 is so large that our model sees each example (corresponding to a block of text from a website) roughly once ever over the entire course of training. Big neural nets trained with SGD are unlikely to memorize something if they only see it once over the course of one million training steps.

lettergram 7y ago

Although those are some great results, I wish I could try it out locally... https://github.com/google-research/text-to-text-transfer-transformer https://github.com/google-research/text-to-text-transfer-tra... It drives me nuts that most of these papers / publications don't have code where I can just run: > python evaluate_model.py Still exciting, just annoying that I'd have to set up google cloud to try this out.

ehsankia 7y ago
They often do setup Python notebooks / Colabs you can simply run, especially with the data hosted on GCloud. Unfortunately not this time.

rrival 7y ago

Where do I take the SuperGLUE test?

riku_iki 7y ago

> T5-11B (11 billion parameters) So, this is largest language model so far?

lucidrains 7y ago
Yes. The last one was 8.3B https://arxiv.org/pdf/1909.08053.pdf https://arxiv.org/pdf/1909.08053.pdf
deleted 7y ago
[deleted]

vonseel 7y ago

I wonder what I would score on this test. Are these things correlated to standardized test scores at all for humans ?

RcouF1uZ4gsC 7y ago

I think classifying this as human level is misleading. Look at the sub-scores on the page. One score that looks very different from humans is AX-b. The SuperGlue paper provides more context about AX-b https://arxiv.org/pdf/1905.00537.pdf https://arxiv.org/pdf/1905.00537.pdf AX-b "is the broad-coverage diagnostic task, scored using Matthews’ correlation (MCC). " This is how the paper describes this test " Analyzing Linguistic and World Knowledge in Models GLUE includes an expert-constructed, diagnostic dataset that automatically tests models for a broad range of linguistic, commonsense, and world knowledge. Each example in this broad-coverage diagnostic is a sentence pair labeled with a three-way entailment relation (entailment, neutral, or contradiction) and tagged with labels that indicate the phenomena that characterize the relationship between the two sentences. Submissions to the GLUE leaderboard are required to include predictions from the submission’s MultiNLI classifier on the diagnostic dataset, and analyses of the results were shown alongside the main leaderboard. Since this broad-coverage diagnostic task has proved difficult for top models, we retain it in SuperGLUE. However, since MultiNLI is not part of SuperGLUE, we collapse contradiction and neutral into a single not_entailment label, and request that submissions include predictions on the resulting set from the model used for the RTE task. We collect non-expert annotations to estimate human performance, following the same procedure we use for the main benchmark tasks (Section 5.2). We estimate an accuracy of 88% and a Matthew’s correlation coefficient (MCC, the two-class variant of the R3 metric used in GLUE) of 0.77. " If you look at the scores, humans are estimated to score 0.77. Google T5 scores -0.4 on the test. How did T5 get such a high score if it scored so abysmally on the AX-b test? The AX scores are not included in the total score. From the paper: "The Avg column is the overall benchmarkscore on non-AX∗ tasks." If the AX scores were included, the gap between humans and machines would be bigger than the current score indicates.

foota 7y ago
I find it strange that they exclude it? Perhaps the reason is related?
craffel 7y ago
Hi, one of the paper's authors here. We didn't submit our model's predictions for the AX-b task yet, we just copied over the predictions from the example submission. We will submit predictions for AX-b in the next few days.

6gvONxR4sf7o 7y ago

One thing to always point out in these cases is that the human baseline isn't "how well people do at this task," like it's often hyped to be. It's "how well does a person quickly and repetitively doing this do, on average." The 'quickly and repetitively' part is important because we all make more boneheaded errors in this scenario. The 'on average' part is important because the errors the algo makes aren't just fewer than people, they're different. The algos often still get certain things wrong that humans almost never would. This is really really super great, let's be clear. It's just not up to the hype "omg super human" usually gets.

jcims 7y ago
Great point! It makes sense in the context of what these algorithms would generally be tasked with.
IshKebab 7y ago
Regarding the type of errors, it seems like the benchmark should be able to take that into account. That is, get a load of humans to do the task on the same specific examples, then for each example you know how hard it is, and what acceptable answers are (I bet a lot of the ground truth is wrong or ambiguous). Then you can benchmark your AI but penalise it more heavily for getting things wrong that are obvious to a human.
- 6gvONxR4sf7o 7y ago
  That would be ideal, if money weren't a factor. Since money is a factor, I wonder what the tradeoff is between labelling each instance N more times versus just getting N times more instances labeled.
TheOtherHobbes 7y ago
It seems to mean "How well does Mechanical Turk do the task?" which is a separate thing again. And yes - error type is at least as revealing as error frequency. I have no idea where the real human baseline is, or how to find it. Also, consider this discussion. GLUE winners may be able to make informed parsing guesses about single text blocks, but they're years away from being able to make a useful contribution to a discussion like this one.

woodgrainz 7y ago

Several of the systems in this leaderboard utilize the BERT model, a clever approach devised by Google for natural language processing. A nice laymen's guide to BERT: https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270 https://towardsdatascience.com/bert-explained-state-of-the-a...

skybrian 7y ago

They came up with the SuperGLUE benchmark because they found that the GLUE benchmark was flawed and too easy to game. There were correlations in the dataset that made it possible to get questions right without real understanding, and so the results didn't generalize. Could the same thing happen again with the better benchmark due to more subtle correlations? These things are tough to judge, so I'd say wait and see if it turns out to be a real result.

enisberk 7y ago

I attended one of the talks(1) of the Sam Bowman. His talk was about "Task-Independent Language Understanding" and he also talked about GLUE and super GLUE; he mentioned that some models are passing an average person in experiments. They did some experiments to understand BERT's performance (2). (similar to article 'NLP's Clever Hans Moment') But they found a different answer to question "what BERT really knows," so he was skeptical about all conclusions. Check these out if you are interested in. (1)[https://www.nyu.edu/projects/bowman/TILU-talk-19-09.pdf https://www.nyu.edu/projects/bowman/TILU-talk-19-09.pdf] (2)[https://arxiv.org/abs/1905.06316 https://arxiv.org/abs/1905.06316]

YeGoblynQueenne 7y ago

Assuming that the baseline human score was set according to the performance of adult humans, then according to these results T5 has a language understanding ability at least as accurate as a human child. In fact it's not just T5 that should be able to understand language as well as a human child, but also BERT++, BERT-mtl and RoBERTa, each of which has a score of 70 or more. There really shouldn't be anything else on the planet that has 70% of human language understanding, other than humans. So if the benchmarks mean what they think they mean, there are currently fully-fledged strongly artificially intelligent systems. That must mean that, in a very short time we should see strong evidence of having created human-like intelligence. Because make no mistake: language understanding is not like image recognition, say, or speech processing. Understanding anything is an AI-complete task, to use a colloquial term. Let's wait and see then. It shouldn't take more than five or six years to figure out what all this means.

YeGoblynQueenne 7y ago
To clarify, I meant this comment as an expression of skepticism- I don't believe that the SuperGLUE benchmark really evaluates language understanding, or that BERT and friends are within a few percents of human language understanding. I think SuperGLUE is just another benchmark that is measuring something else than what it's supposed to be measuring (machine learning benchmarks usually do). It seems that the teams behind the attempts to beat such benchmarks are aware of the weaknesses of the benchmarks though, so that's encouraging.

femto113 7y ago

My experience with image classification benchmarks was that they approached human levels only because the scoring only counts how much they get “right” and doesn’t penalize completely whack answers as much as they should (like getting full credit for being pretty sure a picture of a dog was either a dog or an alligator). I suspect there’s something similar going on in these language benchmarks.

nightnight 7y ago

This clearly demonstrates once again that Google is miles ahead of the competition in AI. I mean, they just have the best data. If you want to have an every day example of Google's AI skills: Switch you phone's keyboard to GBoard, especially all iOS users, and you will face a night and day difference to any other keyboard esepcially the stock one. When using multiple languages at the same time the leap to other keyboards gets even bigger. GBoard is my phone's killer app and if Google dropped it for iOS I'd left the same day to Android.

dingle_thunk 7y ago
Have you tried the iOS 13 keyboard's built in swipe feature? Are you aware of Swiftkey?
- occamrazor 7y ago
  I think GP is talking about predictive text, rather than keyboard ergonomics.
tremon 7y ago
I have the opposite experience. Yes, some of the suggestions from GBoard are useful, but I feel there's an equal number of times where I've typed a complete word, only to hit space and have the word auto-corrected to what GBoard was expecting. As a typing aid, it's almost unusable because of that.
htfu 7y ago
That's how I used to feel, but it's turning into a nuisance. It used to stick to single words or sometimes splitting one if missing a space, but now will sometimes attempt to "correct" the sum of two perfectly valid standalone words after the fact, 97% of the time resulting in nonsense. I cannot for the life of me understand why.

ilaksh 7y ago

The AIs in the benchmark are all trained exclusively on text, correct? My assumption has always been that to get human-level understanding, the AI systems need to be trained on things like visual data in addition to text. This is because there is a fair amount of information that is not encoded at all in text, or at least is not described in enough detail. I mean, humans can't learn to understand language properly without using their other senses. You need something visual or auditory or to associate with the words which are really supposed to represent full systems that are complex and detailed. I think it would be much more obvious if there were questions that involved things like spatial reasoning, or combining image recognition with that and comprehension.

BiasRegularizer 7y ago
I think maybe CLEVR[0] dataset is what you are talking about? Keep in mind that a most of the current ML systems have diverged from biology. A majority of the recent breakthroughs come from mathematics, the rational is that just because human brain does it in a certain way does not necessarily mean it is the only way to do it. [0] https://cs.stanford.edu/people/jcjohns/clevr/ https://cs.stanford.edu/people/jcjohns/clevr/
- ilaksh 7y ago
  In no way do I think that AGI needs to mimic animal/human intelligence. I was just trying to explain why text input alone isn't going to be adequate and that was an example. Thanks for the link, that is one example of the type of thing I was talking about I think.
visarga 7y ago
It's not just grounding the language in vision, but the embodiment, first person perspective and ability to interact with the environment. Humans have had the benefit of slowly evolving in a complex environment which is too expensive to recreate for artificial agents. We can only create very limited sims vs the real world.
deleted 7y ago
[deleted]

nopinsight 7y ago

As someone working in the field, I congratulate the excellent accomplishment but agree with the authors that we shouldn't get too excited yet (their quote below after the four reasons). Here are some reasons: 1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here: http://www.ericswallace.com/triggers http://www.ericswallace.com/triggers 2) T5 was trained with ~750GB of texts or ~150 billion words, which is > 100 times the number of words native English speakers acquire by the age of 20. 3) Most or all of the tests are multiple-choice. Learning complex correlations from sufficient data should help solve most of them. This is useful but human-level understanding is more than correlations. 4) The performance on datasets that require commonsense knowledge, COPA and WSC, are the weakest relative to humans (who score 100.0 on both). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, p.32 https://arxiv.org/pdf/1910.10683.pdf https://arxiv.org/pdf/1910.10683.pdf "Interestingly, on the reading comprehension tasks (MultiRC and ReCoRD) we exceed human performance by a large margin, suggesting the evaluation metrics used for these tasks may be biased towards machine-made predictions. On the other hand, humans achieve 100% accuracy on both COPA and WSC, which is significantly better than our model’s performance. This suggests that there remain linguistic tasks that are hard for our model to perfect, particularly in the low-resource setting." I’d like to emphasize that the work and the paper are excellent. Still, we are quite far from human-level language understanding. --- We may need more advanced tests to probe the actual language understanding ability of AI systems. Here are some ideas: * Test for conceptual understanding in a non-multiple-choice format. Example: Write a summary for a New Yorker article, rather than standard news pieces (which tend to follow repeated patterns). * Commonsense test with longer chains of inference than those needed for solving Winograd Schema and set in non-standard situations (e.g. fantasy world). This should greatly reduce the chance that an approach can simply detect correlations from huge datasets. * Understanding novel, creative metaphors like those used in some essays by professional writers or some of the Economist's title articles.

VikingCoder 7y ago
> 2) T5 was trained with ~750GB of texts or ~150 billion words, which is > 100 times the number of words native English speakers acquire by the age of 20. ...but, humans evolved the ability to use language over hundreds of generations... So... Maybe that's not such a bad thing?

LukeB42 7y ago

http://www.irc.org/history_docs/tao.html http://www.irc.org/history_docs/tao.html

deleted 7y ago

[deleted]

pauljurczak 7y ago

Use of Natural Language Understanding term in context of this benchmark is preposterous. No understanding takes place there. Please stick to NLP (Natural Language Processing) term for the next couple of decades. Thank you.

deleted 7y ago
[deleted]

Al-Khwarizmi 7y ago

This surprised me a bit, on the creation of the corpus they use for training: "We removed any page that contained any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”." I don't understand this decision. This list contains words that can be used in a perfectly objective sense, like "anus", "bastard", "erotic", "eunuch", "fecal", etc. I can understand that they want to avoid websites full of expletives and with no useful content, but outright excluding any website with even one occurrence of such words sounds too radical. If we ask this model a text comprehension question about a legitimized bastard that inherited the throne, or about fecal transplants, I suppose it would easily fail. Strange way of limiting such a powerful model.

Veedrac 7y ago
They say they removed pages, not websites. Having false positives isn't a problem when you're still left with 750GB of data—quality matters more than slightly higher quantity at that point.
- Al-Khwarizmi 7y ago
  Sorry, I was thinking about pages even though I said websites. Native language interference (typically, we use the same term for pages and websites in my language). Anyway, my point is not a matter of quantity. The way they're doing it, they have 750 GB of data, but they have exactly zero data that talks about bastards, fecal transplants, etc. So they may have a hard time answering questions about those specific subjects.

vagab0nd 7y ago

This is cool. Since they released a 11B pre-trained model, can we finally reproduce "unicorn-level" text generation now?

penagwin 7y ago
My understanding is that a lot of these really high performance models that reach for every percentage-point possible require an absurd amount of hardware - specifically an absurd amount of GPU memory. For example I have what I consider a fairly "high end" rig for being a hobbyist individual, with 32GB of RAM, i7 8700k, 1080ti - there's 0 chance their model would fit on my system. So I mean maybe if you have a ton of money? Usually what happens is a slimmer model with not "quite" as high of a score gets released that actually fits on consumer hardware.
- vagab0nd 7y ago
  Maybe I'm oversimplifying, but it seems to me that once you have the model trained, it should be possible to partition it somehow when inferencing, to fit smaller machines. At least for a proof of concept it should be possible.
  - penagwin 7y ago
    The problem is that there is so many weights in the model that they don't fit in memory. You can lower the number of weights, which will lower the effectiveness of the model. The thing is that when you're going for leaderboards you're reaching for every last percentage point, so the efficiency of the model size/performance isn't a concern, you want to ramp up the resource usage to as you have access to. TL;DR - Yeah basically most people will run a "slimmed down" version of the model that isn't "as" performant, but is still an improvement over previous models and actually fits on your machine.
  - nmfisher 7y ago
    I'm not aware of any "partioning" strategies per se (at least during inference), but it's now common practice to distill a larger model to a smaller one by either (a) training a smaller "student" network to replicate the larger "teacher" network, or (b) pruning smaller weights from the larger network to reduce the size. Just brainstorming here, but a vanilla network partition strategy might be to load each layer's weight into memory and perform the forward pass sequentially. I think that would be prohibitively slow - some of these models (e.g. BERT) can already take up to 3-4 seconds to perform a single forward pass on a CPU, and that's with all model weights already loaded into main memory. I suspect fetching/loading each layer separately would blow this out by an order of magnitude.

ArtWomb 7y ago

"Attention is all you need", indeed. Of course, our instinct tells us there is more to language inference than word proximity. And so results approaching or exceeding expert-level human baseline raise more questions than providing cause for popping champagne corks. In Question Answering, which is also advancing rapidly with insights from transformers and denoising auto-encoders, but still far from human baseline. The ease with which these models can answer a sample question such as: "Who was the first human in space", demonstrates both their efficacy and limitations. Pre-trained on a large corpus of text, almost every document that contains the the name "Yuri Gagarin" will in its near vicinity describe him in relation to his pioneering accomplishment for which he became a cultural icon. And for even more generalizable scenarios, such as "what might you find on a Mayan monument"? It becomes imperative that an agent explain its reasoning in natural language as well to enable self-correcting backpropagation of error correction. Language may be considered low-dimensional relatively speaking. And sentence prediction across quotidian tasks manageable in current state-of-the-art architectures. But looking at how difficult it is to predict the next N frames of video given a short input example demonstrates the intractability of the problem in higher dimensional spaces. Neural Models for Speech and Language: Successes, Challenges, and the Relationship to COmputational Models of the Brain - Michael Collins https://www.youtube.com/watch?v=HVnFKmPaU8c https://www.youtube.com/watch?v=HVnFKmPaU8c

mrtweetyhack 7y ago

"T" for terminator. Scary.