8 ms·
Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on te
by The_rationalist 6y ago
Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank
(4.6 vs 20) and with order of magnitudes less parameters
is talked about nowhere on the web...
https://arxiv.org/abs/2003.02645v2 https://arxiv.org/abs/2003.02645v2
- mchusma 6y agoI've seen you post this before and you seem enthusiastic about it. Do you have a good summary of why you feel this paper so important?
- The_rationalist 6y agowhy you feel this paper so important? the best summary are the benchmarks: first place at question answering (on yahoo task) First place on language modeling for pen treebank (by FAR) So it is a totally new model that will probably keep evolving and being applied to more and more kind of NLP tasks. And it seems that it can have the first place on most NLP tasks, its empirically the breakthrough of the year. It achieve this while having an extremely small number of parameters which shows: The model is smarter The model has room for more parameter hence even more accuracy! Finally, theoretically it is a breakthrough as it is a port of a computer vision technology (variational autoencoders) to the NLP world. Actually it might be the successor to the Transformer paradigm. I wonder what pile of incremental improvement will researchers will be able to bring to it like they have on the Transformer paradigm (spanBert, XLnet, etc)
- phreeza 6y agoNot all models scale up well with more parameters. VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015. https://arxiv.org/abs/1511.06349 https://arxiv.org/abs/1511.06349
- The_rationalist 6y agoWhat's new is that they are the first to eliminate the bottleneck that prevented VAES to beat transformers
- phreeza 6y agoWhat kind of bottleneck are you talking about? VAEs as such are really in a different category from transformers. They are primarily a tool to get to more structured latent spaces, which is not something transformers are good at in the first place.
- milquetoastaf 6y agoAny models or code available?
- The_rationalist 6y agohttps://github.com/seraphlabs-ca/SentenceMIM-demo https://github.com/seraphlabs-ca/SentenceMIM-demo
- laretluval 6y agoIt’s only been a few months. If it really does robustly get 4.6 perplexity on PTB (less than I ever thought was possible) then it will receive its due recognition, at the very least via the Hutter prize.
- psb217 6y agoFYI, the MELBO bound in that paper is invalid. Their perplexity numbers using the MELBO bound are also invalid.
- jaredtn 6y agoCan you explain further?
- psb217 6y agoI've added a brief explanation in a reply to a sibling comment.
- The_rationalist 6y agoWhere is the error? How much would that change their score of 4.6?
- psb217 6y agoThe bound is completely invalid, as are the NLL/PPL numbers they report with the MELBO. Look at the equation. If they optimized it directly, it would be trivially driven to 0 by the identity function if we used a latent space equivalent to the input space. The MELBO just adds noiseless autoencoder reconstruction error to a constant offset equal to log of the test set size. This can be driven to zero by evaluating an average bound over test sets of size 1. The mathematical/conceptual error is that they are assuming each test point is added to the "post-hoc aggregated" prior when they evaluate the bound. This is analogous to including a test point in the training set. Another version of this error would be adding a kernel centered on each test point to a kernel density estimator prior to evaluating test set NLL. In this case, obviously the best kernel has variance 0 and assigns arbitrarily high likelihood to the test data.
- The_rationalist 6y agoInteresting, thx!
- lonelappde 6y agoHave they deployed a demo? It's hard to talk about something that no one can see.
- The_rationalist 6y agoThey have: https://github.com/seraphlabs-ca/SentenceMIM-demo https://github.com/seraphlabs-ca/SentenceMIM-demo It's far easier to reproduce than GPT-3 because you won't need a GPU farm, only a powerful one
- canjobear 6y agoThe perplexity numbers are for different tasks. MIM is encoding a sentence into a latent variable and then reconstructing it, and achieves PTB perplexity 4.6. GPT-2 is generating the sentence from scratch, which will on average have higher perplexity numbers. I agree that PTB perplexity 4.6 on autoregressive language modeling would be a huge result.
- p1esk 6y agoI think using AE for text generation is a good idea, and is pretty old one (I tried it myself back in 2015 without particularly good results), but I wouldn't call it a breakthrough. To me a breakthrough/new paradigm would be something like this: https://arxiv.org/abs/1906.05317 https://arxiv.org/abs/1906.05317
- The_rationalist 6y agoThe breakthrough is not that they are the first to use AE for text. It is that they are the first to eliminate the bottleneck that prevented AEs to beat transformers. You linked paper seems interesting on paper but does it bring any new SOTA?
- p1esk 6y agoAEs to beat transformers Can you point me to some example of generated text this model produced? Something similar in quality to that unicorn story from GPT-2? but does it bring any new SOTA? Looking at their tables, seems so. The code is open source, and there's demo at https://mosaickg.apps.allenai.org/ https://mosaickg.apps.allenai.org/