4 ms·

I've seen you post this before and you seem enthusiastic about it. Do you have a good summary of why you feel this paper so important?

The_rationalist 6y ago
why you feel this paper so important? the best summary are the benchmarks: first place at question answering (on yahoo task) First place on language modeling for pen treebank (by FAR) So it is a totally new model that will probably keep evolving and being applied to more and more kind of NLP tasks. And it seems that it can have the first place on most NLP tasks, its empirically the breakthrough of the year. It achieve this while having an extremely small number of parameters which shows: The model is smarter The model has room for more parameter hence even more accuracy! Finally, theoretically it is a breakthrough as it is a port of a computer vision technology (variational autoencoders) to the NLP world. Actually it might be the successor to the Transformer paradigm. I wonder what pile of incremental improvement will researchers will be able to bring to it like they have on the Transformer paradigm (spanBert, XLnet, etc)
- phreeza 6y ago
  Not all models scale up well with more parameters. VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015. https://arxiv.org/abs/1511.06349 https://arxiv.org/abs/1511.06349
  - The_rationalist 6y ago
    What's new is that they are the first to eliminate the bottleneck that prevented VAES to beat transformers
    - phreeza 6y ago
      What kind of bottleneck are you talking about? VAEs as such are really in a different category from transformers. They are primarily a tool to get to more structured latent spaces, which is not something transformers are good at in the first place.

4 ms·

I've seen you post this before and you seem enthusiastic about it. Do you have a good summary of why you feel this paper so important?

by mchusma 6y ago

The_rationalist 6y ago
why you feel this paper so important? the best summary are the benchmarks: first place at question answering (on yahoo task) First place on language modeling for pen treebank (by FAR) So it is a totally new model that will probably keep evolving and being applied to more and more kind of NLP tasks. And it seems that it can have the first place on most NLP tasks, its empirically the breakthrough of the year. It achieve this while having an extremely small number of parameters which shows: The model is smarter The model has room for more parameter hence even more accuracy! Finally, theoretically it is a breakthrough as it is a port of a computer vision technology (variational autoencoders) to the NLP world. Actually it might be the successor to the Transformer paradigm. I wonder what pile of incremental improvement will researchers will be able to bring to it like they have on the Transformer paradigm (spanBert, XLnet, etc)
- phreeza 6y ago
  Not all models scale up well with more parameters. VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015. https://arxiv.org/abs/1511.06349 https://arxiv.org/abs/1511.06349
  - The_rationalist 6y ago
    What's new is that they are the first to eliminate the bottleneck that prevented VAES to beat transformers
    - phreeza 6y ago
      What kind of bottleneck are you talking about? VAEs as such are really in a different category from transformers. They are primarily a tool to get to more structured latent spaces, which is not something transformers are good at in the first place.