3 ms·
Another great blog post on great research by the DeepMind guys, who are also simultaneously releasing a new dataset for long-range language modeling. The post
by cs702 7y ago
Another great blog post on great research by the DeepMind guys, who are also simultaneously releasing a new dataset for long-range language modeling.
The post is worth reading in its entirety.
If I may summarize, the authors propose a transformer augmented with a short-term memory mechanism (analogous to TransformerXL) as well as a long-term memory mechanism (new) that learns to 'compress and memorize' embeddings from the short-term memory. The model is trained on book-length samples (!!!!), and seems to perform significantly better than prior models at generating language with long-range contexts. To my eyes, text generated by the trained model is virtually indistinguishable from human output, and qualitatively superior to GPT2 samples.
- JRKrause 7y agoAgreed that the generated sample is superior to similar outputs from GPT-2. Looking at the additional samples in the publication, my first thought is that the model cannot easily stray from or modify the context. Once a fact is stored within the compressed memory, it seems the model cannot easily generate sentences contradictory to that fact. This is problematic because frequent changes to relational information (e.g. the location a character is standing) is fundamental to story telling.
- deleted 7y ago[deleted]
- scribu 7y agoIn what way is it superior? I can spot several logical contradictions, just like with samples from GPT-2.