5 ms·
This looks like a new iteration on the Pixel-RNN idea: https://arxiv.org/pdf/1601.06759.pdf https://arxiv.org/pdf/1601.06759.pdf GPT2 definitely looks better.
by etaioinshrdlu 6y ago
This looks like a new iteration on the Pixel-RNN idea: https://arxiv.org/pdf/1601.06759.pdf https://arxiv.org/pdf/1601.06759.pdf
GPT2 definitely looks better.
- bigdict 6y agoNo, this is a new iteration on the AIAYN idea.
- elcomet 6y agoWhat's this ?
- ot 6y ago"Attention is all you need"
- bigdict 6y agoAttention is all you need is the title of the 2017 paper that introduced the Transformer architecture. It reflects the evolution of NLP models: (RNN) —> (RNN + attention) —> (attention) and the idea that you don’t need the recurrent component. Just look at all elements of the sequence at once, applying varying weights (attention).
- deleted 6y ago[deleted]
- the8472 6y agoAlso an instance of The Bitter Lesson