3 ms·

Performers look similar to Labda Networks in characteristics, but they don't mention it in the paper (although Lamda Networks are used to model images). I wonde

Performers look similar to Labda Networks in characteristics, but they don't mention it in the paper (although Lamda Networks are used to model images). I wonder what are the main differences in the ideas. Anyways, congratulations for beating Moore's law again!

orbifold 6y ago
A number of groups worked on this at the same time and of course because it is machine learning, everyone had to come up with their own name for it. Even though most of these are basically elementary linear algebra dressed up by fancy language and lots of computationally expensive experiments.
- xiphias2 6y ago
  You are right, but at the same time it's great that finally it seems that we have an efficient unified model for language and image modelling. Having CNNs and RNNs separately was relatively hard to work with (especially RNNs). It would be bad if we would need different hardware accelerator architectures for the two most important / different fields of AI. Also having a unified architecture will help with mixed tasks. https://github.com/lucidrains/lambda-networks https://github.com/lucidrains/lambda-networks https://www.youtube.com/watch?v=3qxJ2WD8p4w https://www.youtube.com/watch?v=3qxJ2WD8p4w https://openreview.net/pdf?id=xTJEN-ggl1b https://openreview.net/pdf?id=xTJEN-ggl1b
gwern 6y ago
The differences are pretty subtle... If you want an overview of the whole zoo, see https://lilianweng.github.io/lil-log/2020/04/07/the-transformer-family.html https://lilianweng.github.io/lil-log/2020/04/07/the-transfor... https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/ https://www.pragmatic.ml/a-survey-of-methods-for-incorporati... https://arxiv.org/abs/2009.06732#google https://arxiv.org/abs/2009.06732#google https://www.gwern.net/GPT-2#efficient-attention https://www.gwern.net/GPT-2#efficient-attention

orbifold 6y ago

A number of groups worked on this at the same time and of course because it is machine learning, everyone had to come up with their own name for it. Even though most of these are basically elementary linear algebra dressed up by fancy language and lots of computationally expensive experiments.

xiphias2 6y ago
You are right, but at the same time it's great that finally it seems that we have an efficient unified model for language and image modelling. Having CNNs and RNNs separately was relatively hard to work with (especially RNNs). It would be bad if we would need different hardware accelerator architectures for the two most important / different fields of AI. Also having a unified architecture will help with mixed tasks. https://github.com/lucidrains/lambda-networks https://github.com/lucidrains/lambda-networks https://www.youtube.com/watch?v=3qxJ2WD8p4w https://www.youtube.com/watch?v=3qxJ2WD8p4w https://openreview.net/pdf?id=xTJEN-ggl1b https://openreview.net/pdf?id=xTJEN-ggl1b

gwern 6y ago

The differences are pretty subtle... If you want an overview of the whole zoo, see https://lilianweng.github.io/lil-log/2020/04/07/the-transformer-family.html https://lilianweng.github.io/lil-log/2020/04/07/the-transfor... https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/ https://www.pragmatic.ml/a-survey-of-methods-for-incorporati... https://arxiv.org/abs/2009.06732#google https://arxiv.org/abs/2009.06732#google https://www.gwern.net/GPT-2#efficient-attention https://www.gwern.net/GPT-2#efficient-attention