3 ms·
First off, GPT-3 is absolutely not a vanilla transformer. The moving pieces here in BigBird are a vector associated with each token in the sequence and a few m
by ianhorn 6y ago
First off, GPT-3 is absolutely not a vanilla transformer.
The moving pieces here in BigBird are a vector associated with each token in the sequence and a few more global vectors that you can think of as latent variables. Those pieces are present at each layer. The vector in layer i+1 at position t in the sequence is a function of a bunch of the vectors in layer i.
If position t depends on all of the other positions, then you end up computing all combinations. A sequence of length N has N^2 combinations. Papers like this are using different patterns.
In this one, position t only depends on local positions in some small window around t, those global/latent-ish variables, and a small number of random positions. The number of combinations is (the size of the local window + the number of random pieces to attend to + the number of global vectors to attend to) times the number of positions, so it's some k * N rather than N^2. That lets you scale to longer sequences.
- YetAnotherNick 6y ago> First off, GPT-3 is absolutely not a vanilla transformer. I thought it had fixed length window. Can you explain how it differs from vanilla transformer other than the size.
- ianhorn 6y agoUnfortunately, they didn't give many details in the paper. It's frustrating, to say the least. Yay reproducibility. They say > We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer. In the referenced paper (sparse transformer) they showed a bunch of different sparsity patterns, and I believe they're referring to either their banded block diagonal sparsity or a true banded diagonal pattern (local windows like bigbird and some other papers). Unfortunately, that paper also was light on details and the repo they open sourced alongside it is inscrutible.
- gwern 6y agohttps://www.reddit.com/r/MachineLearning/comments/hxvts0/d_breaking_the_quadratic_attention_bottleneck_in/fzh7bpd/ https://www.reddit.com/r/MachineLearning/comments/hxvts0/d_b...