4 ms·

Well, scaling is big deal since that is where a lot of the power comes from. The way transformers leverage parallel hardware by processing tokens in parallel al

Well, scaling is big deal since that is where a lot of the power comes from. The way transformers leverage parallel hardware by processing tokens in parallel also makes them more general than just sequence-to-sequence, maybe more like graph-to-graph, which is what has allowed them to also be used for things like vision. There's a common sentiment that data/scale is more important than architecture, and that other architectures would perform just as well if you scaled them up (to degree that is practical of course), but I'm not sure that is totally true. Of course scale is vital to performance, but scaling the wrong architecture isn't going to get you there. For example, you need the ability to attend to specific tokens, and munging the sequence history into single vector the way RNNs do is going to cap performance, which is why you see hybrid architectures like Jamba (Mamba + Transformer). I think there is something special about Transformers - something that the developers accidentally got right! The key-based attention mechanism and the way attention heads in adjacent layers can pair up to form induction heads seems to be what makes them unreasonably effective at learning from language... It's a shame there hasn't been much (any?) discussion of this that I'm aware of - perhaps just because the detailed architecture was more "accidental" than strategic. It seems the history is that Jacob Uszkoreit had the basic idea of language being more hierarchical than sequential, and therefore amenable to parallel processing (with a multi-layer architecture), but in initial implementations wasn't able to get the performance to beat other similar contemporary approaches such as using convolution. Noam Shazeer was apparently the wizard who took the basic idea, threw a lot of inspiration/experience (plus the kitchen sink?) at it, and was able to make it perform - apparently coming up with this specific attention mechanism. There was then an ablation process to simplify the architecture.

VHRanger 3y ago
I have a blog post coming in a few days on this topic. But basically, I agree, architecture doesn't really matter. The number of parameters and the training data matter. The tradeoff transformers make is that you saturate the hardware really efficiently (so: easy to parallelize & scale) but the tradeoff is the n^2 scaling of the attention mechanism. This turns out to be a great tradeoff if what you're doing is scaling models to 7B+ parameters.
- HarHarVeryFunny 3y ago
  Uszkoreit has mentioned that the global/quadratic attention was (paraphrasing) considered as overkill, but the brute force parallelism this simple approach allowed made that irrelevant. But of course that changes when you scale up the context size enough, and the fix is simple since the key insight of the architecture was the hierarchical tree-like nature of language and thus dependence mostly on local (within branch) context, not global context. In Google's Big Bird attention (from their Elmo/Bert Muppet era!) they basically use sliding window local attention, but augmented with a fixed number of global tokens with global attention, and some random attention to further back non-local tokens. This mixture of attention patterns performs almost as well as global attention. Part of the reason (aside from the mostly local nature of language) is that as you ascend the transformer-layer hierarchy, receptive field sizes increase (same as they do in a CNN), so even with the attention gaps of random attention, there is still visibility at higher layers.

VHRanger 3y ago

I have a blog post coming in a few days on this topic. But basically, I agree, architecture doesn't really matter. The number of parameters and the training data matter. The tradeoff transformers make is that you saturate the hardware really efficiently (so: easy to parallelize & scale) but the tradeoff is the n^2 scaling of the attention mechanism. This turns out to be a great tradeoff if what you're doing is scaling models to 7B+ parameters.

HarHarVeryFunny 3y ago
Uszkoreit has mentioned that the global/quadratic attention was (paraphrasing) considered as overkill, but the brute force parallelism this simple approach allowed made that irrelevant. But of course that changes when you scale up the context size enough, and the fix is simple since the key insight of the architecture was the hierarchical tree-like nature of language and thus dependence mostly on local (within branch) context, not global context. In Google's Big Bird attention (from their Elmo/Bert Muppet era!) they basically use sliding window local attention, but augmented with a fixed number of global tokens with global attention, and some random attention to further back non-local tokens. This mixture of attention patterns performs almost as well as global attention. Part of the reason (aside from the mostly local nature of language) is that as you ascend the transformer-layer hierarchy, receptive field sizes increase (same as they do in a CNN), so even with the attention gaps of random attention, there is still visibility at higher layers.