6 ms·
DeepNet: Scaling Transformers to 1k Layers
- cleancoder0 5y ago5 BLEU points more is massive with 4 times less params. The fact that layers themselves are narrower means that training and evaluation of the NN is also much faster.
- yoquan 5y agoActually no. Each layer requires output from previous one, which means sequentially computation. While wider layers can utilize GPU parallel computation better. This is kind of trade-off between less memory (less parameters) vs longer time.
- londons_explore 5y agoIt is notable that they got only 2.5 extra BLEU while translating text to english, and 6.2 extra when translating text from english to another language. Since the network will have seen far more english text than text of other languages, it suggests that performance on limited training data is more improved.
- modeless 5y agoThat's exciting. Getting better performance with more data is trivial, the hard part is getting better performance with less data.
- oofbey 5y agoWhy would evaluation be faster with more narrow layers? If there were fewer tokens it would definitely be faster, because transformers scale by tokens^2, but here "narrow" means number of channels, for presumably the same number of tokens.
- chabons 5y agoIf I remember correctly the fully connected layers after the attention block are [?, a*h] * [a*h, b*h] (for some scalars a,b and hidden size h), which means that transformers also scale with h^2. I don't know what fraction of the total FLOPs that section of the model takes for practical model sizes, but it would indicate that making the model narrower for the same number of params would reduce compute.
- oofbey 5y agoOh yeah, I think you're right. If it's all fully-connected (and parts of a transformer are) then more thinner layers use fewer FLOPs than fewer thicker layers, for the same number of parameters. As long as the layers are wide enough to keep the GPU busy it'll run faster.
- notacanofsoda 5y agoI love the tl;dr for practitioners part. All ML papers should have one.
- lysecret 5y agoWow. This sounds a lot like the Resnet moment for Transformers.
- oofbey 5y agoI hope so, but I'm not convinced. The experimental results don't persuade me that deep transformers are better than the existing extremely wide transformers with tons of parameters. Yes, they show a much better result on the multilingual benchmark (M2M-100), but that's not a super well studied problem, so I don't know that the baseline they're beating is very well optimized. And on OPUS-100 they only beat the baseline by building models with vastly more parameters, which shouldn't really surprise anybody.
- mrfusion 5y agoWhat’s that?
- bckr 5y agoResidual neural networks[] employ skip-layer connections. It's thought that this helps prevent gradient decay in deep networks. It's a very important advancement. []https://www.wikiwand.com/en/Residual_neural_network https://www.wikiwand.com/en/Residual_neural_network
- quocanh 5y agoResnet was when we figured out how to throw a fuckton of perceptrons together and make it actually work. We didn't think networks scaled before that.
- tusharsadhwani 5y agoiirc that was AlexNet, ResNet came 3 years later whose refinements ended up making it better than a human at object detection.
- mlajtos 5y ago
- Reubend 5y agoThis seems really impressive. I would love to see a rough comparison of how much this improves training time compared to past methods.
- rdedev 5y agoAnything that can reduce the number of parameters required for a transformer is good news. Wonder if this approach can be easily plugged into the recent retrieval transformer.
- lucidrains 5y agoyea definitely, the ideas are orthogonal
- albertzeyer 5y agoWhy are the number of parameters relevant? The training and decoding runtime and memory consumption are what is relevant. And the number of parameters is not really connected to that. E.g. I assume this DeepNet with 3.2B params is slower and requires more memory than the M2M-100 with 12B params. The number of layers are more connected to both runtime and memory consumption.
- l33tman 5y agoFirstly the number of parameters is important in itself due to the finite size of VRAM on the GPU cards commonly available. But secondly the bulk of the ops you do are scalar products (arranged as matrix muls) where one parameter means one multiply+accumulate. So regardless of the actual net topology, number of layers etc you should see a rough scaling in the number of parameters, in an ideal case with no other overhead (you can write code that scales badly or weirdly). I didn't look at this DeepNet, maybe it will run slower like you say due to some other choice of architecture.. Also yes an architecture that requires 1000x more training iterations will of course be worse at the training stage even if it has 100x fewer parameters, but maybe the resulting inference model is much faster. Lots of stuff to consider :)
- albertzeyer 5y agoIn a Transformer, the most expensive op is the self attention. So when you have more layers, this will be more expensive. You could also share e.g. all the parameters in every layer (like Universal Transformer), and thus reduce the number of params drastically, but without any change in computing time, and only negligible difference in memory consumption because the hidden activations take most of the memory, not the params.
- mrintellectual 5y agoA lot of previous transformer-based models did not have great depth simply because "shallow" transformers already took forever to train. I am concerned that we are approaching an age where only massive companies and research groups with tons of GPU resources will be able to train next-gen models.
- dtx1 5y agoI'm not too concerned GPUs scale very well horizontally and aren't as much effected by Moore's Law as Single Thread Bound algorithms. We just discovered a new class of algorithms and hardware needs time to catch up from big gpu clusters to end user hardware
- malka 5y ago> only massive companies and research groups with tons of GPU resources will be able to train next-gen models Oh, don't be. We are already at this stage.
- wodenokoto 5y agoThere is still plenty of interesting and relevant ml research available to people who don't have access to CERN-sized compute clusters, just like there is still plenty of interesting and relevant research for physicists outside of CERN.
- dankle 5y agoI'm not particularly concerned about this tbh. What I mean is that ML is highly relevant with less resources as well, and the fact the only a handful companies or research groups have the financial power to push the state of the art isn't really a problem.
- 16890c 5y agoThis study suggests that the number of parameters can be significantly reduced using deeper networks, which is a consistent finding in ML research. So these results could actually help to decrease the amount of computing resources required.
- jurschreuder 5y ago
- primordialsoup 5y agoDoes this apply to non-transformer based architectures as well?