4 ms·
When you train a transformer, you're training what the next expected token is. You can train all positions of the sequence each in parallel rather than having t
by Salgat 3y ago
When you train a transformer, you're training what the next expected token is. You can train all positions of the sequence each in parallel rather than having to sequentially build up the memory state as you generate the sequence with an LSTM. Mind you the inference portion of a transformer is still sequentially bottlenecked since you don't know what the output sequence is supposed to be.
- saynay 3y agoI believe the other big thing is it allowed you to parallelize the training as well, so you could split those colossal training sets across many machines in a way you could not do with LSTMs.
- throwawaymaths 3y agoI don't really believe this because in theory it's possible to split up training of arbitrary neural nets using decoupled neural interfaces (not that it's easy mind you) but at the point where you have openAI scale resources the bottleneck to achieving that is the difficulty of Python.
- golly_ned 3y agoWhat’s a neural interface?
- throwawaymaths 3y agohttps://arxiv.org/abs/1608.05343 https://arxiv.org/abs/1608.05343
- rocho 3y agoIf you train in parallel, how do you combine the weights generated from different trainings in parallel?
- throwawaymaths 3y agoYou take a huge hit.
- Salgat 3y agoBelieve it or not, it's a simple as averaging or adding the gradients of each training result before adding it to the model weights. The same thing happens when you train a model using batches of inputs.
- throwawaymaths 3y agoIt actually isn't. You have to have a synchronizer, batchsize one or else "strange things" can happen and you waste a lot of cycles. Alternatively you can do non-simple changes to your network structure to enable distributed training.
- Salgat 3y agoIt really is that simple. Yes, there's many different approaches to this (which can become quite clever and complex, which is true of training in general), but it all really boils down to adding or averaging the gradients in most cases.