6 ms·
Simplifying Transformer Blocks
- bilsbie 3y agoI wonder why only 15% faster training? Seems like simplying the main element would make a huge difference.
- imjonse 3y ago"While we have demonstrated the efficacy of our simplifications across architectures, datasets, and tasks, the models we have considered (100-300M parameters) are small relative to the largest transformers." University researchers without a big lab's backing cannot try out such experiments on really large models.
- quickthrower2 3y ago300M params will train in a Google Colab most likely, or for a few bucks on vast.ai, or a few hours on a fairly decent card. So that is super small!
- baq 3y agoBy 'fairly decent' do you mean a civilian 4090 or one of the weapons-grade DC cards?
- buildbot 3y agoIn my experience you’d need several weapons grade DC nodes (unless you are fine-tuning with LoRa, totally different game then!)
- quickthrower2 3y ago3080, 4090, etc.
- buildbot 3y agoErrr 20M is a few hours to train 10K iters on 8x V100s for what I’ve been doing. 300M is in the range of 4-8 nodes of that scale.
- quickthrower2 3y agoI'll check my wandb account when I can to see what I got (it was months ago). At the moment I can't log in "An application error occurred. Click to refresh the page." :-o I was using a 3090 though - I remember that because I was using vast.ai and a cheapskate! That card has serious bang for buck.
- quickthrower2 3y agoJust a thought: Are people confusing million and billion?
- buildbot 3y agoOr the library I’ve been using for this work is … not very memory efficient possibly? I’ve been referring to millions at least intentionally
- chessgecko 3y agoNot sure if I read it correctly, but it seems like the skip connections are kinda still present in the skipless block because they added I after the softmax and v= the previous hidden state. Still a cool paper if they really managed to get rid of those projections without degrading the quality.
- samus 3y agoI love reading about papers like these. They raise hopes that novel model architectures might reduce the need for computational resources to train powerful models, which would help lower the barrier of entry.
- heyitsguay 3y agoIf you read "Attention Is All You Need", you'll see that was actually the motive behind the original Transformer too! Bit of a Jevon's paradox situation.
- zozbot234 3y agoTransformer was really about making it feasible to parallelize training of large context windows. More like using large compute resources effectively when available than saving on compute. This is also why it looks quite ad-hoc compared to simpler models like RNN or LSTM, which can be trained in large batches of data (hence still making use of parallelism to some extent) but have to be serial along the context dimension.
- low_tech_love 3y ago“This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable.” I am by no means an expert and I can’t verify the authors’ claims about reduced speed and untrainability, but this reflects an impression I’ve been having on the papers I read and review. The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. It’s basically “we got nice results, and here is the architecture of the model” (proceeds to show a figure with a hundred coloured blocks connected together in some seemingly random complex way). It used to be that such a thing would get backlash from reviewers, and they would require you to actually justify the design. I don’t see that anymore. The problem with this for me is that we fail to build a nice, crisp understanding of the effects of each design decision in the final outcomes, which hurts the actual “science” of it. It also opens up the field for bogus and unreproducible claims. But at least other people are picking up on the thread and doing that in follow-up papers, which is good.
- sytelus 3y agoThere are rarely any design "decisions". Typically, you throw many things at wall and something sticks which becomes paper. Transformer paper has approximately zero "design decisions" apart from attention block. I can imagine they just tried out various combinations, kept adding projections and went with what worked the best.
- galangalalgol 3y agoI hear people say that a lot, but is that really how people at the leading edge of research do this? Those I know who are coming up with new stuff and not just new applications for old architectures, are either building loosely on animal models, or designing based off a traditional algorithm with some room for the training to take advantage of complex interactions the traditional algorithms don't.
- dartos 3y agoMost advances in NLP with transformers over the last 2 years has been random trial and error and just throwing more compute at transformers. Some models like RWKV (which isn’t a transformer model) explain the design decisions in their paper, but generally that’s not the case. Nobody knows why, nobody seems to be looking into it. We’re all just trying to figure out how to improve the outcome atm.
- Bayes7 3y ago"[...] modern neural network (NN) architectures have complex designs with many components [...]" I find the Transformer architecture actually very simple compared to previous models like LSTMs or other recurrent models. You could argue that their vision counterparts like ViT are conceptually maybe even simpler than ConvNets? Also, can someone explain why they are so keen to remove the skip connections? At least when it comes to coding, nothing is simpler than adding a skip connection and computationally the effect should be marginal?
- SuchAnonMuchWow 3y agoSkip connection increase the live range of one intermediate result across the whole part of the network skiped: the tensor at the beginning of a skip connection must be stored in memory for longer while unrelated computation happen: it increase the pressure on the memory hierarchy (either the L2, or scratchpad memory). This is especially true for example for inference for vision transformers, where it decrease the batch size you can use before hitting the L2 capacity wall.
- Bayes7 3y agoOkay, I see that for inference. But for training it shouldn't matter because I need to hold on to all my activations for my backwards pass anyways? But yeah, fair point!
- jksk61 3y agoalso removing skip connections leads to a rougher loss landscape, hence it should be harder to find the optimal weights.
- sdenton4 3y agoYes there's very good theoretical reasons for skip connections. If your initial matrix M is noise centered at 0, then 1+M is a noisy identity operation, while 0+M is a noisy deletion... It's better to do nothing if you don't know what to do, and avoid destroying information. I appreciate the sibling comment perspective that memory pressure is a problem, but that can be mediated by using fewer/longer skip connections across blocks of layers.
- WithinReason 3y agoThis is a nice start, but what would really help is for someone who understands the GPU programming model in-depth to give this a shot, with the goal of reducing DRAM bandwidth and fitting the layers exactly onto a GPU's memory hierarchy (cache levels, local memory and registers). Basically, sizes of a target HW platform's memories should be hyper-parameters of the architecture.
- machinekob 3y agoIndeed it could be a target, but newer and newer co-processor for pure MatMul or even NPU are created and cache/register are changing fast from one architecture to another so it still wouldnt be optimal for most users.
- karmasimida 3y agoThere could be a CISC vs RISC argument for Transformer, which I can see, but my bet is the traditional transformer has established enough fundamentals, it is going to take a long time for alternative architectures to prove they are indeed alternatives on a wide range of tasks. Pretty much at this point, to succeed transformer, one need to showcase, the alternative model could achieve chatgpt level performance with either significant reduction of compute or data requirements.
- Buttons840 3y agoWhat is "Shaped Attention"? They simplified everything except for changing "Attention" to "Shaped Attention".
- Majromax 3y agoThat's discussed at the end of page 4, and it comes from the Noci et al (2023) reference.
- m3kw9 3y agoDo they have a demo of this theory?
- ofou 3y agoAre anyone aware of similar simplifications?