3 ms·
All decent replies but, the truth is that it is mostly a quirk of transformer architecture which scales quadratically in the length of the sequence because tran
by lysecret 4y ago
All decent replies but, the truth is that it is mostly a quirk of transformer architecture which scales quadratically in the length of the sequence because transformers look at all pairwise combinations of the input tokens. So you can get the memory to whatever level you want to just increase the sequence length (also known as context):
- Der_Einzige 4y agoFlash attention gets this down to linear time complexity with no loss of quality.
- sebzim4500 4y agoNo it doesn't, it just improves the constant factor by a lot. Unless you are thinking of block sparse attention, which can chance complexity (assuming you scale sparsity with size) but decreases quality.