23 ms·
This would only be the case if we ignore the multiplication between queries and keys, and the resulting vector being multiple with the values, and also the mult
by GaggiX 2y ago
This would only be the case if we ignore the multiplication between queries and keys, and the resulting vector being multiple with the values, and also the multiple heads.
- ein0p 2y agoNo, that is always the case. Attention is only about one third the ops and qk is a fraction of that. Outside of truly massive sequence lengths it doesn’t matter a whole lot, even though it’s nominally quadratic. It’s trivial to run the numbers on this - you only need to do it for one layer.