3 ms·
Rmsnorm acts like a barrier. No compute on the next network layer can start before all compute in the previous layer is done. Splitting networks across multip
by londons_explore 2y ago
Rmsnorm acts like a barrier. No compute on the next network layer can start before all compute in the previous layer is done.
Splitting networks across multiple GPU's, this means you must wait for the slowest node and the longest latency.
As soon as you can remove most of these barriers, compute over non-latency-guaranteed networks becomes more practical, as does non-homogeneous compute (ie. Mixing different GPU models).
- elcritch 2y agoWhat are other barriers in transformers? Or is the normalization layer the primary one?
- woadwarrior01 2y agodot-product attention is the biggest barrier. This is why there are so many attempts to linearize it.
- amitport 2y agothat fail... linearization is a bad idea. But plenty of other optimizations are done