3 ms·
What are other barriers in transformers? Or is the normalization layer the primary one?
by elcritch 2y ago
What are other barriers in transformers? Or is the normalization layer the primary one?
- woadwarrior01 2y agodot-product attention is the biggest barrier. This is why there are so many attempts to linearize it.
- amitport 2y agothat fail... linearization is a bad idea. But plenty of other optimizations are done