You can only approximate the attention layer if you don't want the O(T^2) speed but no "efficient attention" can beat the quadratic transformer bottleneck in te

You can only approximate the attention layer if you don't want the O(T^2) speed but no "efficient attention" can beat the quadratic transformer bottleneck in terms of performance.