4 ms·
Training compute goes up with approximately the 3rd power of the window size. So turning a 4k window to a 32k window means a 512x increase in compute they'd ne
by londons_explore 4y ago
Training compute goes up with approximately the 3rd power of the window size.
So turning a 4k window to a 32k window means a 512x increase in compute they'd need (just to maintain similar output quality).
I suspect they must have found a better solution to be able to scale the window so big. They haven't announced what it is.
- meghan_rain 4y agoVery interesting, thanks