7 ms·
<4x would be quite optimistic, at ~11x the tokens the amount of compute/memory required would be n^10 (even with the lower starting point of flash attention) so
by f_devd 3y ago
<4x would be quite optimistic, at ~11x the tokens the amount of compute/memory required would be n^10 (even with the lower starting point of flash attention) so unless they are already have excessive margins it wouldn't make much sense to go that low.
- sp332 3y agoI was assuming they used a different architecture to get the increase instead of just letting it eat hardware that way. Especially with the speed numbers in the post.