4 ms·
Well each additional token generated requires rerunning the model right to find the next likely token given the previous one
by space_fountain 3y ago
Well each additional token generated requires rerunning the model right to find the next likely token given the previous one
- newhouseb 3y agoNaively, yes, but you can cache the bulk of that "rerunning" [1]. That said the (non-flash) attention costs go up with the length of the sequence so perhaps this is just a simpler way to approximate these costs. [1] https://kipp.ly/blog/transformer-inference-arithmetic/ https://kipp.ly/blog/transformer-inference-arithmetic/