3 ms·
Sure, but larger models that fit in that 512gb memory are going to take a long time to tokenize/detokenize without hardware-accelerated BLAS.
by bigyabai 2y ago
Sure, but larger models that fit in that 512gb memory are going to take a long time to tokenize/detokenize without hardware-accelerated BLAS.
- microtonal 2y agoWhy would you need BLAS for tokenization/detokenization? Pretty much everyone still uses BBPE which amounts to iteratively applying merges. (Maybe I'm missing something here.)
- ryao 2y agoTokenization/detokenization does not use BLAS.