4 ms·
Why would you need BLAS for tokenization/detokenization? Pretty much everyone still uses BBPE which amounts to iteratively applying merges. (Maybe I'm missing
by microtonal 2y ago
Why would you need BLAS for tokenization/detokenization? Pretty much everyone still uses BBPE which amounts to iteratively applying merges.
(Maybe I'm missing something here.)