3 ms·
It does! I do use CT2! On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me
by omneity 1y ago
It does! I do use CT2!
On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me as a user.
But it’s definitely worth revisiting that. Thanks!
- woodson 1y agoOh, that’s pretty slow. Have you tried using quantization (int8 or int8_float32)? In my experience that can help speed up CT2 execution. Personally, I haven’t had much luck with small-ish decoder-only models (i.e., typical LLMs) for translation. Sure, GPT4 etc. work extremely well, but not so much local models capable of running on small form-factor devices. Perhaps I should revisit that.