3 ms·
This is if we follow the recommendation to use a GPU. But if you don't need 200+ tokens per second and are happy with something like 10-15, a CPU-only solution
by patrakov 3y ago
This is if we follow the recommendation to use a GPU. But if you don't need 200+ tokens per second and are happy with something like 10-15, a CPU-only solution is acceptable and would cost significantly less.