4 ms·
It wouldn’t be useful with your setup, probably 3-4 token per second.
by manmal 5mo ago
It wouldn’t be useful with your setup, probably 3-4 token per second.
- DeathArrow 5mo agoYep, maybe I can open a feature request if it makes sense technically.
- zozbot234 5mo agoArguably it makes more sense technically to get the model support into llama.cpp, which provides many options for GPU+CPU split inference already.