3 ms·
It's technically possible, in the sense that you will eventually get a complete response, but it will be extremely slow. Inference will be something like 1 wor
by loudmax 3y ago
It's technically possible, in the sense that you will eventually get a complete response, but it will be extremely slow. Inference will be something like 1 word per second. Far too slow to use as an assistant for writing code.
If you don't want to rent something with a GPU, you should look into running non-70B models and experiment with varying levels of quantization. Without a GPU you really want to start with 7B models, not 70B models. The Mistral variants in particular are worth a try.