3 ms·
Yes, look at ollama. You'll be able to run CodeLlama 70B at Q6_K with lots of memory to spare. It won't be very fast though.
by petercooper 3y ago
Yes, look at ollama. You'll be able to run CodeLlama 70B at Q6_K with lots of memory to spare. It won't be very fast though.