4 ms·
It will be sort of decent on a 4bit 70B parameter model, like here https://www.youtube.com/watch?v=5ktS0aG3SMc https://www.youtube.com/watch?v=5ktS0aG3SMc (dee
by phonon 1y ago
It will be sort of decent on a 4bit 70B parameter model, like here https://www.youtube.com/watch?v=5ktS0aG3SMc https://www.youtube.com/watch?v=5ktS0aG3SMc
(deepseek-r1:70b Q4_K_M). But yeah, not great.