4 ms·

Isn't that going to be extremely slow? I can only realistically run 7B 5-bit models on my RTX 3060, anything more and it offloads to the CPU. My responses go fr

by adamjc 3y ago

Isn't that going to be extremely slow? I can only realistically run 7B 5-bit models on my RTX 3060, anything more and it offloads to the CPU. My responses go from almost-instantaneous to 3mins+.

_boffin_ 3y ago
It seems like it's running at comparable speeds to GPT4 prior to Turbo. I could be wrong, but what I'm trying to say, it ain't bad at all.
bdavbdav 3y ago
This is where the Mac world shines.
- lamtung 3y ago
  would a 32gb M2 Max be able to run a 34b-model?