3 ms·
> Ie you could technically run something like that today. Yep, on higher end machines it should already be feasible. I can do 2.5-3 tok/sec on a 70B model qua
by rootusrootus 3y ago
> Ie you could technically run something like that today.
Yep, on higher end machines it should already be feasible. I can do 2.5-3 tok/sec on a 70B model quantized at 4 bit today with my MacBook Pro M2 MAX w/96GB. It's a little slower than a 30B, but the difference is less than I had guessed it would be. That's not super fast, but it's usable.
And that's on a machine that isn't designed for this workload. Over the next few years things should improve quite a bit. 200B does not seem like a reach.