3 ms·
It isn't that bad. One user with an M2 studio: > For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second. https://www.r
by acchow 3y ago
It isn't that bad. One user with an M2 studio:
> For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second.
https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment/kc27t0o/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment...
You can't even fit this on 2x RTX 4090.
- resource_waste 3y agoBut why? Like, use the 7B berkley sterling on a freaking 3060 laptop and you are still getting better results on both output and tokens. I don't understand what they are getting out of these large models that perform worse.