3 ms·
This is just Apple marketing, integrated GPUs into CPUs are still CPUs. There is a reason no one ever shows >2000 token reponses.
by resource_waste 3y ago
This is just Apple marketing, integrated GPUs into CPUs are still CPUs.
There is a reason no one ever shows >2000 token reponses.
- acchow 3y agoIt isn't that bad. One user with an M2 studio: > For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second. https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment/kc27t0o/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment... You can't even fit this on 2x RTX 4090.
- resource_waste 3y agoBut why? Like, use the 7B berkley sterling on a freaking 3060 laptop and you are still getting better results on both output and tokens. I don't understand what they are getting out of these large models that perform worse.