4 ms·

Newer Apple computers have unified gpu and cpu memory, so they are BLAZINGLY fast with local models.

• resource_waste 3y ago
  This is just Apple marketing, integrated GPUs into CPUs are still CPUs. There is a reason no one ever shows >2000 token reponses.
  • acchow 3y ago
    It isn't that bad. One user with an M2 studio: > For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second. https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment/kc27t0o/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment... You can't even fit this on 2x RTX 4090.
    • resource_waste 3y ago
      But why? Like, use the 7B berkley sterling on a freaking 3060 laptop and you are still getting better results on both output and tokens. I don't understand what they are getting out of these large models that perform worse.