28 ms·
IMO Berkley Sterling which is a fine-tuned llama. I'm not really sure what you are expecting to do with CPU. You might be able to get some <400 token responses
by resource_waste 3y ago
IMO Berkley Sterling which is a fine-tuned llama.
I'm not really sure what you are expecting to do with CPU. You might be able to get some <400 token responses and have fun, but you aren't going to be doing 2000 token encyclopedia style responses unless you are going to wait ~20 minutes for a response.
Alternatively, you can get like a $800 gaming laptop that has 8gb vram or use something like vastai where you can get 12gb vram for $0.10 an hour.
- dartos 3y agoNewer Apple computers have unified gpu and cpu memory, so they are BLAZINGLY fast with local models.
- resource_waste 3y agoThis is just Apple marketing, integrated GPUs into CPUs are still CPUs. There is a reason no one ever shows >2000 token reponses.
- acchow 3y agoIt isn't that bad. One user with an M2 studio: > For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second. https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment/kc27t0o/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment... You can't even fit this on 2x RTX 4090.
- resource_waste 3y agoBut why? Like, use the 7B berkley sterling on a freaking 3060 laptop and you are still getting better results on both output and tokens. I don't understand what they are getting out of these large models that perform worse.
- acchow 3y agoWhat is the use case of buying a $800 gaming laptop to run a smaller model instead of using GPT 3.5?
- thoughtpeddler 3y agoOne word: privacy