4 ms·
I am glad to see Anthropic get more financial support, or as another person here said, AWS credits. Hopefully not so far topic as to be uninteresting: I am very
by mark_l_watson 3y ago
I am glad to see Anthropic get more financial support, or as another person here said, AWS credits.
Hopefully not so far topic as to be uninteresting: I am very happy with the state of LLM tech. There are open weight models that are very capable on modest home computers (like mistral-7B-instruct-ver-2), and beefed up home computers (I can barely run mixtral-8-7B, but it runs well enough). And there are many open weight specialty models that I find useful.
For commercial offerings, OpenAI, Anthropic, Mistral, etc. have affordable APIs, and there are a bunch of companies like Scaleway for running open weight models too large or inconvenient to run at home.
At the application layer, I like the new emphasis on agents, open source apps that use local models, and highly refined products like Perplexity, OpenAI app/web, Google’s workplace integrations, and all the stuff that Microsoft is doing.
There are problems like the environmental costs of training and running models, but things will get more efficient and people will realize the utility of smaller special purpose models.
- brcmthrowaway 3y agoWhats the best model that can work on a Mac Studio?
- akmittal 3y agoLlama or mistral both are good. Running using ollama
- LeoPanthera 3y agoRAM is the limiting factor, but Mixtral 8x7 is probably the state of the art for self-hosted LLMs right now. If you are low on RAM you can run a quantized version, at the expense of reduced quality.
- resource_waste 3y agoIMO Berkley Sterling which is a fine-tuned llama. I'm not really sure what you are expecting to do with CPU. You might be able to get some <400 token responses and have fun, but you aren't going to be doing 2000 token encyclopedia style responses unless you are going to wait ~20 minutes for a response. Alternatively, you can get like a $800 gaming laptop that has 8gb vram or use something like vastai where you can get 12gb vram for $0.10 an hour.
- dartos 3y agoNewer Apple computers have unified gpu and cpu memory, so they are BLAZINGLY fast with local models.
- resource_waste 3y agoThis is just Apple marketing, integrated GPUs into CPUs are still CPUs. There is a reason no one ever shows >2000 token reponses.
- acchow 3y agoIt isn't that bad. One user with an M2 studio: > For a 34b q8 sending in 6000 context (out of a total of 16384) I get about 4 tokens per second. https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment/kc27t0o/?utm_source=reddit&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18b1qgy/comment... You can't even fit this on 2x RTX 4090.
- resource_waste 3y agoBut why? Like, use the 7B berkley sterling on a freaking 3060 laptop and you are still getting better results on both output and tokens. I don't understand what they are getting out of these large models that perform worse.
- acchow 3y agoWhat is the use case of buying a $800 gaming laptop to run a smaller model instead of using GPT 3.5?
- thoughtpeddler 3y agoOne word: privacy
- woadwarrior01 3y agoYou can run 4-bit quantized Mixtral 8x7B with unquantized MoE gates, if you've got at least 32GB of RAM. The model itself takes up about 24GB of RAM.
- yawnxyz 3y agoI have a 32GB M3 and... Mixtral 8x7B likes to completely crash the machine haha
- woadwarrior01 3y agoWhat software are you using for inference? I hate plugging my own app[1] here, but I know many people on my app's discord that are running 4-bit OmniQuant quantized Mixtral 8x7B on it on >= 32GB M1, M2 and M3 Macs. I run it all the time 64GB M2 Mac Studio and it takes up just under 24GB of RAM. Also runs Yi-34B-Chat, which takes up ~18.15GB of RAM. [1]: https://privatellm.app https://privatellm.app
- mark_l_watson 3y agoI have a M2 Pro 32B memory Mac mini, and I can run mixtral-7-8-Q3 - that is with 3 bit quantization.
- woadwarrior01 3y agoIIRC, you're mentioned once before that you've used Private LLM. :) Please try the 4-bit OmniQuant quantized Mixtral 8x7B Instruct model in it. It runs circles around RTN Q3 models at speed and RTN Q8 models at text generation quality.