3 ms·
Mistral LLM on Apple Silicon Using Apple's MLX Framework Runs Instantaneously
- smoldesu 3y ago"Much Faster" than what?
- behnamoh 3y agohttps://www.reddit.com/r/LocalLLaMA/comments/18bwd1y/comment/kc7c22w/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18bwd1y/comment...
- brucethemoose2 3y agoThats... Not very promising. The thread suggests it doesn't even quantize the model (running it in FP16, so tons of ram usage), and that its slower than the llama.cpp Metal backend anyway? And MLC-LLM was faster than llama.cpp, last I checked. Its hard to keep up with developments.
- behnamoh 3y agoI think llama.cpp is the sweet spot right now, due to its grammar capability and many other features (e.g., multimodal). MLC-LLM is nice but they don't offer uncensored models.
- brucethemoose2 3y ago- A: You can convert models to MLC yourself, just like GGUF models, with relative ease. - B: Yeah, llama.cpp has a killer feature set. And killer integration with other frameworks. MLC is way behind, but is getting more fleshed out every time I take a peek at it. - C: This is a pet peeve of mine, but I've never run into a local model that was really uncensored. For some, if you give them a GPT4 prompt... Of course you get a GPT4 response. But you can just give them a unspeakable system prompt or completion, and they will go right ahead and complete it. I don't really get why people fixate on the "default personality" of models trained on GPT4 data.
- Tiberium 3y agoC. Have you tried OpenHermes 2.5? It's a Mistral chat finetune, but a very good one at that.
- brucethemoose2 3y agoYeah. If you are looking for new models, I would recommend Xaberius and Cybertron, as well as the various OpenHermes DPO finetunes. Personally I run my own Yi 200K DARE merge because I love the long context.
- mark_l_watson 3y agoLlama.cpp is great but I have moved to mostly using Ollama because it is both good on the command line and ‘ollama server’ runs a very convenient to use REST server. In any case, I had fun with MLX today, and I hope it implements 4 bit quantization soon.
- robertritz 3y agoMLX only works for fp16 right now. If it ever works quantized I will almost certainly move my app over to MLX instead of llama.cpp. My app also uses a very small (30MB) PyTorch model and shipping it requires an extra 100MB for PyTorch in the app. Very very stupid. I think its important to remember that last mile inference is still pretty bespoke for most things. If we want to see gen AI stuff take off and now have the big cloud providers in charge this needs to be fixed. Apple is in a good place to solve at least part of the equation.
- kristianp 3y agoIf the model could be made to work with llama.cpp, then https://github.com/abetlen/llama-cpp-python https://github.com/abetlen/llama-cpp-python might be more compact. llama.cpp only supports a limited list of model types though.
- behnamoh 3y agoApple will most likely introduce their own API for an LLM-powered Siri which devs can use in their apps. This way, Apple keeps their control and can fully optimize the SiriGPT (or whatever you call it). Very unlikely that Apple would just provide the hardware, good amount of RAM, a framework competing with Pytorch, all out of the goodness of their heart. I use Apple products but I know they're way past the point of doing things just for the sake of fcking doing something awesome.
- novok 3y agoBecoming the AI dev and inference platform of choice and getting that nvidia profit margin is probably plenty enough of a reason. Right now it's CUDA on linux PCs.
- AceJohnny2 3y agoI don't see Apple taking NVidia's crown for training. Apple's hardware acceleration is fine for low-power on-device inference, which they were designed for, but competitive training requires an order or three magnitude more power, which they aren't.
- bluSCALE4 3y agoI don't really get the purpose of this. Can anyone share some insights?
- jasonjmcghee 3y agoWhat an odd misleading title. From the referenced reddit thread in another comment (https://www.reddit.com/r/LocalLLaMA/comments/18bwd1y/comment/kc7c22w/?utm_source=share&utm_medium=web2x&context=3 https://www.reddit.com/r/LocalLLaMA/comments/18bwd1y/comment...) sounds like it runs slower than llama cpp? 25 t/s. That’s not instantaneous by any measure
- deleted 3y ago[deleted]
- dagmx 3y agoI think you misread the title. The title implies that it took little to no work to port it to this new framework, not that it runs in an instant.
- buffington 3y agoHow does one read "Runs Instantly" and not interpret that to mean anything other than that it runs instantly? "A fire built around a wad of kerosene soaked rags starts instantly." Does that sentence suggest that kerosene starts rags to you, or that kerosene soaked rags help start fires instantly?
- dagmx 3y agoThat comparison is irrelevant because words have meaning based on the context they’re used in. English, being a very complex language, can have the same sentence mean multiple things as well depending on how you interpret the context. There may be better ways to phrase something to remove ambiguity, but the title as presented could be read both ways.
- smoldesu 3y agoOP changed the submission title after this first comment was written. It used to say MLX was "much faster" without any point of comparison. It has since been edited to what it is now.
- luke-stanley 3y agoThe title could be made much clearer by saying "X" tokens per second, with "Y" model,on "Z" hardware, and what the model load time was. The quantisation level and byte size helps too.