3 ms·
Running models on-device on a Mac is immensely annoying though. Figuring out what will work out of BF16, FP8, BF16+FP8, NVFP4, INT8, GGUF ... the list goes on .
by onion2k 3mo ago
Running models on-device on a Mac is immensely annoying though. Figuring out what will work out of BF16, FP8, BF16+FP8, NVFP4, INT8, GGUF ... the list goes on ... is 'non-obvious' at best. Apple do little to support with tooling. There's MLX, but unless you're happy to transform a model to that format yourself you'll be lagging a long way behind.
Apps like LMStudio, Ollama, Draw Things, etc do a great job of simplifying it but it's still a pain.
- kamranjon 3mo agoI dunno I use LMStudio pretty regularly and the MLX folks and the community usually have MLX versions of new model releases up within a day or two.
- onion2k 3mo agoFor some models like the popular coding and chat models, things move faster. For things like images, voice, sound etc they definitely lag a long way behind.
- Barbing 3mo ago& they clearly delineate all the models that’ll work on your exact machine (but guardrails can be disabled in settings)
- throw1234567891 3mo agoConverting a model to mlx is literally a git clone and mlx_lm.convert. 5 minute operation after clone.
- fellowmartian 3mo agoOr just ask any model to do it for you.
- gizajob 3mo agoHow is it a pain exactly? It’s just learning and only takes a day or two to get up to speed. We seem to have forgotten that for the past fifty years doing all kinds of tasks on computers has been tedious and involved and time consuming to even get working. My first computer had 48kb of RAM and to play a game you had to load it off cassette for five minutes. That was annoying. Having LM Studio download a model and load so you can chat or attach an agent it is effortless and easy in comparison.
- prodmod 3mo agoNot to mention you can just ask Codex or Claude Code to set up a local LLM for you. Use one tool to set up the other.
- fghorow 3mo agoCounterpoint, I'm just a happy user of this local LLM [1] from antirez (of Redis fame). [1] https://github.com/antirez/ds4 https://github.com/antirez/ds4
- no_no_no_yes 3mo agoIn general I just don't like not using a cross platform architecture of some kind. MLX is fine, but the cross platform alternatives (ONNX) are terrible, GGUF often lacks and doesn't have the "easy convert" that some of the commenters below say MLX has.
- albrewer 3mo agoWhat problems are you running into? Ive been running multiple qwen coder next agents locally and it works fine.