4 ms·

Just bought a Strix Halo (framework desktop), waffled a long time between that and a Mac Studo but I got tired of waiting for the M5 and don’t really like Apple

by andy99 9mo ago

Just bought a Strix Halo (framework desktop), waffled a long time between that and a Mac Studo but I got tired of waiting for the M5 and don’t really like Apple. I work with ML professionally, almost all in cloud, I just wanted something “off grid” and unmetered, and needed a computer anyway so decided to pay a bit more and get the one I want. It’s “personal” in that it’s exclusively for me, but I have a business and bought it for that. Still figuring out the best software, so far it looks like llama.cpp with Vulcan though I have a lot of experimenting to do and don’t currently find it optimal for what I want.

01092026 9mo ago
Well, Mac chips are badass for training / inference - super underrated. I mean, I've literally run epochs on cloud Nvidia GPU Servers...compared to running them locally (M chip) - and look, not trying to burn any houses down but...eh...Apple does really really well. The good news for you, you can chain like a bunch / couple of them together and run the largest open source models around. But extremely expensive route - but probably the easiest and smoothest way. If you're planning on running this on Apple - you can do some stuff with Metal directly...in PyTorch it's 'mcu' if I remember? I think your llama.cpp route is good - I wouldn't go the Ollama route - I mean great to start, but IMHO: get the models directly, learn the layers and how the heads work as best as you can, make an effort to understand what's going on - well you don't have to, but, I think the models appreciate the effort - respect goes far.
Blue_Cosma 9mo ago
Thanks a lot for sharing. Haven't tested Strix Halo myself. Did you consider DGX Spark as well? What is your target use case? Curious what feels suboptimal about llama.cpp + Vulkan so far.
- andy99 9mo ago
  Re DGX, I’m mostly interested in local inference, it might have been nice to try but it was more expensive for similar performance (or so I think). I do lots of different experiments, synthetic data generation along the lines of Magpie is one of the things I wanted a local machine for, as well as just general access to a decent sized LLM to try different things, without having to spin up a cloud machine each time. I would prefer PyTorch / HF transformers to llama.cpp as I fine the latter less flexible if I want to change anything.

by andy99 9mo ago

01092026 9mo ago

Well, Mac chips are badass for training / inference - super underrated. I mean, I've literally run epochs on cloud Nvidia GPU Servers...compared to running them locally (M chip) - and look, not trying to burn any houses down but...eh...Apple does really really well. The good news for you, you can chain like a bunch / couple of them together and run the largest open source models around. But extremely expensive route - but probably the easiest and smoothest way. If you're planning on running this on Apple - you can do some stuff with Metal directly...in PyTorch it's 'mcu' if I remember? I think your llama.cpp route is good - I wouldn't go the Ollama route - I mean great to start, but IMHO: get the models directly, learn the layers and how the heads work as best as you can, make an effort to understand what's going on - well you don't have to, but, I think the models appreciate the effort - respect goes far.

Blue_Cosma 9mo ago

Thanks a lot for sharing. Haven't tested Strix Halo myself. Did you consider DGX Spark as well? What is your target use case? Curious what feels suboptimal about llama.cpp + Vulkan so far.

andy99 9mo ago
Re DGX, I’m mostly interested in local inference, it might have been nice to try but it was more expensive for similar performance (or so I think). I do lots of different experiments, synthetic data generation along the lines of Magpie is one of the things I wanted a local machine for, as well as just general access to a decent sized LLM to try different things, without having to spin up a cloud machine each time. I would prefer PyTorch / HF transformers to llama.cpp as I fine the latter less flexible if I want to change anything.