5 ms·

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://

I have two 128gb Strix Halos and have been extremely excited about Antirez's (Redis author) work on DS4, especially with 4bit quant using two machines: https://github.com/antirez/ds4 https://github.com/antirez/ds4 Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM https://www.youtube.com/watch?v=PkKXm_mKCCM With a bit more speed and model improvements, local AI becomes a reasonable practical thing! The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... I got my Strix machines for ~2k eur each, best computers this 90s kid has ever owned, but those days are gone :(

rnewme 4mo ago
What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?
- pixelpoet 4mo ago
  IIRC llama.cpp doesn't implement DSv4's compressed attention mechanism, and while it does use (credited) parts of llama.cpp, it's focused on this great model for now. Much of this is covered better in the repo's readme.
  - rnewme 4mo ago
    In repo Readme and antirez reddit comments there was also expressed willingness to upstream.
- francisduvivier 4mo ago
  I think mainly that he can move much faster with specific improvements targeting Deepseek on Systems with unified memory (Mac or Strix). It's a lot easier to optimize if you don't need to worry about all the other architectures. So optimize he did and it's just a lot faster than llama cpp for deepseek v4 pro and flash. Also interesting features are more doable, like SSD streaming, which makes it possible to load MOE weights for a model larger than your VRAM, I don't see that landing in llama cpp anytime soon.
- mkesper 4mo ago
  Currently, llama.cpp clusters don't support tensor parallelism, have a look at Donato Capitella's detailed report: https://m.youtube.com/watch?v=PkKXm_mKCCM https://m.youtube.com/watch?v=PkKXm_mKCCM He also provides rocm toolboxes for Strix Halo: https://strix-halo-toolboxes.com/#about https://strix-halo-toolboxes.com/#about
gruez 4mo ago
>The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from higher egg prices because it makes cooking eggs at home more expensive.
- sdf4j 4mo ago
  You got it wrong. Use appliances instead of eggs. If getting an oven gets more expensive I rather keep going to Dennys. It’s classic capex vs opex. I’d keep paying my openai subscription instead of dropping $3k to run a subpar model. If the thing costs $1k I would consider it.
  - codingrightnow 4mo ago
    The big AI companies already have all of the cheap hardware to run on. It's more like if Denny's bought up all of the eggs when prices were normal, held onto them, then kept buying to keep egg prices up.
    - gruez 4mo ago
      >The big AI companies already have all of the cheap hardware to run on. Have they? Aren't they doing a massive datacenter build-out right now? Moreover the massive profits for Micron and Nvidia must be coming from somewhere, and I doubt it's price-sensitive consumers.
- mkj 4mo ago
  openai etc are going to have a higher utilisation of the hardware so can afford it more than small companies/people. Efficient resource use matters more when they're expensive.
- omgwtfbyobbq 4mo ago
  Dennys can benefit from higher egg prices if they can lock in long term contracts with suppliers for lower egg prices when smaller companies selling directly to consumers can't. I think that realistically, companies compete against each other as individuals and compete against smaller companies and individuals acting more like cartels/monopolies, and that's what OP is referring to in terms of hardware purchasing/contacts/pricing. This also extends outside of tech to investing, so it's likely not just tech responsible for this.
rjzzleep 4mo ago
Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.
- pixelpoet 4mo ago
  I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well. [Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning the wisdom of laptops for high performance computing, including gaming. Everyone insists they need the mobility, and then just leave it plugged in the whole time, absolutely refusing to admit it's about aesthetic preference.]
  - Gareth321 4mo ago
    I’m mostly with you but there are some people who like to use one machine for both laptop and AI work, and it’s much cheaper than buying two separate devices.
  - rzzzt 4mo ago
    For me the smaller footprint, lower power consumption and portability (admittedly between desks only) are the three advantages of using a laptop over a desktop for these purposes.
    - pixelpoet 4mo ago
      The Strix Halo mini PCs use the exact same chip, and have a much smaller footprint than any laptop. Have you seen the size of these machines? I can and have easily popped my daily driver computer into my very small backpack to attend a demoparty for example. With the laptop you probably won't get silent operation at the peak 100-140w, i.e. you've now massively overpaid for lower performance.
Gareth321 4mo ago
I was hoping to buy a competent local model machine later this year but given the prices I’m shelving that for now. Especially because the frontier models are very cheap relative to the cost of building my own setup. Especially because AI specialised hardware and processors are improving very fast, meaning hardware we buy now will become obsolete for this use case much faster than for traditional computer use cases. In 1-3 years the hardware crunch will be over, local distilled models will provide Opus 4.8 like intelligence, and the hardware will exist to provide usable performance.
sspiff 4mo ago
I had a Strix Halo laptop with 128GB which unfortunately died last week. I paid 2800 euro for it. If I buy the same machine today, the sticker price is 7899. The device was not perfect by any means, but the ability to run fairly large models is some kind of magic.
- fennecbutt 4mo ago
  For that price surely you get the rtx pro with like 96GB and vastly superior compute?
  - sspiff 4mo ago
    Yes but it depends what you want. I didn't want to spend 8k (or now 11k) for a video card that I couldn't run without turning my home office into a boiler room and pay through the nose in electricity. I wanted a laptop that could run some medium sized LLMs locally and experiment with. Strix Halo is great for that. But it was also 2800 euros for a 128GB premium laptop. Not 7899 like it is now. Shame it broke out of warranty.
- barbacoa 4mo ago
  >sticker price is 7899. It's not even worth it at that point. You can get a used enterprise grade SXM baseboard with 4-8 V100/A100 GPUs off eBay at a similar price. That will even get you actual HMB ram and NVlink. Along with 10x the AI performance, assuming you don't care about your electricity bill of course.
  - Keyframe 4mo ago
    ignoring the fact one would need a bit of a different setup (chassis, PSU) to run it, I casually looked and there's nothing below $25-50k euros for such a board decked out, depending on the config. TBH even that doesn't sound bad, but I wouldn't even know where to start how to run it.
    - barbacoa 4mo ago
nicman23 4mo ago
if strix was 2k now i would insta buy 1
- a96 4mo ago
  Check https://news.ycombinator.com/item?id=48705076 https://news.ycombinator.com/item?id=48705076 The top model is a little over 2k, though.
fennecbutt 4mo ago
Are they fairly reasonable performance from your experience? I'm about to move house and one of the things I want is better local AI. Ideally running a variety of smaller models at the same time, but occasionally one larger one. 128GB unified is pretty appealing, but for the price I can also get a handful of 5060 ti 16GB which give vastly better compute/inference speed. Seen a few reviews of the strix halo boards from people complaining the tps is too low so definitely keen to hear others' experiences with em.

rnewme 4mo ago

What's the advantage of ds4 over llama.cpp, esp if down the line they upstream his forked kernels?

pixelpoet 4mo ago
IIRC llama.cpp doesn't implement DSv4's compressed attention mechanism, and while it does use (credited) parts of llama.cpp, it's focused on this great model for now. Much of this is covered better in the repo's readme.
- rnewme 4mo ago
  In repo Readme and antirez reddit comments there was also expressed willingness to upstream.
francisduvivier 4mo ago
I think mainly that he can move much faster with specific improvements targeting Deepseek on Systems with unified memory (Mac or Strix). It's a lot easier to optimize if you don't need to worry about all the other architectures. So optimize he did and it's just a lot faster than llama cpp for deepseek v4 pro and flash. Also interesting features are more doable, like SSD streaming, which makes it possible to load MOE weights for a model larger than your VRAM, I don't see that landing in llama cpp anytime soon.
mkesper 4mo ago
Currently, llama.cpp clusters don't support tensor parallelism, have a look at Donato Capitella's detailed report: https://m.youtube.com/watch?v=PkKXm_mKCCM https://m.youtube.com/watch?v=PkKXm_mKCCM He also provides rocm toolboxes for Strix Halo: https://strix-halo-toolboxes.com/#about https://strix-halo-toolboxes.com/#about

gruez 4mo ago

>The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately... You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from higher egg prices because it makes cooking eggs at home more expensive.

sdf4j 4mo ago
You got it wrong. Use appliances instead of eggs. If getting an oven gets more expensive I rather keep going to Dennys. It’s classic capex vs opex. I’d keep paying my openai subscription instead of dropping $3k to run a subpar model. If the thing costs $1k I would consider it.
- codingrightnow 4mo ago
  The big AI companies already have all of the cheap hardware to run on. It's more like if Denny's bought up all of the eggs when prices were normal, held onto them, then kept buying to keep egg prices up.
  - gruez 4mo ago
    >The big AI companies already have all of the cheap hardware to run on. Have they? Aren't they doing a massive datacenter build-out right now? Moreover the massive profits for Micron and Nvidia must be coming from somewhere, and I doubt it's price-sensitive consumers.
mkj 4mo ago
openai etc are going to have a higher utilisation of the hardware so can afford it more than small companies/people. Efficient resource use matters more when they're expensive.
omgwtfbyobbq 4mo ago
Dennys can benefit from higher egg prices if they can lock in long term contracts with suppliers for lower egg prices when smaller companies selling directly to consumers can't. I think that realistically, companies compete against each other as individuals and compete against smaller companies and individuals acting more like cartels/monopolies, and that's what OP is referring to in terms of hardware purchasing/contacts/pricing. This also extends outside of tech to investing, so it's likely not just tech responsible for this.

rjzzleep 4mo ago

Last year you could buy a AI Max 395+ with 128G for 2.5k, now it's almost $4k. Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.

pixelpoet 4mo ago
I think people buying laptops for AI use are, sorry, just plain crazy. You overpay for the screen and keyboard and battery and whatever, plus you get much worse thermal performance because of basic physics (area vs volume). My Framework Desktop has a Noctua cooler which works really well. [Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning the wisdom of laptops for high performance computing, including gaming. Everyone insists they need the mobility, and then just leave it plugged in the whole time, absolutely refusing to admit it's about aesthetic preference.]
- Gareth321 4mo ago
  I’m mostly with you but there are some people who like to use one machine for both laptop and AI work, and it’s much cheaper than buying two separate devices.
- rzzzt 4mo ago
  For me the smaller footprint, lower power consumption and portability (admittedly between desks only) are the three advantages of using a laptop over a desktop for these purposes.
  - pixelpoet 4mo ago
    The Strix Halo mini PCs use the exact same chip, and have a much smaller footprint than any laptop. Have you seen the size of these machines? I can and have easily popped my daily driver computer into my very small backpack to attend a demoparty for example. With the laptop you probably won't get silent operation at the peak 100-140w, i.e. you've now massively overpaid for lower performance.

Gareth321 4mo ago

I was hoping to buy a competent local model machine later this year but given the prices I’m shelving that for now. Especially because the frontier models are very cheap relative to the cost of building my own setup. Especially because AI specialised hardware and processors are improving very fast, meaning hardware we buy now will become obsolete for this use case much faster than for traditional computer use cases. In 1-3 years the hardware crunch will be over, local distilled models will provide Opus 4.8 like intelligence, and the hardware will exist to provide usable performance.

sspiff 4mo ago

I had a Strix Halo laptop with 128GB which unfortunately died last week. I paid 2800 euro for it. If I buy the same machine today, the sticker price is 7899. The device was not perfect by any means, but the ability to run fairly large models is some kind of magic.

fennecbutt 4mo ago
For that price surely you get the rtx pro with like 96GB and vastly superior compute?
- sspiff 4mo ago
  Yes but it depends what you want. I didn't want to spend 8k (or now 11k) for a video card that I couldn't run without turning my home office into a boiler room and pay through the nose in electricity. I wanted a laptop that could run some medium sized LLMs locally and experiment with. Strix Halo is great for that. But it was also 2800 euros for a 128GB premium laptop. Not 7899 like it is now. Shame it broke out of warranty.
barbacoa 4mo ago
>sticker price is 7899. It's not even worth it at that point. You can get a used enterprise grade SXM baseboard with 4-8 V100/A100 GPUs off eBay at a similar price. That will even get you actual HMB ram and NVlink. Along with 10x the AI performance, assuming you don't care about your electricity bill of course.
- Keyframe 4mo ago
  ignoring the fact one would need a bit of a different setup (chassis, PSU) to run it, I casually looked and there's nothing below $25-50k euros for such a board decked out, depending on the config. TBH even that doesn't sound bad, but I wouldn't even know where to start how to run it.
  - barbacoa 4mo ago

nicman23 4mo ago

if strix was 2k now i would insta buy 1

a96 4mo ago
Check https://news.ycombinator.com/item?id=48705076 https://news.ycombinator.com/item?id=48705076 The top model is a little over 2k, though.

fennecbutt 4mo ago

Are they fairly reasonable performance from your experience? I'm about to move house and one of the things I want is better local AI. Ideally running a variety of smaller models at the same time, but occasionally one larger one. 128GB unified is pretty appealing, but for the price I can also get a handful of 5060 ti 16GB which give vastly better compute/inference speed. Seen a few reviews of the strix halo boards from people complaining the tps is too low so definitely keen to hear others' experiences with em.