5 ms·

I absolutely love it. I’ve been up for days playing with it. But there are some bleeding edge issues. I tried to write a balanced article. I would highly recomm

I absolutely love it. I’ve been up for days playing with it. But there are some bleeding edge issues. I tried to write a balanced article. I would highly recommend for people that love to get their hands dirty. Blows away any consumer GPU.

furyofantares 11mo ago
Since the text is obviously LLM output, how much prompting and editing went into this post? Did you have to correct anything that you put into it that it then got wrong or added incorrect output to?
- NathanielK 11mo ago
  Definitely reeks of someone who doesn't know what makes a readable blogpost and hoped the LLM did. I was not familiar with the hardware, so I was disappointed there wasn't a picture of the device. Tried to skim the article and it's a mess. Inconsistent formatting and emoji without a single graph to visualize benchmarks.
  - furyofantares 11mo ago
    I read the whole thing now and it's filled with slop. I don't really care about the emojis and the marketing voice too much. I do care that it's impossible to tell what the author cared about what they didn't, or if any of it is made up or extrapolated. I bet the input to the LLM would have been more interesting.
    - furyofantares 11mo ago
      > Training Performance is Real (When It Works) It looks like it worked? Why's it say this? > Verdict: Inference speed scales proportionally with model size. Author only tried one model size and it's faster than NVIDIA's reported speed at a larger model. Not really a "Verdict". > Verdict: 4-bit quantization is production-viable. That's not really something you can conclude from messing around with it and saying you like the outputs. > GPU Inference is Fundamentally Broken Probably not? It probably just doesn't work in llama.cpp right now? Takes a while reading this to work out they tried ollama and then later llama.cpp, which I'd guess is basically testing llama.cpp twice. Actually I don't even believe that, I'm sure author ran into errors that might be a pain to figure out, but there's no evidence it's worse than that. But then it says this is the "root cause": ARM64 + Blackwell + CUDA 13.0 = Bleeding Edge ↓ Limited production testing ↓ Edge cases in numerical precision (inference) ↓ Memory management issues (training) Am I to believe GPU inference is really fundamentally broken? I'm not seeing the case made here, just claims. At this point the LLM seems to have gotten confused about whether it's talking about the memory fragmentation issue or the GPU inference issue. But it's hard to believe anything from this point on in the post.
enum 11mo ago
+1 I have H100s to myself, and access to more GPUs than I know what to do with in national clusters. The Spark is much more fun. And I’m more productive. With two of them, you can debug shallow NCCL/MPI problems before hitting a real cluster. I sincerely love Slurm, but nothing like a personal computer.
- latchkey 11mo ago
  Your complaint sounds more like the way that you have to access the HPC (via slurm), not the compute itself. After having now tried slurm myself, I don't understand the love for it at all. As for debugging, that's where you should be allowed to spin up a small testing cluster on-demand. Why can't you do that with your slurm access?
  - enum 11mo ago
    I’m not complaining. The clusters are great. The non-Slurm H100s are great. The Spark is more fun.
    - latchkey 11mo ago
      What makes it more fun?
      - enum 11mo ago
        I think that personal computing is more fun than time-shared computing. :) It's remarkable what can now be done on a whisper-quiet little box. I hope the Strix Halo's will be just as much fun, and they should be, so long as Flash Attention works.
        latchkey 11mo ago
        Haven't tried to compile it for SH, but did compile it for MI355x and it worked. LONG compile time though ninja sure helped. Fair, thanks for the answer. The bane of my existence... salloc: Granted job allocation 1978 salloc: Waiting for resource configuration
yunohn 11mo ago
Thanks for this bleeding edge content! But please have your LLM post writer be less verbose and repetitive. This is like the stock output from any LLM, where it describes in detail and then summarizes back and forth over multiple useless sections. Please consider a smarter prompt and post-editing…
- Tepix 11mo ago
  I agree whole-heartedly. Two thirds of the article read like slop.
CompoundEyes 11mo ago
One thing I can’t find anyone mention in reviews - does inference screech to a halt when using large context windows on models? Say if you’re in the 100k range on gpt-oss. I’m not concerned about lightning inference speed overall as I understand the purpose of the spark is to be well rounded / trainer tuner. I just want to know if it becomes unusable vs reasonable slowdown at larger contexts. That’s the thing people are unpleasantly surprised to find about a Mac Studio which has prevented me from going that route.
Tepix 11mo ago
> Blows away any consumer GPU. Nah. Do you have 1st hand experience with Strix Halo? At less than 1600€ for a 128GB configuration it manages >45 tokens/s with gpt-oss 120b. Which is faster than DGX Spark at a fraction of the cost.
- storus 11mo ago
  Strix Halo has awful token prefill speed. Only suitable for very small contexts.

furyofantares 11mo ago

Since the text is obviously LLM output, how much prompting and editing went into this post? Did you have to correct anything that you put into it that it then got wrong or added incorrect output to?

NathanielK 11mo ago
Definitely reeks of someone who doesn't know what makes a readable blogpost and hoped the LLM did. I was not familiar with the hardware, so I was disappointed there wasn't a picture of the device. Tried to skim the article and it's a mess. Inconsistent formatting and emoji without a single graph to visualize benchmarks.
- furyofantares 11mo ago
  I read the whole thing now and it's filled with slop. I don't really care about the emojis and the marketing voice too much. I do care that it's impossible to tell what the author cared about what they didn't, or if any of it is made up or extrapolated. I bet the input to the LLM would have been more interesting.
  - furyofantares 11mo ago
    > Training Performance is Real (When It Works) It looks like it worked? Why's it say this? > Verdict: Inference speed scales proportionally with model size. Author only tried one model size and it's faster than NVIDIA's reported speed at a larger model. Not really a "Verdict". > Verdict: 4-bit quantization is production-viable. That's not really something you can conclude from messing around with it and saying you like the outputs. > GPU Inference is Fundamentally Broken Probably not? It probably just doesn't work in llama.cpp right now? Takes a while reading this to work out they tried ollama and then later llama.cpp, which I'd guess is basically testing llama.cpp twice. Actually I don't even believe that, I'm sure author ran into errors that might be a pain to figure out, but there's no evidence it's worse than that. But then it says this is the "root cause": ARM64 + Blackwell + CUDA 13.0 = Bleeding Edge ↓ Limited production testing ↓ Edge cases in numerical precision (inference) ↓ Memory management issues (training) Am I to believe GPU inference is really fundamentally broken? I'm not seeing the case made here, just claims. At this point the LLM seems to have gotten confused about whether it's talking about the memory fragmentation issue or the GPU inference issue. But it's hard to believe anything from this point on in the post.

enum 11mo ago

+1 I have H100s to myself, and access to more GPUs than I know what to do with in national clusters. The Spark is much more fun. And I’m more productive. With two of them, you can debug shallow NCCL/MPI problems before hitting a real cluster. I sincerely love Slurm, but nothing like a personal computer.

latchkey 11mo ago
Your complaint sounds more like the way that you have to access the HPC (via slurm), not the compute itself. After having now tried slurm myself, I don't understand the love for it at all. As for debugging, that's where you should be allowed to spin up a small testing cluster on-demand. Why can't you do that with your slurm access?
- enum 11mo ago
  I’m not complaining. The clusters are great. The non-Slurm H100s are great. The Spark is more fun.
  - latchkey 11mo ago
    What makes it more fun?
    - enum 11mo ago
      I think that personal computing is more fun than time-shared computing. :) It's remarkable what can now be done on a whisper-quiet little box. I hope the Strix Halo's will be just as much fun, and they should be, so long as Flash Attention works.
      - latchkey 11mo ago
        Haven't tried to compile it for SH, but did compile it for MI355x and it worked. LONG compile time though ninja sure helped. Fair, thanks for the answer. The bane of my existence... salloc: Granted job allocation 1978 salloc: Waiting for resource configuration

yunohn 11mo ago

Thanks for this bleeding edge content! But please have your LLM post writer be less verbose and repetitive. This is like the stock output from any LLM, where it describes in detail and then summarizes back and forth over multiple useless sections. Please consider a smarter prompt and post-editing…

Tepix 11mo ago
I agree whole-heartedly. Two thirds of the article read like slop.

CompoundEyes 11mo ago

One thing I can’t find anyone mention in reviews - does inference screech to a halt when using large context windows on models? Say if you’re in the 100k range on gpt-oss. I’m not concerned about lightning inference speed overall as I understand the purpose of the spark is to be well rounded / trainer tuner. I just want to know if it becomes unusable vs reasonable slowdown at larger contexts. That’s the thing people are unpleasantly surprised to find about a Mac Studio which has prevented me from going that route.

Tepix 11mo ago

> Blows away any consumer GPU. Nah. Do you have 1st hand experience with Strix Halo? At less than 1600€ for a 128GB configuration it manages >45 tokens/s with gpt-oss 120b. Which is faster than DGX Spark at a fraction of the cost.

storus 11mo ago
Strix Halo has awful token prefill speed. Only suitable for very small contexts.