3 ms·

This article keeps getting posted but it runs a thinking model at 3-4 tokens/s. You might as well take a vacation if you ask it a question. It’s a gimmick and

This article keeps getting posted but it runs a thinking model at 3-4 tokens/s. You might as well take a vacation if you ask it a question. It’s a gimmick and not a real solution.

hnuser123456 2y ago
If you value local compute and don't need massive speed, that's still twice as fast as most people can type.
- aurareturn 2y ago
  Human typing speed is magnitudes slower than our eyes scanning for the correct answer. ChatGPT o3 mini high thinks at about 140 tokens/s by my estimation and I sometimes wish it can return answers quicker. Getting a simple prompt answer would take 2-3 minutes using the AMD system and forget about longer context.
- evilduck 2y ago
  Reasoning models spend a whole bunch of time reasoning before returning an answer. I was toying with QWQ 32B last night and ran into one question I gave it where it spent 18 minutes at 13tok/s in the <think> phase before returning a final answer. I value local compute but reasoning models aren’t terribly feasible at this speed since you don’t really need to see the first 90% of their thinking output.
walrus01 2y ago
It's meant to be a test/development setup for people to prepare the software environment and tooling for running the same on more expensive hardware. Not to be fast.
- aurareturn 2y ago
  I remember people trying to run the game Crysis using CPU rendering. They got it to run and move around. People did it for fun and the "cool" factor. But no one actually played the game that way. It's the same thing here. CPUs can run it but only as a gimmick.
  - refulgentis 2y ago
    > It's the same thing here. CPUs can run it but only as a gimmick. No, that's not true. I work on local inference code via llama.cpp, on both GPU and CPU on every platform, and the bottleneck is much more ram / bandwidth than compute. Crappy Pixel Fold 2022 mid-range Android CPU gets you roughly same speed as 2024 Apple iPhone GPU, with Metal acceleration that dozens of very smart people hack on. Additionally, and perhaps more importantly, Arc is a GPU, not a CPU. The headline of the thing you're commenting on, the very first thing you see when you open it, is "Run llama.cpp Portable Zip on Intel GPU" Additionally, the HN headline includes "1 or 2 Arc 7700"
    - aurareturn 2y ago
      It's both compute and bandwidth constrained - just like trying to run Crysis on CPU rendering. A770 has 16GB of RAM. You're shuffling data to the GPU at a rate of 64GB/s, which is magnitudes slower than the internal VRAM of the GPU. Hence, this setup is memory bandwidth constrained. However, once you want to use it to do anything useful like a longer context size, the CPU compute will be a huge bottleneck for time-to-first-token as well as tokens/s. Trying to run a model this large, and a thinking one at that, on CPU RAM is a gimmick.
      - refulgentis 2y ago
        Okay, let's stipulate LLMs are compute and bandwidth sensitive (of course!)... #1, should highlight it up front this time: We are talking about _G_PUs :) #2 You can't get a single consumer GPU that has enough memory to load a 670B parameter model, there's some magic going on here. It's notable and distinct. This is probably due to FlashMoE, given it's prominence in the link. TL;Dr: 1) these are Intel _G_PUs, and 2) it is a remarkable distinct achievement to be loading a 670B parameter model on only one to two cards
        aurareturn 2y ago
        1) This system mostly uses normal DDR RAM, not GPU VRAM. 2) M3 Ultra can load Deepseek R1 671B Q4. Using a very large LLM across the CPU and GPU is not new. It's been done since the beginning of local LLMs.
miklosz 2y ago
Exactly! I run it on my old T7910 Dell workstation (2x 2697A V4, 640GB RAM) that I build for way less than a $1k. But so what, it's about ~2 tokens / s. Just like you said, it's cool that it's run at all, but that's it.

This article keeps getting posted but it runs a thinking model at 3-4 tokens/s. You might as well take a vacation if you ask it a question. It’s a gimmick and not a real solution.

hnuser123456 2y ago

If you value local compute and don't need massive speed, that's still twice as fast as most people can type.

aurareturn 2y ago
Human typing speed is magnitudes slower than our eyes scanning for the correct answer. ChatGPT o3 mini high thinks at about 140 tokens/s by my estimation and I sometimes wish it can return answers quicker. Getting a simple prompt answer would take 2-3 minutes using the AMD system and forget about longer context.
evilduck 2y ago
Reasoning models spend a whole bunch of time reasoning before returning an answer. I was toying with QWQ 32B last night and ran into one question I gave it where it spent 18 minutes at 13tok/s in the <think> phase before returning a final answer. I value local compute but reasoning models aren’t terribly feasible at this speed since you don’t really need to see the first 90% of their thinking output.

walrus01 2y ago

It's meant to be a test/development setup for people to prepare the software environment and tooling for running the same on more expensive hardware. Not to be fast.

aurareturn 2y ago
I remember people trying to run the game Crysis using CPU rendering. They got it to run and move around. People did it for fun and the "cool" factor. But no one actually played the game that way. It's the same thing here. CPUs can run it but only as a gimmick.
- refulgentis 2y ago
  > It's the same thing here. CPUs can run it but only as a gimmick. No, that's not true. I work on local inference code via llama.cpp, on both GPU and CPU on every platform, and the bottleneck is much more ram / bandwidth than compute. Crappy Pixel Fold 2022 mid-range Android CPU gets you roughly same speed as 2024 Apple iPhone GPU, with Metal acceleration that dozens of very smart people hack on. Additionally, and perhaps more importantly, Arc is a GPU, not a CPU. The headline of the thing you're commenting on, the very first thing you see when you open it, is "Run llama.cpp Portable Zip on Intel GPU" Additionally, the HN headline includes "1 or 2 Arc 7700"
  - aurareturn 2y ago
    It's both compute and bandwidth constrained - just like trying to run Crysis on CPU rendering. A770 has 16GB of RAM. You're shuffling data to the GPU at a rate of 64GB/s, which is magnitudes slower than the internal VRAM of the GPU. Hence, this setup is memory bandwidth constrained. However, once you want to use it to do anything useful like a longer context size, the CPU compute will be a huge bottleneck for time-to-first-token as well as tokens/s. Trying to run a model this large, and a thinking one at that, on CPU RAM is a gimmick.
    - refulgentis 2y ago
      Okay, let's stipulate LLMs are compute and bandwidth sensitive (of course!)... #1, should highlight it up front this time: We are talking about _G_PUs :) #2 You can't get a single consumer GPU that has enough memory to load a 670B parameter model, there's some magic going on here. It's notable and distinct. This is probably due to FlashMoE, given it's prominence in the link. TL;Dr: 1) these are Intel _G_PUs, and 2) it is a remarkable distinct achievement to be loading a 670B parameter model on only one to two cards
      - aurareturn 2y ago
        1) This system mostly uses normal DDR RAM, not GPU VRAM. 2) M3 Ultra can load Deepseek R1 671B Q4. Using a very large LLM across the CPU and GPU is not new. It's been done since the beginning of local LLMs.

miklosz 2y ago

Exactly! I run it on my old T7910 Dell workstation (2x 2697A V4, 640GB RAM) that I build for way less than a $1k. But so what, it's about ~2 tokens / s. Just like you said, it's cool that it's run at all, but that's it.