3 ms·

How many people/hackernews can run a 397b param model at home? Probably like 20-30.

by kylehotchkiss 6mo ago

bitbckt 6mo ago
I'm running it on dual DGX Sparks.
- DoctorOetker 6mo ago
  which exact model, and how many tokens per second for generation?
- manyatoms 6mo ago
  I'm interested in your experiences running dual
r-w 6mo ago
OpenRouter.
- parsimo2010 6mo ago
  If you're running it from OpenRouter, you might as well use Qwen3.6 Plus. You don't need to be picky about a particular model size of 3.6. If you just want the 397b version to save money, just pick a cheaper model like M2.7.
- mistercheese 6mo ago
  Yeah I think there’s benefits to third-party providers being able to run the large models and have stronger guarantees about ZDR and knowing where they are hosted! So Open Weights for even the large models we can’t personally serve on our laptops is still useful.
stavros 6mo ago
It doesn't matter how many can run it now, it's about freedom. Having a large open weights model available allows you to do things you can't do with closed models.
kridsdale3 6mo ago
I can (barely, but sustainably) run Q3.5 397B on my Mac Studio with 256GB unified. It cost $10,000 but that's well within reach for most people who are here, I expect.
- toxik 6mo ago
  $10k is well outside my budget for frivolous computer purchases.
  - bdangubic 6mo ago
    99.97% of HN users are nodding… :)
    - hparadiz 6mo ago
      There are way too many good uses of these models for local that I fully expect a standard workstation 10 years from now to start at 128GB of RAM and have at least a workstation inference device.
      - bdangubic 6mo ago
        or if you believe a lot of HN crowd we are in AI bubble and in 10 years inference will be dirt cheap when all of this crashes and we have all this hardware in data centers and it won't make any sense to run monster workstations at home (I work 128GB M4 but not run inference, just too many electron apps running at the same time...) :)
        hparadiz 6mo ago
        Inference will be dirt cheap for things like coding but you'll want much more compute for architectural planning, personal assistants with persistent real time "thinking / memory", as well as real time multimedia. I could put 10 M4s to work right now and it won't be enough for what I've been cooking.
        bigiain 6mo ago
jubilanti 6mo ago
You can rent a cloud H200 with 140GB VRAM in a server with 256GB system ram for $3-4/hr.
- Tamitami2000 6mo ago
  Can you tell me where? I used runpod before, but they don't have systems like that.
ydj 6mo ago
Running the mxfp4 unsloth quant of qwen3.5-397b-a17b, I get 40 tps prefill, 20tps decode. AMD threadripper pro 9965WX, 256gb ddr5 5600, rtx 4090.
adrian_b 6mo ago
The 397B model can be run at home with the weights stored on an SSD (or on 2 SSDs, for double throughput). Probably too slow for chat, but usable as a coding assistant.
- xienze 6mo ago
  I think you have that backwards. Agentic coding is way more demanding than simple chat. The request/response loops (tool calling) are much tighter and more numerous, and the context is waaaaay bigger in general.
  - fragmede 6mo ago
    In processing power, but chat is interactive. Agentic coding, you come up with a plan and sign off on it, and then just let it go for a while. It's the difference between speed and latency.
Nav_Panel 6mo ago
The point is that open weights turns puts inference on the open market, so if your model is actually good and providers want to serve it, it will drive costs down and inference speeds up. Like Cerebras running Qwen 3 235B Instruct at 1.4k tps for cheaper than Claude Haiku (let that tps number sink in for a second. For reference, Claude Opus runs ~30-40 tps, Claude Haiku at ~60. Several orders of magnitude difference). As a company developing models, it means you can't easily capture the inference margins even though I believe you get a small kickback from the providers. So I understand why they wouldn't want to go open weight, but on the other hand, open weight wins you popularity/sentiment if the model is any good, researchers (both academic and other labs) working on your stuff, etc etc. Local-first usage is only part of the story here. My guess is Qwen 3.5 was successful enough that now they want to start reaping the profits. Unfortunately most of Qwen 3.5's success is because it's heavily (and successfully!) optimized for extremely long-context usage on heavily constrained VRAM (i.e. local) systems, as a result of its DeltaNet attention layers.
jmb99 6mo ago
I’ve mentioned this as an option in other discussions, but if you don’t care that much about tok/sec, 4x Xeon E7-8890 v4s with 1TB of DDR3 in a supermicro X10QBi will run a 397b model for <$2k (probably closer to $1500). Power use is pretty high per token but the entry price cannot be beat. Full (non-quantized, non-distilled) DeepSeek runs at 1-2 tok/sec. A model half the size would probably be a little faster. This is also only with the basic NUMA functionality that was in llama.cpp a few months ago, I know they’ve added more interesting distribution mechanisms recently that I haven’t had a chance to test yet.
blurbleblurble 6mo ago
It only has 17b active params, it's a mixture of experts model. So probably a lot more people than you realize!
wolfhumble 6mo ago
This is like saying that Open source is not important because I don't have a machine to run it on right now. Of course it is important. We don't have any state of the art Language models that are open source, but some are still Open Weight. Better than nothing, and the only way to secure some type of privacy and control over own AI use. It is my goal to run these large models locally eventually; if they all go away that is not even a possibility. . .
Tamitami2000 6mo ago
According to this blog (https://kaitchup.substack.com/p/lessons-from-gguf-evaluations-ternary https://kaitchup.substack.com/p/lessons-from-gguf-evaluation...) the UD_IQ2_M quants are quite strong (rel. error to the base is very low), so it's around 120GB of RAM needed, while the experts can be loaded into VRAM and the rest offloaded into system RAM. It's a high end consumer PC, sure, but not unaffordable. For example, I got an older rig with a RTX 6000 ADA (48GB VRAM), 128 GB RAM and a Threadripper, which runs this quant offloaded at 20 tps

How many people/hackernews can run a 397b param model at home? Probably like 20-30.

bitbckt 6mo ago

I'm running it on dual DGX Sparks.

DoctorOetker 6mo ago
which exact model, and how many tokens per second for generation?
manyatoms 6mo ago
I'm interested in your experiences running dual

r-w 6mo ago

OpenRouter.

parsimo2010 6mo ago
If you're running it from OpenRouter, you might as well use Qwen3.6 Plus. You don't need to be picky about a particular model size of 3.6. If you just want the 397b version to save money, just pick a cheaper model like M2.7.
mistercheese 6mo ago
Yeah I think there’s benefits to third-party providers being able to run the large models and have stronger guarantees about ZDR and knowing where they are hosted! So Open Weights for even the large models we can’t personally serve on our laptops is still useful.

stavros 6mo ago

It doesn't matter how many can run it now, it's about freedom. Having a large open weights model available allows you to do things you can't do with closed models.

kridsdale3 6mo ago

I can (barely, but sustainably) run Q3.5 397B on my Mac Studio with 256GB unified. It cost $10,000 but that's well within reach for most people who are here, I expect.

toxik 6mo ago
$10k is well outside my budget for frivolous computer purchases.
- bdangubic 6mo ago
  99.97% of HN users are nodding… :)
  - hparadiz 6mo ago
    There are way too many good uses of these models for local that I fully expect a standard workstation 10 years from now to start at 128GB of RAM and have at least a workstation inference device.
    - bdangubic 6mo ago
      or if you believe a lot of HN crowd we are in AI bubble and in 10 years inference will be dirt cheap when all of this crashes and we have all this hardware in data centers and it won't make any sense to run monster workstations at home (I work 128GB M4 but not run inference, just too many electron apps running at the same time...) :)
      - hparadiz 6mo ago
        Inference will be dirt cheap for things like coding but you'll want much more compute for architectural planning, personal assistants with persistent real time "thinking / memory", as well as real time multimedia. I could put 10 M4s to work right now and it won't be enough for what I've been cooking.
      - bigiain 6mo ago

jubilanti 6mo ago

You can rent a cloud H200 with 140GB VRAM in a server with 256GB system ram for $3-4/hr.

Tamitami2000 6mo ago
Can you tell me where? I used runpod before, but they don't have systems like that.

ydj 6mo ago

Running the mxfp4 unsloth quant of qwen3.5-397b-a17b, I get 40 tps prefill, 20tps decode. AMD threadripper pro 9965WX, 256gb ddr5 5600, rtx 4090.

adrian_b 6mo ago

The 397B model can be run at home with the weights stored on an SSD (or on 2 SSDs, for double throughput). Probably too slow for chat, but usable as a coding assistant.

xienze 6mo ago
I think you have that backwards. Agentic coding is way more demanding than simple chat. The request/response loops (tool calling) are much tighter and more numerous, and the context is waaaaay bigger in general.
- fragmede 6mo ago
  In processing power, but chat is interactive. Agentic coding, you come up with a plan and sign off on it, and then just let it go for a while. It's the difference between speed and latency.

Nav_Panel 6mo ago

The point is that open weights turns puts inference on the open market, so if your model is actually good and providers want to serve it, it will drive costs down and inference speeds up. Like Cerebras running Qwen 3 235B Instruct at 1.4k tps for cheaper than Claude Haiku (let that tps number sink in for a second. For reference, Claude Opus runs ~30-40 tps, Claude Haiku at ~60. Several orders of magnitude difference). As a company developing models, it means you can't easily capture the inference margins even though I believe you get a small kickback from the providers. So I understand why they wouldn't want to go open weight, but on the other hand, open weight wins you popularity/sentiment if the model is any good, researchers (both academic and other labs) working on your stuff, etc etc. Local-first usage is only part of the story here. My guess is Qwen 3.5 was successful enough that now they want to start reaping the profits. Unfortunately most of Qwen 3.5's success is because it's heavily (and successfully!) optimized for extremely long-context usage on heavily constrained VRAM (i.e. local) systems, as a result of its DeltaNet attention layers.

jmb99 6mo ago

I’ve mentioned this as an option in other discussions, but if you don’t care that much about tok/sec, 4x Xeon E7-8890 v4s with 1TB of DDR3 in a supermicro X10QBi will run a 397b model for <$2k (probably closer to $1500). Power use is pretty high per token but the entry price cannot be beat. Full (non-quantized, non-distilled) DeepSeek runs at 1-2 tok/sec. A model half the size would probably be a little faster. This is also only with the basic NUMA functionality that was in llama.cpp a few months ago, I know they’ve added more interesting distribution mechanisms recently that I haven’t had a chance to test yet.

blurbleblurble 6mo ago

It only has 17b active params, it's a mixture of experts model. So probably a lot more people than you realize!

wolfhumble 6mo ago

This is like saying that Open source is not important because I don't have a machine to run it on right now. Of course it is important. We don't have any state of the art Language models that are open source, but some are still Open Weight. Better than nothing, and the only way to secure some type of privacy and control over own AI use. It is my goal to run these large models locally eventually; if they all go away that is not even a possibility. . .

Tamitami2000 6mo ago

According to this blog (https://kaitchup.substack.com/p/lessons-from-gguf-evaluations-ternary https://kaitchup.substack.com/p/lessons-from-gguf-evaluation...) the UD_IQ2_M quants are quite strong (rel. error to the base is very low), so it's around 120GB of RAM needed, while the experts can be loaded into VRAM and the rest offloaded into system RAM. It's a high end consumer PC, sure, but not unaffordable. For example, I got an older rig with a RTX 6000 ADA (48GB VRAM), 128 GB RAM and a Threadripper, which runs this quant offloaded at 20 tps