7 ms·

I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will

I run Q4_K_XL. All it takes to run to get about 6tk/sec is 512gb of ram and 2 3090 GPUs with llama.cpp -cmoe. I also have crappy DDR4, 2400mhz, 3200mhz will bring that speed up to about 9tk/sec. I also have ok 32core epyc CPU, a better 64core would bring it up to about 11tk/sec. I did a budget build before the crazy hardware cost and I regret it everyday. Nevertheless, it's fantastic being able to run this model at home. It's great for planning, one shot prompting once you have a plan or all the context you need. This entire hardware cost $2400 when it was built. If you're willing to be resourceful, you can find ways to run these models at home. I often get the silly question of why, and suggestions about how much I can save using cloud API, but the Fable drama has opened up eyes on why it's good for us to be independent. Thanks team unsloth, Q4_K_XL is solid, if you are going to grab a quant, make sure to get the K_XL variant if it can fit.

redox99 4mo ago
That's crazy good for $2400.
edg5000 4mo ago
Very cool. So it's not just about GPU VRAM which I incorrectly thought. I though you'd need 512 GB GPU VRAM. I don't think it cost only 2400; 512GB ram would be more expensive though back in the day. But not mortgage-grade 200.000 which I estimated myself (which assumed running in 100% VRAM; overkill for a single user probably).
- segmondy 4mo ago
  you can use system ram with a system like llama.cpp which offloads to system ram. token generation is a function of system bandwidth, the faster the bandwidth the better. so I'm on 8 channel 2400mhz. if I had a 12 ddr channel, I would get 1.5x the speed at 2400mhz. of course ddr5 is much faster, so a 12 ddr at 4800mhz will provide 3x the speed for token generation or roughly 18tk/sec. prompt processing is all about compute, so the more cpu cores you have the faster it can do PP.
- nijave 4mo ago
  Well, it's about GPU VRAM if you want something competitive with cloud-hosted offerings at the performance levels showing in benchmarks. This is a heavy quant with quality degradation and significantly lower performance. Cloud offerings are 80-200tk/sec versus single digit tk/sec. That said, I'm also surprised it runs at all locally. I do think it'd be painfully slow for anything interactive so you're relying on another model for a comprehensive design or you're hoping a one-shot with somewhat degraded quality turns out correctly.
  - edg5000 4mo ago
    I see. So not quite usable apart for specific use cases. Maybe in a few years we'll see new hardware players and better prices.
    - nijave 4mo ago
      I think we'll see - better hardware - more efficient model runtime algorithms/code - smarter/more efficient models (same capability with less parameters) So ideally these will all come together and help.
nextaccountic 4mo ago
How can you combine CPU cores and multiple GPU? Are you running some layers in cpu, others in gpu #1, and others in gpu #2? What about the bandwidth and latency between them? Or maybe the model itself only runs at gpus, and the cpu memory only store the weights for experts not corrently activated? If so, then what's the 32 or 64 cpu cores for? I'm a big fan of fully utilizing one's hardware and it's kinda sad that it's not the norm to run things on either gpu, cpu or both, dynamically choosing at runtime, for everyday software
- segmondy 4mo ago
  checkout llama.cpp, the entire point of the project is for us mere mortals and GPU poor.
- nodja 4mo ago
  Pipeline parallelism. Instead of splitting layers by row/column. You split at the layer edges. So instead of having this huge bottleneck of bandwidth you only need to transfer about 4KB per token when changing devices on a model like Qwen 3 30BA3.
- xrd 4mo ago
  This is a good place to start reading about dual gpus. https://github.com/noonghunna/club-3090/blob/master/docs/DUAL_CARD.md https://github.com/noonghunna/club-3090/blob/master/docs/DUA...
  - nextaccountic 4mo ago
    But in this case he used a cpu too
fsuts 4mo ago
6 tokens per second? Can you put up with that? As seems very slow. I aim for 40t/s on a laptop and choose models that deliver that speed over larger slower ones
- segmondy 4mo ago
  I have been putting up with it forever. We are spoiled by MixtureOfExperts. Folks were delighted to run llama3-70B at such speed. We were happy with 15-20tk/sec with 8b models, and if you could run llama3-405B at 1tk/sec you were a god. To each their own. I can live with 6 high quality tokens. If I could get a Fable equivalent model, I'll gladly take 2tk/sec if that's what it took to run it locally.
  - froh 4mo ago
    do you use caveman or similar?
  - manmal 4mo ago
    But what is it doing for you that you couldn’t do yourself at that speed? I‘m really curious and on the fence of partly going local.
    - all2 4mo ago
      Is think you would use it more like email and less like text messages, so the domain of communication shifts drastically. The other part is, you don't have to run just that model, you can offload a lot of chores to smaller models.
    - Mashimo 4mo ago
      Run one task, while you do another? Or while you sleep / eat / rave?
      - manmal 4mo ago
        While my colleagues are running 6 parallel agents at 50-100t/s each, with an actual SOTA model? Don’t you think I‘d get fired after a few weeks of that?
zozbot234 4mo ago
AIUI the llama.cpp implementation for this model is still quite half-baked due to missing the support for DSA sparse attention mechanism. This leads to running the model with a different mechanism that it has not been trained for, which has been shown to lead to lower quality and performance. Anyway, I think GLM 5.2 in many ways is not as interesting as DeepSeek V4 series, which uses an even more advanced attention mechanism and can save a lot of memory capacity for KV cache, especially at larger contexts. Which in turn opens up wide batching especially on consumer platforms. GLM doesn't have that, in some ways it feels broadly similar to Kimi 2.6 wrt. the underlying performance architecture. Both are a bit too heavy to run reasonably at full quality on ordinary hardware.
- trollbridge 4mo ago
  Particularly DeepSeek 4.1, which they appear to be A/B testing on the API and which also seems available on the free chat interface. It also has an input image modality, which is a game changer. The cheap Sinofrontier models have generally been lacking in this regard. Basically, Chinese competition is fierce - DeepSeek set the pricing tier, and the question for each lab now is how to justify charging a little more. MiMo-2.5-Pro has gone with UltraSoeed, pumping out 1000t/s for a 3X price hike. GLM has gone with 5.2, hitting Opus levels of reasoning at a fraction of the cost. DeepSeek will probably keep their pricing model and just keep getting better and better. Qwen-3.7 is the dark horse. Some rumours are Alibaba is simply making these models because they need them internally. The real question is why this level of innovation and competition isn’t happening in America or Europe. In particular I see no reason Europe doesn’t have a lab competing on these terms.
  - SalariedSlave 4mo ago
    Competing and innovating in the fast moving SOTA end of the llm space requires a ruthless disregard for copyright, IP, bureaucracies, formalities, risk assurances and other slowdowns. It requires a risk tolerant, quick and large flowing investment of capital. It requires a scoped focus that is pragmatic and sharp about key concerns, and efficiently dismissive of meaningless details. Europe can provide none of this. They will never be at the frontier of AI tech, for the same reason they were never at the frontier of any tech. I say this as a software engineer from Europe.
dxuh 4mo ago
"All it takes to run" might be fair if you paid $2400, but right now the total price is way closer to $10k (almost 5k for the RAM and 2k each for the GPUs). Today that is a lot of expensive hardware.
- segmondy 4mo ago
  512gb 2400mhz ddr4 ram = $1600 not $5000. https://www.ebay.com/itm/188284985172 https://www.ebay.com/itm/188284985172 You can get creative and source 2-3 2080ti 22gb from China for about $250 a piece. You can either be resourceful and find a way or find a whole bunch of excuses.
  - officialchicken 4mo ago
    > You can either be resourceful and find a way or find a whole bunch of excuses. How about addressing this false dichotomy with the likelihood that someone who is new or interested in a tech isn't willing to drop thousands of dollars on used hardware for a whim or learning exercise.
- deleted 4mo ago
  [deleted]
- Astropolak 4mo ago
  Agree'd here. I am seeing a pattern in this thread - Everyone here seems to justify paying substantially more money to self-host hardware thinking they are saving in the medium-long term. Frankly from the perspective of the company I'm with - lost opportunity cost is more important than trying to future proof on a hunch that AI vendors are going to pull the board out under you. There's alot of competition in this space and we're seeing these frontier models continue to get better for the foreseeable future. As such, you're dumping alot of money upfront on hardware which is not guaranteed to be relevant in a few years time. The benefits to me seem pretty speculative, minus the point of security.
discordance 4mo ago
Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity. Unless you really want privacy or the fuzzy feeling of owning your own, it’s cheaper, more convenient and has much faster tok/s if you pay a hyper scaler. That said, I do like the direction we are heading and look forward to seeing what host your own hardware we get in 2 years.
- SXX 4mo ago
  I guess you missed recent news. Problem is that cloud LLM might just sliently sabotage your work by downgrading output model with no notice. Or cloud LLM might just refuse to sell to you because it dont like your passport.
  - swiftcoder 4mo ago
    This is not really a problem for the open-weight models, you can always give your money to an inference provider in a different jurisdiction
  - yorwba 4mo ago
    So you're buying expensive hardware as insurance for the case that your cloud provider turns against you and you have to switch to another of the twenty offering the same model https://openrouter.ai/z-ai/glm-5.2 https://openrouter.ai/z-ai/glm-5.2 or in the worst case buy the same hardware later? How does that make sense?
    - deleted 4mo ago
      [deleted]
    - drptech 4mo ago
      [dead]
    - brookst 4mo ago
      It’s rationalization for what people want to do anyway. Like buying a new car today and taking on gas, parking, etc, expenses in case the bus route you’re using goes away at some point in the future. It’s not an economic decision, it’s a desire to have the new car dressed up in what-ifs.
radku 4mo ago
I have pretty much almost this exact setup with 2x3090s and with slightly faster DDR4 512GB and 64 core Epyc! [0] I've been enjoying it a lot. Can't wait to give this model a try. Apart of running local models I use this rig as my main remote development platform. All Claude Code sessions are running there in tmux now. And my fingers can't be happier not having to deal with constantly hot laptop. Not to mention that Claude Code is such a battery hog. [0] https://medium.com/@rathko/i-built-an-epyc-64-core-512gb-ram-48gb-vram-llm-server-in-my-apartment-60a8b8a2d355 https://medium.com/@rathko/i-built-an-epyc-64-core-512gb-ram...
effisfor 4mo ago
I applaud all you tinkerers for pushing on the state of the home-brewed art here. Like crypto, AI is drowned out by hucksters, very few people talk about developing resilience. Or the researchers who will push on open source models in efforts to cram them onto an electric toothbrush or tamagotchi. Bravo to you all.
pizza234 4mo ago
LOL, sure this works if one has a time machine or a LOT of money to burn. 32 CPU Epyc (Epyc is required for faster memory access) + 32 GB VRAM + 512 GB RAM is stupid expensive nowadays, and in best case, it will just downgrade to "very" expensive at some point in the future. This makes sense only if 1. one is paranoid about privacy or 2. they have money to smoke or 3. they need to workaround cloud model restrictions, AND they have to do it routinely (because if not, a oneshot cloud bare metal setup is way cheaper, faster, and allows more powerful models, due to VRAM offering). I did spend stupid money as well and yet, the system is 2x slower than cloud providers for comparable performance on vision tasks (I still have to test coding). Oh, and it's hot as hell.
ikari_pl 4mo ago
I can work out max 90GB to the agents. Advise. :)
SwellJoe 4mo ago
6 tokens per second is not fit for interactive use. I find Gemma 4 (QAT 4-bit, MTP) to be tolerable at about 30 tokens per second on my old GPUs. Anything slower than 15 is annoying. I tried DS4 on my Strix halo (1-bit quantization of DeepSeek V4 Flash, the biggest model that can realistically run on 128GB, right now), and it tops out at something like 10 or 11 with a long time to first response, and that's quite painful to use. I'd definitely rather spend money to use the big models on cloud infrastructure. And, the several thousand dollars it costs to run these things unusably slowly buys a lot of tokens on the cheap Chinese models.

redox99 4mo ago

That's crazy good for $2400.

edg5000 4mo ago

Very cool. So it's not just about GPU VRAM which I incorrectly thought. I though you'd need 512 GB GPU VRAM. I don't think it cost only 2400; 512GB ram would be more expensive though back in the day. But not mortgage-grade 200.000 which I estimated myself (which assumed running in 100% VRAM; overkill for a single user probably).

segmondy 4mo ago
you can use system ram with a system like llama.cpp which offloads to system ram. token generation is a function of system bandwidth, the faster the bandwidth the better. so I'm on 8 channel 2400mhz. if I had a 12 ddr channel, I would get 1.5x the speed at 2400mhz. of course ddr5 is much faster, so a 12 ddr at 4800mhz will provide 3x the speed for token generation or roughly 18tk/sec. prompt processing is all about compute, so the more cpu cores you have the faster it can do PP.
nijave 4mo ago
Well, it's about GPU VRAM if you want something competitive with cloud-hosted offerings at the performance levels showing in benchmarks. This is a heavy quant with quality degradation and significantly lower performance. Cloud offerings are 80-200tk/sec versus single digit tk/sec. That said, I'm also surprised it runs at all locally. I do think it'd be painfully slow for anything interactive so you're relying on another model for a comprehensive design or you're hoping a one-shot with somewhat degraded quality turns out correctly.
- edg5000 4mo ago
  I see. So not quite usable apart for specific use cases. Maybe in a few years we'll see new hardware players and better prices.
  - nijave 4mo ago
    I think we'll see - better hardware - more efficient model runtime algorithms/code - smarter/more efficient models (same capability with less parameters) So ideally these will all come together and help.

nextaccountic 4mo ago

How can you combine CPU cores and multiple GPU? Are you running some layers in cpu, others in gpu #1, and others in gpu #2? What about the bandwidth and latency between them? Or maybe the model itself only runs at gpus, and the cpu memory only store the weights for experts not corrently activated? If so, then what's the 32 or 64 cpu cores for? I'm a big fan of fully utilizing one's hardware and it's kinda sad that it's not the norm to run things on either gpu, cpu or both, dynamically choosing at runtime, for everyday software

segmondy 4mo ago
checkout llama.cpp, the entire point of the project is for us mere mortals and GPU poor.
nodja 4mo ago
Pipeline parallelism. Instead of splitting layers by row/column. You split at the layer edges. So instead of having this huge bottleneck of bandwidth you only need to transfer about 4KB per token when changing devices on a model like Qwen 3 30BA3.
xrd 4mo ago
This is a good place to start reading about dual gpus. https://github.com/noonghunna/club-3090/blob/master/docs/DUAL_CARD.md https://github.com/noonghunna/club-3090/blob/master/docs/DUA...
- nextaccountic 4mo ago
  But in this case he used a cpu too

fsuts 4mo ago

6 tokens per second? Can you put up with that? As seems very slow. I aim for 40t/s on a laptop and choose models that deliver that speed over larger slower ones

segmondy 4mo ago
I have been putting up with it forever. We are spoiled by MixtureOfExperts. Folks were delighted to run llama3-70B at such speed. We were happy with 15-20tk/sec with 8b models, and if you could run llama3-405B at 1tk/sec you were a god. To each their own. I can live with 6 high quality tokens. If I could get a Fable equivalent model, I'll gladly take 2tk/sec if that's what it took to run it locally.
- froh 4mo ago
  do you use caveman or similar?
- manmal 4mo ago
  But what is it doing for you that you couldn’t do yourself at that speed? I‘m really curious and on the fence of partly going local.
  - all2 4mo ago
    Is think you would use it more like email and less like text messages, so the domain of communication shifts drastically. The other part is, you don't have to run just that model, you can offload a lot of chores to smaller models.
  - Mashimo 4mo ago
    Run one task, while you do another? Or while you sleep / eat / rave?
    - manmal 4mo ago
      While my colleagues are running 6 parallel agents at 50-100t/s each, with an actual SOTA model? Don’t you think I‘d get fired after a few weeks of that?

zozbot234 4mo ago

AIUI the llama.cpp implementation for this model is still quite half-baked due to missing the support for DSA sparse attention mechanism. This leads to running the model with a different mechanism that it has not been trained for, which has been shown to lead to lower quality and performance. Anyway, I think GLM 5.2 in many ways is not as interesting as DeepSeek V4 series, which uses an even more advanced attention mechanism and can save a lot of memory capacity for KV cache, especially at larger contexts. Which in turn opens up wide batching especially on consumer platforms. GLM doesn't have that, in some ways it feels broadly similar to Kimi 2.6 wrt. the underlying performance architecture. Both are a bit too heavy to run reasonably at full quality on ordinary hardware.

trollbridge 4mo ago
Particularly DeepSeek 4.1, which they appear to be A/B testing on the API and which also seems available on the free chat interface. It also has an input image modality, which is a game changer. The cheap Sinofrontier models have generally been lacking in this regard. Basically, Chinese competition is fierce - DeepSeek set the pricing tier, and the question for each lab now is how to justify charging a little more. MiMo-2.5-Pro has gone with UltraSoeed, pumping out 1000t/s for a 3X price hike. GLM has gone with 5.2, hitting Opus levels of reasoning at a fraction of the cost. DeepSeek will probably keep their pricing model and just keep getting better and better. Qwen-3.7 is the dark horse. Some rumours are Alibaba is simply making these models because they need them internally. The real question is why this level of innovation and competition isn’t happening in America or Europe. In particular I see no reason Europe doesn’t have a lab competing on these terms.
- SalariedSlave 4mo ago
  Competing and innovating in the fast moving SOTA end of the llm space requires a ruthless disregard for copyright, IP, bureaucracies, formalities, risk assurances and other slowdowns. It requires a risk tolerant, quick and large flowing investment of capital. It requires a scoped focus that is pragmatic and sharp about key concerns, and efficiently dismissive of meaningless details. Europe can provide none of this. They will never be at the frontier of AI tech, for the same reason they were never at the frontier of any tech. I say this as a software engineer from Europe.

dxuh 4mo ago

"All it takes to run" might be fair if you paid $2400, but right now the total price is way closer to $10k (almost 5k for the RAM and 2k each for the GPUs). Today that is a lot of expensive hardware.

segmondy 4mo ago
512gb 2400mhz ddr4 ram = $1600 not $5000. https://www.ebay.com/itm/188284985172 https://www.ebay.com/itm/188284985172 You can get creative and source 2-3 2080ti 22gb from China for about $250 a piece. You can either be resourceful and find a way or find a whole bunch of excuses.
- officialchicken 4mo ago
  > You can either be resourceful and find a way or find a whole bunch of excuses. How about addressing this false dichotomy with the likelihood that someone who is new or interested in a tech isn't willing to drop thousands of dollars on used hardware for a whim or learning exercise.
deleted 4mo ago
[deleted]
Astropolak 4mo ago
Agree'd here. I am seeing a pattern in this thread - Everyone here seems to justify paying substantially more money to self-host hardware thinking they are saving in the medium-long term. Frankly from the perspective of the company I'm with - lost opportunity cost is more important than trying to future proof on a hunch that AI vendors are going to pull the board out under you. There's alot of competition in this space and we're seeing these frontier models continue to get better for the foreseeable future. As such, you're dumping alot of money upfront on hardware which is not guaranteed to be relevant in a few years time. The benefits to me seem pretty speculative, minus the point of security.

discordance 4mo ago

Running that full load is at least 600 W, so in a day ~14 kWh. At $0.2 a kWH, that would be $2.80/day or $1k a year of op-ex in electricity. Unless you really want privacy or the fuzzy feeling of owning your own, it’s cheaper, more convenient and has much faster tok/s if you pay a hyper scaler. That said, I do like the direction we are heading and look forward to seeing what host your own hardware we get in 2 years.

SXX 4mo ago
I guess you missed recent news. Problem is that cloud LLM might just sliently sabotage your work by downgrading output model with no notice. Or cloud LLM might just refuse to sell to you because it dont like your passport.
- swiftcoder 4mo ago
  This is not really a problem for the open-weight models, you can always give your money to an inference provider in a different jurisdiction
- yorwba 4mo ago
  So you're buying expensive hardware as insurance for the case that your cloud provider turns against you and you have to switch to another of the twenty offering the same model https://openrouter.ai/z-ai/glm-5.2 https://openrouter.ai/z-ai/glm-5.2 or in the worst case buy the same hardware later? How does that make sense?
  - deleted 4mo ago
    [deleted]
  - drptech 4mo ago
    [dead]
  - brookst 4mo ago
    It’s rationalization for what people want to do anyway. Like buying a new car today and taking on gas, parking, etc, expenses in case the bus route you’re using goes away at some point in the future. It’s not an economic decision, it’s a desire to have the new car dressed up in what-ifs.

radku 4mo ago

I have pretty much almost this exact setup with 2x3090s and with slightly faster DDR4 512GB and 64 core Epyc! [0] I've been enjoying it a lot. Can't wait to give this model a try. Apart of running local models I use this rig as my main remote development platform. All Claude Code sessions are running there in tmux now. And my fingers can't be happier not having to deal with constantly hot laptop. Not to mention that Claude Code is such a battery hog. [0] https://medium.com/@rathko/i-built-an-epyc-64-core-512gb-ram-48gb-vram-llm-server-in-my-apartment-60a8b8a2d355 https://medium.com/@rathko/i-built-an-epyc-64-core-512gb-ram...

effisfor 4mo ago

I applaud all you tinkerers for pushing on the state of the home-brewed art here. Like crypto, AI is drowned out by hucksters, very few people talk about developing resilience. Or the researchers who will push on open source models in efforts to cram them onto an electric toothbrush or tamagotchi. Bravo to you all.

pizza234 4mo ago

LOL, sure this works if one has a time machine or a LOT of money to burn. 32 CPU Epyc (Epyc is required for faster memory access) + 32 GB VRAM + 512 GB RAM is stupid expensive nowadays, and in best case, it will just downgrade to "very" expensive at some point in the future. This makes sense only if 1. one is paranoid about privacy or 2. they have money to smoke or 3. they need to workaround cloud model restrictions, AND they have to do it routinely (because if not, a oneshot cloud bare metal setup is way cheaper, faster, and allows more powerful models, due to VRAM offering). I did spend stupid money as well and yet, the system is 2x slower than cloud providers for comparable performance on vision tasks (I still have to test coding). Oh, and it's hot as hell.

ikari_pl 4mo ago

I can work out max 90GB to the agents. Advise. :)

SwellJoe 4mo ago

6 tokens per second is not fit for interactive use. I find Gemma 4 (QAT 4-bit, MTP) to be tolerable at about 30 tokens per second on my old GPUs. Anything slower than 15 is annoying. I tried DS4 on my Strix halo (1-bit quantization of DeepSeek V4 Flash, the biggest model that can realistically run on 128GB, right now), and it tops out at something like 10 or 11 with a long time to first response, and that's quite painful to use. I'd definitely rather spend money to use the big models on cloud infrastructure. And, the several thousand dollars it costs to run these things unusably slowly buys a lot of tokens on the cheap Chinese models.