4 ms·

GLM 5.3 is probably the sweet spot open weights model if you want to go beyond deepseek flash or the new glm flash. I used it with pi and had a fairly good time

GLM 5.3 is probably the sweet spot open weights model if you want to go beyond deepseek flash or the new glm flash. I used it with pi and had a fairly good time, especially since it’s less touchy about cyber and whatnot than the US guys. It’s slightly behind Kimi in ability but it’s a lot easier to run it, I’d expect prices (and speed!) from third parties to be noticeably better. Assuming you’re willing to drop a fat stack of cash on the upcoming Mac m5 ultra with 512 gb unified memory, you can even run it locally, quantized to 4 bit. Whether it’s even slightly reasonable, well, my wife would probably skin me alive but maybe yours is more understanding.

walrus01 29d ago
One could also run it locally on a used dual xeon (or amd-equivalent) server with 512GB RAM, albeit slower, if you have a useful workflow for it that's like "take this day's efforts and run it through various analysis agents", combined with giving it one-shot tasks/modules to build overnight. You would want a place like a garage or basement to put the server because it'll be loud.
- dataplumb3r 29d ago
  You'd also likely spend far more in electricity than the API cost of processing the prompt(s)
  - walrus01 29d ago
    yes, though for some uses, not sending data anywhere to third parties has its own value which is harder to measure.
- peri-cl 29d ago
  > "dual xeon" Does inference make full use of the memory bandwidth in a NUMA system?
  - walrus01 29d ago
    Memory throughput performance is better than many equivalently priced (talking about the cheaper end of the used/refurb market here) single socket systems because each CPU is talking to its own banks of RAM. Using an example of llama-server on a 256GB (each CPU is attached to 128GB) system running qwen 3.8-flash-next Q8, which takes up about 188GB with the model and context, it's not that the inference engine 'knows' about NUMA, but if you use --load-mode mlock to load the whole thing into memory at once at the time llama-server launches, the kernel will distribute it relatively evenly across the memory connected to both CPUs.
0xdeadbeefbabe 29d ago
Well if you did get the m5 ultra could you obliterate the guardrails and then your wife can ask it pertinent but unsafe questions about how to punish you. Seems doable.
lnenad 29d ago
I have just built an Epyc with 512gb DDR4 3200 RAM for a "reasonable" price and I'm hoping to have a setup with GLM as the architect and Qwen 27b/Next Flash as the implementer. This is 1/5 of the price of the Mac, but also probably 1/5 of the speed lol.
- jchw 29d ago
  Honestly I suspect neither of them will be performing terribly well but with DDR4 3200 RAM I wonder if you'll be counting tokens per second or seconds per token. I mean, you do at least get a lot of memory channels at least, compared to consumer PCs. I am curious to hear what performance you get, I feel there is not enough information out there on what different setups manage to eek out.
  - Philpax 29d ago
    The fastest I was able to get my Threadripper 3960X + 2x 3090s + 256GB DDR4-3200 to run a 2-bit quant of GLM-5.2 was 8 TPS. I would expect to be in seconds-per-token territory for a pure-CPU 4-bit quant.
    - jchw 29d ago
      One thing I'd like to try is MoE offloading: I have 2x32 GiB of VRAM and 128 GiB of DDR5 running at 4800 MT/s (only 2 channels though). I've seen people post difficult to believe MoE offloading results albeit a decently long time ago with older models. Maybe there is a quant that would fit with MoE offloading? That said, I am guessing my problem is not enough RAM - but this poor consumer platform struggles to do memory training with 128 GiB as it is. Now I surely regret not having gotten Threadripper and 256 GiB of RAM in the before-times.
      - Philpax 29d ago
        My measurement was with MoE offloading, but there's only so much you can keep on-GPU with a 200GB quant and 48GB of VRAM. It's hard to overcome the CPU/RAM bottleneck. For what it's worth, all of my hardware was used; I think, all-in, I'm probably at around 3k-4k USD? Not cheap, but also not the worst for something relatively versatile.
Der_Einzige 29d ago
[flagged]
- CookieCrisp 29d ago
  You are allowed to have an opinion on her 20k bag addiction
- HighGoldstein 29d ago
  Not every tech worker is making top-tier US salaries. For some (I suspect not few) people on HN that $20,000 Mac is almost a year's salary.
  - tough 29d ago
    and even if you were making such a salary, the quesiton of if the investment on hardware to run llm's locally is still a big if, its OK if you buy the HW cause you'll use it and you get the extra capability as a nice extra, but doesnt make sense to spend so much when you could just get 200$ subs with almost infinite SOTA tokens a month etc (if you dont need the local/privacy aspects of it)
sickcodebruh 29d ago
When we consider: * LLM usage is new for the world * Models are evolving quickly with high worldwide competition * Hardware is evolving despite RAM shortages Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what you’d spend otherwise on serverless or dedicated GPU spend…
- rkangel 29d ago
  It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years. However there are other reasons (e.g. privacy) that might make it worth running locally for some people.
  - WASDx 29d ago
    I think the privacy argument that keeps coming up is overrepresented. Certainly ZDR is enough for an absolute majority of use cases? I see so much talk about local inference but I doubt most of it has privacy as a valid argument (not arguing it doesn't exist). It's fun to do things locally though. I've tried it as well but cloud is just faster and cheaper.
    - rkangel 29d ago
      These companies have displayed zero respect for everyone's intellectual property getting these models trained. I think not giving them your complete trust is reasonable! I'm not saying zero trust, and ZDR is fine for most things but I understand the people who don't want to stream their whole codebase out token by token.
      - WASDx 29d ago
        Then use other providers hosting open models. Companies and individuals already put their whole code base on the cloud. I'm genuinely interested in privacy-oriented use cases where ZDR is not enough.
crossroadsguy 29d ago
Its reasoning leaves a lot to be desired :( Though I appreciate how good it is at "solid" grunt work and at that price (in fact I am paying the grandfathered subscription price; mostly). I am planning to let go for my Claude AI subscription which I now use only for "planning" and maybe use that via Open Router as PAYG (at to try how it ends up). But god glm is bad at "talking" and "responding" anything prose. Not only quality but it's almost impossible to tune it and make it let go of its habits and biases and enthusiasms which often result in too many too and fro. So I sometimes wonder at what point that starts becoming the cost and mental hassle. Maybe it's not there for me yet.
colingauvin 29d ago
In terms of pure tokens per dollar, absolutely not worth it. That said, when I bought my pair of Sparks, the best model I could run on it was GPT OSS 120B. That has an AA score of 24. Today, the best model I can run on them is GLM 5.3 Flash at Q4, AA score 57. Just still out on GLM 5.3 mixed quant. So from that perspective, they are many times better value than when I bought them, and will likely continue to increase in value.
- lostmsu 28d ago
  > GLM 5.3 Flash at Q4, AA score 57 That AA score is for the original model only
  - nolist_policy 28d ago
    Then take DeepSeek V4 flash with AA score 52. Runs unquantized on 2x DGX spark with 1M context.
    - rubin55 28d ago
      Or Qwen 3.8 27B, AA score 52 (which is utterly insane given the size of this model); I have been testing Qwen 3.8 27B since a week now, as an intensive GLM-5.2 and Opus 5 user - I can say that I just can't believe my eyes i.r.t. to how good this model is.
    - lostmsu 28d ago
      5 points is a huge regression
epolanski 29d ago
I get some appeal of running locally, but isn't it just easier to rent cloud hardware and run whatever model you want to run?
- Gigachad 29d ago
  There's certainly a mental difference between a computer you can use as much as you want for a fixed upfront cost vs a rented server you are being billed by the hour for. But the cost difference between api and self hosted is so incredibly large now it's almost impossible to ignore the fact self hosting is a terrible deal. I'm waiting things out until the dust settles on what the required specs are and consumer hardware gets cheaper/more capable.
johnnyApplePRNG 29d ago
It IS crazy to drop big money on any AI rig right now imho... the size of models and the cost to run them is falling through the floor as we speak. I'm happy with all of the competition in the APIs on openrouter... I watch that like I used to watch the stock markets, lol. It's great fun.
- deleted 29d ago
  [deleted]
pprotas 28d ago
”less touchy” is an understatement, it happily complies with running exploits, reverse engineering and decompiling. Asking Claude to do that will give you an error and make you feel like you’re about to get banned.
subarctic 28d ago
I was going to try out pi and I set it up with some of the recent big open weights models, but then I realized that if I just use pi the way I use claude code, it doesn't have an auto mode so it's basically just --dangerously-skip-permissions and you're supposed to sandbox everything yourself. What's your sandboxing setup like?

walrus01 29d ago

One could also run it locally on a used dual xeon (or amd-equivalent) server with 512GB RAM, albeit slower, if you have a useful workflow for it that's like "take this day's efforts and run it through various analysis agents", combined with giving it one-shot tasks/modules to build overnight. You would want a place like a garage or basement to put the server because it'll be loud.

dataplumb3r 29d ago
You'd also likely spend far more in electricity than the API cost of processing the prompt(s)
- walrus01 29d ago
  yes, though for some uses, not sending data anywhere to third parties has its own value which is harder to measure.
peri-cl 29d ago
> "dual xeon" Does inference make full use of the memory bandwidth in a NUMA system?
- walrus01 29d ago
  Memory throughput performance is better than many equivalently priced (talking about the cheaper end of the used/refurb market here) single socket systems because each CPU is talking to its own banks of RAM. Using an example of llama-server on a 256GB (each CPU is attached to 128GB) system running qwen 3.8-flash-next Q8, which takes up about 188GB with the model and context, it's not that the inference engine 'knows' about NUMA, but if you use --load-mode mlock to load the whole thing into memory at once at the time llama-server launches, the kernel will distribute it relatively evenly across the memory connected to both CPUs.

0xdeadbeefbabe 29d ago

Well if you did get the m5 ultra could you obliterate the guardrails and then your wife can ask it pertinent but unsafe questions about how to punish you. Seems doable.

lnenad 29d ago

I have just built an Epyc with 512gb DDR4 3200 RAM for a "reasonable" price and I'm hoping to have a setup with GLM as the architect and Qwen 27b/Next Flash as the implementer. This is 1/5 of the price of the Mac, but also probably 1/5 of the speed lol.

jchw 29d ago
Honestly I suspect neither of them will be performing terribly well but with DDR4 3200 RAM I wonder if you'll be counting tokens per second or seconds per token. I mean, you do at least get a lot of memory channels at least, compared to consumer PCs. I am curious to hear what performance you get, I feel there is not enough information out there on what different setups manage to eek out.
- Philpax 29d ago
  The fastest I was able to get my Threadripper 3960X + 2x 3090s + 256GB DDR4-3200 to run a 2-bit quant of GLM-5.2 was 8 TPS. I would expect to be in seconds-per-token territory for a pure-CPU 4-bit quant.
  - jchw 29d ago
    One thing I'd like to try is MoE offloading: I have 2x32 GiB of VRAM and 128 GiB of DDR5 running at 4800 MT/s (only 2 channels though). I've seen people post difficult to believe MoE offloading results albeit a decently long time ago with older models. Maybe there is a quant that would fit with MoE offloading? That said, I am guessing my problem is not enough RAM - but this poor consumer platform struggles to do memory training with 128 GiB as it is. Now I surely regret not having gotten Threadripper and 256 GiB of RAM in the before-times.
    - Philpax 29d ago
      My measurement was with MoE offloading, but there's only so much you can keep on-GPU with a 200GB quant and 48GB of VRAM. It's hard to overcome the CPU/RAM bottleneck. For what it's worth, all of my hardware was used; I think, all-in, I'm probably at around 3k-4k USD? Not cheap, but also not the worst for something relatively versatile.

Der_Einzige 29d ago

[flagged]

CookieCrisp 29d ago
You are allowed to have an opinion on her 20k bag addiction
HighGoldstein 29d ago
Not every tech worker is making top-tier US salaries. For some (I suspect not few) people on HN that $20,000 Mac is almost a year's salary.
- tough 29d ago
  and even if you were making such a salary, the quesiton of if the investment on hardware to run llm's locally is still a big if, its OK if you buy the HW cause you'll use it and you get the extra capability as a nice extra, but doesnt make sense to spend so much when you could just get 200$ subs with almost infinite SOTA tokens a month etc (if you dont need the local/privacy aspects of it)

sickcodebruh 29d ago

When we consider: * LLM usage is new for the world * Models are evolving quickly with high worldwide competition * Hardware is evolving despite RAM shortages Is investing a huge sum of money in equipment for local inference a wise use of money? Or are M5 Ultra and equivalently priced local inference hardware future-proof enough to be worth it relative to how the market is evolving? Maybe it’s all a question of what you’d spend otherwise on serverless or dedicated GPU spend…

rkangel 29d ago
It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years. However there are other reasons (e.g. privacy) that might make it worth running locally for some people.
- WASDx 29d ago
  I think the privacy argument that keeps coming up is overrepresented. Certainly ZDR is enough for an absolute majority of use cases? I see so much talk about local inference but I doubt most of it has privacy as a valid argument (not arguing it doesn't exist). It's fun to do things locally though. I've tried it as well but cloud is just faster and cheaper.
  - rkangel 29d ago
    These companies have displayed zero respect for everyone's intellectual property getting these models trained. I think not giving them your complete trust is reasonable! I'm not saying zero trust, and ZDR is fine for most things but I understand the people who don't want to stream their whole codebase out token by token.
    - WASDx 29d ago
      Then use other providers hosting open models. Companies and individuals already put their whole code base on the cloud. I'm genuinely interested in privacy-oriented use cases where ZDR is not enough.

crossroadsguy 29d ago

Its reasoning leaves a lot to be desired :( Though I appreciate how good it is at "solid" grunt work and at that price (in fact I am paying the grandfathered subscription price; mostly). I am planning to let go for my Claude AI subscription which I now use only for "planning" and maybe use that via Open Router as PAYG (at to try how it ends up). But god glm is bad at "talking" and "responding" anything prose. Not only quality but it's almost impossible to tune it and make it let go of its habits and biases and enthusiasms which often result in too many too and fro. So I sometimes wonder at what point that starts becoming the cost and mental hassle. Maybe it's not there for me yet.

colingauvin 29d ago

In terms of pure tokens per dollar, absolutely not worth it. That said, when I bought my pair of Sparks, the best model I could run on it was GPT OSS 120B. That has an AA score of 24. Today, the best model I can run on them is GLM 5.3 Flash at Q4, AA score 57. Just still out on GLM 5.3 mixed quant. So from that perspective, they are many times better value than when I bought them, and will likely continue to increase in value.

lostmsu 28d ago
> GLM 5.3 Flash at Q4, AA score 57 That AA score is for the original model only
- nolist_policy 28d ago
  Then take DeepSeek V4 flash with AA score 52. Runs unquantized on 2x DGX spark with 1M context.
  - rubin55 28d ago
    Or Qwen 3.8 27B, AA score 52 (which is utterly insane given the size of this model); I have been testing Qwen 3.8 27B since a week now, as an intensive GLM-5.2 and Opus 5 user - I can say that I just can't believe my eyes i.r.t. to how good this model is.
  - lostmsu 28d ago
    5 points is a huge regression

epolanski 29d ago

I get some appeal of running locally, but isn't it just easier to rent cloud hardware and run whatever model you want to run?

Gigachad 29d ago
There's certainly a mental difference between a computer you can use as much as you want for a fixed upfront cost vs a rented server you are being billed by the hour for. But the cost difference between api and self hosted is so incredibly large now it's almost impossible to ignore the fact self hosting is a terrible deal. I'm waiting things out until the dust settles on what the required specs are and consumer hardware gets cheaper/more capable.

johnnyApplePRNG 29d ago

It IS crazy to drop big money on any AI rig right now imho... the size of models and the cost to run them is falling through the floor as we speak. I'm happy with all of the competition in the APIs on openrouter... I watch that like I used to watch the stock markets, lol. It's great fun.

deleted 29d ago
[deleted]

pprotas 28d ago

”less touchy” is an understatement, it happily complies with running exploits, reverse engineering and decompiling. Asking Claude to do that will give you an error and make you feel like you’re about to get banned.

subarctic 28d ago

I was going to try out pi and I set it up with some of the recent big open weights models, but then I realized that if I just use pi the way I use claude code, it doesn't have an auto mode so it's basically just --dangerously-skip-permissions and you're supposed to sandbox everything yourself. What's your sandboxing setup like?