4 ms·
My quickie: MoE model heavily optimized for coding agents, complex reasoning, and tool use. 358B/32B active. vLLM/SGLang only supported on the main branch of th
by jtrn 10mo ago
My quickie: MoE model heavily optimized for coding agents, complex reasoning, and tool use. 358B/32B active. vLLM/SGLang only supported on the main branch of these engines, not the stable releases. Supports tool calling in OpenAI-style format. Multilingual English/Chinese primary. Context window: 200k. Claims Claude 3.5 Sonnet/GPT-5 level performance. 716GB in FP16, probably ca 220GB for Q4_K_M.
My most important takeaway is that, in theory, I could get a "relatively" cheap Mac Studio and run this locally, and get usable coding assistance without being dependent on any of the large LLM providers. Maybe utilizing Kimik2 in addition. I like that open-weight models are nipping at the feet of the proprietary models.
- embedding-shape 10mo ago> Supports tool calling in OpenAI-style format So Harmony? Or something older? Since Z.ai also claim the thinking mode does tool calling and reasoning interwoven, would make sense it was straight up OpenAI's Harmony. > in theory, I could get a "relatively" cheap Mac Studio and run this locally In practice, it'll be incredible slow and you'll quickly regret spending that much money on it instead of just using paid APIs until proper hardware gets cheaper / models get smaller.
- reissbaker 10mo agoNo, it's not Harmony; Z.ai has their own format, which they modified slightly for this release (by removing the required newlines from their previous format). You can see their tool call parsing code here: https://github.com/sgl-project/sglang/blob/34013d9d5a591e3c05b4ccf01e3bdc67898dc3e9/python/sglang/srt/function_call/glm47_moe_detector.py#L127 https://github.com/sgl-project/sglang/blob/34013d9d5a591e3c0...
- embedding-shape 10mo agoMan, really? Why, just why? If it's similar, why not just the same? It's like they're purposefully adding more work for the ecosystem to support their special model instead of just trying to add more value to the ecosystem.
- reissbaker 10mo agoThe parser is a small part of running an LLM, and Zai's format is superior to Harmony: it avoids having the model escape JSON in most cases by using XML, so e.g. long code edits are more in-domain compared to pretraining data (where code is typically not nested in JSON and isn't JSON-escaped). FWIW almost everyone has their own format. Also, Harmony is a mess. The common API specs adopted by the open-source community don't have developer roles, so including one is just bloat for the Responses API no one outside of OpenAI adopted. And why are there two types of hidden CoT reasoning? Harmony tool definition syntax invents a novel programming language that the model has never seen in training, so you need even more post-training to get it to work (Zai just uses JSON Schema). Etc etc. It's just bad. Re: removing newlines from their old format, it's slightly annoying, but it does give a slight speed boost, since it removes one token per call and one token per argument. Not a huge difference, but not nothing, especially with parallel tool calls.
- embedding-shape 10mo agoSometimes worse is better, I don't really care what the specific format is, just that providers/model releasers would use more of the same, because compatibility sucks when everyone has their very own format. Conveniently for them, it gets harder to compare models when everyone has different formats too.
- biddit 10mo ago> In practice, it'll be incredible slow and you'll quickly regret spending that much money on it instead of just using paid APIs until proper hardware gets cheaper / models get smaller. Yes, as someone who spent several thousand $ on a multi-GPU setup, the only reason to run local codegen inference right now is privacy or deep integration with the model itself. It’s decidedly more cost efficient to use frontier model APIs. Frontier models trained to work with their tightly-coupled harnesses are worlds ahead of quantized models with generic harnesses.
- theLiminator 10mo agoYeah, I think without a setup that costs 10k+ you can't even get remotely close in performance to something like claude code with opus 4.5.
- cmrdporcupine 10mo ago10k wouldn't even get you 1/4 of the way there. You couldn't even run this or DeepSeek 3.2 etc for that. Esp with RAM prices now spiking.
- coder543 10mo ago$10k gets you a Mac Studio with 512GB of RAM, which definitely can run GLM-4.7 with normal, production-grade levels of quantization (in contrast to the extreme quantization that some people talk about). The point in this thread is that it would likely be too slow due to prompt processing. (M5 Ultra might fix this with the GPU's new neural accelerators.)
- benjiro 10mo ago> $10k gets you a Mac Studio with 512GB of RAM Because Apple has not adjusted their pricing yet for the new ram pricing reality. The moment they do, its not going to be a $10k system anymore but in the $15k+... The amount of wafers going to AI is insane and will influence not just memory prices. Do not forget, the only reason why Apple is currently immunity to this, is because they tend to make long term contracts but the moment those expire ... then will push the costs down consumers.
- rz2k 10mo agoIn practice the 4bit MLX version runs at 20t/s for general chat. Do you consider that too slow for practical use? What example tasks would you try?
- embedding-shape 10mo agoWhenever reasoning/thinking is involved, 20t/s is way too slow for most non-async tasks, yeah. Translation, classification, whatever. If the response is 300 tokens for the reasoning and 50 tokens for the final reply, you're sitting and waiting 17,5 seconds for processing one item. In practice, you're also forgetting about prefill, prompt processing, tokenization and such. Please do share all relevant numbers :)
- reissbaker 10mo agos/Sonnet 3.5/Sonnet 4.5 The model output also IMO look significantly more beautiful than GLM-4.6; no doubt in part helped by ample distillation data from the closed-source models. Still, not complaining, I'd much prefer a cheap and open-source model vs. a more-expensive closed-source one.
- __natty__ 10mo agoI can imagine someone from the past reading this comment and having a moment of doubt
- deleted 10mo ago[deleted]
- mft_ 10mo agoI’m never clear, for these models with only a proportion active (32B here) to what extentt this reduces the RAM a system needs, if at all?
- l9o 10mo agoRAM requirements stay the same. You need all 358B parameters loaded in memory, as which experts activate depends on each token dynamically. The benefit is compute: only ~32B params participate per forward pass, so you get much faster tok/s than a dense 358B would give you.
- atq2119 10mo agoThe benefit is also RAM bandwidth. That probably adds to the confusion, but it matters a lot for decode. But yes, RAM capacity requirements stay the same.
- deepsquirrelnet 10mo agoFor mixture of experts, it primarily helps with time to first token latency, throughput generation and context length memory usage. You still have to have enough RAM/VRAM to load the full parameters, but it scales much better for memory consumed from input context than a dense model of comparable size.
- noahbp 10mo agoIt doesn't reduce the amount of RAM you need at all. It does reduce the amount of VRAM/HBM you need, however, since having all parameters/experts in one pass loaded on your GPU substantially increases token processing and generation speed, even if you have to load different experts for the next pass. Technically you don't even need to have enough RAM to load the entire model, as some inference engines allow you to offload some layers to disk. Though even with top of the line SSDs, this won't be ideal unless you can accept very low single-digit token generation rates.
- aurohacker 10mo agoGreat answers here, in that, for MoE, there's compute saving but no memory savings even tho the network is super-sparse. Turns out, there is a paper on the topic of predicting in advance the experts to be used in the next few layers, "Accelerating Mixture-of-Experts language model inference via plug-and-play lookahead gate on a single GPU". As to its efficacy, I'd love to know...
- hasperdi 10mo agoI bought a second‑hand Mac Studio Ultra M1 with 128 GB of RAM, intending to run an LLM locally for coding. Unfortunately, it's just way too slow. For instance, an 4‑bit quantized model of GLM 4.6 runs very slowly on my Mac. It's not only about tokens per second speed but also input processing, tokenization, and prompt loading; it takes so much time that it's testing my patience. People often mention about the TPS numbers, but they neglect to mention the input loading times.
- Reubend 10mo agoAnything except a 3bit quant of GLM 4.6 will exceed those 128 GB of RAM you mentioned, so of course it's slow for you. If you want good speeds, you'll at least need to store the entire thing in memory.
- mechagodzilla 10mo agoI've been running the 'frontier' open-weight LLMs (mainly deepseek r1/v3) at home, and I find that they're best for asynchronous interactions. Give it a prompt and come back in 30-45 minutes to read the response. I've been running on a dual-socket 36-core Xeon with 768GB of RAM and it typically gets 1-2 tokens/sec. Great for research questions or coding prompts, not great for text auto-complete while programming.
- tyre 10mo agoGiven the cost of the system, how long would it take to be less expensive than, for example, a $200/mo Claude Max subscription with Opus running?
- mechagodzilla 10mo agoIt's not really an apples-to-apples comparison - I enjoy playing around with LLMs, running different models, etc, and I place a relatively high premium on privacy. The computer itself was $2k about two years ago (and my employer reimbursed me for it), and 99% of my usage is for research questions which have relatively high output per input token. Using one for a coding assistant seems like it can run through a very high number of tokens with relatively few of them actually being used for anything. If I wanted a real-time coding assistant, I would probably be using something that fit in the 24GB of VRAM and would have very different cost/performance tradeoffs.
- whimsicalism 10mo agocommentators here are oddly obsessed with local serving imo, it's essentially never practical. it is okay to have to rent a GPU, but open weights are definitely good and important.
- retr0rocket 10mo ago[dead]
- nutjob2 10mo agoIt's not odd, people don't want to be dependent and restricted by vendors, especially if they're running a business based on the tool. What do you do when your vendor arbitrarily cuts you off from their service?
- whimsicalism 10mo agoi am not saying the desire to be uncoupled from token vendors is unreasonable, but you can rent cloud GPUs and run these models there. running on your own hardware is what seems a little fantastical at least for a reasonable TPS
- pixelpoet 10mo agoI don't understand what is going on with people willing to give up their computing sovereignty. You should be able to own and run your own computation, permissionlessly as much as your electricity bill and reasonable usage goes. If you can't do it today, you should aim for it tomorrow. Stop giving infinite power to these rent-seeking ghouls! Be grateful that open models / open source and semi-affordable personal computing still exists, and support it. Pertinent example: imagine if two Strix Halo machines (2x128 GB) can run this model locally over fast ethernet. Wouldn't that be cool, compared to trying to get 256 GB of Nvidia-based VRAM in the cloud / on a subscription / whatever terms Nv wants?
- RickyLahey 10mo agoi don't understand what is going on with people not training their own models
- Tepix 10mo agoI‘m going to try running it on two Strix Halo systems (256GB RAM total) networked via 2 USB4/TB3 ports.
- cmrdporcupine 10mo agoCurious to see how this works out for you. Let us know.
- pixelpoet 10mo agoAlso curious with two Strix Halo machines at the ready for exactly this kind of usage
- Tepix 10mo agoDon't wait for me. Donato Capitella has done this and created videos on his youtube channel at https://www.youtube.com/@donatocapitella https://www.youtube.com/@donatocapitella
- cmrdporcupine 10mo agoThat's GLM 4.6 tho, not 4.7? Still, informative. And stupidly I'd seen this video before. It sounds like the TLDR is: not quite.
- Tepix 10mo agoIt will probably be very similar in terms of speed.
- sa-code 10mo agoThis is true assuming there will be updates consistently. One of the advantages of the proprietary models is that the are updated often EKG and the cutoff date moves into the future This is important because libraries change, introduce new functionality, deprecate methods and rename things all the time, e.g. Polars.
- DeathArrow 10mo agoI think you will be much better with a couple of RTX 5090,4090 or 3090. I think Macs will be too slow for inference.
- andai 10mo ago>heavily optimized for coding agents I tested the previous one GLM-4.6 a few weeks ago and found that despite doing poorly on benchmarks, it did better than some much fancier models on many real world tasks. Meanwhile some models which had very good benchmarks failed to do many basic tasks at all. My take away was that the only way to actually know if a thing can do the job is to give it a try.
- lumost 10mo agoThis model is much stronger than 3.5 sonnet, 3.5 sonnet scored 49% on swe-bench verified vs. 72% here. This model is about 4 points ahead of sonnet4, but behind sonnet 4.5 by 4 points. If I were to guess, we will see a convergence on measurable/perceptible coding ability sometime early next year without substantially updated benchmarks.