3 ms·
I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RA
by SwellJoe 3mo ago
I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
- woadwarrior01 3mo agoPerf should be fairly straightforward to ballpark. You'll need to transfer roughly 2 . hidden_size . num_shards bytes over the network per token during autoregressive decoding. And divide that number by chunk size during prefill.
- i386 3mo agoThis was done on my home lab simulating 5ms latency and jitter between machines. Splits work quite well if you your nodes are over WAN at metro latency’s but not super fast on global WAN. The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others. I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
- zdw 3mo agoWhat hardware (CPU/GPU/memory) and network was used for this? What quantization for GLM 5.2? How much tuning of the split was needed?
- i386 3mo agoThe lab features two Mac Studios: an Apple M3 Ultra (32 CPU cores, 80 GPU cores, 256 GB unified memory) and an Apple M1 Ultra (20 CPU cores, 48 GPU cores, 128 GB unified memory), both connected via 1Gbit Ethernet. We use a customized Q2 quantization that preserves sensitive tensors at Q8. To reduce compute time per layer, we are developing a custom GLM DSA Metal graph. While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2. Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.
- wlesieutre 3mo agoEquivalent M3 machines no longer for sale from Apple (only up to 96 GB) but can be had on eBay for around $14,000 each
- drawnwren 3mo agoIt's notable that they're so valuable because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5, and exactly what makes llm inference fast.
- embedding-shape 3mo ago> because they feature 800Gbps of memory bandwidth. About twice what's available on the top end of M5 Ouch, about half of the memory bandwidth of a dedicated GPU though :/ Running LLMs on Apple hardware still doesn't make any sense to me.
- khalic 3mo agoM5 max has 614GB/s, you mean the m4?
- drawnwren 3mo agoOh, I was looking at every M5 except for the 40-core M5 Max. They have 460.
- SwellJoe 3mo agoThat sounds cool, but it's still pretty meaningless without information about what your home lab looks like. A few DGX Sparks wired up with their fancy super fast network is much different than a few laptops on wifi.
- deleted 3mo ago[deleted]
- imrehg 3mo agoThat's about the speed I get on a AMD Ryzen AI 9 HX 370 (inside a Framework 13), with Qwen3.6-35B-A3B, so doing the same on that much larger model...
- stymaar 3mo ago> I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk) Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why: - when offloading the weights to RAM or NVMe, you need to transfer the massive weights from your slow storage to the GPU for each layer being processed for each token. And as such you are being bottlenecked by the transfer bandwidth (which is either the men bandwidth of your DRAM or the read speed of your disk) - when using a distributed setup, the weights stay in the VRAM on each machine, the it's the GPU memory bandwidth that matters for the weights, and it's much higher than the two other bandwidth discussed above and as such the bottleneck isn't here. You need to tranfert data from a group of layers sitting on one device to the next one another device, but the amount of data is much smaller than the weights (we're talking about kilobytes of data, not gigabytes) so the network throughput isn't a limiting factor. The limiting factor is the network latency: if you split your model between 4 devices, you'll have 3 times the network latency per token. If you're on a network with 1ms latency, that means 3ms of latency per token. Which means the theoretical upper bound for your inference speed without speculative decoding is 30tps (this theoretical limit assumes the computation itself is instantaneous). So this is unlikely to be practical over the internet (too high of a latency) but on a local/enterprise network with speculative decoding it could totally work. Edit: note that all of the above is about token generation, for prefill/prompt-processing the distributed setup will almost certainly win (because in this case, the network latency doesn't add up)
- deleted 3mo ago[deleted]
- SwellJoe 3mo agoAh, that's interesting. I though there was more data crossing the network. So, why does a DGX Spark come with super fast network if 10Gbps ethernet would be sufficient for splitting a model? I never bought a second Strix Halo on the assumption that the pipe between them would be a limiting factor to using larger models, so obviously there's something I don't understand.