6 ms·
I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
by anonzzzies 6mo ago
I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
- underlines 6mo agodepends on format, compute type, quantization and kv cache size.
- mottosso 6mo agoSpecs for whatever they used to achieve the benchmarks would be a good start.
- Aurornis 6mo agoThe benchmarks are from the unquantized model they release. This will only run on server hardware, some workstation GPUs, or some 128GB unified memory systems. It’s a situation where if you have to ask, you can’t run the exact model they released. You have to wait for quantizations to smaller sizes, which come in a lot of varieties and have quality tradeoffs.
- deleted 6mo ago[deleted]
- bityard 6mo agoThis would likely run fine in just 96 GB of VRAM, by my estimation. Well within the ability of an enthusiastic hobbyist with a few thousand dollars of disposable income. Quantizations are already out: https://huggingface.co/unsloth/Qwen3.6-27B-GGUF https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
- bityard 6mo agoThe benchmarks in the model card are purported to be measurements of model quality (ability to perform tasks with few errors), not speed. They almost certainly run these benchmarks on their own cloud infrastructure (Alibaba afaik), which is typically not hardware that even the most enthusiastic homelab hobbyist can afford.
- seffignoz 6mo ago[dead]
- Aurornis 6mo agoThe 27B model they release directly would require significant hardware to run natively at 16-bit: A Mac or Strix Halo 128GB system, multiple high memory consumer GPUs, or an RTX 6000 workstation card. This is why they don’t advertise which consumer hardware it can run on: Their direct release that delivers these results cannot fit on your average consumer system. Most consumers don’t run the model they release directly. They run a quantized model that uses a lower number of bits per weight. The quantizations come with tradeoffs. You will not get the exact results they advertise using a quantized version, but you can fit it on smaller hardware. The previous 27B Qwen3.5 model had reasonable performance down to Q5 or Q4 depending on your threshold for quality loss. This was usable on a unified memory system (Mac, Strix Halo) with 32GB of extra RAM, so generally a 64GB Mac. They could also be run on an nVidia 5090 with 32GB RAM or a pair of 16GB or 24GB GPUs, which would not run as fast due to the split. Watch out for some of the claims about running these models on iPhones or smaller systems. You can use a lot of tricks and heavy quantization to run it on very small systems but the quality of output will not be usable. There is a trend of posting “I ran this model and this small hardware” repos for social media bragging rights but the output isn’t actually good.
- muyuu 6mo agoi have a Strix Halo machine typically those dense models are too slow on Strix Halo to be practical, expect 5-7 tps you can get an idea by looking at other dense benchmarks here: https://strixhalo.zurkowski.net/experiments https://strixhalo.zurkowski.net/experiments - i'd expect this model to be tested here soon, i don't think i will personally bother
- hedgehog 6mo agoThis one is around 250 t/s prefill and 12.4 generation in my testing.
- anonym29 6mo agosimilar numbers here - slightly higher PP. slightly better peak speed and retention w/ q8_0 kv cache quants too. llama-bench results here, cba to format for hn: https://pastebin.com/raw/zgJeqRbv https://pastebin.com/raw/zgJeqRbv GTR 9 Pro, "performance" profile in BIOS, GTT instead of GART, Fedora 44
- benob 6mo agoI get ~5 tokens/s on an M4 with 32G of RAM, using: llama-server \ -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \ --no-mmproj \ --fit on \ -np 1 \ -c 65536 \ --cache-ram 4096 -ctxcp 2 \ --jinja \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --reasoning on \ --chat-template-kwargs '{"preserve_thinking": true}' 35B-A3B model is at ~25 t/s. For comparison, on an A100 (~RTX 3090 with more memory) they fare respectively at 41 t/s and 97 t/s. I haven't tested the 27B model yet, but 35B-A3B often gets off rails after 15k-20k tokens of context. You can have it to do basic things reliably, but certainly not at the level of "frontier" models.
- dunb 6mo agoWhy use --fit on on an M4? My understanding was that given the unified memory, you should push all layers to the GPU with --n-gpu-layers all. Setting --flash-attn on and --no-mmap may also get you better results.
- halJordan 6mo agoMeaningless question, fit will put everything on the gpu if it fits. Fa is default on. No-mmap is not an inference tradeoff and if you do turn it off you need to turn on direct io via -dio What he should actually do is enable speculative decoding
- danielhanchen 6mo agoWe also made some dynamic MLX ones if they help - it might be faster for Macs, but llama-server definitely is improving at a fast pace. https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-4bit https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-4bit
- DarmokJalad1701 6mo agoWhat exactly does the .sh file install? How does it compare to running the same model in, say, omlx?
- jjcm 6mo agoFwiw, huggingface does this on the page where you download the weights. Slightly different format though - you put all the hardware you have, and it shows which quants you can run.
- chrsw 6mo agoThese might help if the provider doesn't offer the same details themselves. Of course, we have to wait for the newly released models to get added to these sites. https://llmfit.io/ https://llmfit.io/ https://modelfit.io/ https://modelfit.io/
- proxysna 6mo agoQwen3.5-27B with a 4bit quant can be run on a 24G card with no problem. With 2 Nvidia L4 cards and some additional vllm flags, i am serving 10 developers at 20-25tok/sek, off-peak is around 40tok/sek. Developers are ok with that performance, but ofc they requested more GPU's for added throughput.
- tandr 6mo agoWhat would be these additional vllm flags, if you don't mind sharing?
- proxysna 6mo agoThis is from an example from my Nomad cluster with two a5000's, which is a bit different what i have at work, but it will mostly apply to most modern 24G vram nvidia gpu. "--tensor-parallel-size", "2" - spread the LLM weights over 2 GPU's available "--max-model-len", "90000" - I've capped context window from ~256k to 90k. It allows us to have more concurrency and for our use cases it is enough. "--kv-cache-dtype", "fp8_e4m3", - On an L4 cuts KV cache size in half without a noticeable drop in quality, does not work on a5000, as it has no support for native FP8. Use "auto" to see what works for your gpu or try "tq3" once vllm people merge into the nightly. "--enable-prefix-caching" - Improves time to first output. "--speculative-config", "{\"method\":\"qwen3_next_mtp\",\"num_speculative_tokens\":2}", - Speculative mutli-token prediction. Qwen3.5 specific feature. In some cases provides a speedup of up to 40%. "--language-model-only" - does not load vision encoder. Since we are using just the LLM part of the model. Frees up some VRAM.
- czl 6mo ago> "--speculative-config", Regarding that last option: speculation helps max concurrency when it replaces many memory-expensive serial decode rounds with fewer verifier rounds, and the proposer is cheap enough. It hurts when you are already compute-saturated or the acceptance rate is too low. Good idea to benchmark a workload with and without speculative decoding.
- tandr 6mo ago
- ekojs 6mo agoAs this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.
- binary132 6mo agoThat seems awfully speculative without at least some anecdata to back it up.
- ekojs 6mo agoNot at all, I actually run ~30B dense models for production and have tested out 5090/3090 for that. There are gotchas of course, but the speed/quality claims should be roughly there.
- arcanemachiner 6mo agoSure, go get some. This isn't the first open-weight LLM to be released. People tend to get a feel for this stuff over time. Let me give you some more baseless speculation: Based on the quality of the 3.5 27B and the 3.6 35B models, this model is going to absolutely crush it.
- Aurornis 6mo ago> As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless The 4-bit quants are far from lossless. The effects show up more on longer context problems. > You can probably even go FP8 with 5090 (though there will be tradeoffs) You cannot run these models at 8-bit on a 32GB card because you need space for context. Typically it would be Q5 on a 32GB card to fit context lengths needed for anything other than short answers.
- ekojs 6mo ago> You cannot run these models at 8-bit on a 32GB card because you need space for context You probably can actually. Not saying that it would be ideal but it can fit entirely in VRAM (if you make sure to quantize the attention layers). KV cache quantization and not loading the vision tower would help quite a bit. Not ideal for long context, but it should be very much possible. I addressed the lossless claim in another reply but I guess it really depends on what the model is used for. For my usecases, it's nearly lossless I'd say.
- arcanemachiner 6mo agoDivide the value before the B by 2, and there's your answer if you get a Q4_K_M quant. Plus a bit of room for KV cache. TLDR: If you have 14GB of VRAM, you can try out this model with a 4-bit quant. Tokens per second is an unreasonable ask since every card is different, are you using GGUF or not, CUDA or ROCm or Vulkan or MLX, what optimizations are in your version of your inference software, flags are you running, etc. Note that it's a dense model (the Qwen models have another value at the end of the MoE model names, e.g. A3B) so it will not run very well in RAM, whereas with a MoE model, you can spill over into RAM if you don't have enough VRAM, and still have reasonable performance. Using these models requires some technical know-how, and there's no getting around that.
- rubiquity 6mo agoAt 8-bit quantization (q8_0) I get 20 tokens per second on a Radeon R9700.
- UncleOxidant 6mo agoFor Qwen3.5-27b I'm getting in the 20 to 25 tok/sec range on a 128GB Strix Halo box (Framework Desktop). That's with the 8-bit quant. It's definitely usable, but sometimes you're waiting a bit, though I'm not finding it problematic for the most part. I can run the Qwen3-coder-next (80b MoE) at 36tok/sec - hoping they release a Qwen3.6-coder soon.
- bityard 6mo agoI have a Framework Desktop too and 20-25 t/s is a lot better than I was expecting for such a large dense model. I'll have to try it out tonight. Are you using llama.cpp?
- UncleOxidant 6mo agoLMStudio, but it uses llama.cpp to run inference, so yeah. This is with the vulkan backend, not ROCm.
- petu 6mo ago> Qwen3.5-27b 8-bit quant 20 to 25 tok/sec It that with some kind of speculative decoding? Or total throughput for parallel requests?
- lambda 6mo agoThat sounds high for a Strix Halo with a dense 27b model. Are you talking about decode (prompt eval, which can happend in parallel) or generation when you quote tokens per second? Usually if people quote only one number they're quoting generation speed, and I would be surprised if you got that for generation speed on a Strix Halo.
- jauntywundrkind 6mo agoI would detest the time/words it takes to hand hold through such a review, of teaching folks the basics about LLM like this. It's also a section that, with hope, becomes obsolete sometime semi soon-ish.
- xngbuilds 6mo agoFor Apple Mac, there is https://omlx.ai/benchmarks https://omlx.ai/benchmarks
- random3 6mo agoCheck out https://www.canirun.ai/ https://www.canirun.ai/ (and https://news.ycombinator.com/item?id=47363754 https://news.ycombinator.com/item?id=47363754)
- esskay 6mo agoCaniRun's not a great tool - look how long its been since it's been updated. It's not got any of the qwen3.6 models on the list nor the new kimi one. In fact it's missing many of the "popular" models.
- bityard 6mo agoThere are infinite combinations of CPU/GPU capable of running LLMs locally. What most people do is buy the system they can afford and roughly meets their goals and then ball-park VRAM usage by looking at the model size and quantization. For more a detailed analysis, there are several online VRAM calculators. Here's one: https://smcleod.net/vram-estimator/ https://smcleod.net/vram-estimator/ If you have a huggingface account, you can set your system configuration and then you get little icons next to each quant in the sidebar. (Green: will likely fit, Yellow: Tight fit, Red: will not fit) Further, t/s depends greatly on a lot of different factors, the best you might get is a guess based on context size. One thing about running local LLMs right now, is that there are tradeoffs literally everywhere and you have to choose what to optimize for down to the individual task.
- zargon 6mo agoThese calculators are almost entirely useless. They don't understand specific model architectures. Even the ones that try to support only specific models (like the apxml one) get it very wrong a lot of the time. For example, the one you linked, when I provide a Qwen3.5 27B Q_4_M GGUF [0], says that it will require 338 GB of memory with 16-bit kv cache. That is wrong by over an order of magnitude. [0] https://huggingface.co/bartowski/Qwen_Qwen3.5-27B-GGUF/resolve/main/Qwen_Qwen3.5-27B-Q4_K_M.gguf https://huggingface.co/bartowski/Qwen_Qwen3.5-27B-GGUF/resol...
- gdevenyi 6mo agoMine does https://github.com/gdevenyi/huggingface-estimate https://github.com/gdevenyi/huggingface-estimate
- zargon 6mo agoExcellent job with this! I tried a few combinations that completely fail on other calculators and yours gets VRAM usage pretty much spot on, and even the performance estimate is in the ballpark to what I see with mixed VRAM / RAM workloads. It's a shame that search is so polluted these days that it's impossible to find good tools like yours.
- CamperBob2 6mo ago27B dense is not a screamer, even on an RTX 6000, but it will run at full precision with (more than) enough room for context at the model's own capacity. You can expect about 30 tokens/second after prompt processing. Quants will likely run similarly well on the 16/24/32 GB consumer GPUs. The 3.5 27B model was a strong and capable reasoner, so I have high hopes for this one. Thanks to the team at Qwen for keeping competition in this space alive.
- scosman 6mo agothis is what the unsloth post is for
- neonstatic 6mo agoI just ran Unsloth's Q4_K_M quant on Macbook Pro M1 Max (2021), 32 GB RAM. Getting about 9 tok/s. Reported RAM usage 15.9 GB. It's okay, but a bit slow for my taste. An example of a query and a response is below. It was done with 9.09 tok/s, 893 tokens, 7.56 seconds to first token, thought for 1 minute and 9 seconds. Question: I gave the model this truth table and asked to tell me what boolean function would produce the output given A, B as inputs. | A | B | Output | | :--- | --- | ---: | | (T) | (F) | (T) | | (T) | (T) | (F) | | (F) | (F) | (F) | | (F) | (T) | (F) | Answer: (note this is part of a larger conversation, where I was getting it to help me with some boolean logic): The boolean condition for your table is simply: ```rust A && !B ``` ### Why? Your table shows the output is `true` **only** when `A` is true and `B` is false. That's exactly what `A && !B` expresses. | A | B | `A && !B` | Output | |---|---|-----------|--------| | (T) | (F) | `T && T = T` | | | (T) | (T) | `T && F = F` | | | (F) | (F) | `F && T = F` | | | (F) | (T) | `F && F = F` | | ### In Rust: ```rust if A && !B { // output is true } else { // output is false } ``` This is the most direct and idiomatic way to express that logic. Let me know if you need it adapted back to your `live_ticks` / `on_disconnect` context!
- jen20 6mo agohttps://github.com/AlexsJones/llmfit https://github.com/AlexsJones/llmfit gives a good approximation of that.
- SlavikCA 6mo agoI'm getting 30 t/s on RTX 4090D (using 42 out of 48GB VRAM) with UD-Q6_K_XL https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/discussions/7 https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/discussions/...
- maxloh 6mo agoI thought Q4_K_M is the standard. Why did you choose the 6-bit variant? Does it generate better input?
- SlavikCA 6mo agoThere is no standard. The higher quantization - the better results, but more memory is needed. Q8 is the best.
- SV_BubbleTime 6mo agoFP32 is best, although I wonder if there isn’t something better I don’t know about. Q8 is for the most part equal to FP16 in practical terms by being smart about what is quantized, but iirc always slower than FP16 and FP8.
- yorhodes 6mo ago[dead]
- gdevenyi 6mo agoYou can point at the GGUF files and figure it out with your hardware here. https://github.com/gdevenyi/huggingface-estimate https://github.com/gdevenyi/huggingface-estimate
- Havoc 6mo agoYou can often sorta estimate it but multiply it with like 2/3rds give or take a lot to work out how much vram you need. 27B will fit onto a 24gb card with decent context and a couple GB for operating system to spare at Q4. tok/s doesn't really have a good way to eyeball it