5 ms·

> This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM for ~3500 USD).

> This model? You can run it at Q4 with 70GB of VRAM. This is approaching consumer level territory (you can get a Mac Studio with 128GB of RAM for ~3500 USD). The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains enough quality (models have different sensitivities to quantization) and how fast it runs. For running async work and background tasks the prompt processing and token generation speeds matter less, but a lot of Mac Studio buyers have discovered the hard way that it's not going to be as responsive as working with a model hosted in the cloud on proper hardware. For most people without hard requirements for on-site processing, the best use case for this model would be going through one of the OpenRouter hosted providers for it and paying by token. > This beats the latest Sonnet while running locally Almost every open weight model launch this year has come with claims that it matches or exceeds Sonnet. I've been trying a lot of them and I have yet to see it in practice, even when the benchmarks show a clear lead.

zozbot234 6mo ago
Cloud hardware is not inherently more "proper" than what's being proposed here, there's nothing wrong per se about targeting slower inference speeds in an on prem single-user context.
- Aurornis 6mo ago
  > Cloud hardware is not inherently more "proper" than what's being proposed here Cloud hardware can run the original model. Quantization will reduce quality. The quality drop to Q4 is not trivial. Cloud hardware is also massively faster in time to first token and token generation speed. > there's nothing wrong per se about targeting slower inference speeds in a local single-user context. If that's what the user wants and expects then it's fine Most people working interactively with an LLM would suffer from slower turns.
  - zozbot234 6mo ago
    > Cloud hardware can run the original model. Quantization will reduce quality. New models are often being released in quantized format to begin with. This is true of both Kimi and the new DeepSeek V4 series. There is no "original model", the model is generated using Quantization Aware Training (QAT).
    - Aurornis 6mo ago
      > There is no "original model", the model is generated using Quantization Aware Training (QAT). The original model is the model used for the benchmarks People will say "You can run it locally!" then show the benchmarks of the original model, but what they really mean is that you can run a heavily quantized adaptation of the model which has difference performance characteristics.
      - zozbot234 6mo ago
        That remark was specific to newer models like Kimi 2.x and DeepSeek V4 series, and this is clearly stated in my comment. As for other models, we quantize them because we are generally constrained by the model's total footprint in bytes, and running a larger model that's been quantized to fit in the same footprint as a smaller one improves performance compared to a smaller original, generally up to Q4 or so, with even tighter quantizations (up to Q2) being usable for some uses such as general Q&A chat.
        hu3 6mo ago
        When you say DeepSeek v4... you do realise it is a 1.6T param model right? What kind of consumer hardware can run it reasonably in your mind?
      - DANmode 6mo ago
        I wish “performance” didn’t cover speed and quality, here.
- cbg0 6mo ago
  The quantization for some models can be very detrimental and their quality can drop considerably from the posted benchmarks which are probably at bf16, this is why having considerable RAM can be important.
simjnd 6mo ago
> The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains enough quality (models have different sensitivities to quantization) and how fast it runs. Very valid. This is an active area of research, and there are a lot of options to try out already today. - People have successfully used TurboQuant to quantize model weights (TQ3_4S), not just the context KV, to achieve smaller sizes than Q4 (~3.5 bpw) with much better PPL and faster decoding. - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. - DFlash (block diffusion for speculative decoding) needs a good drafting model compatible with the big model, but can provide an uplift up to 5x in decoding (although usually in the 2-2.5x range) - Forcing a model's thinking to obey a simple grammar has been shown to improve results with drastically lower thinking output (faster effective result generation) although that has been more impactful on smaller models. We should be skeptical, but it's definitely trending in the right direction and I wouldn't be surprised if we are indeed able to run it at acceptable speeds. > Almost every open weight model launch this year has come with claims that it matches or exceeds Sonnet. I've been trying a lot of them and I have yet to see it in practice, even when the benchmarks show a clear lead. This hasn't been my experience. After Anthropic's started their shenanigans I've switched to exclusively using open-weights models via OpenRouter and OpenCode and I can't really tell a difference (for better or for worse).
- sroussey 6mo ago
  Super interesting! > - People have successfully used TurboQuant to quantize model weights (TQ3_4S), not just the context KV, to achieve smaller sizes than Q4 (~3.5 bpw) with much better PPL and faster decoding. Where can I find more info on this? I’d like to convert models to onnx this way. > - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. Where can I find more info on this? I’d like to convert models to onnx this way. The most difficult environment for small models is in the browser. Would be great to push the SOTA in that environment.
  - hadlock 6mo ago
Computer0 6mo ago
Sure but for a casual conversational use case I have not found speed to be a huge barrier. I chatted with a 100b model using ddr5 only on a plane recently and it was fine. It's mainly that I cannot do data classification and coding tasks in a timely manner.
nijave 6mo ago
>Almost every open weight model launch this year has come with claims that it matches or exceeds Sonnet. I've been trying a lot of them and I have yet to see it in practice, even when the benchmarks show a clear lead. This has been my experience as well. I've been testing an agent built with Strands Agents which receives a load balancer latency alert and is expected to query logs with AWS Athena (Trino) then drill down with Datadog spans/traces to find the root cause. Admittedly, "devops" domain knowledge is important here My notes so far: "us.anthropic.claude-sonnet-4-6" # working, good results "us.anthropic.claude-sonnet-4-20250514-v1:0" # has problems following the prompt instructions "us.anthropic.claude-sonnet-4-5-20250929-v1:0" # working, good results "us.anthropic.claude-opus-4-5-20251101-v1:0" "us.anthropic.claude-opus-4-6-v1" # best results, slower, more expensive "amazon.nova-pro-v1:0" # completely fails "openai.gpt-oss-120b-1:0" # tool calling broken "zai.glm-5" # seems to work pretty well, a little slow, more expensive than Sonnet "minimax.minimax-m2.5" # didn't diagnose correctly "zai.glm-4.7" # good results but high tool call count, more expensive than Sonnet "mistral.mistral-large-3-675b-instruct" # misdiagnosed--somehow claimed a Prometheus scrape issue was involved "moonshotai.kimi-k2.5" # identified the right endpoints but interpreted trace data/root cause incorrectly "moonshot.kimi-k2-thinking" # identified endpoint, 1 correct root cause, 1 missing index hallucination Using models on AWS Bedrock. I let Claude Code w/ Opus 4.7 iterate over the agent prompt but didn't try to optimize per model. Really the only thing that came close to Sonnet 4.5 was GLM-5. The real kicker is, Sonnet is also the cheapest since it supports prompt caching The Kimi ones were close to working but didn't quite make the mark
- pbgcp2026 6mo ago
  " it supports prompt caching" May I ask if you checked that? I use "{"cachePoint": { "type": "default" }" and I found 2 things: * 1) even if stated in the Doco, Bedrock Converse API does not allow 1hr expiry time, only 5m - gives error when attempted; * 2) Bedrock Converse API does accept up to 4 cachePoint's but does NOT cache and returns zeroes. LOL. It was confirmed by some other people on Github. (Note: VertexAI does cache properly reducing the bill drastically, so I use Vertex instead of OpenRouter.)
  - nijave 6mo ago
parsimo2010 6mo ago
> being able to run a model and being able to run a model fast are two very different thresholds Specifically speaking, on my Strix Halo machine with (theoretical) memory bandwidth of 256 GB/s, a 70 GB model can't generate faster than 256/70= 3.65 t/s. The logic here is that a dense model must do a full read of the weights for each token. So even if the GPU can keep up, the memory bandwidth is limiting. A Mac M5 Pro is faster with a bandwidth of 307 GB/s, but that's only a little faster. This thing is going to be slow on consumer hardware. Maybe that is useful for someone, but I probably prefer a faster model in most cases even if the model isn't quite as smart. Qwen3.6 35B-A3B generates about 50 t/s on my machine, so it can make mistakes, be corrected, and try again in the same time that this model would still be thinking about its first response.
- zozbot234 6mo ago
  Recent models support multi-token prediction, which can guess multiple future tokens in a single decode step (using some subset of the model itself, not a separate drafting model) and then verify them all at once. It's an emerging feature still (not widely supported) and it's only useful for speeding up highly predictable token runs, but it's one way to do better in practice than the common-sense theoretical limit might suggest.
  - pbgcp2026 6mo ago
    It seems to me it's only Grok 4.20 that does this currently? Which other models did you have in mind, if I may ask?
    - phamilton 6mo ago
      Gemma4, qwen3.6, deepseek v4, mimo, glm 5/5.1 all do MTP.
      - pbgcp2026 6mo ago
        Thank you, I just realised we are talking about MTP. It seems that it's not that clear though. "Currently, the MTP capabilities are primarily accessible through Google's proprietary LiteRT framework, rather than the open-weights versions... Despite the missing MTP heads in the open release, Gemma 4 (specifically the 26B-A4B variant) still demonstrates high efficiency"
notatoad 6mo ago
being able to run a model fast is definitely more useful, but being able to run a model slowly for free is still super useful. agentic workflows are maturing all the time. yes, if i'm directly interacting with the LLM, i want it to be reasonably fast. but lately i've been queueing up a bunch of things when i go for lunch, or leaving things running when i go home at the end of the day. and claude doesn't keep working on that all night, it runs for an hour or so, gets to a point where it needs more input from me, and gives me some stuff to review in the morning. that could run 16x slower and still be just as useful for me.

zozbot234 6mo ago

Cloud hardware is not inherently more "proper" than what's being proposed here, there's nothing wrong per se about targeting slower inference speeds in an on prem single-user context.

Aurornis 6mo ago
> Cloud hardware is not inherently more "proper" than what's being proposed here Cloud hardware can run the original model. Quantization will reduce quality. The quality drop to Q4 is not trivial. Cloud hardware is also massively faster in time to first token and token generation speed. > there's nothing wrong per se about targeting slower inference speeds in a local single-user context. If that's what the user wants and expects then it's fine Most people working interactively with an LLM would suffer from slower turns.
- zozbot234 6mo ago
  > Cloud hardware can run the original model. Quantization will reduce quality. New models are often being released in quantized format to begin with. This is true of both Kimi and the new DeepSeek V4 series. There is no "original model", the model is generated using Quantization Aware Training (QAT).
  - Aurornis 6mo ago
    > There is no "original model", the model is generated using Quantization Aware Training (QAT). The original model is the model used for the benchmarks People will say "You can run it locally!" then show the benchmarks of the original model, but what they really mean is that you can run a heavily quantized adaptation of the model which has difference performance characteristics.
    - zozbot234 6mo ago
      That remark was specific to newer models like Kimi 2.x and DeepSeek V4 series, and this is clearly stated in my comment. As for other models, we quantize them because we are generally constrained by the model's total footprint in bytes, and running a larger model that's been quantized to fit in the same footprint as a smaller one improves performance compared to a smaller original, generally up to Q4 or so, with even tighter quantizations (up to Q2) being usable for some uses such as general Q&A chat.
      - hu3 6mo ago
        When you say DeepSeek v4... you do realise it is a 1.6T param model right? What kind of consumer hardware can run it reasonably in your mind?
    - DANmode 6mo ago
      I wish “performance” didn’t cover speed and quality, here.
cbg0 6mo ago
The quantization for some models can be very detrimental and their quality can drop considerably from the posted benchmarks which are probably at bf16, this is why having considerable RAM can be important.

simjnd 6mo ago

> The one thing I would want everyone curious about local LLMs to know is that being able to run a model and being able to run a model fast are two very different thresholds. You can get these models to run on a 128GB Mac, but we need to first tell if Q4 retains enough quality (models have different sensitivities to quantization) and how fast it runs. Very valid. This is an active area of research, and there are a lot of options to try out already today. - People have successfully used TurboQuant to quantize model weights (TQ3_4S), not just the context KV, to achieve smaller sizes than Q4 (~3.5 bpw) with much better PPL and faster decoding. - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. - DFlash (block diffusion for speculative decoding) needs a good drafting model compatible with the big model, but can provide an uplift up to 5x in decoding (although usually in the 2-2.5x range) - Forcing a model's thinking to obey a simple grammar has been shown to improve results with drastically lower thinking output (faster effective result generation) although that has been more impactful on smaller models. We should be skeptical, but it's definitely trending in the right direction and I wouldn't be surprised if we are indeed able to run it at acceptable speeds. > Almost every open weight model launch this year has come with claims that it matches or exceeds Sonnet. I've been trying a lot of them and I have yet to see it in practice, even when the benchmarks show a clear lead. This hasn't been my experience. After Anthropic's started their shenanigans I've switched to exclusively using open-weights models via OpenRouter and OpenCode and I can't really tell a difference (for better or for worse).

sroussey 6mo ago
Super interesting! > - People have successfully used TurboQuant to quantize model weights (TQ3_4S), not just the context KV, to achieve smaller sizes than Q4 (~3.5 bpw) with much better PPL and faster decoding. Where can I find more info on this? I’d like to convert models to onnx this way. > - Importance-weighted quantization (e.g. IQ4) also provides way better PPL, KDL, etc. at the same size as a Q4. Where can I find more info on this? I’d like to convert models to onnx this way. The most difficult environment for small models is in the browser. Would be great to push the SOTA in that environment.
- hadlock 6mo ago

Computer0 6mo ago

Sure but for a casual conversational use case I have not found speed to be a huge barrier. I chatted with a 100b model using ddr5 only on a plane recently and it was fine. It's mainly that I cannot do data classification and coding tasks in a timely manner.

nijave 6mo ago

>Almost every open weight model launch this year has come with claims that it matches or exceeds Sonnet. I've been trying a lot of them and I have yet to see it in practice, even when the benchmarks show a clear lead. This has been my experience as well. I've been testing an agent built with Strands Agents which receives a load balancer latency alert and is expected to query logs with AWS Athena (Trino) then drill down with Datadog spans/traces to find the root cause. Admittedly, "devops" domain knowledge is important here My notes so far: "us.anthropic.claude-sonnet-4-6" # working, good results "us.anthropic.claude-sonnet-4-20250514-v1:0" # has problems following the prompt instructions "us.anthropic.claude-sonnet-4-5-20250929-v1:0" # working, good results "us.anthropic.claude-opus-4-5-20251101-v1:0" "us.anthropic.claude-opus-4-6-v1" # best results, slower, more expensive "amazon.nova-pro-v1:0" # completely fails "openai.gpt-oss-120b-1:0" # tool calling broken "zai.glm-5" # seems to work pretty well, a little slow, more expensive than Sonnet "minimax.minimax-m2.5" # didn't diagnose correctly "zai.glm-4.7" # good results but high tool call count, more expensive than Sonnet "mistral.mistral-large-3-675b-instruct" # misdiagnosed--somehow claimed a Prometheus scrape issue was involved "moonshotai.kimi-k2.5" # identified the right endpoints but interpreted trace data/root cause incorrectly "moonshot.kimi-k2-thinking" # identified endpoint, 1 correct root cause, 1 missing index hallucination Using models on AWS Bedrock. I let Claude Code w/ Opus 4.7 iterate over the agent prompt but didn't try to optimize per model. Really the only thing that came close to Sonnet 4.5 was GLM-5. The real kicker is, Sonnet is also the cheapest since it supports prompt caching The Kimi ones were close to working but didn't quite make the mark

pbgcp2026 6mo ago
" it supports prompt caching" May I ask if you checked that? I use "{"cachePoint": { "type": "default" }" and I found 2 things: * 1) even if stated in the Doco, Bedrock Converse API does not allow 1hr expiry time, only 5m - gives error when attempted; * 2) Bedrock Converse API does accept up to 4 cachePoint's but does NOT cache and returns zeroes. LOL. It was confirmed by some other people on Github. (Note: VertexAI does cache properly reducing the bill drastically, so I use Vertex instead of OpenRouter.)
- nijave 6mo ago

parsimo2010 6mo ago

> being able to run a model and being able to run a model fast are two very different thresholds Specifically speaking, on my Strix Halo machine with (theoretical) memory bandwidth of 256 GB/s, a 70 GB model can't generate faster than 256/70= 3.65 t/s. The logic here is that a dense model must do a full read of the weights for each token. So even if the GPU can keep up, the memory bandwidth is limiting. A Mac M5 Pro is faster with a bandwidth of 307 GB/s, but that's only a little faster. This thing is going to be slow on consumer hardware. Maybe that is useful for someone, but I probably prefer a faster model in most cases even if the model isn't quite as smart. Qwen3.6 35B-A3B generates about 50 t/s on my machine, so it can make mistakes, be corrected, and try again in the same time that this model would still be thinking about its first response.

zozbot234 6mo ago
Recent models support multi-token prediction, which can guess multiple future tokens in a single decode step (using some subset of the model itself, not a separate drafting model) and then verify them all at once. It's an emerging feature still (not widely supported) and it's only useful for speeding up highly predictable token runs, but it's one way to do better in practice than the common-sense theoretical limit might suggest.
- pbgcp2026 6mo ago
  It seems to me it's only Grok 4.20 that does this currently? Which other models did you have in mind, if I may ask?
  - phamilton 6mo ago
    Gemma4, qwen3.6, deepseek v4, mimo, glm 5/5.1 all do MTP.
    - pbgcp2026 6mo ago
      Thank you, I just realised we are talking about MTP. It seems that it's not that clear though. "Currently, the MTP capabilities are primarily accessible through Google's proprietary LiteRT framework, rather than the open-weights versions... Despite the missing MTP heads in the open release, Gemma 4 (specifically the 26B-A4B variant) still demonstrates high efficiency"

notatoad 6mo ago

being able to run a model fast is definitely more useful, but being able to run a model slowly for free is still super useful. agentic workflows are maturing all the time. yes, if i'm directly interacting with the LLM, i want it to be reasonably fast. but lately i've been queueing up a bunch of things when i go for lunch, or leaving things running when i go home at the end of the day. and claude doesn't keep working on that all night, it runs for an hour or so, gets to a point where it needs more input from me, and gives me some stuff to review in the morning. that could run 16x slower and still be just as useful for me.