4 ms·

Didn't expect it to beat 3.8 27B so cleanly. Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.

Squarex 1mo ago
I don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.
- LaurensBER 1mo ago
  If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet. Self-learning/improving would be even better but that's still a long way to go.
  - redox99 1mo ago
    Search results suck because the web sucks these days. The big models from OpenAI/Anthropic have every book in existence baked into them
    - jameshart 1mo ago
      I don’t think that’s the right way to think about LLM ‘knowledge’. They don’t have absolute recall of everything in the training set. They have been trained so that they have weights that can predict what those books might say - that is, if they read them they would find the contents unsurprising. That doesn’t mean it wouldn’t be helpful to pull relevant passages of text directly into context for a particular task.
    - infp_arborist 1mo ago
      Does it really matter? What about including all relevant and up-to-date literature as skills for local models? I have no experience with this but I am pretty sure someone has already thought about it.
- rohansood15 1mo ago
  For world knowledge, you'd want it to find and reference the source material to be sure. At that point, it doesn't matter if the knowledge is embedded.
  - quev 1mo ago
    Keep in mind a web search might not include scanned books baked in the weights ;)
    - skeledrew 1mo ago
      Are Chinese labs also acquiring and scanning books?
  - hedora 1mo ago
    I think the big models have adequate recall, so tool use is probably unnecessary, but the user said the correctness of my response is important. Let me look up the data instead of relying on my memory.
  - dist-epoch 1mo ago
    World knowledge also means knowing the various algorithms and ways particular programming problems are solved. You can't search what you don't even know exists.
    - serf 1mo ago
      >You can't search what you don't even know exists. that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire. a lot of smaller agentic models and a lot of harnesses live on that premise.
      - water-drummer 1mo ago
        Path finding is a very closed and well defined problem.
- horsawlarway 1mo ago
  In a lot of spaces, this is actually preferable. Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc. Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model. Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained. --- Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store. We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value). I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please. Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store. If I'm doing auto-repair... "cars for dummies (version 3)". etc...
  - donmcronald 1mo ago
    > Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store. This is what I've been trying to focus on with local AI for now. I've been trying to build all new documentation so it's more AI friendly. It's been pretty interesting. Qwen-35BA3B with a small prompt does a good job of surfacing what I'd consider institutional knowledge. I've been trying to silo the docs I write from the model with a prompt that tells it not to use general knowledge unless asked to. From the anecdotal testing I did, Qwen-35BA3B is great for it. It does a really good job of following the prompt and calling tools, so I've been able to play around a lot to see what seems to work best. Ultimately, I think one of the most effective uses of AI will be having a distinct knowledge store combined with an opinionated agent (and sub-agent) setup along with different models for each task. Who owns the knowledge store is going to be the big caveat. Right now I think the big online models are trying for generic, persistent memory and I'd be very hesitant to let that happen. Think of having someone with a perfect memory following you around forever, but someone else has the ability to make them disappear. That's not a good situation.
  - 1mo ago
- redox99 1mo ago
  At 125B + 51B I'd expect it to have some degree of world knowledge, clearly in the middle between small models like qwen 27B, and huge trillion parameter models.
RobertasTa 1mo ago
[flagged]
gruez 1mo ago
>Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy. How much memory does this translate to and what quantization (if any) were applied?
- rohansood15 1mo ago
  128GB, 4-bit quantized.
user43928 1mo ago
For comparison with hosted models, GPT 5.6 Luna scores 67% on DeepSWE, compared to 59% here for Qwen. Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.
- rohansood15 1mo ago
  This is a good counter argument. But you have to note that this is after OpenAI cut Luna costs by 80%. If you compare launch pricing, Qwen probably comes out ahead on a cost-performance basis.
  - jrflo 1mo ago
    The luna cost cuts were real though, not a one time promotion or something, due to some optimization (probably distillation?) that openai did.
    - QwenGlazer9000 1mo ago
      Was it? Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus. It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
    - throwaw12 1mo ago
      what if it was because of quantization and they haven't released the new benchmarks for it? Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
      - dannyw 1mo ago
        ARC Prize has retested Luna after the discount and validated identical performance. (Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT). These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing. Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).
dist-epoch 1mo ago
It's a much bigger model, with a next-gen architecture. It's expected to be much better.
hedora 1mo ago
My AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now. Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
- aftbit 1mo ago
  I believe you're underestimating the lag inherent in the economy. Even if we grant the idea that the political party controlling the US House/Senate has a significant impact on the economy, and that the current party is BAD and the next one would be GOOD, I would still expect that things will continue getting WORSE for a good 4 to 8 years before they get better again. And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
- NewJazz 1mo ago
  You know the memory cartel isn't even close to being broken, right?
overgard 1mo ago
Curious, how are you running it and what quantization are you using? I've mostly been using MTPLX; 125B sort of looks like it'd be right at the limits of my 128GB MacBook once you factor in KV cache and context window.. wondering if it's worth it compared to the 27B model which gives me a lot of headroom or even a 72B model.

Squarex 1mo ago

I don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.

LaurensBER 1mo ago
If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet. Self-learning/improving would be even better but that's still a long way to go.
- redox99 1mo ago
  Search results suck because the web sucks these days. The big models from OpenAI/Anthropic have every book in existence baked into them
  - jameshart 1mo ago
    I don’t think that’s the right way to think about LLM ‘knowledge’. They don’t have absolute recall of everything in the training set. They have been trained so that they have weights that can predict what those books might say - that is, if they read them they would find the contents unsurprising. That doesn’t mean it wouldn’t be helpful to pull relevant passages of text directly into context for a particular task.
  - infp_arborist 1mo ago
    Does it really matter? What about including all relevant and up-to-date literature as skills for local models? I have no experience with this but I am pretty sure someone has already thought about it.
rohansood15 1mo ago
For world knowledge, you'd want it to find and reference the source material to be sure. At that point, it doesn't matter if the knowledge is embedded.
- quev 1mo ago
  Keep in mind a web search might not include scanned books baked in the weights ;)
  - skeledrew 1mo ago
    Are Chinese labs also acquiring and scanning books?
- hedora 1mo ago
  I think the big models have adequate recall, so tool use is probably unnecessary, but the user said the correctness of my response is important. Let me look up the data instead of relying on my memory.
- dist-epoch 1mo ago
  World knowledge also means knowing the various algorithms and ways particular programming problems are solved. You can't search what you don't even know exists.
  - serf 1mo ago
    >You can't search what you don't even know exists. that's not really entirely true -- one can google for "fast pathfinding' and stumble upon A-star , all that had to be queried was the intent/desire. a lot of smaller agentic models and a lot of harnesses live on that premise.
    - water-drummer 1mo ago
      Path finding is a very closed and well defined problem.
horsawlarway 1mo ago
In a lot of spaces, this is actually preferable. Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc. Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a new model. Another classic example of this back in the day was to ask who the president of the US was, and watch different models happily give different answers based on the date they were trained. --- Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store. We're vaguely there with the ability for models to go search the web, but I think the reliability of that path is going to continue declining (more and more spam content, less and less genuine value). I kinda want a paradigm where I can pick and engine and a knowledge bank, and combine them as I please. Ex - if I'm doing gardening, I can pick "gardening for models (version 32)" as my knowledge store. If I'm doing auto-repair... "cars for dummies (version 3)". etc...
- donmcronald 1mo ago
  > Personally, I'm really interested to see if we're headed towards a spot where the model is entirely distinct from the knowledge store. This is what I've been trying to focus on with local AI for now. I've been trying to build all new documentation so it's more AI friendly. It's been pretty interesting. Qwen-35BA3B with a small prompt does a good job of surfacing what I'd consider institutional knowledge. I've been trying to silo the docs I write from the model with a prompt that tells it not to use general knowledge unless asked to. From the anecdotal testing I did, Qwen-35BA3B is great for it. It does a really good job of following the prompt and calling tools, so I've been able to play around a lot to see what seems to work best. Ultimately, I think one of the most effective uses of AI will be having a distinct knowledge store combined with an opinionated agent (and sub-agent) setup along with different models for each task. Who owns the knowledge store is going to be the big caveat. Right now I think the big online models are trying for generic, persistent memory and I'd be very hesitant to let that happen. Think of having someone with a perfect memory following you around forever, but someone else has the ability to make them disappear. That's not a good situation.
- 1mo ago
redox99 1mo ago
At 125B + 51B I'd expect it to have some degree of world knowledge, clearly in the middle between small models like qwen 27B, and huge trillion parameter models.

RobertasTa 1mo ago

[flagged]

gruez 1mo ago

>Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy. How much memory does this translate to and what quantization (if any) were applied?

rohansood15 1mo ago
128GB, 4-bit quantized.

user43928 1mo ago

For comparison with hosted models, GPT 5.6 Luna scores 67% on DeepSWE, compared to 59% here for Qwen. Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.

rohansood15 1mo ago
This is a good counter argument. But you have to note that this is after OpenAI cut Luna costs by 80%. If you compare launch pricing, Qwen probably comes out ahead on a cost-performance basis.
- jrflo 1mo ago
  The luna cost cuts were real though, not a one time promotion or something, due to some optimization (probably distillation?) that openai did.
  - QwenGlazer9000 1mo ago
    Was it? Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus. It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
  - throwaw12 1mo ago
    what if it was because of quantization and they haven't released the new benchmarks for it? Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
    - dannyw 1mo ago
      ARC Prize has retested Luna after the discount and validated identical performance. (Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT). These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing. Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).

dist-epoch 1mo ago

It's a much bigger model, with a next-gen architecture. It's expected to be much better.

hedora 1mo ago

My AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now. Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.

aftbit 1mo ago
I believe you're underestimating the lag inherent in the economy. Even if we grant the idea that the political party controlling the US House/Senate has a significant impact on the economy, and that the current party is BAD and the next one would be GOOD, I would still expect that things will continue getting WORSE for a good 4 to 8 years before they get better again. And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
NewJazz 1mo ago
You know the memory cartel isn't even close to being broken, right?

overgard 1mo ago

Curious, how are you running it and what quantization are you using? I've mostly been using MTPLX; 125B sort of looks like it'd be right at the limits of my 128GB MacBook once you factor in KV cache and context window.. wondering if it's worth it compared to the 27B model which gives me a lot of headroom or even a 72B model.