4 ms·

The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago. That's not "nothing" and is plenty good enough f

The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago. That's not "nothing" and is plenty good enough for everyday work.

reilly3000 9mo ago
Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…
- teaearlgraycold 9mo ago
  Which while expensive is dirt cheap compared to a comparable NVidia or AMD system.
  - blharr 9mo ago
    What speed are you getting at that level of hardware though?
  - SchemaLoad 9mo ago
    It's still very expensive compared to using the hosted models which are currently massively subsidised. Have to wonder what the fair market price for these hosted models will be after the free money dries up.
    - cactusplant7374 9mo ago
      Inference is profitable. Maybe we hit a limit and we don't need as many expensive training runs in the future.
      - teaearlgraycold 9mo ago
        For sure Claude Code isn’t profitable
        bdangubic 9mo ago
        Neither was Uber and … and …
        plagiarist 9mo ago
        Businesses will desire me for my insomnia once Anthropics starts charging congestion pricing.
        bdangubic 9mo ago
        that is coming for sure to replace the "500" errors
      - paxys 9mo ago
        Inference APIs are probably profitable, but I doubt the $20-$100 monthly plans are.
        cactusplant7374 9mo ago
        I wouldn’t be so sure. Most users aren’t going to use up their quota every week.
    - whatsupdog 9mo ago
      I wonder if the "distributed AI computing" touted by some of the new crypto projects [0] works and is relatively cheaper. 0. https://www.daifi.ai/ https://www.daifi.ai/
- PlatoIsADisease 9mo ago
  [flagged]
  - zozbot234 9mo ago
    70B dense models are way behind SOTA. Even the aforementioned Kimi 2.5 has fewer active parameters than that, and then quantized at int4. We're at a point where some near-frontier models may run out of the box on Mac Mini-grade hardware, with perhaps no real need to even upgrade to the Mac Studio.
    - PlatoIsADisease 9mo ago
      >may I'm completely over these hypotheticals and 'testing grade'. I know Nvidia VRAM works, not some marketing about 'integrated ram'. Heck look at /r/locallama/ There is a reason its entirely Nvidia.
      - K0balt 9mo ago
        Mmmm, not really. I have both a4x 3090 box and a Mac m1 with 64 gb. I find that the Mac performs about the same as a 2x 3090. That’s nothing stellar, but you can run 70b models at decent quants with moderate context windows. Definitely useful for a lot of stuff.
        PlatoIsADisease 9mo ago
        >quants >moderate context windows Really had to modify the problem to make it seem equal? Not that quants are that bad, but the context windows thing is the difference between useful and not useful.
        K0balt 9mo ago
        Equal to the 2x3090? Yeah it’s about equal in every way, context windows included. As for useful at that scale? I use mine for coding a fair bit, and I don’t find it a detractor overall. It enforces proper API discipline, modularity, and hierarchal abstraction. Perhaps the field of application makes that more important though. (Writing firmware and hardware drivers). It also brings the advantage of focusing exclusively on the problems that are presented in the limited context, and not wandering off on side quests that it makes up. I find it works well up to about 1KLOC at a time. I wouldn’t imply they were equal to commercial models, but I would definitely say that local models are very useful tools. They are also stable, which is not something I can say for SOTA models. You cal learn how to get the best results from a model and the ground doesn’t move underneath you just when you’re on a roll.
- bigyabai 9mo ago
  "Full quality" being a relative assessment, here. You're still deeply compute constrained, that machine would crawl at longer contexts.
- 0xbadcafebee 9mo ago
  Most benchmarks show very little improvement of "full quality" over a quantized lower-bit model. You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use.
  - MuffinFlavored 9mo ago
    > You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use. Are there a lot of options how "how far" do you quantize? How much VRAM does it take to get the 92-95% you are speaking of?
    - bigyabai 9mo ago
      > Are there a lot of options how "how far" do you quantize? So many: https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods/ https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie... > How much VRAM does it take to get the 92-95% you are speaking of? For inference, it's heavily dependent on the size of the weights (plus context). Quantizing an f32 or f16 model to q4/mxfp4 won't necessarily use 92-95% less VRAM, but it's pretty close for smaller contexts.
      - MuffinFlavored 9mo ago
        Thank you. Could you give a tl;dr on "the full model needs ____ this much VRAM and if you do _____ the most common quantization method it will run in ____ this much VRAM" rough estimate please?
        omneity 9mo ago
        It’s a trivial calculation to make (+/- 10%). Number of params == “variables” in memory VRAM footprint ~= number of params * size of a param A 4B model at 8 bits will result in 4GB vram give or take, same as params. At 4 bits ~= 2GB and so on. Kimi is about 512GB at 4 bits.
- deaux 9mo ago
  And that's at unusable speeds - it takes about triple that amount to run it decently fast at int4. Now as the other replies say, you should very likely run a quantized version anyway.
- deleted 9mo ago
  [deleted]
- polynomial 9mo ago
  Depending on what your usage requirements are, Mac Minis running UMA over RDMA is becoming a feasible option. At roughly 1/10 of the cost you're getting much much more than 1/10 the performance. (YMMV) https://buildai.substack.com/i/181542049/the-mac-mini-moment https://buildai.substack.com/i/181542049/the-mac-mini-moment
  - danw1979 9mo ago
    I did not expect this to be a limiting factor in the mac mini RDMA setup ! - > Thermal throttling: Thunderbolt 5 cables get hot under sustained 15GB/s load. After 10 minutes, bandwidth drops to 12GB/s. After 20 minutes, 10GB/s. Your 5.36 tokens/sec becomes 4.1 tokens/sec. Active cooling on cables helps but you’re fighting physics. Thermal throttling of network cables is a new thing to me…
    - cat_plus_plus 9mo ago
      I admire patience of anyone who runs dense models on unified memory. Personally, I would rather feed an entire programming book or code directory to a sparse model and get an answer in 30 seconds and then use cloud in rare cases it's not enough.
    - polynomial 9mo ago
      Luckily we're having a record cold winter and your setup can double as a personal space heater.
teaearlgraycold 9mo ago
Having used K2.5 I’d judge it to be a little better than that. Maybe as good as proprietary models from last June?
paxys 9mo ago
LOCAL models. No one is running Kimi 2.5 on their Macbook or RTX 4090.
- DennisP 9mo ago
  On Macbooks, no. But there are a few lunatics like this guy: https://www.youtube.com/watch?v=bFgTxr5yst0 https://www.youtube.com/watch?v=bFgTxr5yst0
  - HarHarVeryFunny 9mo ago
    Wow! I've never heard of this guy before, but I see he's got 5M YouTube subscribers, which I guess is the clout you need to have Apple loan (I assume) you $50K worth of Mac Studios! I'll be interesting to see how model sizes, capability, and local compute prices evolve. A bit off topic, but I was in best buy the other day and was shocked to see 65" TVs selling for $300 ... I can remember the first large flat screen TVs (plasma?) selling for 100x that ($30K) when they first came out.
  - danw1979 9mo ago
    He must be mad, accepting $50k of free (probably loaned?) hardware from Apple ! Great demo video though. Nice to see some benchmarks of Exo with this cluster across various models.
- deaux 9mo ago
  Some people spend $50k on a new car, others spend it on running Kimi K2.5 at good speeds locally. No one's running Sonnet/Gemini/GPT-5 locally though.
corysama 9mo ago
The article mentions https://unsloth.ai/docs/basics/claude-codex https://unsloth.ai/docs/basics/claude-codex I'll add on https://unsloth.ai/docs/models/qwen3-coder-next https://unsloth.ai/docs/models/qwen3-coder-next The full model is supposedly comparable to Sonnet 4.5 But, you can run the 4 bit quant on consumer hardware as long as your RAM + VRAM has room to hold 46GB. 8 bit needs 85.
- deleted 9mo ago
  [deleted]
0xbadcafebee 9mo ago
Kimi K2.5 is fourth place for intelligence right now. And it's not as good as the top frontier models at coding, but it's better than Claude 4.5 Sonnet. https://artificialanalysis.ai/models https://artificialanalysis.ai/models
Aurornis 9mo ago
> The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago Kimi K2.5 is a trillion parameter model. You can't run it locally on anything other than extremely well equipped hardware. Even heavily quantized you'd still need 512GB of unified memory, and the quantization would impact the performance. Also the proprietary models a year ago were not that good for anything beyond basic tasks.

The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago. That's not "nothing" and is plenty good enough for everyday work.

reilly3000 9mo ago

Which takes a $20k thunderbolt cluster of 2 512GB RAM Mac Studio Ultras to run at full quality…

teaearlgraycold 9mo ago
Which while expensive is dirt cheap compared to a comparable NVidia or AMD system.
- blharr 9mo ago
  What speed are you getting at that level of hardware though?
- SchemaLoad 9mo ago
  It's still very expensive compared to using the hosted models which are currently massively subsidised. Have to wonder what the fair market price for these hosted models will be after the free money dries up.
  - cactusplant7374 9mo ago
    Inference is profitable. Maybe we hit a limit and we don't need as many expensive training runs in the future.
    - teaearlgraycold 9mo ago
      For sure Claude Code isn’t profitable
      - bdangubic 9mo ago
        Neither was Uber and … and …
        plagiarist 9mo ago
        Businesses will desire me for my insomnia once Anthropics starts charging congestion pricing.
        bdangubic 9mo ago
        that is coming for sure to replace the "500" errors
    - paxys 9mo ago
      Inference APIs are probably profitable, but I doubt the $20-$100 monthly plans are.
      - cactusplant7374 9mo ago
        I wouldn’t be so sure. Most users aren’t going to use up their quota every week.
  - whatsupdog 9mo ago
    I wonder if the "distributed AI computing" touted by some of the new crypto projects [0] works and is relatively cheaper. 0. https://www.daifi.ai/ https://www.daifi.ai/
PlatoIsADisease 9mo ago
[flagged]
- zozbot234 9mo ago
  70B dense models are way behind SOTA. Even the aforementioned Kimi 2.5 has fewer active parameters than that, and then quantized at int4. We're at a point where some near-frontier models may run out of the box on Mac Mini-grade hardware, with perhaps no real need to even upgrade to the Mac Studio.
  - PlatoIsADisease 9mo ago
    >may I'm completely over these hypotheticals and 'testing grade'. I know Nvidia VRAM works, not some marketing about 'integrated ram'. Heck look at /r/locallama/ There is a reason its entirely Nvidia.
    - K0balt 9mo ago
      Mmmm, not really. I have both a4x 3090 box and a Mac m1 with 64 gb. I find that the Mac performs about the same as a 2x 3090. That’s nothing stellar, but you can run 70b models at decent quants with moderate context windows. Definitely useful for a lot of stuff.
      - PlatoIsADisease 9mo ago
        >quants >moderate context windows Really had to modify the problem to make it seem equal? Not that quants are that bad, but the context windows thing is the difference between useful and not useful.
        K0balt 9mo ago
        Equal to the 2x3090? Yeah it’s about equal in every way, context windows included. As for useful at that scale? I use mine for coding a fair bit, and I don’t find it a detractor overall. It enforces proper API discipline, modularity, and hierarchal abstraction. Perhaps the field of application makes that more important though. (Writing firmware and hardware drivers). It also brings the advantage of focusing exclusively on the problems that are presented in the limited context, and not wandering off on side quests that it makes up. I find it works well up to about 1KLOC at a time. I wouldn’t imply they were equal to commercial models, but I would definitely say that local models are very useful tools. They are also stable, which is not something I can say for SOTA models. You cal learn how to get the best results from a model and the ground doesn’t move underneath you just when you’re on a roll.
bigyabai 9mo ago
"Full quality" being a relative assessment, here. You're still deeply compute constrained, that machine would crawl at longer contexts.
0xbadcafebee 9mo ago
Most benchmarks show very little improvement of "full quality" over a quantized lower-bit model. You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use.
- MuffinFlavored 9mo ago
  > You can shrink the model to a fraction of its "full" size and get 92-95% same performance, with less VRAM use. Are there a lot of options how "how far" do you quantize? How much VRAM does it take to get the 92-95% you are speaking of?
  - bigyabai 9mo ago
    > Are there a lot of options how "how far" do you quantize? So many: https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods/ https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overvie... > How much VRAM does it take to get the 92-95% you are speaking of? For inference, it's heavily dependent on the size of the weights (plus context). Quantizing an f32 or f16 model to q4/mxfp4 won't necessarily use 92-95% less VRAM, but it's pretty close for smaller contexts.
    - MuffinFlavored 9mo ago
      Thank you. Could you give a tl;dr on "the full model needs ____ this much VRAM and if you do _____ the most common quantization method it will run in ____ this much VRAM" rough estimate please?
      - omneity 9mo ago
        It’s a trivial calculation to make (+/- 10%). Number of params == “variables” in memory VRAM footprint ~= number of params * size of a param A 4B model at 8 bits will result in 4GB vram give or take, same as params. At 4 bits ~= 2GB and so on. Kimi is about 512GB at 4 bits.
deaux 9mo ago
And that's at unusable speeds - it takes about triple that amount to run it decently fast at int4. Now as the other replies say, you should very likely run a quantized version anyway.
deleted 9mo ago
[deleted]
polynomial 9mo ago
Depending on what your usage requirements are, Mac Minis running UMA over RDMA is becoming a feasible option. At roughly 1/10 of the cost you're getting much much more than 1/10 the performance. (YMMV) https://buildai.substack.com/i/181542049/the-mac-mini-moment https://buildai.substack.com/i/181542049/the-mac-mini-moment
- danw1979 9mo ago
  I did not expect this to be a limiting factor in the mac mini RDMA setup ! - > Thermal throttling: Thunderbolt 5 cables get hot under sustained 15GB/s load. After 10 minutes, bandwidth drops to 12GB/s. After 20 minutes, 10GB/s. Your 5.36 tokens/sec becomes 4.1 tokens/sec. Active cooling on cables helps but you’re fighting physics. Thermal throttling of network cables is a new thing to me…
  - cat_plus_plus 9mo ago
    I admire patience of anyone who runs dense models on unified memory. Personally, I would rather feed an entire programming book or code directory to a sparse model and get an answer in 30 seconds and then use cloud in rare cases it's not enough.
  - polynomial 9mo ago
    Luckily we're having a record cold winter and your setup can double as a personal space heater.

teaearlgraycold 9mo ago

Having used K2.5 I’d judge it to be a little better than that. Maybe as good as proprietary models from last June?

paxys 9mo ago

LOCAL models. No one is running Kimi 2.5 on their Macbook or RTX 4090.

DennisP 9mo ago
On Macbooks, no. But there are a few lunatics like this guy: https://www.youtube.com/watch?v=bFgTxr5yst0 https://www.youtube.com/watch?v=bFgTxr5yst0
- HarHarVeryFunny 9mo ago
  Wow! I've never heard of this guy before, but I see he's got 5M YouTube subscribers, which I guess is the clout you need to have Apple loan (I assume) you $50K worth of Mac Studios! I'll be interesting to see how model sizes, capability, and local compute prices evolve. A bit off topic, but I was in best buy the other day and was shocked to see 65" TVs selling for $300 ... I can remember the first large flat screen TVs (plasma?) selling for 100x that ($30K) when they first came out.
- danw1979 9mo ago
  He must be mad, accepting $50k of free (probably loaned?) hardware from Apple ! Great demo video though. Nice to see some benchmarks of Exo with this cluster across various models.
deaux 9mo ago
Some people spend $50k on a new car, others spend it on running Kimi K2.5 at good speeds locally. No one's running Sonnet/Gemini/GPT-5 locally though.

corysama 9mo ago

The article mentions https://unsloth.ai/docs/basics/claude-codex https://unsloth.ai/docs/basics/claude-codex I'll add on https://unsloth.ai/docs/models/qwen3-coder-next https://unsloth.ai/docs/models/qwen3-coder-next The full model is supposedly comparable to Sonnet 4.5 But, you can run the 4 bit quant on consumer hardware as long as your RAM + VRAM has room to hold 46GB. 8 bit needs 85.

deleted 9mo ago
[deleted]

0xbadcafebee 9mo ago

Kimi K2.5 is fourth place for intelligence right now. And it's not as good as the top frontier models at coding, but it's better than Claude 4.5 Sonnet. https://artificialanalysis.ai/models https://artificialanalysis.ai/models

Aurornis 9mo ago

> The best open models such as Kimi 2.5 are about as smart today as the big proprietary models were one year ago Kimi K2.5 is a trillion parameter model. You can't run it locally on anything other than extremely well equipped hardware. Even heavily quantized you'd still need 512GB of unified memory, and the quantization would impact the performance. Also the proprietary models a year ago were not that good for anything beyond basic tasks.