5 ms·

This is great as a first look, but the author is not a developer, so we don't yet know whether a dev can be as productive with local models on M5 Mac Studio com

This is great as a first look, but the author is not a developer, so we don't yet know whether a dev can be as productive with local models on M5 Mac Studio compared to a 20x subscription plan. I'm also curious about any new low hanging optimization opportunities in the kernels for this new hardware. It's already clear to me that M5 Mac Studio is more cost-effective than anything you can run on open router, assuming decent utilization. The M5 Mac Studio will be the most cost effective way to run uncensored cyber capable open agents. An exciting tipping point will be if programmers can get an Astra-Ultra like experience all week with this hardware. That would be a real sense where this hardware exceeds the value of even 20x cloud subscriptions.

slowin 7d ago
> This is great as a first look, but the author is not a developer, so we don't yet know whether a dev can be as productive with local models on M5 Mac Studio compared to a 20x subscription plan. Local models are definitely not as productive as SOTA, sadly it's not close yet. I do think someday they will be "good enough" to use, but they aren't today. Even the SOTA models barely code well, with Opus 4.5 being the first, good coding model. That being said, I think it's absolutely imperative that we keep pushing local model performance. We need to continue to advance technology there and ensure that the model labs don't do regulatory capture in the name of "safety" (or anything else).
- nowittyusername 7d ago
  With the latest codex (weekly quota burn) fiasco I tried open weight alternatives for the first time. And tyeah... open weight models cant compete with likes of astra yet. But, my hope is that by the time I get my Mac studio at end of november an open weight models would have closed the gap (which i think is realistic at the speed of progress). Now its true a better gpt version will also be available then but it also seems the gap is shrinking with time so theres that.
  - Octoth0rpe 7d ago
    > And tyeah... open weight models cant compete with likes of astra yet I think this is true, but also misses that a lot of us are just doing basic flask apps with a react front end. We don't need astra; Something sonnet 4.6 level locally is perfectly sufficient 95% of the time, and maybe 99% of the time.
- _hugerobots_ 7d ago
  Local models can be widely used as productive assets. Yes the infrastructure of SOTA API models is engineered specifically for you to be that utility, but the blanket statement that local isn't up to par is intensely short sighted. Billions of tokens per month on local pays for the hardware when compared to sota costs per month.
  - slowin 7d ago
    I believe they can currently be used productively for non-coding tasks (classification, light summary)... but they definitely are not even close to SOTA when it comes to software development.
    - _hugerobots_ 7d ago
      Defining productivity is a use-case scenario, and a wildly generalized assumption for most people in this argument. Local infrastructure doesn't need to be sota for absolutely every single need for a dev lab, but it absolutely can be delivered with non-api frontier class models.
      - slowin 7d ago
        Just to be clear, I'm specifically talking about coding. I think local models can help with productivity today, just not coding. I'm also a huge fan of local models and think it's absolutely imperative that they continue to advance so we can move off of the Anthropic/OpenAI hosted models. It's important to accurately asses where we are in that journey though.
        srcreigh 7d ago
        I think the issue is generalization, if you were more specific about which local models aren’t good enough for which tasks compared to which frontier models in your experience, it’d be a lot more informative
        7d ago
zozbot234 7d ago
Astra-Ultra? Even the largest open model to date (Kimi K3) is nowhere close to Astra level, and it will be quite slow even on the highest-spec M5 Ultra, with achievable speeds of about 0.5 tok/s at most due to having to stream weights from SSD (~13 GB/s on the highest storage capacity M5 Max machines so far). This is OK for doing simple Q&A in the background but it's far from a genuine coding experience. You'd have to test batching of multiple thinking streams in order to try and raise overall tok/s via layer-wise reuse of the streamed weights (and this is where the "Ultra" part sort of becomes relevant; Kimi series models have good support for agent swarms) but this would decrease single-session performance even further. It would only be usable for background jobs, though the hardware would then have a chance of paying for itself if it was fully used on a 24/7 basis.
- srcreigh 7d ago
  > You'd have to test batching of multiple thinking streams in order to try and raise overall tok/s via layer-wise reuse of the streamed weights isn’t this very straightforward to do..? I thought batching for Qwen models is already proven out. > but this would decrease single-session performance even further Well let’s take Qwen 3.8 27B. Throughput for M3 at 8 agents is 4x compared to single agent. [1] It’s really not clear to me that 8 concurrent agents at half speed will be worse task completion latency than 1 agent. And that’s M3 studio benchmarks, not even M5 ultra, and without the many software improvements we will see If you haven’t tried Qwen 3.8 27B xhigh on a task you might not get the hype. Idk. If you’ve tried doing this and don’t like it sure, and be specific about what isn’t effective, but let’s not speculate. [1]: https://omlx.ai/benchmarks/performance/69kzkrv8?utm_source=chatgpt.com https://omlx.ai/benchmarks/performance/69kzkrv8?utm_source=c...

slowin 7d ago

> This is great as a first look, but the author is not a developer, so we don't yet know whether a dev can be as productive with local models on M5 Mac Studio compared to a 20x subscription plan. Local models are definitely not as productive as SOTA, sadly it's not close yet. I do think someday they will be "good enough" to use, but they aren't today. Even the SOTA models barely code well, with Opus 4.5 being the first, good coding model. That being said, I think it's absolutely imperative that we keep pushing local model performance. We need to continue to advance technology there and ensure that the model labs don't do regulatory capture in the name of "safety" (or anything else).

nowittyusername 7d ago
With the latest codex (weekly quota burn) fiasco I tried open weight alternatives for the first time. And tyeah... open weight models cant compete with likes of astra yet. But, my hope is that by the time I get my Mac studio at end of november an open weight models would have closed the gap (which i think is realistic at the speed of progress). Now its true a better gpt version will also be available then but it also seems the gap is shrinking with time so theres that.
- Octoth0rpe 7d ago
  > And tyeah... open weight models cant compete with likes of astra yet I think this is true, but also misses that a lot of us are just doing basic flask apps with a react front end. We don't need astra; Something sonnet 4.6 level locally is perfectly sufficient 95% of the time, and maybe 99% of the time.
_hugerobots_ 7d ago
Local models can be widely used as productive assets. Yes the infrastructure of SOTA API models is engineered specifically for you to be that utility, but the blanket statement that local isn't up to par is intensely short sighted. Billions of tokens per month on local pays for the hardware when compared to sota costs per month.
- slowin 7d ago
  I believe they can currently be used productively for non-coding tasks (classification, light summary)... but they definitely are not even close to SOTA when it comes to software development.
  - _hugerobots_ 7d ago
    Defining productivity is a use-case scenario, and a wildly generalized assumption for most people in this argument. Local infrastructure doesn't need to be sota for absolutely every single need for a dev lab, but it absolutely can be delivered with non-api frontier class models.
    - slowin 7d ago
      Just to be clear, I'm specifically talking about coding. I think local models can help with productivity today, just not coding. I'm also a huge fan of local models and think it's absolutely imperative that they continue to advance so we can move off of the Anthropic/OpenAI hosted models. It's important to accurately asses where we are in that journey though.
      - srcreigh 7d ago
        I think the issue is generalization, if you were more specific about which local models aren’t good enough for which tasks compared to which frontier models in your experience, it’d be a lot more informative
        7d ago

zozbot234 7d ago

Astra-Ultra? Even the largest open model to date (Kimi K3) is nowhere close to Astra level, and it will be quite slow even on the highest-spec M5 Ultra, with achievable speeds of about 0.5 tok/s at most due to having to stream weights from SSD (~13 GB/s on the highest storage capacity M5 Max machines so far). This is OK for doing simple Q&A in the background but it's far from a genuine coding experience. You'd have to test batching of multiple thinking streams in order to try and raise overall tok/s via layer-wise reuse of the streamed weights (and this is where the "Ultra" part sort of becomes relevant; Kimi series models have good support for agent swarms) but this would decrease single-session performance even further. It would only be usable for background jobs, though the hardware would then have a chance of paying for itself if it was fully used on a 24/7 basis.

srcreigh 7d ago
> You'd have to test batching of multiple thinking streams in order to try and raise overall tok/s via layer-wise reuse of the streamed weights isn’t this very straightforward to do..? I thought batching for Qwen models is already proven out. > but this would decrease single-session performance even further Well let’s take Qwen 3.8 27B. Throughput for M3 at 8 agents is 4x compared to single agent. [1] It’s really not clear to me that 8 concurrent agents at half speed will be worse task completion latency than 1 agent. And that’s M3 studio benchmarks, not even M5 ultra, and without the many software improvements we will see If you haven’t tried Qwen 3.8 27B xhigh on a task you might not get the hype. Idk. If you’ve tried doing this and don’t like it sure, and be specific about what isn’t effective, but let’s not speculate. [1]: https://omlx.ai/benchmarks/performance/69kzkrv8?utm_source=chatgpt.com https://omlx.ai/benchmarks/performance/69kzkrv8?utm_source=c...