4 ms·

Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly s

by paxys 2mo ago

Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly subsidized right now.

monatron 2mo ago
I think your last point is exactly why I'm so interested in local models. The current landscape doesn't feel sustainable. The last few months we've seen the big providers (OpenAI, Anthropic) start to play with usage limits, resets, banked resets, pulling models, etc. I think local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference. It is nice to know that I have hardware under my desk that I control with open weight models that I can interact with on my terms.
- julianlam 2mo ago
  > local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference In my opinion, 98% of the work most devs would send to an AI can be capably achieved with a local model and a frontier-level model is overkill. The goalpost moving feeds right into Anthropic and OpenAI's interests.
- htrp 2mo ago
  yes, you have to unfortunately adopt a 2025 mindset when working with smaller local models because that's where they are compared to the frontier.
- polnoner 2mo ago
  It is certainly not sustainable but that is exactly why I want to use $400-$800 of resources a month for $20 while the deal last. It is really a once in a lifetime deal. Once the deal is over the local models will be better than what I am using now anyway and the hardware will be all the better than what I can get now for the price.
deleted 2mo ago
[deleted]
apparent 2mo ago
> AI subscriptions are too highly subsidized right now I've been running into annoying limits with Claude recently. It gives me like 5 questions over the course of 15 mins and then tells me to wait 5 hours. When companies can change things up to make the base subscription nearly useless (the last question always gets messed up, too), then you realize the value of owning your own infrastructure.
- adastra22 2mo ago
  On the $200/mo plan I have never hit a five hour limit, and I struggle to use my full credits each week. $200/month is vastly cheaper than owning and operating comparable hardware.
  - apparent 2mo ago
    You're right that $200/mo is much cheaper than comparable infrastructure. OTOH, you don't get to have a computer that can also be used for other applications, or which works when the internet is down. Also, you remain tethered to whatever pricing the AI companies want to charge. If AI pricing goes like Uber/Lyft did when the VC cash ran out, then we'll be paying much more in a few years' time. We could look back and think "I wish I'd bought my own setup back in 2026" if it's going to be inevitable. And all this before you get into privacy/security/compliance stuff.
    - wolvoleo 2mo ago
      Ohhh trust me we're going to pay a hell of a lot more. It'll be nothing like Uber's VC running out. Someone has to be make back all those trillions. What we're paying now is just akin to the drug dealer's free starting shot.
  - 0x457 2mo ago
    > On the $200/mo plan I have never hit a five hour limit, and I struggle to use my full credits each week. I started tasking fable with huge projects over the weekend and now I hit at very least fable limit by monday.
nowittyusername 1mo ago
Yep, main reason I ordered my Mac studio was because of privacy and control. I know no one will see the work being done on that machine and also the model will never be silently swapped out for a different lower quality model like anthropic does or a lower quant like openAI does with their models. Also having the inference engine has huge benefits because of speculative re-generation capabilities for voice agents. Anyways people who care about cost are simply bad at math if they think thell be saving any money running locally versus cloud providers.

by paxys 2mo ago

Local setups aren't going to make sense purely from a cost perspective, and definitely not when you are buying Apple hardware. AI subscriptions are too highly subsidized right now.

monatron 2mo ago

I think your last point is exactly why I'm so interested in local models. The current landscape doesn't feel sustainable. The last few months we've seen the big providers (OpenAI, Anthropic) start to play with usage limits, resets, banked resets, pulling models, etc. I think local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference. It is nice to know that I have hardware under my desk that I control with open weight models that I can interact with on my terms.

julianlam 2mo ago
> local models are close to the point where, with a sufficiently well-architected harness, you can get results that are on par with the experience you'd have with cloud inference In my opinion, 98% of the work most devs would send to an AI can be capably achieved with a local model and a frontier-level model is overkill. The goalpost moving feeds right into Anthropic and OpenAI's interests.
htrp 2mo ago
yes, you have to unfortunately adopt a 2025 mindset when working with smaller local models because that's where they are compared to the frontier.
polnoner 2mo ago
It is certainly not sustainable but that is exactly why I want to use $400-$800 of resources a month for $20 while the deal last. It is really a once in a lifetime deal. Once the deal is over the local models will be better than what I am using now anyway and the hardware will be all the better than what I can get now for the price.

deleted 2mo ago

[deleted]

apparent 2mo ago

> AI subscriptions are too highly subsidized right now I've been running into annoying limits with Claude recently. It gives me like 5 questions over the course of 15 mins and then tells me to wait 5 hours. When companies can change things up to make the base subscription nearly useless (the last question always gets messed up, too), then you realize the value of owning your own infrastructure.

adastra22 2mo ago
On the $200/mo plan I have never hit a five hour limit, and I struggle to use my full credits each week. $200/month is vastly cheaper than owning and operating comparable hardware.
- apparent 2mo ago
  You're right that $200/mo is much cheaper than comparable infrastructure. OTOH, you don't get to have a computer that can also be used for other applications, or which works when the internet is down. Also, you remain tethered to whatever pricing the AI companies want to charge. If AI pricing goes like Uber/Lyft did when the VC cash ran out, then we'll be paying much more in a few years' time. We could look back and think "I wish I'd bought my own setup back in 2026" if it's going to be inevitable. And all this before you get into privacy/security/compliance stuff.
  - wolvoleo 2mo ago
    Ohhh trust me we're going to pay a hell of a lot more. It'll be nothing like Uber's VC running out. Someone has to be make back all those trillions. What we're paying now is just akin to the drug dealer's free starting shot.
- 0x457 2mo ago
  > On the $200/mo plan I have never hit a five hour limit, and I struggle to use my full credits each week. I started tasking fable with huge projects over the weekend and now I hit at very least fable limit by monday.

nowittyusername 1mo ago

Yep, main reason I ordered my Mac studio was because of privacy and control. I know no one will see the work being done on that machine and also the model will never be silently swapped out for a different lower quality model like anthropic does or a lower quant like openAI does with their models. Also having the inference engine has huge benefits because of speculative re-generation capabilities for voice agents. Anyways people who care about cost are simply bad at math if they think thell be saving any money running locally versus cloud providers.