5 ms·

> doesn't make financial sense to self-host I guess that's debatable. I regularly run out of quota on my claude max subscription. When that happens, I can sort

> doesn't make financial sense to self-host I guess that's debatable. I regularly run out of quota on my claude max subscription. When that happens, I can sort of kind of get by with my modest setup (2x RTX3090) and quantized Qwen3. And this does not even account for privacy and availability. I'm in Canada, and as the US is slowly consumed by its spiral of self-destruction, I fully expect at some point a digital iron curtain will go up. I think it's prudent to have alternatives, especially with these paradigm-shattering tools.

deleted 8mo ago
[deleted]
jsheard 8mo ago
I think AI may be the only place you could get away with calling a 2x350W GPU rig "modest". That's like ten normal computers worth of power for the GPUs alone.
- kataklasm 8mo ago
  Did you even try to read and understand the parent comment? They said they regularly run out of quota on the exact subscription you're advising they subscribe to.
  - h3half 8mo ago
    Pot, kettle
- dymk 8mo ago
  That's maybe a few dollars to tens of dollars in electricity per month depending on where in the US you live
  - GeorgeOldfield 8mo ago
    the upfront cost
- bigyabai 8mo ago
  > That's like ten normal computers worth of power for the GPUs alone. Maybe if your "computer" in question is a smartphone? Remember that the M3 Ultra is a 300w+ chip that won't beat one of those 3090s in compute or raster efficiency.
  - jsheard 8mo ago
    I wouldn't class the M3 Ultra as a "normal" computer either. That's a big-ass workstation. I was thinking along the lines of a typical Macbook or Mac Mini or Windows laptop, which are fine for 99% of anyone who isn't looking to play games or run gigantic AI models locally.
    - bigyabai 8mo ago
      Those aren't "normal" computers, either. They're iPad chips running in the TDP envelope of a tablet, usually with iPad-level performance to match.
flaviolivolsi 8mo ago
How does your quantized Qwen3 compares in code quality to Opus?
- Aurornis 8mo ago
  Not the person you’re responding to, but my experience with models up through Qwen3-coder-next is that they’re not even close. They can do a lot of simple tasks in common frameworks well. Doing anything beyond basic work will just burn tokens for hours while you review and reject code.
- btbuildem 8mo ago
  It's just as fast, but not nearly as clever. I can push the context size to 120k locally, but quality of the work it delivers starts to falter above say 40k. Generally you have to feed it more bite-sized pieces, and keep one chat to one topic. It's definitely a step down from SOTA.
mythz 8mo ago
Did the napkin math on M3 Ultra ROI when DeepSeek V3 launched: at $0.70/2M tokens and 30 tps, a $10K M3 Ultra would take ~30 years of non-stop inference to break even - without even factoring in electricity. Clearly people aren't self-hosting to save money. I've got a lite GLM sub $72/yr which would require 138 years to burn through the $10K M3 Ultra sticker price. Even GLM's highest cost Max tier (20x lite) at $720/yr would buy you ~14 years.
- retr0rocket 8mo ago
  [dead]
- wongarsu 8mo ago
  And it's worth noting that you can get DeepSeek at those prices from DeepSeek (Chinese), DeepInfra (US with Bulgarian founder), NovitaAI (US), AtlasCloud (US with Chinese founder), ParaSail (US), etc. There is no shortage of companies offering inference, with varying levels of trustworthiness, certificates and promises around (lack of) data retention. You just have to pick one you trust
- oceanplexian 8mo ago
  Doing inference with a Mac Mini to save money is more or less holding it wrong. Of course if you buy some overpriced Apple hardware it’s going to take years to break even. Buy a couple real GPUs and do tensor parallelism and concurrent batch requests with vllm and it becomes extremely cost competitive to run your own hardware.
  - mythz 8mo ago
    > Doing inference with a Mac Mini to save money is more or less holding it wrong. No one's running these large models on a Mac Mini. > Of course if you buy some overpriced Apple hardware it’s going to take years to break even. Great, where can I find cheaper hardware that can run GLM 5's 745B or Kimi K2.5 1T models? Currently it requires 2x M3 Ultras (1TB VRAM) to run Kimi K2.5 at 24 tok/s [1] What are the better value alternatives? [1] https://x.com/alexocheema/status/2016404573917683754 https://x.com/alexocheema/status/2016404573917683754
wongarsu 8mo ago
Self-hosting training (or gaming) makes a lot of sense, and once you have the hardware self-hosting inference on it is an easy step. But if you have to factor in hardware costs self-hosting doesn't seem attractive. All the models I can self-host I can browse on openrouter and instantly get a provider who can get great prices. With most of the cost being in the GPUs themselves it just makes more sense to have others do it with better batching and GPU utilization
- zozbot234 8mo ago
  If you can get near 100% utilization for your own GPUs (i.e. you're letting requests run overnight and not insisting on any kind of realtime response) it starts to make sense. OpenRouter doesn't have any kind of batched requests API that would let you leverage that possibility.
  - sowbug 8mo ago
    In Silicon Valley we pay PG&E close to 50 cents per kWh. An RTX 6000 PC uses about 1 kW at full load, and renting such a machine from vast.ai costs 60 cents/hour as of this morning. It's very hard for heavy-load local AI to make sense here.
    - Imustaskforhelp 8mo ago
      And you are forgetting the fact that things like vast.ai subscriptions would STILL be more expensive than Openrouter's api pricing and even more so in the case of AI subscriptions which actively LOSE money for the company. So I would still point out the GP (Original comment) where yes, it might not make financial sense to run these AI Models [They make sense when you want privacy etc, which are all fair concerns but just not financial sense] But the fact that these models are open source still means that they can be run when maybe in future the dynamics might shift and it might make sense running such large models locally. Even just giving this possibility and also the fact that multiple providers could now compete in say openrouter etc. as well. All facts included, definitely makes me appreciate GLM & Kimi compared to proprietory counterparts. Edit: I highly recommend this video a lot https://www.youtube.com/watch?v=SmYNK0kqaDI https://www.youtube.com/watch?v=SmYNK0kqaDI [AI subscription vs H100] This video is honestly one of the best in my opinion about this topic that I watched.
7thpower 8mo ago
Unless you already had those cards, it probably still doesn’t make sense from a purely financial perspective unless you have other things you’re discounting for. Doesn’t mean you shouldn’t do it though.
visarga 8mo ago
Your $5,000 PC with 2 GPUs could have bought you 2 years of Claude Max, a model much more powerful and with longer context. In 2 years you could make that investment back in pay raise.
- dymk 8mo ago
  This claim has so many assumptions mixed in it's utterly useless
- benterix 8mo ago
  > In 2 years you could make that investment back in pay raise. Could you elaborate? I fail to grasp the implication here.
- tw1984 8mo ago
  > In 2 years you could make that investment back in pay raise. you can't be a happy uber driver making more money in the next 24 months by having a fancy car fitted with the best FSD in town when all cars in your town have the same FSD.
  - visarga 8mo ago
    But they don't have the same human in the loop though.
    - tw1984 8mo ago
      that software is called autonomous agents, the term autonomous has nothing to do with human in the loop, it is the complete opposite.
      - drdaeman 8mo ago
        Nothing changed since ’87. Machines still can’t be accountable and still shouldn’t make managerial decisions. Acceptance control is one of those decisions, and all the technical knowledge still matters to form a well-informed one. It may change, of course, but I have an impression that those who try otherwise seem to not fare well after the initial vibecoding honeymoon period. Of course, it varies from case to case - sometimes machines get things right, but long-term luck seems to eventually run out.
Aurornis 8mo ago
> I regularly run out of quota on my claude max subscription. When that happens, I can sort of kind of get by with my modest setup (2x RTX3090) and quantized Qwen3. When talking about fallback from Claude plans, The correct financial comparison would be the same model hosted on OpenRouter. You could buy a lot of tokens for the price of a pair of 3090s and a machine to run them.
- bigyabai 8mo ago
  > You could buy a lot of tokens for the price of a pair of 3090s and a machine to run them. That's a subjective opinion, to which the answer is "no you can't" for many people.
int_19h 8mo ago
Anthropic has very tight limits, so you're basically using the worst (pricing-wise) SOTA cloud model as your baseline. I have $200 subs for both Claude and OpenAI, and I also bump into limits with Claude all the time, whether coding or research. With Codex, I ran into the limit once so far, and that's in a month of very heavy (sometimes literally 24 hours around the clock, leaving long-running tasks overnight) use.
- sheepscreek 8mo ago
  I bought the Gemini Ultra to try for a month (at the discounted price). I have been using it non-stop for Opus 4.6 Thinking, which is much better than Gemini 3 Pro (High) and it's been a blast. The most I've managed to consume is 60% of my 5 hourly quota. That was with 2-3 instances in parallel. I hope too many of us won't be doing this and cause Google to add limits! My hope is Google sees the benefit in this and goes all in - continues to let people decide which Google hosted model to use, including their own.
  - doctoboggan 8mo ago
    Can you use the models you get through Gemini Ultra in Claude Code? If not, what coding tool do you use?
    - meeq 8mo ago
      Not OP, but I am pretty sure they are using Opencode with a certain antigravity plugin. Not going to link it, since it technically allows breaking TOS. If you‘re not using Opencode yet, I wholeheartedly recommend the switch.
    - rnewme 8mo ago
      Claude code router
    - btbuildem 8mo ago
      Getting CC to work with other models is quite straightforward -- setting a few env vars, and a thin proxy that rewrites the requests/responses to be in the expected format.

deleted 8mo ago

[deleted]

jsheard 8mo ago

I think AI may be the only place you could get away with calling a 2x350W GPU rig "modest". That's like ten normal computers worth of power for the GPUs alone.

kataklasm 8mo ago
Did you even try to read and understand the parent comment? They said they regularly run out of quota on the exact subscription you're advising they subscribe to.
- h3half 8mo ago
  Pot, kettle
dymk 8mo ago
That's maybe a few dollars to tens of dollars in electricity per month depending on where in the US you live
- GeorgeOldfield 8mo ago
  the upfront cost
bigyabai 8mo ago
> That's like ten normal computers worth of power for the GPUs alone. Maybe if your "computer" in question is a smartphone? Remember that the M3 Ultra is a 300w+ chip that won't beat one of those 3090s in compute or raster efficiency.
- jsheard 8mo ago
  I wouldn't class the M3 Ultra as a "normal" computer either. That's a big-ass workstation. I was thinking along the lines of a typical Macbook or Mac Mini or Windows laptop, which are fine for 99% of anyone who isn't looking to play games or run gigantic AI models locally.
  - bigyabai 8mo ago
    Those aren't "normal" computers, either. They're iPad chips running in the TDP envelope of a tablet, usually with iPad-level performance to match.

flaviolivolsi 8mo ago

How does your quantized Qwen3 compares in code quality to Opus?

Aurornis 8mo ago
Not the person you’re responding to, but my experience with models up through Qwen3-coder-next is that they’re not even close. They can do a lot of simple tasks in common frameworks well. Doing anything beyond basic work will just burn tokens for hours while you review and reject code.
btbuildem 8mo ago
It's just as fast, but not nearly as clever. I can push the context size to 120k locally, but quality of the work it delivers starts to falter above say 40k. Generally you have to feed it more bite-sized pieces, and keep one chat to one topic. It's definitely a step down from SOTA.

mythz 8mo ago

Did the napkin math on M3 Ultra ROI when DeepSeek V3 launched: at $0.70/2M tokens and 30 tps, a $10K M3 Ultra would take ~30 years of non-stop inference to break even - without even factoring in electricity. Clearly people aren't self-hosting to save money. I've got a lite GLM sub $72/yr which would require 138 years to burn through the $10K M3 Ultra sticker price. Even GLM's highest cost Max tier (20x lite) at $720/yr would buy you ~14 years.

retr0rocket 8mo ago
[dead]
wongarsu 8mo ago
And it's worth noting that you can get DeepSeek at those prices from DeepSeek (Chinese), DeepInfra (US with Bulgarian founder), NovitaAI (US), AtlasCloud (US with Chinese founder), ParaSail (US), etc. There is no shortage of companies offering inference, with varying levels of trustworthiness, certificates and promises around (lack of) data retention. You just have to pick one you trust
oceanplexian 8mo ago
Doing inference with a Mac Mini to save money is more or less holding it wrong. Of course if you buy some overpriced Apple hardware it’s going to take years to break even. Buy a couple real GPUs and do tensor parallelism and concurrent batch requests with vllm and it becomes extremely cost competitive to run your own hardware.
- mythz 8mo ago
  > Doing inference with a Mac Mini to save money is more or less holding it wrong. No one's running these large models on a Mac Mini. > Of course if you buy some overpriced Apple hardware it’s going to take years to break even. Great, where can I find cheaper hardware that can run GLM 5's 745B or Kimi K2.5 1T models? Currently it requires 2x M3 Ultras (1TB VRAM) to run Kimi K2.5 at 24 tok/s [1] What are the better value alternatives? [1] https://x.com/alexocheema/status/2016404573917683754 https://x.com/alexocheema/status/2016404573917683754

wongarsu 8mo ago

Self-hosting training (or gaming) makes a lot of sense, and once you have the hardware self-hosting inference on it is an easy step. But if you have to factor in hardware costs self-hosting doesn't seem attractive. All the models I can self-host I can browse on openrouter and instantly get a provider who can get great prices. With most of the cost being in the GPUs themselves it just makes more sense to have others do it with better batching and GPU utilization

zozbot234 8mo ago
If you can get near 100% utilization for your own GPUs (i.e. you're letting requests run overnight and not insisting on any kind of realtime response) it starts to make sense. OpenRouter doesn't have any kind of batched requests API that would let you leverage that possibility.
- sowbug 8mo ago
  In Silicon Valley we pay PG&E close to 50 cents per kWh. An RTX 6000 PC uses about 1 kW at full load, and renting such a machine from vast.ai costs 60 cents/hour as of this morning. It's very hard for heavy-load local AI to make sense here.
  - Imustaskforhelp 8mo ago
    And you are forgetting the fact that things like vast.ai subscriptions would STILL be more expensive than Openrouter's api pricing and even more so in the case of AI subscriptions which actively LOSE money for the company. So I would still point out the GP (Original comment) where yes, it might not make financial sense to run these AI Models [They make sense when you want privacy etc, which are all fair concerns but just not financial sense] But the fact that these models are open source still means that they can be run when maybe in future the dynamics might shift and it might make sense running such large models locally. Even just giving this possibility and also the fact that multiple providers could now compete in say openrouter etc. as well. All facts included, definitely makes me appreciate GLM & Kimi compared to proprietory counterparts. Edit: I highly recommend this video a lot https://www.youtube.com/watch?v=SmYNK0kqaDI https://www.youtube.com/watch?v=SmYNK0kqaDI [AI subscription vs H100] This video is honestly one of the best in my opinion about this topic that I watched.

7thpower 8mo ago

Unless you already had those cards, it probably still doesn’t make sense from a purely financial perspective unless you have other things you’re discounting for. Doesn’t mean you shouldn’t do it though.

visarga 8mo ago

Your $5,000 PC with 2 GPUs could have bought you 2 years of Claude Max, a model much more powerful and with longer context. In 2 years you could make that investment back in pay raise.

dymk 8mo ago
This claim has so many assumptions mixed in it's utterly useless
benterix 8mo ago
> In 2 years you could make that investment back in pay raise. Could you elaborate? I fail to grasp the implication here.
tw1984 8mo ago
> In 2 years you could make that investment back in pay raise. you can't be a happy uber driver making more money in the next 24 months by having a fancy car fitted with the best FSD in town when all cars in your town have the same FSD.
- visarga 8mo ago
  But they don't have the same human in the loop though.
  - tw1984 8mo ago
    that software is called autonomous agents, the term autonomous has nothing to do with human in the loop, it is the complete opposite.
    - drdaeman 8mo ago
      Nothing changed since ’87. Machines still can’t be accountable and still shouldn’t make managerial decisions. Acceptance control is one of those decisions, and all the technical knowledge still matters to form a well-informed one. It may change, of course, but I have an impression that those who try otherwise seem to not fare well after the initial vibecoding honeymoon period. Of course, it varies from case to case - sometimes machines get things right, but long-term luck seems to eventually run out.

Aurornis 8mo ago

> I regularly run out of quota on my claude max subscription. When that happens, I can sort of kind of get by with my modest setup (2x RTX3090) and quantized Qwen3. When talking about fallback from Claude plans, The correct financial comparison would be the same model hosted on OpenRouter. You could buy a lot of tokens for the price of a pair of 3090s and a machine to run them.

bigyabai 8mo ago
> You could buy a lot of tokens for the price of a pair of 3090s and a machine to run them. That's a subjective opinion, to which the answer is "no you can't" for many people.

int_19h 8mo ago

Anthropic has very tight limits, so you're basically using the worst (pricing-wise) SOTA cloud model as your baseline. I have $200 subs for both Claude and OpenAI, and I also bump into limits with Claude all the time, whether coding or research. With Codex, I ran into the limit once so far, and that's in a month of very heavy (sometimes literally 24 hours around the clock, leaving long-running tasks overnight) use.

sheepscreek 8mo ago
I bought the Gemini Ultra to try for a month (at the discounted price). I have been using it non-stop for Opus 4.6 Thinking, which is much better than Gemini 3 Pro (High) and it's been a blast. The most I've managed to consume is 60% of my 5 hourly quota. That was with 2-3 instances in parallel. I hope too many of us won't be doing this and cause Google to add limits! My hope is Google sees the benefit in this and goes all in - continues to let people decide which Google hosted model to use, including their own.
- doctoboggan 8mo ago
  Can you use the models you get through Gemini Ultra in Claude Code? If not, what coding tool do you use?
  - meeq 8mo ago
    Not OP, but I am pretty sure they are using Opencode with a certain antigravity plugin. Not going to link it, since it technically allows breaking TOS. If you‘re not using Opencode yet, I wholeheartedly recommend the switch.
  - rnewme 8mo ago
    Claude code router
  - btbuildem 8mo ago
    Getting CC to work with other models is quite straightforward -- setting a few env vars, and a thin proxy that rewrites the requests/responses to be in the expected format.