3 ms·
Perhaps this is a helpful model, rather than worrying about the "billions spent" and whether its inference vs training. How much would it cost you to deploy a
by program_whiz 9mo ago
Perhaps this is a helpful model, rather than worrying about the "billions spent" and whether its inference vs training.
How much would it cost you to deploy a model that you and maybe a few coworkers could effectively use? $400k probably to buy all the hardware required to host a top-tier model that could do a few hundred tokens per second for 10 concurrent users? That's $40k per person. Ammortize the hardware over 5 years, thats $8k per person per year (roughly), with no training costs (that's just you buying hardware and running it yourself). So that means, you need ~$800 per user monthly just to cover hardware to run the model (this is with no staffing costs, internet, taxes, electricity, hosting, housing, etc).
So just food for thought, but $200 claude code is probably still losing money even just on inference.
Since they are in the software realm, they are probably shooting for a 90% profit margin. Using the above example, it would be ($800 + R&D + opex) x 10. My guess is assuming no more training (which probably can never be profitable at current rates), they need $20k per month per user, which is why that number was floated by OpenAI previously.
- Blemiono 9mo agoA RTX 6000 pro costs 9k. You can run good models with 96gb of memory. It can easily serve 10 people or more depending on the overall usage pattern (coding vs everything else). So now imagine Hardware getting better every year, models getting better too and everything overall gets more efficient. M1 vs M4 apple increased performance by 100% in 4 years And there are inferencing optimized chips like groq. Don't forget kv cache and overall optimizations. I think your math is off. And ai is already better than interns. An intern costs you at least 1k per month probably 2k. For me the math works just fine.
- estimator7292 9mo agoAfter a hypothetical AI crash, the cost of hardware will plummet. It will suddenly become quite affordable to spin up a GPU or five on-prem to host a couple of models for internal use. The only reason hardware is so expensive now is to scalp the hyperscalers. Once that demand crashes, the supply will skyrocket and prices will crash.
- program_whiz 8mo agoFair point -- but my overall point about how much users may have to pay to make these companies profitable stands. Maybe if prices stay depressed for years, but these companies are doing buildout at current prices, and they need to make returns on hardware they are buying now. I suppose they could bank on prices coming down in 2 to 3 years by a factor of 10, then current price ($200 per month) might be profitable (disregarding training, employees, power, etc).