3 ms·
You got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.
by adastra22 5mo ago
You got numbers? Because it seems perfectly possible to me. OpenAI and Anthropic’s marginal cost for inference is certainly far less than their API pricing.
- ai_fry_ur_brain 5mo agoHow can you say that with such certainty? You have no idea what it costs to run a 10T parameter model at extremely high concurrency. These 1T param models running at <$3.00 per 1mm are certainly not profitable.
- adastra22 5mo agoBecause I’ve looked at what it would cost my company to self-host a SOTA sized model. For us it wasn’t worth it because the hardware is all bought up by frontier labs and we can’t get any supply. But if we could, at the prices they’re paying, it would pay for itself in 10-ish months. I assume further that they have economies of scale on top of what I was estimating.
- callmeal 5mo agoSee: https://www.wheresyoured.at/ https://www.wheresyoured.at/ He's been "numbering" for quite a while now.
- tempay 5mo agoEverything there is extremely speculative and I don't see anything that contradicts that inference itself could be profitable at massive scale. See https://youtu.be/xmkSf5IS-zw https://youtu.be/xmkSf5IS-zw for example. If the companies as a whole are destined to be profitable, or worth their valuations is a very different question. The only people who can truely answer that have time machines.