4 ms·

It seems like every 6 months they come out with a new model that's as good as the previous generation but with a fraction of the inference cost. Inference cost

It seems like every 6 months they come out with a new model that's as good as the previous generation but with a fraction of the inference cost. Inference cost for a given level of quality has been dropping fast. E.g. GPT-4.1 nano outperforms the old GPT-4 model but costs 300x less. Right now, the API cost for asking a single question costs about a fiftieth of a cent on their cheap 4.1-nano model, up to about 2 cents on their o3 model. This is pretty affordable. On the other end of the spectrum, if you're maxing out the context window on o3 or 4.1, it'll cost you around $0.50 to $2. Pricy, but on 4.1, this is like inputting several novels worth of text (maybe around 2000 pages).

janalsncm 1y ago
That would be great news for OpenAI if they could somehow prevent people from buying their own computers. Because as inference costs come down for OpenAI, their customers also get access to better and better commodity hardware (and also better models to run on it). And commodity models become more and more capable all the time, even if they’re not the best.
- nearbuy 1y ago
  It counters the claim that "inference is too damn expensive". You can argue that cheap inference is actually bad for OpenAI because it makes it easier to run models on commodity hardware, but then you're arguing against your point that inference was too expensive for OpenAI to be sustainable. Which is it? Is inference too expensive or too cheap? For now, OpenAI's models are closed source, so if you find their models offer the best value for your use case, you don't have the option of running it on your own hardware. If a competitor releases better products for cheaper, OpenAI will fail, just like any other company would.
  - janalsncm 1y ago
    It’s both. For an increasing number of tasks, it’s too cheap. People will be able to do simple things on their own hardware. Things OpenAI would love to charge high margins on. There’s probably an 80/20 rule on the horizon. And for others, it’s too expensive. The frontier is constantly being pushed, so they can’t stop improving or they will fall behind. Google at least makes their own chips so they can control their costs somewhat.
    - disgruntledphd2 1y ago
      Honestly, it's the training costs that will kill them. AFAIK, training costs have not come down anywhere near as much as inference costs. And the models don't last long. So you have a rapidly depreciating capital asset that you need to provide your services, not really a recipe for a sustainable business (certainly not with the fat software margins tech companies are used to).
    - nearbuy 1y ago
      So you suspect their cheap models have few customers because people prefer to run open source models on their own computers, and their high-end models have either very thin margins over the inference cost or few customers because the costs are too high. Do you have any evidence for any of this?
yorwba 1y ago
If you're looking at it from the perspective of business sustainability (i.e. what the article is about) "we keep lowering our prices" doesn't sound so great. The question is whether GPT-4.1 nano costs OpenAI 300x less than GPT-4 to run or not. If it costs exactly that much less, that still means demand needs to grow by more than 300x just to keep revenue constant. And if it does, then total inference cost correspondingly goes up again.
- nearbuy 1y ago
  I think that's convoluted logic trying to show that a business lowering its costs is actually bad for the business. You can make this assumptions about any business. It's not a good assumption. If a car company discovered a way to lower its production costs and started selling their cars cheaper, would you argue that maybe this is actually bad for the car company because maybe they lowered their price so much that it cancelled out all the benefit? You're just hoping they're really bad at pricing their cars. The different GPT models are for different use cases. The existence of the nano model does not imply everyone who used the formerly cutting-edge GPT-4 will switch over to the cheaper nano model. Most of them will switch to the smarter models like 4o, 4.1, or o3. Nano allows the creation of new tools where the other models are either too pricey or too slow to respond to be viable.
  - yorwba 1y ago
    It's funny that you bring up cars as a point of comparison, because constantly releasing new models while dropping prices is exactly what made a bunch of Chinese EV companies go bankrupt in recent years. It's not that they were stupid and priced their cars badly, it's that they didn't have much choice. Developing a new car model has significant upfront costs that require a correspondingly large number of cars to be sold for the company as a whole to turn a profit, each individual sale being profitable is not enough. But in an environment with lots of competitors constantly releasing new models for cheaper, any single company had little choice but to also release a new model and lower their prices in order to sell any cars at all, eating a loss on the previous model. And eventually some companies couldn't take those losses any more and folded. OpenAI is definitely losing money overall, presumably because training new large language models is expensive. But can they stop doing that to turn a profit? If OpenAI announced that the funding for their next frontier training run fell through and they're laying off research staff to focus on inference, and then competitors come out with better, cheaper models, how long will OpenAI be able to stay relevant? I guess they hope everyone else gives up first and they won't have to find out.

janalsncm 1y ago

That would be great news for OpenAI if they could somehow prevent people from buying their own computers. Because as inference costs come down for OpenAI, their customers also get access to better and better commodity hardware (and also better models to run on it). And commodity models become more and more capable all the time, even if they’re not the best.

nearbuy 1y ago
It counters the claim that "inference is too damn expensive". You can argue that cheap inference is actually bad for OpenAI because it makes it easier to run models on commodity hardware, but then you're arguing against your point that inference was too expensive for OpenAI to be sustainable. Which is it? Is inference too expensive or too cheap? For now, OpenAI's models are closed source, so if you find their models offer the best value for your use case, you don't have the option of running it on your own hardware. If a competitor releases better products for cheaper, OpenAI will fail, just like any other company would.
- janalsncm 1y ago
  It’s both. For an increasing number of tasks, it’s too cheap. People will be able to do simple things on their own hardware. Things OpenAI would love to charge high margins on. There’s probably an 80/20 rule on the horizon. And for others, it’s too expensive. The frontier is constantly being pushed, so they can’t stop improving or they will fall behind. Google at least makes their own chips so they can control their costs somewhat.
  - disgruntledphd2 1y ago
    Honestly, it's the training costs that will kill them. AFAIK, training costs have not come down anywhere near as much as inference costs. And the models don't last long. So you have a rapidly depreciating capital asset that you need to provide your services, not really a recipe for a sustainable business (certainly not with the fat software margins tech companies are used to).
  - nearbuy 1y ago
    So you suspect their cheap models have few customers because people prefer to run open source models on their own computers, and their high-end models have either very thin margins over the inference cost or few customers because the costs are too high. Do you have any evidence for any of this?

yorwba 1y ago

If you're looking at it from the perspective of business sustainability (i.e. what the article is about) "we keep lowering our prices" doesn't sound so great. The question is whether GPT-4.1 nano costs OpenAI 300x less than GPT-4 to run or not. If it costs exactly that much less, that still means demand needs to grow by more than 300x just to keep revenue constant. And if it does, then total inference cost correspondingly goes up again.

nearbuy 1y ago
I think that's convoluted logic trying to show that a business lowering its costs is actually bad for the business. You can make this assumptions about any business. It's not a good assumption. If a car company discovered a way to lower its production costs and started selling their cars cheaper, would you argue that maybe this is actually bad for the car company because maybe they lowered their price so much that it cancelled out all the benefit? You're just hoping they're really bad at pricing their cars. The different GPT models are for different use cases. The existence of the nano model does not imply everyone who used the formerly cutting-edge GPT-4 will switch over to the cheaper nano model. Most of them will switch to the smarter models like 4o, 4.1, or o3. Nano allows the creation of new tools where the other models are either too pricey or too slow to respond to be viable.
- yorwba 1y ago
  It's funny that you bring up cars as a point of comparison, because constantly releasing new models while dropping prices is exactly what made a bunch of Chinese EV companies go bankrupt in recent years. It's not that they were stupid and priced their cars badly, it's that they didn't have much choice. Developing a new car model has significant upfront costs that require a correspondingly large number of cars to be sold for the company as a whole to turn a profit, each individual sale being profitable is not enough. But in an environment with lots of competitors constantly releasing new models for cheaper, any single company had little choice but to also release a new model and lower their prices in order to sell any cars at all, eating a loss on the previous model. And eventually some companies couldn't take those losses any more and folded. OpenAI is definitely losing money overall, presumably because training new large language models is expensive. But can they stop doing that to turn a profit? If OpenAI announced that the funding for their next frontier training run fell through and they're laying off research staff to focus on inference, and then competitors come out with better, cheaper models, how long will OpenAI be able to stay relevant? I guess they hope everyone else gives up first and they won't have to find out.