4 ms·

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and

Which is like saying, “If all we did is charge people money and didn’t have any COGS, we’d be a very profitable company.” That’s a truism of every business and therefore basically meaningless.

deleted 1y ago
[deleted]
dcre 1y ago
The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.
- prasadjoglekar 1y ago
  Well, only if the one training model continued to function as a going business. Their amortization window for the training cost is 2 months or so. They can't just keep that up and collect $. They have to build the next model, or else people will go to someone else.
  - dcre 1y ago
    Why two months? It was almost a year between Claude 3.5 and 4. (Not sure how much it costs to go from 3.5 to 3.7.)
    - Jalad 1y ago
      Even being generous, and saying it's a year, most capital expenditures depreciate over a period of 5-7 years. To state the obvious, training one model a year is not a saving grace
      - dcre 1y ago
        I don't understand why the absolute time period matters — all that matters is that you get enough time making money on inference to make up for the cost of training.
    - oblio 1y ago
      Don't they need to accelerate that, though? Having a 1 year old model isn't really great, it's just tolerable.
      - dcre 1y ago
        I think this is debatable as more models become good enough for more tasks. Maybe a smaller proportion of tasks will require SOTA models. On the other hand, the set of tasks people want to use LLMs for will expand along with the capabilities of SOTA models.
- ToucanLoucan 1y ago
  So is OpenAI capable of not making a new model at some point? They've been training the next model continuously as long as they've existed AFAIK. Our software house spends a lot on R&D sure, but we're still incredibly profitable all the same. If OpenAI is in a position where they effectively have to stop iterating the product to be profitable, I wouldn't call that a very good place to be when you're on the verge of having several hundred billion in debt.
  - dcre 1y ago
    I think at that point there is strong financial pressure to figure out how to continuously evolve models instead of changing new ones, for example by building models out of smaller modules that can be trained individually and swapped out. Jeff Dean and Noam Shazeer talked about that a bit in their interview with Dwarkesh: https://www.dwarkesh.com/p/jeff-dean-and-noam-shazeer https://www.dwarkesh.com/p/jeff-dean-and-noam-shazeer
  - DenisM 1y ago
    There’s still untapped value in deeper integrations. They might hit a jackpot of exponentially increasing value from network effects caused by tight integration with e.g. disjoint business processes. We know that businesses with tight network effects can grow to about 2 trillion in valuation.
    - oblio 1y ago
      How would that look with at least 3 US companies, probably 2 Chinese ones and at least 1 European company developing state of the art LLMs?
      - drob518 1y ago
        Like a very over-served market, I think. I see perhaps three survivors long term, or at most one gorilla, two chimps, and perhaps a few very small niche-focused monkeys.
      - DenisM 1y ago
- drob518 1y ago
  So,if they stopped training they’d be profitable? Only in some incremental sense, ignoring all sunk costs.
gomox 1y ago
I can't imagine the hoops an accountant would have to go through to argue training cost is COGS. In the most obvious stick-figures-for-beginners interpretation, as in, "If I had to explain how a P&L statement works to an AI engineer", training is R&D cost and inference cost is COGS.
- jgalt212 1y ago
  there's not a bright line there, though.
- drob518 1y ago
  I wasn’t using COGS in a GAAP sense, but rather as a synonym for unspecified “costs.” My bad. I suppose you would classify training as development and ongoing datacenter and GPU costs as actual GAAP COGS. My point was, if all you focus on is revenue and ignore the costs of creating your business and keeping it running, it’s pretty easy for any business to be “profitable.”
  - DenisM 1y ago
    It’s generally useful to consider unit economy separate from whole company. If your unit economy is negative thing are very bleak. If it’s positive, your chance are going up by a lot - scaling the business amortizes fixed (non-unit) costs, such as admin and R&D, and slightly improves unit margins as well. However this does not work as well if your fixed (non-unit) cost is growing exponentially. You can’t get out of this unless your user base grows exponentially or the customer value (and price) per user grows exponentially. I think this is what Altman is saying - this is an unusual situation: unit economy is positive but fixed costs are exploding faster than economy if scale can absorb it. You can say it’s splitting hair, but insightful perspective often requires teasing things apart.
    - drob518 1y ago
      It’s splitting a hair, but a pretty important hair. Does anyone think that models won’t need continuous retraining? Does anyone think models won’t continue to try to scale? Personally, I think we’re reaching diminishing returns with scaling, which is probably good because we’ve basically run out of content to train on, and so perhaps that does stop or at least slow down drastically. But I don’t see a scenario where constant retraining isn’t the norm, even if the rough amount of content we’re using for it grows only slightly.

deleted 1y ago

[deleted]

dcre 1y ago

The Amodei quote in my other reply explains why this is wrong. The point is not to compare the training of the current model to inference on the current model. The thing that makes them lose so much money is that they are training the next model while making back their training cost on the current model. So it's not COGS at all.

prasadjoglekar 1y ago
Well, only if the one training model continued to function as a going business. Their amortization window for the training cost is 2 months or so. They can't just keep that up and collect $. They have to build the next model, or else people will go to someone else.
- dcre 1y ago
  Why two months? It was almost a year between Claude 3.5 and 4. (Not sure how much it costs to go from 3.5 to 3.7.)
  - Jalad 1y ago
    Even being generous, and saying it's a year, most capital expenditures depreciate over a period of 5-7 years. To state the obvious, training one model a year is not a saving grace
    - dcre 1y ago
      I don't understand why the absolute time period matters — all that matters is that you get enough time making money on inference to make up for the cost of training.
  - oblio 1y ago
    Don't they need to accelerate that, though? Having a 1 year old model isn't really great, it's just tolerable.
    - dcre 1y ago
      I think this is debatable as more models become good enough for more tasks. Maybe a smaller proportion of tasks will require SOTA models. On the other hand, the set of tasks people want to use LLMs for will expand along with the capabilities of SOTA models.
ToucanLoucan 1y ago
So is OpenAI capable of not making a new model at some point? They've been training the next model continuously as long as they've existed AFAIK. Our software house spends a lot on R&D sure, but we're still incredibly profitable all the same. If OpenAI is in a position where they effectively have to stop iterating the product to be profitable, I wouldn't call that a very good place to be when you're on the verge of having several hundred billion in debt.
- dcre 1y ago
  I think at that point there is strong financial pressure to figure out how to continuously evolve models instead of changing new ones, for example by building models out of smaller modules that can be trained individually and swapped out. Jeff Dean and Noam Shazeer talked about that a bit in their interview with Dwarkesh: https://www.dwarkesh.com/p/jeff-dean-and-noam-shazeer https://www.dwarkesh.com/p/jeff-dean-and-noam-shazeer
- DenisM 1y ago
  There’s still untapped value in deeper integrations. They might hit a jackpot of exponentially increasing value from network effects caused by tight integration with e.g. disjoint business processes. We know that businesses with tight network effects can grow to about 2 trillion in valuation.
  - oblio 1y ago
    How would that look with at least 3 US companies, probably 2 Chinese ones and at least 1 European company developing state of the art LLMs?
    - drob518 1y ago
      Like a very over-served market, I think. I see perhaps three survivors long term, or at most one gorilla, two chimps, and perhaps a few very small niche-focused monkeys.
    - DenisM 1y ago
drob518 1y ago
So,if they stopped training they’d be profitable? Only in some incremental sense, ignoring all sunk costs.

gomox 1y ago

I can't imagine the hoops an accountant would have to go through to argue training cost is COGS. In the most obvious stick-figures-for-beginners interpretation, as in, "If I had to explain how a P&L statement works to an AI engineer", training is R&D cost and inference cost is COGS.

jgalt212 1y ago
there's not a bright line there, though.
drob518 1y ago
I wasn’t using COGS in a GAAP sense, but rather as a synonym for unspecified “costs.” My bad. I suppose you would classify training as development and ongoing datacenter and GPU costs as actual GAAP COGS. My point was, if all you focus on is revenue and ignore the costs of creating your business and keeping it running, it’s pretty easy for any business to be “profitable.”
- DenisM 1y ago
  It’s generally useful to consider unit economy separate from whole company. If your unit economy is negative thing are very bleak. If it’s positive, your chance are going up by a lot - scaling the business amortizes fixed (non-unit) costs, such as admin and R&D, and slightly improves unit margins as well. However this does not work as well if your fixed (non-unit) cost is growing exponentially. You can’t get out of this unless your user base grows exponentially or the customer value (and price) per user grows exponentially. I think this is what Altman is saying - this is an unusual situation: unit economy is positive but fixed costs are exploding faster than economy if scale can absorb it. You can say it’s splitting hair, but insightful perspective often requires teasing things apart.
  - drob518 1y ago
    It’s splitting a hair, but a pretty important hair. Does anyone think that models won’t need continuous retraining? Does anyone think models won’t continue to try to scale? Personally, I think we’re reaching diminishing returns with scaling, which is probably good because we’ve basically run out of content to train on, and so perhaps that does stop or at least slow down drastically. But I don’t see a scenario where constant retraining isn’t the norm, even if the rough amount of content we’re using for it grows only slightly.