4 ms·

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost

by vb-8448 2mo ago

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

amelius 2mo ago
> Without training cost you can infer only the marginal cost of serving this kind of models. Which is by far the most interesting number of the two. > Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?) If you get close in output quality, then does that matter?
- embedding-shape 2mo ago
  > If you get close in output quality, then does that matter? When you're trying to estimate/infer the costs of serving the tokens and even include the cost of training the weights in order to output tokens then yeah, why wouldn't that matter?
  - jvuygbbkuurx 2mo ago
    That only matters if you are an investor not if you are a consumer.
    - embedding-shape 2mo ago
      Well, or if you're participating in a discussion on HN where the sub-topic happens to be "if labs are subsidising tokens on API pricing" and literally the cost of serving the tokens is relevant to the sub-topic people are trying to discuss...
    - solenoid0937 2mo ago
      Consumers want better models too, of course it matters
- vb-8448 2mo ago
  > Which is by far the most interesting number of the two. Only if you don't have to continuously train new models, and you are not at a runway risk.
energy123 2mo ago
Gross margins are insanely important, possibly the most important single metric if for some reason you were forced to choose one.
- vb-8448 2mo ago
  Only if you assume that at some point, for any reason, there will be "the model" that doesn't need costly retraining. I guess this is one of the reason Anthropic i so "active" for asking for a development break.
  - trollbridge 2mo ago
    "I want my competitors to stop competing" is an interesting ask for someone who's currently charging the highest prices in the entire industry.
NitpickLawyer 2mo ago
Training cost is directly impacted by inference cost nowadays. Most of the gains come from RL these days, and that is highly dependant on inference (~7:1 inference:training in units of compute). That's mainly because you want many roll-outs for each training scenario. Of course inference efficiency is dictated by model architecture, size, etc. You can still guesstimate some of those and have an idea about cost/serve at several size tiers.
- deleted 2mo ago
  [deleted]
- jeremyjh 2mo ago
  That isn't really relevant to GP's point. We still don't know what training costs because we don't know how much RL is done.
criley2 2mo ago
>No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Are you talking about Kimi's training cost or the training cost of the model(s) that Kimi distilled? Because Moonshot didn't even incur the majority of the training costs either
- tiagod 2mo ago
  I believe they are talking about the closed models' training costs. I other words, the providers that will be offering K3 inference don't have any training costs to offset, so they are only charging for the inference itself. OAI/Anthropic would need to offset their R&D and training costs in order to not be selling API access at a loss.
- nl 2mo ago
  > the training cost of the model(s) that Kimi distilled? The distillation process involves getting conversation traces from the model you are distilling from, and then training your model against them. You still have to do the training!
lelanthran 2mo ago
> No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Still useful; "are the labs marginally profitable just on the marginal inference costs?" is still a useful question to answer. After all, if they aren't even profitable on inference in isolation, then we can expect to see large price increases. If they are able to turn a marginal profit on inference alone, then perhaps the price increases won't be so severe (or perhaps they expand the time between generations so that they spend less on training but take longer to complete training). "Are the labs profitable at all?" is, of course, a much more useful question, but that doesn't mean that the first question is completely useless.
- solenoid0937 2mo ago
  We know labs make money on inference, and we know they lose a lot of money on inference+training.
  - vb-8448 2mo ago
    Just out of curiosity, based on what we know for sure they(OAI+A) make money on pure inference and lose on inference+training?
    - nl 2mo ago
      OpenAI's financials leaked and showed this pretty convincingly. Anthropic was probably profitable last quarter, without training costs: https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-propel-anthropic-into-its-first-profitable-quarter-7edbf2f4 https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...
  - lelanthran 2mo ago
    > We know labs make money on inference We don't really know that, for OpenAI and Anthropic. We suspect that, but as far as I know, even they have stopped claiming that they are profitable on inference.
    - 2mo ago

by vb-8448 2mo ago

amelius 2mo ago

> Without training cost you can infer only the marginal cost of serving this kind of models. Which is by far the most interesting number of the two. > Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?) If you get close in output quality, then does that matter?

embedding-shape 2mo ago
> If you get close in output quality, then does that matter? When you're trying to estimate/infer the costs of serving the tokens and even include the cost of training the weights in order to output tokens then yeah, why wouldn't that matter?
- jvuygbbkuurx 2mo ago
  That only matters if you are an investor not if you are a consumer.
  - embedding-shape 2mo ago
    Well, or if you're participating in a discussion on HN where the sub-topic happens to be "if labs are subsidising tokens on API pricing" and literally the cost of serving the tokens is relevant to the sub-topic people are trying to discuss...
  - solenoid0937 2mo ago
    Consumers want better models too, of course it matters
vb-8448 2mo ago
> Which is by far the most interesting number of the two. Only if you don't have to continuously train new models, and you are not at a runway risk.

energy123 2mo ago

Gross margins are insanely important, possibly the most important single metric if for some reason you were forced to choose one.

vb-8448 2mo ago
Only if you assume that at some point, for any reason, there will be "the model" that doesn't need costly retraining. I guess this is one of the reason Anthropic i so "active" for asking for a development break.
- trollbridge 2mo ago
  "I want my competitors to stop competing" is an interesting ask for someone who's currently charging the highest prices in the entire industry.

NitpickLawyer 2mo ago

Training cost is directly impacted by inference cost nowadays. Most of the gains come from RL these days, and that is highly dependant on inference (~7:1 inference:training in units of compute). That's mainly because you want many roll-outs for each training scenario. Of course inference efficiency is dictated by model architecture, size, etc. You can still guesstimate some of those and have an idea about cost/serve at several size tiers.

deleted 2mo ago
[deleted]
jeremyjh 2mo ago
That isn't really relevant to GP's point. We still don't know what training costs because we don't know how much RL is done.

criley2 2mo ago

>No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Are you talking about Kimi's training cost or the training cost of the model(s) that Kimi distilled? Because Moonshot didn't even incur the majority of the training costs either

tiagod 2mo ago
I believe they are talking about the closed models' training costs. I other words, the providers that will be offering K3 inference don't have any training costs to offset, so they are only charging for the inference itself. OAI/Anthropic would need to offset their R&D and training costs in order to not be selling API access at a loss.
nl 2mo ago
> the training cost of the model(s) that Kimi distilled? The distillation process involves getting conversation traces from the model you are distilling from, and then training your model against them. You still have to do the training!

lelanthran 2mo ago

> No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Still useful; "are the labs marginally profitable just on the marginal inference costs?" is still a useful question to answer. After all, if they aren't even profitable on inference in isolation, then we can expect to see large price increases. If they are able to turn a marginal profit on inference alone, then perhaps the price increases won't be so severe (or perhaps they expand the time between generations so that they spend less on training but take longer to complete training). "Are the labs profitable at all?" is, of course, a much more useful question, but that doesn't mean that the first question is completely useless.

solenoid0937 2mo ago
We know labs make money on inference, and we know they lose a lot of money on inference+training.
- vb-8448 2mo ago
  Just out of curiosity, based on what we know for sure they(OAI+A) make money on pure inference and lose on inference+training?
  - nl 2mo ago
    OpenAI's financials leaked and showed this pretty convincingly. Anthropic was probably profitable last quarter, without training costs: https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-propel-anthropic-into-its-first-profitable-quarter-7edbf2f4 https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...
- lelanthran 2mo ago
  > We know labs make money on inference We don't really know that, for OpenAI and Anthropic. We suspect that, but as far as I know, even they have stopped claiming that they are profitable on inference.
  - 2mo ago