4 ms·

The marginal cost of inference per token is lower than what OpenAI charges you (IIRC about 2x cheaper), they make a loss because of the enormous costs of R&D an

The marginal cost of inference per token is lower than what OpenAI charges you (IIRC about 2x cheaper), they make a loss because of the enormous costs of R&D and training new models.

diggan 2y ago
Did OpenAI publish concrete numbers regarding this, or where are you getting this data from?
- lukeschlather 2y ago
  https://news.ycombinator.com/item?id=41833287 https://news.ycombinator.com/item?id=41833287 This says 506 tokens/second for Llama 405B on a machine with 8x H200s which you can rent for $4/GPU so probably $40/hour for a server with enough GPUs. And so it can do ~1.8M tokens per hour. OpenAI charges $10/1M output tokens for GPT4o. (input tokens and cached tokens are cheaper, but this is just ballpark estimates.) So if it were 405B it might cost $20/1M output tokens. Now, OpenAI is a little vague, but they have implied that GPT4o is actually only 60B-80B parameters. So they're probably selling it with a reasonable profit margin assuming it can do $5/1M output tokens at approximately 100B parameters. And even if they were selling it at cost, I wouldn't be worried because a couple years from now Nvidia will release H300s that are at least 30% more efficient and that will cause a profit margin to materialize without raising prices. So if I have a use case that works with today's models, I will be able to rent the same thing a year or two from now for roughly the same price.
ignoramous 2y ago
> The marginal cost of inference per token is lower than what OpenAI charges you Unlike most Gen AI shops, OpenAI also incurs a heavy cost for traning base models gunning for SoTA, which involves drawing power from a literal nuclear reactor inside data centers.
- fransje26 2y ago
  > from a literal nuclear reactor inside data centers. No.
  - Tostino 2y ago
    Their username is fitting though.
    - ignoramous 2y ago
      Bully. I wrote "inside" to mean that those mini reactors (300MW+) are meant to be used solely for the DCs. (noun: https://www.collinsdictionary.com/dictionary/english-thesaurus/inside https://www.collinsdictionary.com/dictionary/english-thesaur... / https://en.wikipedia.org/wiki/Heterosemy https://en.wikipedia.org/wiki/Heterosemy) Replace it with nearby if that's makes you feel good about anyone's username.
      - Tostino 2y ago
        You are right, that wasn't a charitable reading of your comment. Should have kept it to myself. Sorry for being rude.
- candiddevmike 2y ago
  > literal nuclear reactor inside data centers This is fascinating to think about. Wonder what kind of shielding/environmental controls/all other kinds of changes you'd need for this to actually work. Would rack-sized SMR be contained enough not to impact anything? Would datacenter operators/workers need to follow NRC guidance?
  - talldayo 2y ago
    I think the simple answer is that it doesn't make sense. Nuclear power plants generate a byproduct that inherently limits the performance of computers; heat. Having either a cooling system, reactor or turbine located inside a datacenter is immediately rendered pointless because you end up managing two competing thermal systems at once. There is no reason to localize a reactor inside a datacenter when you could locate it elsewhere and pipe the generated electricity into it via preexisting high voltage lines.
tempusalaria 2y ago
It’s not clear this is true because reported numbers don’t disaggregate paid subscription revenue (certainly massively GP positive) vs free usage (certainly negative) vs API revenue (probably GP negative). Most of their revenue is the subscription stuff, which makes it highly likely they lose money per token on the api (not surprising as they are are in price war with Google et al) If you have an enterprise ChatGPT sub you have to consume around 5mln tokens a month to match the cost of using the api on GPT4o. At 100 words per minute that’s 35 days on continuous typing which shows how ridiculous the costs of api vs subscription are.
- seizethecheese 2y ago
  In summary, the original point of this thread is wrong. There’s essentially no future where these tools disappear or become unavailable at reasonable cost for consumers. Much more likely is they get way better.
  - jazzyjackson 2y ago
    I mean use to be I could get an Uber across Manhattan for $5 From my view chatbots are still in the "selling dollars for 90 cents" category of product, of course it sells like discounted hotcakes...
    - sebzim4500 2y ago
      The difference is that Uber was making a loss on those journeys whereas OpenAI aren't making a loss on chatgpt subscriptions. They make a loss overall because they spend a ton on R&D.
    - seizethecheese 2y ago
      … this is conflating two things, marginal and average cost/revenue. They are very very different.
  - tempusalaria 2y ago
    Definitely they will. OpenAI’s potential issue is that if Google offers tokens at a 10% gross margin, OpenAI won’t be able to offer api tokens at a positive gross margin at all. Their only chance really is building a big subscription business. No way they can compete with a hyperscaler on api cost long run
deleted 2y ago
[deleted]

The marginal cost of inference per token is lower than what OpenAI charges you (IIRC about 2x cheaper), they make a loss because of the enormous costs of R&D and training new models.

diggan 2y ago

Did OpenAI publish concrete numbers regarding this, or where are you getting this data from?

lukeschlather 2y ago
https://news.ycombinator.com/item?id=41833287 https://news.ycombinator.com/item?id=41833287 This says 506 tokens/second for Llama 405B on a machine with 8x H200s which you can rent for $4/GPU so probably $40/hour for a server with enough GPUs. And so it can do ~1.8M tokens per hour. OpenAI charges $10/1M output tokens for GPT4o. (input tokens and cached tokens are cheaper, but this is just ballpark estimates.) So if it were 405B it might cost $20/1M output tokens. Now, OpenAI is a little vague, but they have implied that GPT4o is actually only 60B-80B parameters. So they're probably selling it with a reasonable profit margin assuming it can do $5/1M output tokens at approximately 100B parameters. And even if they were selling it at cost, I wouldn't be worried because a couple years from now Nvidia will release H300s that are at least 30% more efficient and that will cause a profit margin to materialize without raising prices. So if I have a use case that works with today's models, I will be able to rent the same thing a year or two from now for roughly the same price.

ignoramous 2y ago

> The marginal cost of inference per token is lower than what OpenAI charges you Unlike most Gen AI shops, OpenAI also incurs a heavy cost for traning base models gunning for SoTA, which involves drawing power from a literal nuclear reactor inside data centers.

fransje26 2y ago
> from a literal nuclear reactor inside data centers. No.
- Tostino 2y ago
  Their username is fitting though.
  - ignoramous 2y ago
    Bully. I wrote "inside" to mean that those mini reactors (300MW+) are meant to be used solely for the DCs. (noun: https://www.collinsdictionary.com/dictionary/english-thesaurus/inside https://www.collinsdictionary.com/dictionary/english-thesaur... / https://en.wikipedia.org/wiki/Heterosemy https://en.wikipedia.org/wiki/Heterosemy) Replace it with nearby if that's makes you feel good about anyone's username.
    - Tostino 2y ago
      You are right, that wasn't a charitable reading of your comment. Should have kept it to myself. Sorry for being rude.
candiddevmike 2y ago
> literal nuclear reactor inside data centers This is fascinating to think about. Wonder what kind of shielding/environmental controls/all other kinds of changes you'd need for this to actually work. Would rack-sized SMR be contained enough not to impact anything? Would datacenter operators/workers need to follow NRC guidance?
- talldayo 2y ago
  I think the simple answer is that it doesn't make sense. Nuclear power plants generate a byproduct that inherently limits the performance of computers; heat. Having either a cooling system, reactor or turbine located inside a datacenter is immediately rendered pointless because you end up managing two competing thermal systems at once. There is no reason to localize a reactor inside a datacenter when you could locate it elsewhere and pipe the generated electricity into it via preexisting high voltage lines.

tempusalaria 2y ago

It’s not clear this is true because reported numbers don’t disaggregate paid subscription revenue (certainly massively GP positive) vs free usage (certainly negative) vs API revenue (probably GP negative). Most of their revenue is the subscription stuff, which makes it highly likely they lose money per token on the api (not surprising as they are are in price war with Google et al) If you have an enterprise ChatGPT sub you have to consume around 5mln tokens a month to match the cost of using the api on GPT4o. At 100 words per minute that’s 35 days on continuous typing which shows how ridiculous the costs of api vs subscription are.

seizethecheese 2y ago
In summary, the original point of this thread is wrong. There’s essentially no future where these tools disappear or become unavailable at reasonable cost for consumers. Much more likely is they get way better.
- jazzyjackson 2y ago
  I mean use to be I could get an Uber across Manhattan for $5 From my view chatbots are still in the "selling dollars for 90 cents" category of product, of course it sells like discounted hotcakes...
  - sebzim4500 2y ago
    The difference is that Uber was making a loss on those journeys whereas OpenAI aren't making a loss on chatgpt subscriptions. They make a loss overall because they spend a ton on R&D.
  - seizethecheese 2y ago
    … this is conflating two things, marginal and average cost/revenue. They are very very different.
- tempusalaria 2y ago
  Definitely they will. OpenAI’s potential issue is that if Google offers tokens at a 10% gross margin, OpenAI won’t be able to offer api tokens at a positive gross margin at all. Their only chance really is building a big subscription business. No way they can compete with a hyperscaler on api cost long run

deleted 2y ago

[deleted]