4 ms·

Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perce

Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perceived value to the user, that might make them less willing to pay for it. Also, inference costs are bound to go way down with more optimized architectures. GPUs are fundamentally not great at inference. No platform where the weights are streamed from a large pool of memory is. If the models ever quiet down, there will be massive step changes in cost/token, energy/token and tokens/second, as models are etched into silicon ala https://chatjimmy.ai/ https://chatjimmy.ai/

golem14 4mo ago
Why would any company brag about their margins ? Yet they do, to attract investors.
- Tuna-Fish 4mo ago
  The key AI labs are not public companies, they are at liberty to brag about their margins to potential investors in private.
  - bwhiting2356 4mo ago
    this is changing soon
    - joelthelion 4mo ago
      Not really, how much of a public company are you when 5% of your capital is public ?
      - tverbeure 4mo ago
        The percentage is irrelevant for this discussion. As soon as you’re public, you need to report detailed financial numbers.
        overgard 4mo ago
        Plus, you have to do real GAAP accounting, not their made up metrics.
        fakedang 4mo ago
        That's changing with this administration though. Reduced reporting cycles reduce transparency.
        deleted 4mo ago
        [deleted]
        mrosett 4mo ago
        It won't impact the disclosure of key business details because it doesn't reduce the level of disclosure needed in the S-1 or the 10-K.
        jimnotgym 4mo ago
        This is an interesting anomaly in the US. In the civilised world all corporations have to file public accounts, as the price for their limited liability. The detail and audit requirements depend on the size, turnover, staff numbers etc. This is because the shareholders are not the only stakeholder. The companies creditors, for instance, who are exposed to the limited liability have a right to see what they are lending to. To answer the sibling comment, all of these public accounts follow local GAAP or IFRS. The US still astounds me with its willingness to allow corporations to rip people off!
        kortilla 4mo ago
        Creditors in the US can make visibility into financials a requirement for financing if they want. Protecting creditors isn’t a good argument for public reporting.
        VBprogrammer 4mo ago
        What are the arguments against public reporting? As a consumer you are often sending deposits or even the full cost of goods to companies some time before you receive those goods (in effect you become a creditor). You are also dependent upon some of those companies for service and repairs. It seems reasonable that you can check the finances of a company you are creating a business relationship with, I know in the past I've checked company statements. You are unlikely to have significant enough sway to force that kind of disclosure. Small businesses as consumers have less legal protection and are similarly unlikely to be able to make disclosure a precondition of a deal.
      - Tuna-Fish 4mo ago
        That doesn't matter for the legal requirements. The short and only kind of wrong version is: In the US, companies are not allowed to unfairly privilege some investors over others by giving them access to secret information that would let them judge the future prospects of the company. (Except in all the ways they can, but these usually involve some kinds of insider trading rules.) Private companies can handle giving out secrets to investors by literally writing and memo and mailing it to all their investors, if they want to give out some secrets to one of them. Public companies cannot do that, even if they knew who all their investors were, but must instead consider every member of the public a potential investor, even if they don't already own the stock. Because of this, when public companies want to reveal material information about their future prospects, they must reveal it to everyone.
      - daemin 4mo ago
        Isn't there a limit on the public markets where if a company has less than a certain percentage of its ownership traded publicly then it is no longer a public company and therefore de-listed? I remember hearing about a guy trying to squeeze out short sellers of his own company but ended up effectively taking his company private because he bought out like 95% of all the shares. I wonder how that aligns to these small releases of stock for the public.
        extraextra 4mo ago
        There is no legal minimum free float requirement before deregistration in US, however, different exchanges have different rules Essentially, a stock has to stay above 1$ per share, have a minimum market cap of $15m, minimum 400 shareholders and "adequate" liquidity If it meets those 4 criteria, it's essentially not at risk of deregistration
  - SiempreViernes 4mo ago
    And investors will leak such claims quickly enough that this reasoning cannot plausibly hide big secrets.
    - Tuna-Fish 4mo ago
      It's not a big secret. If you just do the math yourself, it's easy to compute that inference doesn't cost all that much. People just see all the capital investment going around and all the new data centers being built, see that it's spent on "AI", put two and two together and get a three, or "clearly serving AI requests costs an arm and a leg". The 1 they were missing is that AI requires both training and inference, and training is by far the expensive part. And that in principle you can stop training at any point and keep using the models as they are. (But that means that if other companies keep improving their models, you'll be left behind...) In contrast, inference is fairly cheap and all the providers have great margins on it. Eventually either investment in training stops having commensurate impact on model quality, and people stop doing that and instead concentrate on making inference faster and even more efficient. Or if that doesn't happen, things will get very weird very quickly.
      - ethin 4mo ago
        > If you just do the math yourself, it's easy to compute that inference doesn't cost all that much. Show us your work, then. If it's so easy to do, this should be a trivial request to accommodate, no?
        mediaman 4mo ago
        Just look at large open weights models being served by inference providers. Kimi 2.6 is a 1 trillion total / 32B active parameter model that's something comparable to Sonnet. Sonnet's API pricing is $5 in, $15 out per million tokens. Deepinfra serves Kimi at $0.75 in, $3.50 out, and about the same at openrouter. So you're looking at a 4-7x multiple that Anthropic is charging compared to market rates that any plebe can get with a credit card.
        4mo ago
- lmm 4mo ago
  Growing companies don't brag about their margins, they brag about their growth and revenue. Margin talk is for when you're a mature company squeezing out every bit of profitability you can - if anything it would be a negative sign to be worrying about your margins when you're supposed to still be growing and innovating.
  - byzantinegene 4mo ago
    [dead]
- amarant 4mo ago
  I mean, did anyone expect them to not have margins? Why keep it secret?
jimbokun 4mo ago
I doubt having to replace every single chip in your data center every time you release a new model will bring down costs.
overgard 4mo ago
A couple of years ago Altman was saying the price of AI compute is going to drop 90% year over year or something like that, so I don't think they're nervous about talking about lowering their costs. They probably just haven't been able to lower their costs. You have to keep in mind that about 99% of their announcements are targeted towards investors (their most important revenue source..), so they're not going to be afraid to mention metrics that make the business look better.
- mcmcmc 4mo ago
  Ah yes, Sam “Not Consistently Candid” Altman
  - pixelready 4mo ago
    Oh, is that the guy that sold Loopt by claiming it had hundreds of thousands of users and it turned out to have 500 DAU after his exit?
    - chipsrafferty 4mo ago
      Yep, the very same scammer. Wonder if he's lying about OpenAI too? Maybe about a person blowing a metal instrument?
- whateveracct 4mo ago
  he lied. he's good at that.
- bwhiting2356 4mo ago
  Jevons paradox. Cheaper tokens does not mean we will spend less.
  - Skinney 4mo ago
    Cheaper tokens means the company's margins increase, which would be valuable for investors to hear
  - 4mo ago
neltnerb 4mo ago
Because companies that want to go public need to look profitable or potentially profitable. And before they go public they have to release real, actual, legally demonstrable numbers for their costs and revenue anyway.
- extraextra 4mo ago
  When they will actually file to go public, their numbers will be intensely scrutinized. That's all that global headlines will be talking about for weeks on end. Why would they create forward expectations before it's necessary? Of course they don't want to create forward expectations in a volatile macro environment, with the public listing being 6 months out.
  - deleted 4mo ago
    [deleted]
DrewADesign 4mo ago
Because they can think more than one quarter into the future? Why on earth would someone adopt something into their core workflow that was fantastically unprofitable? Uncertainty and business don’t mix. Most people aren’t hype-eating bacteria that only care about maximizing their next paycheck.
- wheresmylogin 4mo ago
  One reason is that all the code you write with this goes in your private git. If using AI no longer is possible because of cost, you can still profit a lot from what you did with it before.
  - DrewADesign 4mo ago
    For consultants? Sure. What percentage of contractors are consultants? And is that better than going with something in your stack that’s sustainable even if it’s not totally optimal? I’d wager most would say no.
- nradov 4mo ago
  Regardless of profitability there will always be multiple good LLM vendors as well as open-source alternatives (slightly worse but still pretty good). If one vendor fails then it's easy to switch your core workflow to a competitor.
  - DrewADesign 4mo ago
    On an individual basis for coding? Sure. If you’re a significant business with agents that do more nuanced work, which is the only kind of customer that will let any of these companies pay back those trillions of dollars as quickly as they need to to stay alive, these are not fungible services.
etempleton 4mo ago
Because the most important thing for any pure play AI company right now is to prove they are a viable company. And sure they have proved they can make billions, but also that they can lose billions more. They are going to need even more money and to prove to the next round of investors at an even higher valuation that they are a viable business they need to show not that they can generate revenue, but that they can one day turn a healthy profit. And that is the trillion dollar question.
kopirgan 4mo ago
If inference costs drop 90% or whatever, that would be a massive write-off of hardware even before they gave any returns for it?! Given Chinese and others are snapping at the heels and would also benefit from such reduction in cost.
kopirgan 4mo ago
Went to that URL asked one question - "how is this different from other AI" and it took 598/6144 tokens, not sure what that means.
- philipswood 4mo ago
  Not super clear from the site itself, but this LLM is running on specialized silicon implementing just it. So has super low energy use and blazing speed. See https://taalas.com/products/ https://taalas.com/products/ Edit: updated link
  - kopirgan 4mo ago
    Incredible increase over Nvidia! Need to read more.. Thanks!
solarkraft 4mo ago
> Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? Investor confidence. They have a bit of a need for cash (also an interesting part of the profitability discussion of course). > Also, inference costs are bound to go way down with more optimized architectures I agree. Jimmy is incredible, I wonder what non-toy use cases they have. Surely they’ll come out with updated chips soon. That said, I was apparently a bit over-excited for Groq and Cerebras. I thought they’d quickly dethrone Nvidia for inference, but not so far. Even the GPT spark trial isn’t seeming to go far.
Yoric 4mo ago
> Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perceived value to the user, that might make them less willing to pay for it. Wouldn't they be bragging about it to investors? It feels like something that would matter a lot to them, and at least OpenAI kinda feels desperate to find them. There's also the small question about whether a drop in inference cost would actually change anything about profitability, when training seems to get exponentially more expensive.
m463 4mo ago
I wonder if inference costs will go down... or will it be like microsoft office, where the software bloats to use/fill current hardware? (and in this case bloats might mean better thinking or pulling in more data)

golem14 4mo ago

Why would any company brag about their margins ? Yet they do, to attract investors.

Tuna-Fish 4mo ago
The key AI labs are not public companies, they are at liberty to brag about their margins to potential investors in private.
- bwhiting2356 4mo ago
  this is changing soon
  - joelthelion 4mo ago
    Not really, how much of a public company are you when 5% of your capital is public ?
    - tverbeure 4mo ago
      The percentage is irrelevant for this discussion. As soon as you’re public, you need to report detailed financial numbers.
      - overgard 4mo ago
        Plus, you have to do real GAAP accounting, not their made up metrics.
      - fakedang 4mo ago
        That's changing with this administration though. Reduced reporting cycles reduce transparency.
        deleted 4mo ago
        [deleted]
        mrosett 4mo ago
        It won't impact the disclosure of key business details because it doesn't reduce the level of disclosure needed in the S-1 or the 10-K.
      - jimnotgym 4mo ago
        This is an interesting anomaly in the US. In the civilised world all corporations have to file public accounts, as the price for their limited liability. The detail and audit requirements depend on the size, turnover, staff numbers etc. This is because the shareholders are not the only stakeholder. The companies creditors, for instance, who are exposed to the limited liability have a right to see what they are lending to. To answer the sibling comment, all of these public accounts follow local GAAP or IFRS. The US still astounds me with its willingness to allow corporations to rip people off!
        kortilla 4mo ago
        Creditors in the US can make visibility into financials a requirement for financing if they want. Protecting creditors isn’t a good argument for public reporting.
        VBprogrammer 4mo ago
        What are the arguments against public reporting? As a consumer you are often sending deposits or even the full cost of goods to companies some time before you receive those goods (in effect you become a creditor). You are also dependent upon some of those companies for service and repairs. It seems reasonable that you can check the finances of a company you are creating a business relationship with, I know in the past I've checked company statements. You are unlikely to have significant enough sway to force that kind of disclosure. Small businesses as consumers have less legal protection and are similarly unlikely to be able to make disclosure a precondition of a deal.
    - Tuna-Fish 4mo ago
      That doesn't matter for the legal requirements. The short and only kind of wrong version is: In the US, companies are not allowed to unfairly privilege some investors over others by giving them access to secret information that would let them judge the future prospects of the company. (Except in all the ways they can, but these usually involve some kinds of insider trading rules.) Private companies can handle giving out secrets to investors by literally writing and memo and mailing it to all their investors, if they want to give out some secrets to one of them. Public companies cannot do that, even if they knew who all their investors were, but must instead consider every member of the public a potential investor, even if they don't already own the stock. Because of this, when public companies want to reveal material information about their future prospects, they must reveal it to everyone.
    - daemin 4mo ago
      Isn't there a limit on the public markets where if a company has less than a certain percentage of its ownership traded publicly then it is no longer a public company and therefore de-listed? I remember hearing about a guy trying to squeeze out short sellers of his own company but ended up effectively taking his company private because he bought out like 95% of all the shares. I wonder how that aligns to these small releases of stock for the public.
      - extraextra 4mo ago
        There is no legal minimum free float requirement before deregistration in US, however, different exchanges have different rules Essentially, a stock has to stay above 1$ per share, have a minimum market cap of $15m, minimum 400 shareholders and "adequate" liquidity If it meets those 4 criteria, it's essentially not at risk of deregistration
- SiempreViernes 4mo ago
  And investors will leak such claims quickly enough that this reasoning cannot plausibly hide big secrets.
  - Tuna-Fish 4mo ago
    It's not a big secret. If you just do the math yourself, it's easy to compute that inference doesn't cost all that much. People just see all the capital investment going around and all the new data centers being built, see that it's spent on "AI", put two and two together and get a three, or "clearly serving AI requests costs an arm and a leg". The 1 they were missing is that AI requires both training and inference, and training is by far the expensive part. And that in principle you can stop training at any point and keep using the models as they are. (But that means that if other companies keep improving their models, you'll be left behind...) In contrast, inference is fairly cheap and all the providers have great margins on it. Eventually either investment in training stops having commensurate impact on model quality, and people stop doing that and instead concentrate on making inference faster and even more efficient. Or if that doesn't happen, things will get very weird very quickly.
    - ethin 4mo ago
      > If you just do the math yourself, it's easy to compute that inference doesn't cost all that much. Show us your work, then. If it's so easy to do, this should be a trivial request to accommodate, no?
      - mediaman 4mo ago
        Just look at large open weights models being served by inference providers. Kimi 2.6 is a 1 trillion total / 32B active parameter model that's something comparable to Sonnet. Sonnet's API pricing is $5 in, $15 out per million tokens. Deepinfra serves Kimi at $0.75 in, $3.50 out, and about the same at openrouter. So you're looking at a 4-7x multiple that Anthropic is charging compared to market rates that any plebe can get with a credit card.
        4mo ago
lmm 4mo ago
Growing companies don't brag about their margins, they brag about their growth and revenue. Margin talk is for when you're a mature company squeezing out every bit of profitability you can - if anything it would be a negative sign to be worrying about your margins when you're supposed to still be growing and innovating.
- byzantinegene 4mo ago
  [dead]
amarant 4mo ago
I mean, did anyone expect them to not have margins? Why keep it secret?

jimbokun 4mo ago

I doubt having to replace every single chip in your data center every time you release a new model will bring down costs.

overgard 4mo ago

A couple of years ago Altman was saying the price of AI compute is going to drop 90% year over year or something like that, so I don't think they're nervous about talking about lowering their costs. They probably just haven't been able to lower their costs. You have to keep in mind that about 99% of their announcements are targeted towards investors (their most important revenue source..), so they're not going to be afraid to mention metrics that make the business look better.

mcmcmc 4mo ago
Ah yes, Sam “Not Consistently Candid” Altman
- pixelready 4mo ago
  Oh, is that the guy that sold Loopt by claiming it had hundreds of thousands of users and it turned out to have 500 DAU after his exit?
  - chipsrafferty 4mo ago
    Yep, the very same scammer. Wonder if he's lying about OpenAI too? Maybe about a person blowing a metal instrument?
whateveracct 4mo ago
he lied. he's good at that.
bwhiting2356 4mo ago
Jevons paradox. Cheaper tokens does not mean we will spend less.
- Skinney 4mo ago
  Cheaper tokens means the company's margins increase, which would be valuable for investors to hear
- 4mo ago

neltnerb 4mo ago

Because companies that want to go public need to look profitable or potentially profitable. And before they go public they have to release real, actual, legally demonstrable numbers for their costs and revenue anyway.

extraextra 4mo ago
When they will actually file to go public, their numbers will be intensely scrutinized. That's all that global headlines will be talking about for weeks on end. Why would they create forward expectations before it's necessary? Of course they don't want to create forward expectations in a volatile macro environment, with the public listing being 6 months out.
- deleted 4mo ago
  [deleted]

DrewADesign 4mo ago

Because they can think more than one quarter into the future? Why on earth would someone adopt something into their core workflow that was fantastically unprofitable? Uncertainty and business don’t mix. Most people aren’t hype-eating bacteria that only care about maximizing their next paycheck.

wheresmylogin 4mo ago
One reason is that all the code you write with this goes in your private git. If using AI no longer is possible because of cost, you can still profit a lot from what you did with it before.
- DrewADesign 4mo ago
  For consultants? Sure. What percentage of contractors are consultants? And is that better than going with something in your stack that’s sustainable even if it’s not totally optimal? I’d wager most would say no.
nradov 4mo ago
Regardless of profitability there will always be multiple good LLM vendors as well as open-source alternatives (slightly worse but still pretty good). If one vendor fails then it's easy to switch your core workflow to a competitor.
- DrewADesign 4mo ago
  On an individual basis for coding? Sure. If you’re a significant business with agents that do more nuanced work, which is the only kind of customer that will let any of these companies pay back those trillions of dollars as quickly as they need to to stay alive, these are not fungible services.

etempleton 4mo ago

Because the most important thing for any pure play AI company right now is to prove they are a viable company. And sure they have proved they can make billions, but also that they can lose billions more. They are going to need even more money and to prove to the next round of investors at an even higher valuation that they are a viable business they need to show not that they can generate revenue, but that they can one day turn a healthy profit. And that is the trillion dollar question.

kopirgan 4mo ago

If inference costs drop 90% or whatever, that would be a massive write-off of hardware even before they gave any returns for it?! Given Chinese and others are snapping at the heels and would also benefit from such reduction in cost.

kopirgan 4mo ago

Went to that URL asked one question - "how is this different from other AI" and it took 598/6144 tokens, not sure what that means.

philipswood 4mo ago
Not super clear from the site itself, but this LLM is running on specialized silicon implementing just it. So has super low energy use and blazing speed. See https://taalas.com/products/ https://taalas.com/products/ Edit: updated link
- kopirgan 4mo ago
  Incredible increase over Nvidia! Need to read more.. Thanks!

solarkraft 4mo ago

> Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? Investor confidence. They have a bit of a need for cash (also an interesting part of the profitability discussion of course). > Also, inference costs are bound to go way down with more optimized architectures I agree. Jimmy is incredible, I wonder what non-toy use cases they have. Surely they’ll come out with updated chips soon. That said, I was apparently a bit over-excited for Groq and Cerebras. I thought they’d quickly dethrone Nvidia for inference, but not so far. Even the GPT spark trial isn’t seeming to go far.

Yoric 4mo ago

> Why on earth would AI labs be bragging about how little the product they sell actually costs them to make? You don't want to do anything that reduces it's perceived value to the user, that might make them less willing to pay for it. Wouldn't they be bragging about it to investors? It feels like something that would matter a lot to them, and at least OpenAI kinda feels desperate to find them. There's also the small question about whether a drop in inference cost would actually change anything about profitability, when training seems to get exponentially more expensive.

m463 4mo ago

I wonder if inference costs will go down... or will it be like microsoft office, where the software bloats to use/fill current hardware? (and in this case bloats might mean better thinking or pulling in more data)