4 ms·

Is pricing due to AWS massively overcharging because they know Enterprise might pay it due to ease of use with purchasing? Or is this realistic pricing for LLM

Is pricing due to AWS massively overcharging because they know Enterprise might pay it due to ease of use with purchasing? Or is this realistic pricing for LLM coding assistance and AWS has decided to be upfront with actual costs?

gjsman-1000 1y ago
Well, having personally used over $120 in Claude API credit on my $200/mo. Claude Code subscription... in a single day, without parallel tasks, yeah, it sounds like the actual price. (And keep in mind, Claude's API is still running on zero-margin, if not even subsidized, AWS prices for GPUs; combined with Anthropic still lighting money on fire and presumably losing money on the API pricing.) The future is not that AI takes over. It's when the accountants realize for a $120K a year developer, if it makes them even 20% more efficient (doubt that), you have a ceiling of $2000/mo. on AI spend before you break even. Once the VC subsidies end, it could easily cost that much. When that happens... who cares if you use AI? Some developers might use it, others might not, it doesn't matter anymore. This is also assuming Anthropic or OpenAI don't lose any of their ongoing lawsuits, and aren't forced to raise prices to cover settlement fees. For example, Anthropic is currently in the clear on the fair use "transformative" argument; but they are in hot water over the book piracy from LibGen (illegal regardless of use case). The worst case scenario in that lawsuit, although unlikely, is $150,000 per violation * 5 million books = $750B in damages.
- sdesol 1y ago
  > 120K a year developer, if it makes them even 20% more efficient (doubt that), you have a ceiling of $2000/mo I don't think businesses sees it this way. They sort of want you to be 20% more efficient by being 20% better (with no added cost). I'm sure the math is, if their efficiency is increased by 20% then than means we can reduce head count by 20% or not hire new developers.
  - hobs 1y ago
    Oh its much worse than that - they think that most developers don't do anything and the core devs are just supported by the ancillary devs, 80% of the work in core devs and 20% otherwise. In many workplaces this is true. That means an "ideal" workspace is 20% of the size of its current setup, with AI doing all the work that the non-core devs used to do.
- tsvetkov 1y ago
  > Claude's API is still running on zero-margin, if not even subsidized, AWS prices for GPUs; combined with Anthropic still lighting money on fire and presumably losing money on the API pricing. Source? Dario claims API inference is already “fairly profitable”. They have been optimizing models and inference, while keeping prices fairly high. > dario recently told alex kantrowitz the quiet part out loud: "we make improvements all the time that make the models, like, 50% more efficient than they are before. we are just the beginning of optimizing inference... for every dollar the model makes, it costs a certain amount. that is actually already fairly profitable." https://ethanding.substack.com/p/openai-burns-the-boats https://ethanding.substack.com/p/openai-burns-the-boats
  - JCM9 1y ago
    Most of these “we’re profitable on inference” comments are glossing over the depreciation cost of developing the model, which is essentially a capital expense. Given the short lifespan of models it seems unlikely that fully loaded cost looks pretty. If you can sweat a model for 5 years then the financials would likely look decent. With new models every few months, it’s likely really ugly.
    - AnimalMuppet 1y ago
      Interesting. But it would depend on how much of model X is salvaged in creating model X+1. I suspect that the answer is almost all of the training data, and none of the weights (because the new model has a different architecture, rather than some new pieces bolted on to the existing architecture). So then the question becomes, what is the relative cost of the training data vs. actually training to derive the weights? I don't know the answer to that; can anyone give a definitive answer?
      - JCM9 1y ago
        There are some transferable assets but the challenge is the commoditization of everything that means others have easy access to “good enough” assets to build upon. There’s very little moat to build in this business and that’s making all the money dumped into it looking a bit froth and ready to implode. GPT-5 is a bellwether there. OpenAI had a huge head start and basically access to whatever money and resources they needed and after a ton of hype released a pile of underwhelming meh. With the pace of advances slowing rapidly the pressure will be on to make money from what’s there now (which is well short of what the hype had promised). In the language of Gartner’s hype curve, we’re about to rapidly fall into the “trough of disillusionment.”
Insanity 1y ago
Yeah, honestly, the 'upstarts' in the space need to get users and burn VC money to do so, and part of that is selling below a reasonable market price. Amazon / AWS might not want (or need) to play that game.
- verdverm 1y ago
  What about MS and Copilot?
  - stackskipton 1y ago
    It's possible they have decided they are doing more VC route of burning money trying to gain market share in hopes they can lock everyone in then jack up the prices. Obviously, they are not taking VC money but using revenue from other parts of the company.
JCM9 1y ago
Likely both. AWS was often 70+% more expensive on GPUs for a while and remains typically the most expensive provider when put in competitive scenarios. Remember that for Microsoft, Google, and others cloud is their low margin fun side business… for AWS/Amazon cloud is their high margin business that makes money. Whereas others are willing to loose a bit to get ahead in a competitive space like GenAI, AWS has been freaking out since if the cloud gets driven to a low margin commodity then Amazon is in big trouble. This they’ll do everything possible to keep the margins high, which translates to “realistic” pricing based on actual costs in this case. And thus yes seemingly hoping enterprises will buy some expensive second rate product so they can say they have customers here and hoping they don’t notice better cheaper offerings are readily available. This is also a signal what what the “real cost” of these services will be once the VC subsidies dry up.
- ipython 1y ago
  > This is also a signal what what the “real cost” of these services will be once the VC subsidies dry up. Sooooo true. Waiting for AI’s “uber” moment aka when uber suddenly needed to actually turn a profit and overnight drivers were paid less while prices rose 3x.
- stackskipton 1y ago
  Azure/Office365 is Microsoft money maker as well. It's why if you compare their prices, they are pretty lock step. GCP is only big three that is much cheaper, clearly a play to get customers since they are in distant third.

gjsman-1000 1y ago

Well, having personally used over $120 in Claude API credit on my $200/mo. Claude Code subscription... in a single day, without parallel tasks, yeah, it sounds like the actual price. (And keep in mind, Claude's API is still running on zero-margin, if not even subsidized, AWS prices for GPUs; combined with Anthropic still lighting money on fire and presumably losing money on the API pricing.) The future is not that AI takes over. It's when the accountants realize for a $120K a year developer, if it makes them even 20% more efficient (doubt that), you have a ceiling of $2000/mo. on AI spend before you break even. Once the VC subsidies end, it could easily cost that much. When that happens... who cares if you use AI? Some developers might use it, others might not, it doesn't matter anymore. This is also assuming Anthropic or OpenAI don't lose any of their ongoing lawsuits, and aren't forced to raise prices to cover settlement fees. For example, Anthropic is currently in the clear on the fair use "transformative" argument; but they are in hot water over the book piracy from LibGen (illegal regardless of use case). The worst case scenario in that lawsuit, although unlikely, is $150,000 per violation * 5 million books = $750B in damages.

sdesol 1y ago
> 120K a year developer, if it makes them even 20% more efficient (doubt that), you have a ceiling of $2000/mo I don't think businesses sees it this way. They sort of want you to be 20% more efficient by being 20% better (with no added cost). I'm sure the math is, if their efficiency is increased by 20% then than means we can reduce head count by 20% or not hire new developers.
- hobs 1y ago
  Oh its much worse than that - they think that most developers don't do anything and the core devs are just supported by the ancillary devs, 80% of the work in core devs and 20% otherwise. In many workplaces this is true. That means an "ideal" workspace is 20% of the size of its current setup, with AI doing all the work that the non-core devs used to do.
tsvetkov 1y ago
> Claude's API is still running on zero-margin, if not even subsidized, AWS prices for GPUs; combined with Anthropic still lighting money on fire and presumably losing money on the API pricing. Source? Dario claims API inference is already “fairly profitable”. They have been optimizing models and inference, while keeping prices fairly high. > dario recently told alex kantrowitz the quiet part out loud: "we make improvements all the time that make the models, like, 50% more efficient than they are before. we are just the beginning of optimizing inference... for every dollar the model makes, it costs a certain amount. that is actually already fairly profitable." https://ethanding.substack.com/p/openai-burns-the-boats https://ethanding.substack.com/p/openai-burns-the-boats
- JCM9 1y ago
  Most of these “we’re profitable on inference” comments are glossing over the depreciation cost of developing the model, which is essentially a capital expense. Given the short lifespan of models it seems unlikely that fully loaded cost looks pretty. If you can sweat a model for 5 years then the financials would likely look decent. With new models every few months, it’s likely really ugly.
  - AnimalMuppet 1y ago
    Interesting. But it would depend on how much of model X is salvaged in creating model X+1. I suspect that the answer is almost all of the training data, and none of the weights (because the new model has a different architecture, rather than some new pieces bolted on to the existing architecture). So then the question becomes, what is the relative cost of the training data vs. actually training to derive the weights? I don't know the answer to that; can anyone give a definitive answer?
    - JCM9 1y ago
      There are some transferable assets but the challenge is the commoditization of everything that means others have easy access to “good enough” assets to build upon. There’s very little moat to build in this business and that’s making all the money dumped into it looking a bit froth and ready to implode. GPT-5 is a bellwether there. OpenAI had a huge head start and basically access to whatever money and resources they needed and after a ton of hype released a pile of underwhelming meh. With the pace of advances slowing rapidly the pressure will be on to make money from what’s there now (which is well short of what the hype had promised). In the language of Gartner’s hype curve, we’re about to rapidly fall into the “trough of disillusionment.”

Insanity 1y ago

Yeah, honestly, the 'upstarts' in the space need to get users and burn VC money to do so, and part of that is selling below a reasonable market price. Amazon / AWS might not want (or need) to play that game.

verdverm 1y ago
What about MS and Copilot?
- stackskipton 1y ago
  It's possible they have decided they are doing more VC route of burning money trying to gain market share in hopes they can lock everyone in then jack up the prices. Obviously, they are not taking VC money but using revenue from other parts of the company.

JCM9 1y ago

Likely both. AWS was often 70+% more expensive on GPUs for a while and remains typically the most expensive provider when put in competitive scenarios. Remember that for Microsoft, Google, and others cloud is their low margin fun side business… for AWS/Amazon cloud is their high margin business that makes money. Whereas others are willing to loose a bit to get ahead in a competitive space like GenAI, AWS has been freaking out since if the cloud gets driven to a low margin commodity then Amazon is in big trouble. This they’ll do everything possible to keep the margins high, which translates to “realistic” pricing based on actual costs in this case. And thus yes seemingly hoping enterprises will buy some expensive second rate product so they can say they have customers here and hoping they don’t notice better cheaper offerings are readily available. This is also a signal what what the “real cost” of these services will be once the VC subsidies dry up.

ipython 1y ago
> This is also a signal what what the “real cost” of these services will be once the VC subsidies dry up. Sooooo true. Waiting for AI’s “uber” moment aka when uber suddenly needed to actually turn a profit and overnight drivers were paid less while prices rose 3x.
stackskipton 1y ago
Azure/Office365 is Microsoft money maker as well. It's why if you compare their prices, they are pretty lock step. GCP is only big three that is much cheaper, clearly a play to get customers since they are in distant third.