7 ms·

Corporate America Is Starting to Ration AI as Cost Skyrockets

188 points by 1vuio0pswjnm7 5mo ago

gonzalohm 5mo ago
In my opinion, the problem is not even the cost. The problem is that people are using AI for running recurrent stuff instead of writing code to automate it. For example. Imagine that you are comparing two documents (let's assume diff doesn't exist). You could ask an AI to compare the differences from you or you could use AI to write a tool to do it. For whatever reason, people are starting to go with the former not realizing that now they basically have to pay to compare documents.
- CompoundEyes 5mo ago
  Agreed. I’ve been telling my team to build up internal packages so we can push all that ad hoc reinvention into something more tangible and deterministic. Invest the $$$ in inference into something the agent can reach for next time that’s neutral and consumable by other code to reduce future spend.
  - trollbridge 5mo ago
    Yes. Build compact CLI-driven tools, write a skill for it (you can use your agent to do most of this work for you). It just requires being willing to think instead of mashing prompts into a keyboard.
- avereveard 5mo ago
  Same, even opus favor short term solution and scripts with a billion flags that constabtly require rescanning to understand how to launch it is a constant struggle to get it to build sane default and reusable scripts that run with minimal parameters
  - gonzalohm 5mo ago
    Yeah, and what's up with adding dry run to everything? I saw some code that doesn't write anything but still the AI added a dry run which had a completely different codebase
    - duskdozer 5mo ago
      Because dry run is in a lot of scripts in its training data. It's not "thinking" about the script or the concept of a dry run.
    - lanstin 5mo ago
      And everything configurable gets an environment variable. Editing the first few lines in the script is a fine way to configure things in Python.
- plmpsu 5mo ago
  AI can do things around semantic analysis that a deterministic diff tool cannot. I understand and agree with your point though.
  - bilekas 5mo ago
    I'm curious if you could give me an example of something that couldn't be down deterministically. We have fuzzy search/matching too ? Regex is a monster when used correctly.
    - SpicyLemonZest 5mo ago
      I sometimes find myself with thousands of log lines from a problematic execution and a known good reference, wondering nonspecifically if "something weird" happened in the first one. I don't think there's any matching-based solution there; you need a scan process that understands variations in execution time, object identifiers, etc. aren't meaningful.
      - bilekas 5mo ago
        You would need specific domain knowledge and a very clever parser, I've done one for a ridiculously over engineered system but a pain. That's fair but how often would you need it? Certainly not token maxing amounts!
        SpicyLemonZest 5mo ago
        It's a spectrum. Could it be worth it to run that as a first pass on every report of anything going wrong, just in case it produces a useful insight? Depends on how much engineering time it saves! In practice, I've found the answer seems to be "not much", because human triage is still required to understand whether the insight is correct and useful. But I'm not sure that was obvious in retrospect.
    - plmpsu 5mo ago
      Pretty much anything for which you'd need intelligence of any kind. Questions such as: Do these two paragraphs have the same semantic meaning? Do they have the same sentiment? Do these two methods have the same contract? etc. Not all documents our code and even with code deterministic tools gets you only so far.
- r_lee 5mo ago
  it's all about cost at the end of the day. if you're allowed and encouraged to tokenmaxx, then of course this'll happen.
- jgalt212 5mo ago
  I agree, but even this use case isn't the most wasteful. The interwebs says Agentic consumes 50% of token use, but I'd hazard this number is north of 90% for many shops. My cynical view of Agentic is its sole purpose is to make "number go up".
  - id 5mo ago
    Look at me! I'm the smartest guy. I've wasted 10M tokens! No one has wasted more!
- bluejay2387 5mo ago
  I have exposure to AI initiatives at several companies including a few F500's. I have seen teams dump huge logs into frontier models that took hours to get so-so results that we were able to replace with a few lines of python code at 1000 times the speed and 100% accuracy. When asked why they were doing this they literally said "because we don't understand the subject matter so we were depending on the AI". I saw one team file a complaint with a vendor about a frontier backed coding harness and it's inability to consistently format headers because they were using it as a reporting engine. When I recommended they just use the coding tool to write code to generate reports you would have thought I had just cured cancer from their response. I frequently see people complain about the fact that AI is going to take their jobs and then see them gripe about the fact that AI is 'worthless' because it can't do more of their job than it already does. It's easy to see the difference between the people seeing 10x productivity gains from leveraging AI and those who aren't and it's not the AI.
  - sbarre 5mo ago
    I've heard this framed as "AI raises the floor by 2x or less but raises the ceiling by 10x or more"
  - irishcoffee 5mo ago
    Someone asked me if I was using models for fantasy sports, and if it was smart enough to help make decisions about drafting. My answer: no, but it was able to help me find the website and social handles for every beat writer for every team, and generate a simple website where I can do a daily skim of teams/players and draw my own conclusions. LLMs are a tool, not a panacea.
  - mxschumacher 5mo ago
    i have trouble understanding these situations, e.g. the AI itself would presumably make the suggestion to write a python script for such a task. It seems to me that there two huge problems right now * understanding which category of problems an LLM is an appropriate solution for (rather than throwing LLMs at any and all problems) * matching model capability (and therefore cost) to the problem at hand. You can easily overspend massively by using a model that's too powerful
- cyanydeez 5mo ago
  Oh no! People are doing what they've been told to do!
  - cindyllm 5mo ago
    [dead]
- throwatdem12311 5mo ago
  Laziness, pure and simple. The inevitable consequence of “the LLm is the compiler now”. And what do you even expect people to do when they are forced at threat of termination to use AI for everything as much as possible? Not to mention people are being pressured to do insane thing like review hundreds of pull requests per day and deliver like 15 features per week so OBVIOUSLY there isn’t time to build out proper tooling. Just shove everything in a prompt and call it a day. Some people have families to feed, just do what you’re told.
- bilekas 5mo ago
  It's this and worse. To use your example, it's like people using AI to write a diff algorithm, incorrectly, then using AI to fix it, because they don't know that diff exists already. Lazyness and starting development with a very low level of understanding. People think lowering the barrier to entry is a good thing, when in reality there are just fundamentals and things you just have to know before you can start using a tool like llms properly.
- dawnerd 5mo ago
  Same with writing boilerplate code. It’s been a solved problem yet here we are.
- rich_sasha 5mo ago
  Isn't that the supposed point of it though? At least how it is marketed/hyped. Don't use your brain, you don't need one, spend all your thinking energy on... dunno, something else, and leave all the "mundane" stuff to AI. Just pay for the tokens, it's going to make you 10x more efficient, the $1000/month is worth it.
- jerojero 5mo ago
  Because you look at the work from the perspective of a programmer, not the perspective of a regular person. Normal people have never gone around automating their work. The most automation they do is dynamic tables on excel sheets. I obviously know building a tool that can programmatically do something is a better solution, but I think that requires a fundamental shift in how people work. People need to be told by someone "this is how you should be using the AI" but right now they're simple told "use the AI".
  - gonzalohm 5mo ago
    I'm talking about programmers doing this. That's what's sad. These were normal people before,but it feels like they have some kind of AI schizophrenia now. They don't use their brains anymore
- m3nu 5mo ago
  100% this. For my own company I mostly build deterministic workflows that may have a simple AI step in the middle using an appropriate Chinese model in a very limited way. I wouldn't want to burn tokens to satisfy some metric. With this AI is a fallback and not the default. Sounds like large companies have it backwards.
- gib444 5mo ago
  Recurrent expensive inefficient processes? Sounds familiar ! This is why we have business analysts and software developers. To help identify inefficiencies and to build technical solutions.
shanewei 5mo ago
As a developer, I don’t think it’s just that costs are going up. I’m also seeing more people lately talk about “vibe slop”.
- BearOso 5mo ago
  I've noticed as well. A lot of pull requests are just agents running constantly, hoping to have produced something of value. Entropy is at an all-time high, though.
  - deleted 5mo ago
    [deleted]
  - shanewei 5mo ago
    The bottleneck becomes human review, not code generation. A PR can look plausible and still add more entropy than value.
- OptionOfT 5mo ago
  We have some dude at work who runs their own agent that makes constant commits. We're supposed to review the agent's output.
  - shanewei 5mo ago
    Exactly. The bottleneck becomes human review, not code generation. Agents can generate commits faster than humans can verify whether those commits should exist.
scronkfinkle 5mo ago
On the one hand, organizations are without question using LLM's well beyond what is actually necessary, and as reality kicks in they're forced to scale back accordingly. However at the same time, on intervals counted in months, we're seeing breakthroughs both in hardware and software that dramatically reduce the cost of inference. Between corporate FOMO and the rapidly decreasing costs of actually running LLM's I'm interested to see at which side of the spectrum these two meet
deleted 5mo ago
[deleted]
amazingamazing 5mo ago
AI is overhyped. I have yet to see an end user product that in itself isnt a wrapper around LLMs that is impressive created by LLM assistance. I have also yet to see dramatic increases of revenue of companies using LLMs that don't involve selling things in its supply chain. Is it a nice affordance? Sure. 1T capex good? No. If it was so good I would expect to see 2005-2015 advancements yearly. Meanwhile China is blowing past the world with real improvements in the real world- solar, EVs, etc. meanwhile people keep making their fancy sans serif websites about todo apps, faster than ever before. Useless.
- dawnerd 5mo ago
  Productivity gains seem like it’s at best a wash when you factor in the massive tech debt cleanup and additional time needed to spec and review.
  - trollbridge 5mo ago
    Misuse of AI tools because of continuing a fundamentally broken software development process.
- criddell 5mo ago
  > I have yet to see an end user product that in itself isnt a wrapper around LLMs that is impressive created by LLM assistance. I don’t disagree that AI is overhyped. But I think you are probably looking in the wrong place. I think most software that is written isn’t really a product, at least not a public product. It’s an in-house tool or a one-off project needed to complete some larger task. People everywhere are always writing small programs that make their life or job just a bit easier (and explains why so many corporate projects are little more than an excel spreadsheet). And there are a lot of people who have made custom software just for themselves with AI. Not a product, just a tool or project that finally made sense to build.
  - pessimizer 5mo ago
    But where's the revenue from those? It has to add up to a couple trillion dollars to break even on the capital spending.
elevation 5mo ago
Another reason to favor using AI to build automation instead of relying on it in prod: the risk of war and global instability. If LLMs are genuinely helpful or even decisive in a military engagement, you can expect any host country to commandeer whatever data centers they need, leaving commercial entities to bid up the prices on the leftover capacity. Another risk is that data centers are a great target for cyber warfare. It’s ideal if your business can leverage LLMs when they’re online but continue to operate profitably when they’re offline.
- lanstin 5mo ago
  Even regular warfare, if the Middle East AWS regions are an indication. The giant and arguably excessive data centers being built are not hardened physically.
ChrisArchitect 5mo ago
Some related discussions: https://news.ycombinator.com/item?id=48268871 https://news.ycombinator.com/item?id=48268871 https://news.ycombinator.com/item?id=48238896 https://news.ycombinator.com/item?id=48238896 https://news.ycombinator.com/item?id=48307098 https://news.ycombinator.com/item?id=48307098
feverzsj 5mo ago
LLM doesn't work, let alone profit.
- r_lee 5mo ago
  elaborate please, how does it not work?
  - bigstrat2003 5mo ago
    They cannot be trusted to produce output that works (let alone works well) because they are just statistical models, without any actual understanding of what they produce. That means that you have to carefully review every single line of code they produce, because you don't know where the hallucinations will be. But by the time you do that, you have saved no time at all (indeed, in my experience you lose time), because typing the code was never the part that took time. It was understanding the problem. So if you use an LLM, you spend a bunch of money for zero gain in productivity, or you sacrifice quality and pray there aren't nasty bugs lurking. I certainly think it's fair to call that state of affairs "it doesn't work".
- ninkendo 5mo ago
  Yesterday I updated our dependency on the sqlx crate and put up a PR, and it failed in the CI build in a way I couldn’t reproduce locally. I asked codex to take a look, and it: - Grabbed the CI logs on its own to figure out what the CI error was - Looked at my local setup - Looked at the changes in sqlx from 0.8 to 0.9 And figured out that sqlx depends on an updated version of the “whoami” crate but doesn’t specify default features, which causes it to fall back on a stub implementation that makes the default user “anonymous”, which was failing to authenticate to the UNIX socket we use in our CI Postgres server. It patched the environment variable for our docker container to explicitly specify a username and the issue was fixed. It would’ve taken me probably several hours to figure this out on my own. It took codex maybe 5 minutes. Tell me again how LLM’s “don’t work”?
  - pocksuppet 5mo ago
    You used it in a way where the result was simple and you could verify its correctness. You used it as a super-search tool, it's good at that. It's a different use case than having it generate a lot of code from scratch.
marcosdumay 5mo ago
There's a paywall, but it's an interesting question how much of the recent explosion of the AI companies revenues is because of the explosion in prices, and how much their customers will accept the increased prices.
tyingq 5mo ago
The abrupt swing in many non-technology company IT departments from "hey developer, you aren't using enough tokens" to this is just too funny. And I'm seeing almost no self-awareness from leaders. They are making decisions about things that they just don't understand. And are completely unworried about it. Just blindly following whatever the news cycle is about AI.
- onlyrealcuzzo 5mo ago
  The actual cost is going to drop 99% in ~4 years. How much that makes it into enterprise pricing is TBD, since none of the hyper scalers are making money yet of selling AI inference. Almost all businesses are ahead of the gun. For most of their use cases, AI is either not yet good enough on its own, or good enough but too expensive. No one wants to get left behind, so everyone's trying to get onto it now, even though it's not ready for what most enterprises want to do with it. It's easy for them to look at a small startup without billions of lines of legacy business logic debt and see them having success and wonder why they can't have just as much - or more - why they're bigger so they should have better and more success, right??? Wrong... But when it gets ~99% cheaper for local inference over the next 4 years, at the same time the price per watt improve 4x -> a lot of those cases will start to pencil out.
  - datakan 5mo ago
    What makes you think prices will drop? Everyone I’ve spoken to believes they will only skyrocket. Genuinely curious
    - onlyrealcuzzo 5mo ago
      The technology already exists now on the algorithmic front for the next 10x drop between everyone adopting DeepSeek's MLA, MoE (mostly already done), Medusa (a better version of Google's speculative decoding), Kimi's Attn Residuals, and Mimo's Sliding Window Attn, and (possibly) Microsoft's 1.58b (this may be a nothing burger). Historic trends, every 18 months, performance for the same level of quality has gone down 90%. See: https://www.reddit.com/r/LocalLLaMA/comments/1gpr2p4/llms_co https://www.reddit.com/r/LocalLLaMA/comments/1gpr2p4/llms_co... And Chart 13 here: https://www.rdworldonline.com/ais-great-compression-20-chart https://www.rdworldonline.com/ais-great-compression-20-chart... And here: https://epoch.ai/data-insights/llm-inference-price-trends https://epoch.ai/data-insights/llm-inference-price-trends Historically, algorithmic gains are only ~30% of the pie, but there's enough out there to get to 10x, with just what's available already. The other ~70% of the pie is better training data (often synthetic) and distilling frontier knowledge. There's no sign we are tapped out on that front. Additionally, GRAM (from ~10 days ago) is likely to be a 5-10x on its own (if not substantially more for smaller models). It's unlikely within 4 years LeCun's JEPA ideas and similar ideas like GRAM applied to LLMs have ZERO impact. The preliminary results are absolutely astounding (5000x better reasoning - this is not peanuts). Further, that's not even counting that cost per watt is still dropping ~2x every 2 years on its own on the hardware front. If you look at the "cost" of inference. People think it's electricity - but it's currently almost ~80% hardware amortization. The memory shortage is not going to last, nor are Nvidia's ~80-90% margins. The human brain is still 8-10 orders of magnitude more efficient than the best LLMs of today. With ~1/10th of global capex riding on AI, if you don't think they're going to knock of 2 orders of magnitude more, when it's this obvious and easy... I don't know what to tell you... Sure, it might take 6 years instead of 4. My crystal ball isn't perfect.
throwatdem12311 5mo ago
It will be interesting to see to see Anthropic’s “revenue bubble” pop as this happens. At least it should hopefully free up some capacity.
1970-01-01 5mo ago
Would have been nice to see 'soaring costs' with numbers. WSJ could do better here. Hundreds of thousands of dollars a month is nothing compared to how much they take with better financial models.
cs702 5mo ago
There's an old saying, "in the land of the blind, the one-eyed man is king." Here we have the opposite: In the land of the one-eyed, the blind are leading. The blind in this case are all those executives and managers who don't understand much about AI's current potential and limitations, and so far have treated it like a magic button that will solve everything. The one-eyed are rank-and-file employees who maybe sort of know a little more about AI.
- pocksuppet 5mo ago
  Executives and managers are the ones who correctly understood which game was being played. The game we are playing is not one of making good products, it's one of getting money from people who both have more money and are stupider than you. They're succeeding at that. We're also doing it, but we're not getting as much money.
  - bastawhiz 5mo ago
    In many cases, the people who have more money and are stupider than you are other executives. Sam Altman is arguably one of the executives who know how the game is played. OpenAI is at the front. Microsoft's executives are an example of the ones who got played.
wg0 5mo ago
The other day we (wrongly) concluded that product market fit has been achieved and now the rivers of hot molten milk chocolate and honey are all that's in the future etc.
dangus 5mo ago
I’ve seen comments on other threads on this subject the general idea that these article headlines are overstating the pullback from AI. In other words, the news cycle is looking for an AI story that lands with readers, and that the example of Uber blowing through its AI budget and Microsoft discontinuing use of Claude internally are not good indicators. I agree that those aren’t good indicators. However, at some point we have to remember that CEOs and boards of directors are just regular morons who read the news the same way everyone else does. At some point, if a lot of corporate leaders associate AI with mediocre results, high costs, and public backlash, they might just start saying “this juice isn’t worth the squeeze.”
Majeh905 5mo ago
Don't have a subscription to wsj. Only thing I can say AI was useful for, in a corporate environment, was learning a new coding language on the fly. Gives me a baseline to work off of and fix. But I can learn without it, too. A nice tool, but not a need.
- dude250711 5mo ago
  > Don't have a subscription to wsj. An ironic analogy sort of, once media started hiding behind a paywall, I just stopped reading them rather than paying. Same with LLMs - usable if cheap/free.
glass1122 5mo ago
[dead]
dgellow 5mo ago
The cost is a problem, but IMHO more important is delegating so much of your internal knowledge, thinking, and systems to a 3rd party. We are very close to the point where if Claude and ChatGPT APIs are down, companies cannot function. How is that introduced so quickly into so many critical places without taking that specific fact in consideration? What is the plan for all those companies whose workflows now depend heavily on a remote LLM whenever the services get cut? What if your company account gets banned? In some ways it is worth than depending on a company for hosting, because even your debugging tools are based on AI. MCP is great to go through datadog, sentry, until your agent or the MCP server are down and you don't know how to look for the issue yourself because you do not actually understand how your systems work.
- new_account_102 5mo ago
  [dead]
- duskdozer 5mo ago
  Those sound like problems for another quarter. The people making the decisions ride the AI hype wave, and if in the worst case the company tanks one day, they take their severance package and leave.
- HappMacDonald 5mo ago
  > We are very close to the point where if Claude and ChatGPT APIs are down, companies cannot function. Contrast with Gmail/Gsuite/Outlook365/QuickbooksOnline/etc are down, though. What you cite here isn't a direct attack on AI but on centralized service provision in general. Unfortunately that battle has been lost for decades, now.
  - dgellow 5mo ago
    None of those are doing the actual development. Here we are talking about a technology people delegate judgement, technical expertise to. It’s way, way deeper of an integration than a standard saas
ath3nd 5mo ago
[dead]
lumost 5mo ago
They are likely also starting to realize that the end result of their anthropic contract is that nobody but anthropic knows how to run their business. Why would anthropic not treat their business like a utility in the future?
UltraSane 5mo ago
These articles are weird because rationing consumption based on price is one of the most fundamental concepts in economics.
Havoc 5mo ago
Corporate or corporate in programming space? 90%+ of corporate people are not programmers. 1 programmers can cause the same token damage with a bunch of concurrent agents as a couple thousand Karens in compliance asking a chatbot questions It's much easier to deliver incremental AI ROI on the later even if it's hard to measure/quantify. A 1000 tokens might point this compliance person in the right direction on a key problem. Meanwhile 1000 tokens doesn't get you anything useful on coding
nickvec 5mo ago
https://archive.ph/v2dwg https://archive.ph/v2dwg
sbochins 5mo ago
This phase that all these companies went through doesn’t seem that bad. Before these places had a big problem where all their employees didn’t understand how to us ai for their work. Now they’ve overspent and tokenmaxxed and haven’t seen much from it. The next phase is to set the goalpost lower and set quotas based on who uses ai more effectively. Eventually the folks that use it well and are productive will bring in roi. Then you can fire all the folks that aren’t using it effectively and replace them with people that know how to use it. We’re already starting to see this.
yowo 5mo ago
- Global economy on the verge of depression. - ChatGPT drops, AI is perfect to be our savior. - AI glorified as the great messiah. - Everyone worships stocks even remotely related to AI. - Execs desperate for relevance boast about tokenmaxxing. - SHTF - burst - last year flagship GPUs and DRAM are sold used for the price of a burger. - Laidoff people start using local AI as hardware price drops to make actual useful stuff - New round of bootstrapped tech bros that eventually give birth to the new metaverse/NFT/etc.. hype.
blitzar 5mo ago
Where is the tokenmaxxing chad / chadette that burnt a half a billion dollars in a single month?
486sx33 5mo ago
[dead]
OutOfHere 5mo ago
Tokenmaxxing is absurd. Using a fixed cost monthly plan seems sensible. Taking time to review the generated code is a good thing.
kermatt 5mo ago
My org has a monthly team plan, and because I don't use what I would call unsupervised agents, I rarely come close to exceeding limits. I guess I am one of those "chat only users" that so many articles limit my output. A split terminal with Vim on the left snd Claude on the right has been a great combination. Neanderthalic AI. Personally I'd consider efficient and economical use of AI to be a key skill for a good developer. Using AI for everything at full throttle would appear to be a crutch. I wonder if this will eventually become a hiring criteria for companies without unlimited budgets (most of them).
halamadrid 5mo ago
Is Anthropic rushing to an IPO because investors also fear it’s the peak? I guess it can grow even after the IPO but is there a fear that the spending will rationalize and hurt the valuations? Like a few other commenters, other than reasonably making a software developers work easier I haven’t seen a lot of value creation or revenue boost from AI. I know AI companies make money, but the companies consuming these services, are they meaningfully making a revenue to justify the continued spend?
deleted 5mo ago
[deleted]
adam_rida 5mo ago
This feels like the natural next phase. A lot of companies went from “use as much AI as possible” to suddenly realizing that not every call needs frontier-model intelligence. In production, many calls are just repeated classification, tagging, routing, extraction, etc. The interesting problem is not banning usage. It is figuring out which parts of the traffic are safe to handle cheaply, and which parts should still go to the expensive model.

188 points by 1vuio0pswjnm7 5mo ago

In my opinion, the problem is not even the cost. The problem is that people are using AI for running recurrent stuff instead of writing code to automate it. For example. Imagine that you are comparing two documents (let's assume diff doesn't exist). You could ask an AI to compare the differences from you or you could use AI to write a tool to do it. For whatever reason, people are starting to go with the former not realizing that now they basically have to pay to compare documents.

CompoundEyes 5mo ago
Agreed. I’ve been telling my team to build up internal packages so we can push all that ad hoc reinvention into something more tangible and deterministic. Invest the $$$ in inference into something the agent can reach for next time that’s neutral and consumable by other code to reduce future spend.
- trollbridge 5mo ago
  Yes. Build compact CLI-driven tools, write a skill for it (you can use your agent to do most of this work for you). It just requires being willing to think instead of mashing prompts into a keyboard.
avereveard 5mo ago
Same, even opus favor short term solution and scripts with a billion flags that constabtly require rescanning to understand how to launch it is a constant struggle to get it to build sane default and reusable scripts that run with minimal parameters
- gonzalohm 5mo ago
  Yeah, and what's up with adding dry run to everything? I saw some code that doesn't write anything but still the AI added a dry run which had a completely different codebase
  - duskdozer 5mo ago
    Because dry run is in a lot of scripts in its training data. It's not "thinking" about the script or the concept of a dry run.
  - lanstin 5mo ago
    And everything configurable gets an environment variable. Editing the first few lines in the script is a fine way to configure things in Python.
plmpsu 5mo ago
AI can do things around semantic analysis that a deterministic diff tool cannot. I understand and agree with your point though.
- bilekas 5mo ago
  I'm curious if you could give me an example of something that couldn't be down deterministically. We have fuzzy search/matching too ? Regex is a monster when used correctly.
  - SpicyLemonZest 5mo ago
    I sometimes find myself with thousands of log lines from a problematic execution and a known good reference, wondering nonspecifically if "something weird" happened in the first one. I don't think there's any matching-based solution there; you need a scan process that understands variations in execution time, object identifiers, etc. aren't meaningful.
    - bilekas 5mo ago
      You would need specific domain knowledge and a very clever parser, I've done one for a ridiculously over engineered system but a pain. That's fair but how often would you need it? Certainly not token maxing amounts!
      - SpicyLemonZest 5mo ago
        It's a spectrum. Could it be worth it to run that as a first pass on every report of anything going wrong, just in case it produces a useful insight? Depends on how much engineering time it saves! In practice, I've found the answer seems to be "not much", because human triage is still required to understand whether the insight is correct and useful. But I'm not sure that was obvious in retrospect.
  - plmpsu 5mo ago
    Pretty much anything for which you'd need intelligence of any kind. Questions such as: Do these two paragraphs have the same semantic meaning? Do they have the same sentiment? Do these two methods have the same contract? etc. Not all documents our code and even with code deterministic tools gets you only so far.
r_lee 5mo ago
it's all about cost at the end of the day. if you're allowed and encouraged to tokenmaxx, then of course this'll happen.
jgalt212 5mo ago
I agree, but even this use case isn't the most wasteful. The interwebs says Agentic consumes 50% of token use, but I'd hazard this number is north of 90% for many shops. My cynical view of Agentic is its sole purpose is to make "number go up".
- id 5mo ago
  Look at me! I'm the smartest guy. I've wasted 10M tokens! No one has wasted more!
bluejay2387 5mo ago
I have exposure to AI initiatives at several companies including a few F500's. I have seen teams dump huge logs into frontier models that took hours to get so-so results that we were able to replace with a few lines of python code at 1000 times the speed and 100% accuracy. When asked why they were doing this they literally said "because we don't understand the subject matter so we were depending on the AI". I saw one team file a complaint with a vendor about a frontier backed coding harness and it's inability to consistently format headers because they were using it as a reporting engine. When I recommended they just use the coding tool to write code to generate reports you would have thought I had just cured cancer from their response. I frequently see people complain about the fact that AI is going to take their jobs and then see them gripe about the fact that AI is 'worthless' because it can't do more of their job than it already does. It's easy to see the difference between the people seeing 10x productivity gains from leveraging AI and those who aren't and it's not the AI.
- sbarre 5mo ago
  I've heard this framed as "AI raises the floor by 2x or less but raises the ceiling by 10x or more"
- irishcoffee 5mo ago
  Someone asked me if I was using models for fantasy sports, and if it was smart enough to help make decisions about drafting. My answer: no, but it was able to help me find the website and social handles for every beat writer for every team, and generate a simple website where I can do a daily skim of teams/players and draw my own conclusions. LLMs are a tool, not a panacea.
- mxschumacher 5mo ago
  i have trouble understanding these situations, e.g. the AI itself would presumably make the suggestion to write a python script for such a task. It seems to me that there two huge problems right now * understanding which category of problems an LLM is an appropriate solution for (rather than throwing LLMs at any and all problems) * matching model capability (and therefore cost) to the problem at hand. You can easily overspend massively by using a model that's too powerful
cyanydeez 5mo ago
Oh no! People are doing what they've been told to do!
- cindyllm 5mo ago
  [dead]
throwatdem12311 5mo ago
Laziness, pure and simple. The inevitable consequence of “the LLm is the compiler now”. And what do you even expect people to do when they are forced at threat of termination to use AI for everything as much as possible? Not to mention people are being pressured to do insane thing like review hundreds of pull requests per day and deliver like 15 features per week so OBVIOUSLY there isn’t time to build out proper tooling. Just shove everything in a prompt and call it a day. Some people have families to feed, just do what you’re told.
bilekas 5mo ago
It's this and worse. To use your example, it's like people using AI to write a diff algorithm, incorrectly, then using AI to fix it, because they don't know that diff exists already. Lazyness and starting development with a very low level of understanding. People think lowering the barrier to entry is a good thing, when in reality there are just fundamentals and things you just have to know before you can start using a tool like llms properly.
dawnerd 5mo ago
Same with writing boilerplate code. It’s been a solved problem yet here we are.
rich_sasha 5mo ago
Isn't that the supposed point of it though? At least how it is marketed/hyped. Don't use your brain, you don't need one, spend all your thinking energy on... dunno, something else, and leave all the "mundane" stuff to AI. Just pay for the tokens, it's going to make you 10x more efficient, the $1000/month is worth it.
jerojero 5mo ago
Because you look at the work from the perspective of a programmer, not the perspective of a regular person. Normal people have never gone around automating their work. The most automation they do is dynamic tables on excel sheets. I obviously know building a tool that can programmatically do something is a better solution, but I think that requires a fundamental shift in how people work. People need to be told by someone "this is how you should be using the AI" but right now they're simple told "use the AI".
- gonzalohm 5mo ago
  I'm talking about programmers doing this. That's what's sad. These were normal people before,but it feels like they have some kind of AI schizophrenia now. They don't use their brains anymore
m3nu 5mo ago
100% this. For my own company I mostly build deterministic workflows that may have a simple AI step in the middle using an appropriate Chinese model in a very limited way. I wouldn't want to burn tokens to satisfy some metric. With this AI is a fallback and not the default. Sounds like large companies have it backwards.
gib444 5mo ago
Recurrent expensive inefficient processes? Sounds familiar ! This is why we have business analysts and software developers. To help identify inefficiencies and to build technical solutions.

shanewei 5mo ago

As a developer, I don’t think it’s just that costs are going up. I’m also seeing more people lately talk about “vibe slop”.

BearOso 5mo ago
I've noticed as well. A lot of pull requests are just agents running constantly, hoping to have produced something of value. Entropy is at an all-time high, though.
- deleted 5mo ago
  [deleted]
- shanewei 5mo ago
  The bottleneck becomes human review, not code generation. A PR can look plausible and still add more entropy than value.
OptionOfT 5mo ago
We have some dude at work who runs their own agent that makes constant commits. We're supposed to review the agent's output.
- shanewei 5mo ago
  Exactly. The bottleneck becomes human review, not code generation. Agents can generate commits faster than humans can verify whether those commits should exist.

scronkfinkle 5mo ago

On the one hand, organizations are without question using LLM's well beyond what is actually necessary, and as reality kicks in they're forced to scale back accordingly. However at the same time, on intervals counted in months, we're seeing breakthroughs both in hardware and software that dramatically reduce the cost of inference. Between corporate FOMO and the rapidly decreasing costs of actually running LLM's I'm interested to see at which side of the spectrum these two meet

deleted 5mo ago

[deleted]

amazingamazing 5mo ago

AI is overhyped. I have yet to see an end user product that in itself isnt a wrapper around LLMs that is impressive created by LLM assistance. I have also yet to see dramatic increases of revenue of companies using LLMs that don't involve selling things in its supply chain. Is it a nice affordance? Sure. 1T capex good? No. If it was so good I would expect to see 2005-2015 advancements yearly. Meanwhile China is blowing past the world with real improvements in the real world- solar, EVs, etc. meanwhile people keep making their fancy sans serif websites about todo apps, faster than ever before. Useless.

dawnerd 5mo ago
Productivity gains seem like it’s at best a wash when you factor in the massive tech debt cleanup and additional time needed to spec and review.
- trollbridge 5mo ago
  Misuse of AI tools because of continuing a fundamentally broken software development process.
criddell 5mo ago
> I have yet to see an end user product that in itself isnt a wrapper around LLMs that is impressive created by LLM assistance. I don’t disagree that AI is overhyped. But I think you are probably looking in the wrong place. I think most software that is written isn’t really a product, at least not a public product. It’s an in-house tool or a one-off project needed to complete some larger task. People everywhere are always writing small programs that make their life or job just a bit easier (and explains why so many corporate projects are little more than an excel spreadsheet). And there are a lot of people who have made custom software just for themselves with AI. Not a product, just a tool or project that finally made sense to build.
- pessimizer 5mo ago
  But where's the revenue from those? It has to add up to a couple trillion dollars to break even on the capital spending.

elevation 5mo ago

Another reason to favor using AI to build automation instead of relying on it in prod: the risk of war and global instability. If LLMs are genuinely helpful or even decisive in a military engagement, you can expect any host country to commandeer whatever data centers they need, leaving commercial entities to bid up the prices on the leftover capacity. Another risk is that data centers are a great target for cyber warfare. It’s ideal if your business can leverage LLMs when they’re online but continue to operate profitably when they’re offline.

lanstin 5mo ago
Even regular warfare, if the Middle East AWS regions are an indication. The giant and arguably excessive data centers being built are not hardened physically.

ChrisArchitect 5mo ago

Some related discussions: https://news.ycombinator.com/item?id=48268871 https://news.ycombinator.com/item?id=48268871 https://news.ycombinator.com/item?id=48238896 https://news.ycombinator.com/item?id=48238896 https://news.ycombinator.com/item?id=48307098 https://news.ycombinator.com/item?id=48307098

feverzsj 5mo ago

LLM doesn't work, let alone profit.

r_lee 5mo ago
elaborate please, how does it not work?
- bigstrat2003 5mo ago
  They cannot be trusted to produce output that works (let alone works well) because they are just statistical models, without any actual understanding of what they produce. That means that you have to carefully review every single line of code they produce, because you don't know where the hallucinations will be. But by the time you do that, you have saved no time at all (indeed, in my experience you lose time), because typing the code was never the part that took time. It was understanding the problem. So if you use an LLM, you spend a bunch of money for zero gain in productivity, or you sacrifice quality and pray there aren't nasty bugs lurking. I certainly think it's fair to call that state of affairs "it doesn't work".
ninkendo 5mo ago
Yesterday I updated our dependency on the sqlx crate and put up a PR, and it failed in the CI build in a way I couldn’t reproduce locally. I asked codex to take a look, and it: - Grabbed the CI logs on its own to figure out what the CI error was - Looked at my local setup - Looked at the changes in sqlx from 0.8 to 0.9 And figured out that sqlx depends on an updated version of the “whoami” crate but doesn’t specify default features, which causes it to fall back on a stub implementation that makes the default user “anonymous”, which was failing to authenticate to the UNIX socket we use in our CI Postgres server. It patched the environment variable for our docker container to explicitly specify a username and the issue was fixed. It would’ve taken me probably several hours to figure this out on my own. It took codex maybe 5 minutes. Tell me again how LLM’s “don’t work”?
- pocksuppet 5mo ago
  You used it in a way where the result was simple and you could verify its correctness. You used it as a super-search tool, it's good at that. It's a different use case than having it generate a lot of code from scratch.

marcosdumay 5mo ago

There's a paywall, but it's an interesting question how much of the recent explosion of the AI companies revenues is because of the explosion in prices, and how much their customers will accept the increased prices.

tyingq 5mo ago

The abrupt swing in many non-technology company IT departments from "hey developer, you aren't using enough tokens" to this is just too funny. And I'm seeing almost no self-awareness from leaders. They are making decisions about things that they just don't understand. And are completely unworried about it. Just blindly following whatever the news cycle is about AI.

onlyrealcuzzo 5mo ago
The actual cost is going to drop 99% in ~4 years. How much that makes it into enterprise pricing is TBD, since none of the hyper scalers are making money yet of selling AI inference. Almost all businesses are ahead of the gun. For most of their use cases, AI is either not yet good enough on its own, or good enough but too expensive. No one wants to get left behind, so everyone's trying to get onto it now, even though it's not ready for what most enterprises want to do with it. It's easy for them to look at a small startup without billions of lines of legacy business logic debt and see them having success and wonder why they can't have just as much - or more - why they're bigger so they should have better and more success, right??? Wrong... But when it gets ~99% cheaper for local inference over the next 4 years, at the same time the price per watt improve 4x -> a lot of those cases will start to pencil out.
- datakan 5mo ago
  What makes you think prices will drop? Everyone I’ve spoken to believes they will only skyrocket. Genuinely curious
  - onlyrealcuzzo 5mo ago
    The technology already exists now on the algorithmic front for the next 10x drop between everyone adopting DeepSeek's MLA, MoE (mostly already done), Medusa (a better version of Google's speculative decoding), Kimi's Attn Residuals, and Mimo's Sliding Window Attn, and (possibly) Microsoft's 1.58b (this may be a nothing burger). Historic trends, every 18 months, performance for the same level of quality has gone down 90%. See: https://www.reddit.com/r/LocalLLaMA/comments/1gpr2p4/llms_co https://www.reddit.com/r/LocalLLaMA/comments/1gpr2p4/llms_co... And Chart 13 here: https://www.rdworldonline.com/ais-great-compression-20-chart https://www.rdworldonline.com/ais-great-compression-20-chart... And here: https://epoch.ai/data-insights/llm-inference-price-trends https://epoch.ai/data-insights/llm-inference-price-trends Historically, algorithmic gains are only ~30% of the pie, but there's enough out there to get to 10x, with just what's available already. The other ~70% of the pie is better training data (often synthetic) and distilling frontier knowledge. There's no sign we are tapped out on that front. Additionally, GRAM (from ~10 days ago) is likely to be a 5-10x on its own (if not substantially more for smaller models). It's unlikely within 4 years LeCun's JEPA ideas and similar ideas like GRAM applied to LLMs have ZERO impact. The preliminary results are absolutely astounding (5000x better reasoning - this is not peanuts). Further, that's not even counting that cost per watt is still dropping ~2x every 2 years on its own on the hardware front. If you look at the "cost" of inference. People think it's electricity - but it's currently almost ~80% hardware amortization. The memory shortage is not going to last, nor are Nvidia's ~80-90% margins. The human brain is still 8-10 orders of magnitude more efficient than the best LLMs of today. With ~1/10th of global capex riding on AI, if you don't think they're going to knock of 2 orders of magnitude more, when it's this obvious and easy... I don't know what to tell you... Sure, it might take 6 years instead of 4. My crystal ball isn't perfect.

throwatdem12311 5mo ago

It will be interesting to see to see Anthropic’s “revenue bubble” pop as this happens. At least it should hopefully free up some capacity.

1970-01-01 5mo ago

Would have been nice to see 'soaring costs' with numbers. WSJ could do better here. Hundreds of thousands of dollars a month is nothing compared to how much they take with better financial models.

cs702 5mo ago

There's an old saying, "in the land of the blind, the one-eyed man is king." Here we have the opposite: In the land of the one-eyed, the blind are leading. The blind in this case are all those executives and managers who don't understand much about AI's current potential and limitations, and so far have treated it like a magic button that will solve everything. The one-eyed are rank-and-file employees who maybe sort of know a little more about AI.

pocksuppet 5mo ago
Executives and managers are the ones who correctly understood which game was being played. The game we are playing is not one of making good products, it's one of getting money from people who both have more money and are stupider than you. They're succeeding at that. We're also doing it, but we're not getting as much money.
- bastawhiz 5mo ago
  In many cases, the people who have more money and are stupider than you are other executives. Sam Altman is arguably one of the executives who know how the game is played. OpenAI is at the front. Microsoft's executives are an example of the ones who got played.

wg0 5mo ago

The other day we (wrongly) concluded that product market fit has been achieved and now the rivers of hot molten milk chocolate and honey are all that's in the future etc.

dangus 5mo ago

I’ve seen comments on other threads on this subject the general idea that these article headlines are overstating the pullback from AI. In other words, the news cycle is looking for an AI story that lands with readers, and that the example of Uber blowing through its AI budget and Microsoft discontinuing use of Claude internally are not good indicators. I agree that those aren’t good indicators. However, at some point we have to remember that CEOs and boards of directors are just regular morons who read the news the same way everyone else does. At some point, if a lot of corporate leaders associate AI with mediocre results, high costs, and public backlash, they might just start saying “this juice isn’t worth the squeeze.”

Majeh905 5mo ago

Don't have a subscription to wsj. Only thing I can say AI was useful for, in a corporate environment, was learning a new coding language on the fly. Gives me a baseline to work off of and fix. But I can learn without it, too. A nice tool, but not a need.

dude250711 5mo ago
> Don't have a subscription to wsj. An ironic analogy sort of, once media started hiding behind a paywall, I just stopped reading them rather than paying. Same with LLMs - usable if cheap/free.

glass1122 5mo ago

[dead]

dgellow 5mo ago

The cost is a problem, but IMHO more important is delegating so much of your internal knowledge, thinking, and systems to a 3rd party. We are very close to the point where if Claude and ChatGPT APIs are down, companies cannot function. How is that introduced so quickly into so many critical places without taking that specific fact in consideration? What is the plan for all those companies whose workflows now depend heavily on a remote LLM whenever the services get cut? What if your company account gets banned? In some ways it is worth than depending on a company for hosting, because even your debugging tools are based on AI. MCP is great to go through datadog, sentry, until your agent or the MCP server are down and you don't know how to look for the issue yourself because you do not actually understand how your systems work.

new_account_102 5mo ago
[dead]
duskdozer 5mo ago
Those sound like problems for another quarter. The people making the decisions ride the AI hype wave, and if in the worst case the company tanks one day, they take their severance package and leave.
HappMacDonald 5mo ago
> We are very close to the point where if Claude and ChatGPT APIs are down, companies cannot function. Contrast with Gmail/Gsuite/Outlook365/QuickbooksOnline/etc are down, though. What you cite here isn't a direct attack on AI but on centralized service provision in general. Unfortunately that battle has been lost for decades, now.
- dgellow 5mo ago
  None of those are doing the actual development. Here we are talking about a technology people delegate judgement, technical expertise to. It’s way, way deeper of an integration than a standard saas

ath3nd 5mo ago

[dead]

lumost 5mo ago

They are likely also starting to realize that the end result of their anthropic contract is that nobody but anthropic knows how to run their business. Why would anthropic not treat their business like a utility in the future?

UltraSane 5mo ago

These articles are weird because rationing consumption based on price is one of the most fundamental concepts in economics.

Havoc 5mo ago

Corporate or corporate in programming space? 90%+ of corporate people are not programmers. 1 programmers can cause the same token damage with a bunch of concurrent agents as a couple thousand Karens in compliance asking a chatbot questions It's much easier to deliver incremental AI ROI on the later even if it's hard to measure/quantify. A 1000 tokens might point this compliance person in the right direction on a key problem. Meanwhile 1000 tokens doesn't get you anything useful on coding

nickvec 5mo ago

https://archive.ph/v2dwg https://archive.ph/v2dwg

sbochins 5mo ago

This phase that all these companies went through doesn’t seem that bad. Before these places had a big problem where all their employees didn’t understand how to us ai for their work. Now they’ve overspent and tokenmaxxed and haven’t seen much from it. The next phase is to set the goalpost lower and set quotas based on who uses ai more effectively. Eventually the folks that use it well and are productive will bring in roi. Then you can fire all the folks that aren’t using it effectively and replace them with people that know how to use it. We’re already starting to see this.

yowo 5mo ago

- Global economy on the verge of depression. - ChatGPT drops, AI is perfect to be our savior. - AI glorified as the great messiah. - Everyone worships stocks even remotely related to AI. - Execs desperate for relevance boast about tokenmaxxing. - SHTF - burst - last year flagship GPUs and DRAM are sold used for the price of a burger. - Laidoff people start using local AI as hardware price drops to make actual useful stuff - New round of bootstrapped tech bros that eventually give birth to the new metaverse/NFT/etc.. hype.

blitzar 5mo ago

Where is the tokenmaxxing chad / chadette that burnt a half a billion dollars in a single month?

486sx33 5mo ago

[dead]

OutOfHere 5mo ago

Tokenmaxxing is absurd. Using a fixed cost monthly plan seems sensible. Taking time to review the generated code is a good thing.

kermatt 5mo ago

My org has a monthly team plan, and because I don't use what I would call unsupervised agents, I rarely come close to exceeding limits. I guess I am one of those "chat only users" that so many articles limit my output. A split terminal with Vim on the left snd Claude on the right has been a great combination. Neanderthalic AI. Personally I'd consider efficient and economical use of AI to be a key skill for a good developer. Using AI for everything at full throttle would appear to be a crutch. I wonder if this will eventually become a hiring criteria for companies without unlimited budgets (most of them).

halamadrid 5mo ago

Is Anthropic rushing to an IPO because investors also fear it’s the peak? I guess it can grow even after the IPO but is there a fear that the spending will rationalize and hurt the valuations? Like a few other commenters, other than reasonably making a software developers work easier I haven’t seen a lot of value creation or revenue boost from AI. I know AI companies make money, but the companies consuming these services, are they meaningfully making a revenue to justify the continued spend?

deleted 5mo ago

[deleted]

adam_rida 5mo ago

This feels like the natural next phase. A lot of companies went from “use as much AI as possible” to suddenly realizing that not every call needs frontier-model intelligence. In production, many calls are just repeated classification, tagging, routing, extraction, etc. The interesting problem is not banning usage. It is figuring out which parts of the traffic are safe to handle cheaply, and which parts should still go to the expensive model.