16 ms·

Tokens are getting more expensive

363 points by admp 1y ago

raincole 1y ago
[flagged]
michaelbuckbee 1y ago
A major current problem is that we're smashing gnats with sledgehammers via undifferentiated model use. Not every problem needs a SOTA generalist model, and as we get systems/services that are more "bundles" of different models with specific purposes I think we will see better usage graphs.
- mustyoshi 1y ago
  Yeah this is the thing people miss a lot. 7,32b models work perfectly fine for a lot of things, and run on previously high end consumer hardware. But we're still in the hype phase, people will come to their senses once the large model performance starts to plateau
  - _heimdall 1y ago
    I expect people to come to their senses when LLM companies stop subsidizing cost and start charging customers what it actually costs them to train and run these models.
    - gunalx 1y ago
      I mean, there is no reason for a inference provider og open models to subsidice you. And costs there is usually cheaper than Claude API pricing.
      - _heimdall 1y ago
        Its still a market though, there is always the incentive to subsidize if all the competition is keeping prices artificially low.
  - bakugo 1y ago
    > 7,32b models work perfectly fine for a lot of things Like what? People always talk about how amazing it is that they can run models on their own devices, but rarely mention what they actually use them for. For most use cases, small local models will always perform significantly worse than even the most inexpensive cloud models like Gemini Flash.
    - totaa 1y ago
      Gemma 3n E4B has been crazy good for me - fine tune running on Google Cloud Run via Ollama, completely avoiding token based pricing at the cost of throughput limitations
      - pigeonhole123 1y ago
        What kind of applications are you using it for?
  - zamadatix 1y ago
    People don't want to guess which sized model is right for a task and current systems are neither good or efficient at trying to estimate that automatically. I see only the power users tweaking more and more as performance plateaus and the average user only changing when it's automatic.
- simonjgreen 1y ago
  Completely agree. It’s worth spending time to experiment too. A reasonably simple chat support system I build recently uses 5 different models dependent on the function it it’s in. Swapping out different models for different things makes a huge difference to cost, user experience, and quality.
- alecco 1y ago
  If there was an option to have Claude Opus guide Sonnet I'd use it for most interactions. Doing it manually is a hassle and breaks the flow, so I end up using Opus too often. This shouldn't be that expensive even for large prompts since input is cheaper due to parallel processing.
  - isoprophlex 1y ago
    You can define subagents that are forced to run on eg. Sonnet, and call these from your main Opus backed agent. /agent in CC for more info...
    - danielbln 1y ago
      That's what I do. I used to use Opus for the dumbest stuff, writing commits and such, but now that' all subagent business that run on Sonnet (or even Haiku sometimes). Same for running tests, executing services, docker etc. All Sonnet subagents. Positive side effect: my Opus allotment lasts a lot longer.
      - illusive4080 1y ago
        I’m just sitting here on my $20 subscription hoping one day we will get to use Opus
        edg5000 1y ago
        You can just get your own account right? Just pay out of pocket.
- nateburke 1y ago
  generalist = fungible? In the food industry is it more profitable to sell whole cakes or just the sweetener? The article makes a great point about replit and legacy ERP systems. The generative in generative AI will not replace storage, storage is where the margins live. Unless the C in CRUD can eventually replace the R and U, with the D a no-op.
  - marcosdumay 1y ago
    > In the food industry is it more profitable to sell whole cakes or just the sweetener? I really don't understand where you are trying to get. But on that example, cakes have a higher profit margin, and sweeteners have larger scale.
- deleted 1y ago
  [deleted]
- matusp 1y ago
  Yeah, but the juiciest tasks are still far from solved. The amount of tasks where people are willing to accept low accuracy answers is not that high. It is maybe true for some text processing pipelines, but all the user facing use cases require good performance.
- benreesman 1y ago
  Because none of them are good enough yet to trust completely with any task. Even the absolute best ones still fart out at surprising times, and for most stuff I have an AI that's always on, it requires no cognitive overhead to delegate to my own brain. So to delegate, it has to be a reliable win: I'm not here to make AI look good, I'm here to make my own performance be good, only a sure thing is a candidate for reflexive delegation. AI companies advertise peak AI performance, users select AI tools on worst case AI fuckups: hence, only SOTA is ever in demand. TFA illustrates this well. AI will be judged on it's worst performance, just like people are fired for their worst showing, not their best. No one cares about AI performance in ideal (read: carefully contrived) settings. We care how bad it fucks up when we take our eyes off it for 2 seconds.
- jdietrich 1y ago
  Isn't this just MoE?
- nijave 1y ago
  This is a place testing and benchmarking can definitely save you money. It's the same as compute--you can skip testing and throw money at the problem but you're going to end up paying more. We have some pretty basic guidelines at work and I think that's a decent starting point. They amount to a few example prompts/problem types and which OpenAI model to try using first for best bang for your buck. I think some of it also comes down to scale. Buying a 5 pack of sledgehammers isn't a terrible value when everything comes in a "5 pack" and you only need <= 5 tools total. Or more practically, on the small end it's more economical to run general purpose models than tailor more specific models. Once you start invoking them enough, there's a break even and flip point where spending more time on the tailored or custom model is cheaper.
comrade1234 1y ago
I'm kind of curious what IntelliJ's deal is with the different providers. I usually just keep it set to Claude but there are others that you can pick. I don't pay extra for the AI assistant - it's part of my regular subscription. I don't think I use the AI features as heavily as many others, but it does feed my code base to whoever I'm set to...
- louthy 1y ago
  Are you sure you don’t pay extra? I’m on Rider and it’s an additional cost. Unless us C# and F# devs are subsidising everyone else :D Edit: It says on the Jetbrains website: “The AI Assistant plugin is not bundled and is not enabled in IntelliJ IDEA by default. AI Assistant will not be active and will not have access to your code unless you install the plugin, acquire a JetBrains AI Service license and give your explicit consent to JetBrains AI Terms of Service and JetBrains AI Acceptable Use Policy while installing the plugin.”
  - comrade1234 1y ago
    When they first added the assistant it was $100/yr to enable it. However, it's now part of the subscription and they even reimbursed me a portion of the $100 that I paid.
    - terminalbraid 1y ago
      You're one of the lucky ones. They just outright stole from many of the people who did pay for it.
  - double051 1y ago
    If you pay for the all products subscription, their AI features are now bundled in. I believe that may be a relatively recent change, and I would not have known about it if I hadn't been curious and checked.
    - louthy 1y ago
      I’ve just checked. I have the ‘dotUltimate’ bundle, which now appears to include ‘AI Pro’. They didn’t cancel my existing ‘AI Pro’ subscription though, and have just let it keep running with no refunds. Thanks, Jetbrains. You get worse every day.
ath3nd 1y ago
Mathematics are not relevant when we have hype and vibes. We can't have facts and projections and no path to profitability distract us from our final goal. Which, of course, is to donate money to Sama so he can create AGI and be less lonely with his robotic girlfriend, I mean...change the world for the better somehow. /s
- NitpickLawyer 1y ago
  I get your point but I think it's debatable. As long as the capabilities increase (and they have, IMO) cost isn't really relevant. If you can reasonably solve problems of a given difficulty (and we're starting to see that), then suddenly you can do stuff that you simply can't with humans. You can "hire" 100 agents / servers / API bundles, whatever and "solve" all tasks with difficulty x in your business. Then you cancel and your bottom is suddenly raised. You can't do that with humans. You can't suddenly hire 100 entry-level SWEs and fire them after 3 months. Then you can think about automated labs. If things pan out, we can have the same thing in chemistry/bio/physics. Having automated labs definitely seems closer now than 2.5 years ago. Is cost relevant when you can have a lab test formulas 24/7/365? Is cost a blocker when you can have a cure to cancer_type_a? And then _b_c...etc? Also, remember that costs go down within a few generations. There's no reason to think this will stop.
  - ath3nd 1y ago
    > You can "hire" 100 agents / servers / API bundles, whatever and "solve" all tasks with difficulty x in your business. In that bright AGI future, who does my business serve, like who actually are my actual paying clients? Like, the robots are farming, the robots are driving, the robots are "creating" and robots are "thinking", right? In that awesome future, what paid jobs do us humans have, so my clients can afford my amazing entrepreneurial business that I just bootstrapped with the help of 100s of agents? And how did I get the money to hire those 100s of agents in the first place? > Is cost a blocker when you can have a cure to cancer_type_a? And then _b_c...etc? Yes, it very much is. The fact that even known and long discovered solutions like insulin for diabetes management are being sold to people at 9x its actual price should speak to you volumes that while it's great to have cures for X, Y and Z, it's the control over the production and development of these cures that is equally, if not much more important for the cure to actually reach people. In this rosy world of yours, do you think Zuck will give you his LLAMAGI-generated cancer cure out of the goodness of his heart? We are talking about the same dude that helped a couple of genocides and added ads in Whatsapp to squeeze the last cent of the people who are trapped with an app that gets progressively worse and more invasive. https://www.rand.org/news/press/2024/02/01/index1.html https://www.rand.org/news/press/2024/02/01/index1.html https://systemicjustice.org/article/facebook-and-genocide-how-facebook-contributed-to-genocide-in-myanmar-and-why-it-will-not-be-held-accountable/ https://systemicjustice.org/article/facebook-and-genocide-ho... > Also, remember that costs go down within a few generations. There's no reason to think this will stop. The destruction of the natural world, the fires all around us, the rise of fascism and nationalism, the wars that are spawning all over the place and the fact that white and blue collar jobs are being automated out while soil erosion and PFAS make our land infertile point to a different future. But yeah, I am simply ecstatic at the possibility that the costs of generating a funny picture Ghibli style with a witty caption could go down by 10 to 30%.
flyinglizard 1y ago
The truth is we're brute forcing some problems via tremendous amount of compute. Especially for apps that use AI backends (rather than chats where you interface with the LLM directly), there needs to be hybridization. I haven't used Claude Code myself but I did a screenshare session with someone who does and I think I saw it running old fashioned keyword search on the codebase. That's much more effective than just pushing more and more raw data into the chat context. On one of the systems I'm developing I'm using LLMs to compile user intents to a DSL, without every looking at the real data to be examined. There are ways; increased context length is bad for speed, cost and scalability.
mark_l_watson 1y ago
I have already thought a lot about the large packaged inference companies hitting a financial brick wall, but I was surprised by material near the end of the article: the discussions of lock in for companies that can’t switch and about Replit making money on the whole stack. Really interesting. I managed a deep learning team at Capital One and the lock-in thing is real. Replit is an interesting case study for me because after a one week free agent trial I signed up for a one year subscription, had fun the their agent LLM-based coding assistant for a few weeks, and almost never used their coding agent after that, but I still have fun with Replit as an easy way to spin up Nix based coding environments. Replit seems to offer something for everyone.
raincole 1y ago
First of all the title is click-bait. Tokens are getting cheaper and cheaper. People just use more and more tokens. And everything, I mean everything after the title is only a downhill: > saying "this car is so much cheaper now!" while pointing at a 1995 honda civic misses the point. sure, that specific car is cheaper. but the 2025 toyota camry MSRPs at $30K. Cars got cheaper. The only reason you don't feel it is trade barrier that stops BYD from flooding your local dealers. > charge 10x the price point > $200/month when cursor charges $20. start with more buffer before the bleeding begins. What does this even mean? The cheapest Cursor plan is $20, just like Claude Code. And the most expensive Cursor plan is $200, just like Claude Code. So clearly they're at the exact same price point. > switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains. > they almost certainly built this behavior directly into the model weights, which is a paradigm shift we’ll probably see a lot more of "I don't know how Claude built their models and I have no insider knowledge, but I have very strong opinions." > 3. offload processing to user machines What? > ten. billion. tokens. that's 12,500 copies of war and peace. in a month. Unironically quoting data from viberank leaderboard, which is just user-submitted number... > it's that there is no flat subscription price that works in this new world. The author doesn't know what throttling is...? I've stopped reading here. I should've just closed the tab when I saw the first letter in each sentence isn't capitalized. This is so far the most glaring signal of slop. More than the overuse of em-dash and lists.
- WA 1y ago
  All good points, but: > when I saw the first letter in each sentence isn't capitalized. This is so far the most glaring signal of slop. How so? It's the exact opposite imho. Lowercase everything with a staccato writing style to differentiate from AI slop, because LLMs usually don't write lowercase.
  - lelanthran 1y ago
    I think GP is drawing a distinction between "slop" and "AI slop". This comes across as sloppily written, but not sloppily generated.
  - 1y ago
djhworld 1y ago
Over the past year or two I've just been paying for the API access and using open source frontends like LibreChat to access these models. This has been working great for the occasional use, I'd probably top up my account by $10 every few months. I figured the amount of tokens I use is vastly smaller than the packaged plans so it made sense to go with the cheaper, pay-as-you-go approach. But since I've started dabbling in tooling like Claude Code, hoo-boy those tokens burn _fast_, like really fast. Yesterday I somehow burned through $5 of tokens in the space of about 15 minutes. I mean, sure, the Code tool is vastly different to asking an LLM about a certain topic, but I wasn't expecting such a huge leap, a lot of the token usage is masked from you I guess wrapped up in the ever increasing context + back/forth tool orchestration, but still
- TechDebtDevin 1y ago
  $20.00 via Deepseek's api (Yes China, can have my code idc), has lasted me almost a year. Its slow, but better quality output than any of the independently hosted Deepseek models (ime). I don't really use agents or anything tho.
  - vitaflo 1y ago
    Agreed, I'm still trying to use up my first $5 on Deepseek. The best thing is the off-peak rate is during the US work day and is only 55 cents per million tokens. Great for use with agents cuz you never have to worry about cost or throttling. Everyone complains about the prices of other models but there are much cheaper alternatives out there and DS is no slouch either.
- zurfer 1y ago
  The simple reason for this is that Claude Code uses way more context and repetitions than what you would use in a typical chat.
senko 1y ago
Insisting on flaunting English spelling rules (by not starting a sentence with a capital letter) in a think piece is a dead giveaway that the author thinks too highly of themselves, and results in me automatically discounting whatever they're saying. If I (and billions others) can be bothered to learn your damn language so we can all communicate, do us a service and actually use it properly, FFS.
- deleted 1y ago
  [deleted]
- SoftTalker 1y ago
  Well his name is Ethan so…
- berdario 1y ago
  > Insisting on flaunting English spelling rules... To flaunt: > display (something) ostentatiously, especially in order to provoke envy or admiration or to show defiance. To flout: > openly disregard (a rule, law, or convention). (I'm also a non-native speaker)
  - dankwizard 1y ago
    Way to flaunt that in his face!
  - feintruled 1y ago
    I was wondering if it was a way to 'flaunt' his avoidance of LLMs. Could be in future genuine human conversation will be so rare that we will seize upon mistakes and typos like finding a shiny jewel in mud. (But then I saw he used the formation - 'Honestly?' which made me think he WAS using LLMs!)
- int_19h 1y ago
  there's no official authority on what constitutes "proper english". it is a cultural convention, and conventions change over time.
furyofantares 1y ago
> claude code has had to roll back their original unlimited $200/mo tier this week The article repeats this throughout but isn't it a straight lie? The plan was named 20x because it's 20x usage limits, it always had enforced 5 hour session limits, it always had (unenforced? soft?) 50 session per month limits. It was limited, but not enough and very very probably still isn't, judging by my own usage. So I don't think the argument would even suffer from telling the truth.
- Aurornis 1y ago
  You’re right, the Max plan was never advertised as unlimited. I can’t believe how many comments and articles I’ve read that assume it was unlimited. It’s like it has been repeated so many times that it’s assumed to be true.
- adastra22 1y ago
  Also the “walkback” of the unlimited Max plan was anything but. The plan is still exactly the same for 95% of the subscribers. It just turns out that less than 5% were doing things like scripting claude to run multiple sessions 24/7 with the same login credentials. The Max plan was supposed to be a single user plan, not an infinite-parallel-sessions plan.
- t14000 1y ago
  Probably they asked AI if it was unlimited and it responded something like "oh wise and sagacious user what an amazingly insightful question! Yes, yes Max is unlimited! Would you like me to help you use an infinite amount of tokens?"
robertclaus 1y ago
My team is debating this exact question for a new product we have in early access. Ultimately we realized the issue early on, so even our plans option would include at-cost usage limits.
- arnon 1y ago
  that's a good idea but the best idea is to decouple the product you're selling from the tokens it consumes. you shouldn't be pricing compute directly (by charging for tokens yourself)
ysofunny 1y ago
and the AIs stupider! I am seeing problems with formatting that seemed 'solved' already. I mean, I have seen "the same" model get better and worse already. clearly somebody is calibrating the stupidity level relative to energy cost and monetary gain
Havoc 1y ago
The combination of "thinking models" plus the blind focus on incremental benchmarking gains was a mistake for practical use. You definitely want that for some tasks, but for the majority of tasks there is a lot of space for cheap & cheerful (and non-thinking)
mystraline 1y ago
From the article: > consumers hate metered billing. they'd rather overpay for unlimited than get surprised by a bill. Yes and no. Take Amazon. You think your costs are known and WHAMMO surprise bill. Why do you get a surprise bill? Because you cannot say 'Turn shit off at X money per month'. Can't do it. Not an option. All of these 'Surprise Net 30' offerings are the same. You think you're getting a stable price until GOTAHCA. Now, metered billing can actually be good, when the user knows exactly where they stand on the metering AND can set maximums so their budget doesn't go over. Taken realistically, as an AI company, you provide a 'used tokens/total tokens' bar graph, tokens per response, and estimated amount of responses before exceeding. Again, don't surprise the user. But that's an anathema to companies who want to hide tokens to dollars, the same way gambling companies obfuscate 'corporate bux' to USD.
- ikari_pl 1y ago
  I often find Amazon pricing to be vague and cryptic, sometimes there's literally no way to tell ehy, for example, your database cost is fluctuating all the time
  - joseda-hg 1y ago
    Amazon pricing is nice if you compare it to Azure...
  - crinkly 1y ago
    Yeah that. We moved to AWS using their best practices and enterprise cost estimation stuff and got a 6x cost increment on something that was supposed to be cheaper and now we’re fucked because we can’t get out. It’s nearly impossible to tell what the hell is going where and we are mostly surviving on enterprise discounts from negotiations. The worst thing is they worked out you can blend costs in using AWS marketplace without having to raise due diligence on a new vendor or PO. So up it goes even more. Not my department or funeral fortunately. Our AWS account is about $15 a month.
    - bufferoverflow 1y ago
abtinf 1y ago
Lack of proper capitalization makes the text unreadable for me.
- blamestross 1y ago
  https://convertcase.net/browser-extension/ https://convertcase.net/browser-extension/ This extension might make the internet more accessible for you!
  - machomaster 1y ago
    If the writer is that lazy to press Shift and do it manually, then it is him who should have used autocapitalization software.
- machomaster 1y ago
  You are not the only one. I really don't understand this trend of wanting to share opinions, but purposefully making it harder/impossible for other to read it. Might as well write with dark-grey font over black background, just to make readers struggle extra hard. If you don't care to trivially make your text readable, then we for sure don't care to spend time to struggle through your text to see if there is any useful substance there.
deleted 1y ago
[deleted]
happytoexplain 1y ago
While reading this, every time I started a paragraph and saw a lowercase, my brain and eyes were stalling or jumping up, to reflexively look for the text that got cut off. My brain has been trained for decades that, when reading full prose, a paragraph starting with lowercase means I'm starting in the middle of a sentence, and something happened in the layout or HTML to interrupt it. And, I know this seems dramatic, but besides being cognitively distracting, it also makes me feel sad. Chatroom formatting in published writings is clearly a developing trend at this point, and I love my language so much. Not in a linguistic capacity - I'm not an English expert or anything, nor do I follow every rule - I mean in an emotional capacity. I'm not trying to be condescending. This is a style choice, not "bad writing" in the typical sense. I realize there is often a lot of low-quality bitterness on both sides about this kind of thing. Edit: I also fear that this is exactly the kind of thing where any opinion in opposition to this style will feel like the kind of attack that makes a writer want to push back in a "oh yeah? fuck you" kind of way. I.e. even just my writing this opinion may give an author using the style in question the desire to "double down". Though this conundrum is appropriate (ironic?) - the intensely personal nature of language is part of why I love it.
- simianwords 1y ago
  It’s to draw contrast against extremely polished and sterile looking slop content. Think of it like avoiding em dash but going a bit far.
  - rafram 1y ago
    > all content here is generated by ai
  - tanseydavid 1y ago
    It is lazy.
- egypturnash 1y ago
  IT COULD BE WORSE, YOU COULD BE READING A LENGTHY ESSAY PRESENTED ENTIRELY IN ALL CAPS WITH MINIMAL PUNCTUATION TO BREAK IT UP SEARCH FOR “FILM CRIT HULK” FOR SOME EXAMPLES
strangescript 1y ago
We haven't reached a peak on scaling/performance, so even if an old model can be commoditized, a new one will be created to take advantage of the newly freed infra. Until we hit a ceiling on scaling, tokens are going to remain expensive relative to what people are trying to do with them because the underlying compute is expensive.
dcre 1y ago
Vibes-based analysis. We have no idea how much these models cost to serve.
Waterluvian 1y ago
On the topic of cost per token, is it accurate to represent a token as, ideally, a composable atomic unit of information. But because we’re (often) using English as the encoding format, it can only be as efficient as English can encode the data. Does this mean that other languages might offer better information density per token? And does this mean that we could invent a language that’s more efficient for these purposes, and something humans (perhaps only those who want a job as a prompt engineer) could be taught? Kevin speak good? https://youtu.be/_K-L9uhsBLM?si=t3zuEAmspuvmefwz https://youtu.be/_K-L9uhsBLM?si=t3zuEAmspuvmefwz
- deleted 1y ago
  [deleted]
- deegles 1y ago
  Human speech has a bit rate of around 39 bits per second, no matter how quickly you speak. assuming reading is similar, I guess more "dense" tokens would just take longer for humans to read. https://www.science.org/content/article/human-speech-may-have-universal-transmission-rate-39-bits-second https://www.science.org/content/article/human-speech-may-hav...
  - __s 1y ago
    Sure, but that link has Japanese at 5 bits per syllable & Vietnamese at 8 bits per syllable, so if billing was based on syllables per prompt you'd want Vietnamese prompts Granted English is probably going to have better quality output based on training data size
- r_lee 1y ago
  Sure, for example Korean is unicode heavy, e.g. 경찰 = police, but its just 2 unicode chars. Not too familiar with how things are encoded but it could be more efficient
- joseda-hg 1y ago
  IIRC, in linguistics there's a hypothesis for "Uniform Information density" languages seem to follow on a human level (Denser languages slow down, sparse languages speed up) so you might have to go for an Artificial encoding, that maps effectively to english English (And any of the dominant languages that you could use in it's place) work significantly better than other languages purely by having significantly larger bodies of work for the LLM to work from
GiorgioG 1y ago
I tried Gemini CLI and in 2 hours somehow spent $22 just messing around with a very small codebase. I didn’t find out until the next day from Google’s billing system. That was enough for me - I won’t touch it again.
- adrianbooth17 1y ago
  Isn't Gemini CLI free? Or did you BYOK?
  - GiorgioG 1y ago
    BYOK
  - squirrellous 1y ago
    The free usage gets exhausted really fast. I had a similar experience as the parent where I easily burned through $70+ in two days with two requests in a medium size codebase. Then I switched to free, and it couldn’t finish a single request end to end. On top of this Gemini CLI still doesn’t support paying through the Google AI subscription. I assume it’s some sort of bureaucratic reason that’s preventing them from moving quickly.
- JofArnold 1y ago
  Same. 30m tokens in a few hours (of which many were cached it seems and only a few output tokens). I use Gemini to solve only specific problems and in my case $20+ was worth it.
ankit219 1y ago
Interesting article, full of speculation and some logical follows, but feels like it feels short of admitting what the true conclusion is. Model building companies can build thinner wrapper / harness and can offer better prices than third party companies (the article assumes it costs anthropic same price per token as it does for their customers) because their costs per token is lower than app layer companies. Anthropic has a decent margin (likely higher than openai) on sale of every token, and with more scale, they can sell at a lower cost (or some unlimited plans with limits that keeps out 1%-5% of the power users). I don't agree with the Cognition conclusion either. Enterprises are fighting super hard to not have a long term buying contract when they know SOTA (app or model) is different every 6 months. They are keeping their switching costs low and making sure they own the workflow, not the tool. This is even more prominent after Slack restricted API usage for enterprise customers. Making money on the infra is possible, but that again misunderstands the pricing power of Anthropic. Lovable, Replit etc. work because of Claude. Openai had codex, google had jules, both aren't as good in terms of taste compared to Claude. It's not the cli form factor which people love, it's the outcome they like. When Anthropic sees the money being left on the table in infra play, they will offer the same (at presumably better rates given Amazon is an investor) and likely repeat this strategy. Abstraction is a good play, only if you abstract it to the maximum possible levels.
xrd 1y ago
This is the moment an open source solution could pop in and say just "uv add aider" and then make sure you have a 24gb card for Qwen3 for each dev, and you are future proofed for at least the next year. It seems like the only way out.
- bufferoverflow 1y ago
  [dead]
ej88 1y ago
The article just isn't that coherent for me. > when a new model is released as the SOTA, 99% of the demand immediately shifts over to it 99% is in the wrong ballpark. Lots of users use Sonnet 4 over Opus 4, despite Opus being 'more' SOTA. Lots of users use 4o over o3 or Gemini over Claude. In fact it's never been a closer race on who is the 'best': https://openrouter.ai/rankings https://openrouter.ai/rankings >switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains. they almost certainly built this behavior directly into the model weights ??? Overall the article seems to argue that companies are running into issues with usage-based pricing due to consumers not accepting or being used to usage based pricing and it's difficult to be the first person to crack and switch to usage based. I don't think it's as big of an issue as the author makes it out to be. We've seen this play out before in cloud hosting. - Lots of consumers are OK with a flat fee per month and using an inferior model. 4o is objectively inferior to o3 but millions of people use it (or don't know any better). The free ChatGPT is even worse than 4o and the vast majority of chatgpt visitors use it! - Heavy users or businesses consume via API and usage based pricing (see cloud). This is almost certainly profitable. - Fundamentally most of these startups are B2B, not B2C
- motorest 1y ago
  > In fact it's never been a closer race on who is the 'best' Thank you for pointing out that fact. Sometimes it's very hard to keep perspective. Sometimes I use Mistral as my main LLM. I know it's not lauded as the top performing LLM but the truth of the matter is that it's results are just as useful as the best models that ChatGPT/Gemini/Claude outputs, and it is way faster. There is indeed diminished returns on the current blend of commercial LLMs. Deep seek already proved that cost can be a major factor and quality can even improve. I think we're very close to see competition based on price, which might be the reason there is so much talk about mixture of experts approaches and how specialized models can drive down cost while improving targeted output.
  - torginus 1y ago
    Yeah, my biggest problem with CC is that it's slow, prone to generating tons of bullshit exposition, and often goes down paths that I can tell almost immediately will yield no useful result. It's great if you can leave it unattended, but personally, coding's an active thing for me, and watching it go is really frustrating.
jsnell 1y ago
> now look at the actual pricing history of frontier models, the ones that 99% of the demand is for at any given time: The meaningful frontier isn't scalar on just the capability, it's on capability for a given cost. The highest capability models are not where 99% of the demand is on. Actually the opposite. To get an idea of what point on the frontier people prefer, have a look at the OpenRouter statistics (https://openrouter.ai/rankings https://openrouter.ai/rankings). Claude Opus 4 has about 1% of their total usage, not 99%. Claude Sonnet 4 is the single most popular model at about 18%. The runners up in volume are Gemini Flash 2.0 and 2.5, which are in turn significantly cheaper than Sonnet 4.
- bakugo 1y ago
  This is true. I agree with the overall premise of the article, but claiming that Opus is more used than Sonnet is just wrong. One of the graphs even lists a "Claude 3.5 Opus", which does not exist. After 3.5 Sonnet was released, 3 Opus largely fell into irrelevance until they decided to finally release another big, expensive model with Opus 4, which still isn't anywhere near as popular as Sonnet 4 with users who pay API prices.
codr7 1y ago
This is such a nice setup! They can deliver pretty much whatever they feel like. Who can tell a trash token from an hallucination? And tracking token usage is a pita. Sum it up and it translates to: sell whatever you feel like at whatever price you feel like. Nice!
sshine 1y ago
> nobody opens claude and thinks, "you know what? let me use the shitty version" Sure I do! I will consistently pick the fastest and cheapest model that will do the job. Sonnet > Opus when coding Haiku > Sonnet when fusing kitchen recipes, or answering questions where search results deliver the bulk of the value, and the LLM part is really just for summarizing.
esafak 1y ago
This is wrong. People are not dropping old models when new ones come out. I'm always on the lookout for cost effective models. The logical thing is to use the cheapest model that gets the job done, and you get a sense for that once you the model for a while. It is standard practice with some coding agents to have different models for different tasks, like building and planning.
farkin88 1y ago
Even though tokens are getting cheaper, I think the real killer of "unlimited" LLM plans isn't token costs themselves, it's the shape of the usage curve that's unsustainable. These products see a Zipf-like distribution: thousands of casual users nibble a few-hundred tokens a day while a tiny group of power automations devour tens of millions. Flat pricing works fine until one of those whales drops a repo-wide refactor or a 100 MB PDF into chat and instantly torpedoes the margin. Unless vendors turn those extreme loops into cheaper, purpose-built primitives (search, static analyzers, local quantized models, etc.), every "all-you-can-eat" AI subscription is just a slow-motion implosion waiting for its next whale.
- andyferris 1y ago
  I actually think Anthropic's plans with capped usage per 5-hour period and per week is good, for exactly this problem. I'd prefer it just specify a number of tokens rather than be variable on demand - I see that lets them be more generous during low periods but the opacity of it all sucks. I have 5-minute time-of-use pricing on my electricty and can look up the current rate on my phone in an insant - why not simply provide an API to look up the current "demand factor" for Claude (along with the rules for how the demand factor can change - min and max values for example) and let it be fully transparent?
  - farkin88 1y ago
    Anthropic still relies on quotas (5-hour rolling + weekly coming at the end of the month) so the next step is dynamic per-token pricing. But, even with transparent off-peak rates only batch jobs will shift and history suggests variable pricing usually smooths rather than sharpens peaks. The long-term fix stays the same: route 100 MB PDFs, repo refactors and other whale jobs through retrieval or analysis pipelines and keep the flagship chat model for real-time conversation.
blotfaba 1y ago
We're not going to be using tokens forever, and inevitably specialized hardware will solve this bottleneck. Underestimating how much proprietary advancements are loaded into Google TPUs is sort of like thinking the best we've got are Acura TSXs when somebody's driving around in a Ferrari.
jstummbillig 1y ago
This is silly? The important metric is value per token, which is obviously increasing, and thus the relative token is getting cheaper because you need far less of them to produce anything of value. Which then might lead to you using a lot more, because it offsets some other thing that costs even more still, like your time.
- acedTrex 1y ago
  "Which is obviously increasing" With the primary advancements over the past two years being Chain Of Thought which absolutely obliterates token counts in what world would the "per token" value of a model be going up...
  - jstummbillig 1y ago
    If you are able to cogently explain how you would instruct GPT 3.5 with ANY amount of tokens to do what Sonnet 4 is able to do, I am sure there's a lot of wealthy people that would be very interested in having a talk with you.
    - imtringued 1y ago
      You build a reasoning variant of GPT 3.5. Please deposit 1 billion dollars in my bank account for this advice.
_0ffh 1y ago
Dunno, I'm happy to pay for API access based on token usage. I'd never so much as look at flat pricing, but maybe that's just me.
torginus 1y ago
First of all, do they shoot you in San Francisco, if you use capital letters and punctuation? Second, why are SV people obsessed with fake exponentials? It's very clear that AI progress has only been exponential in the sense that people are throwing a lot more resources at AI then they did a couple years ago.
- 369548684892826 1y ago
  > First of all, do they shoot you in San Francisco, if you use capital letters and punctuation? Is it done like this just to show it wasn't written by a LLM?
  - handfuloflight 1y ago
    The Substack tagline for the account is "all content here is generated by ai"
  - SweetSoftPillow 1y ago
    It doesn't show that. You can prompt AI to omit capital letters and punctuation.
- luqtas 1y ago
  oh no! i can't deal with the natural morphing a lingua-franca has! /j Thou needst to live in the archaic.
- Lucasoato 1y ago
  If you go in Tenderloin or Mission Street yes, but even if you don't use capital letters and punctuation.
mensetmanusman 1y ago
They will soon be subsidized by ads or people will run their own.
jongjong 1y ago
There are a few forces at play and there is a much bigger overarching problem. Even before AI, it felt like the value of intelligence and knowledge had been dropping over time. This makes sense as the internet has democratized access to information and promoted intellectual self-improvement. The supply of intelligence increased dramatically but demand for it struggled to keep up (in spite of the tech boom). Now demand for intelligence has plateaued; This is one way to look at current tech layoffs. It got to a point that intelligence is almost worthless now. 'Earning' money is mostly about social connections, not intelligence. So all these use cases which people are using AI for are pointless in terms of earning money in the current system. The current system rewards money-acquisition, it does not reward value-creation. You don't need intelligence to acquire money in this system; you need social connections. AI does not give you social connections; if anything, it takes away social connections. The people using AI to build themselves an amazing second internet will have nobody to share it with; no users, no investment. The oversupply of intelligence means that it cannot find any serious avenues to earn a financial return, so instead it turns to political manipulations because system reform (or manipulation) is the shortest path to earning monetary returns... Though often this manipulation only further decouples value creation from money-acquisition.
reilly3000 1y ago
I’m really liking the experience of using Cline with OpenRouter. You get detailed token count and context size for every Task as it goes, along with the actual price in realtime. That combined with the ability to use virtually any model with a single key makes it easier to determine what work to give to what model, and if it’s worth it to you. Use the best models for the most important work, cheap/free ones for the rest, and do the rest yourself. That way you drive your model, not when the model decides enough is enough.
johnfn 1y ago
Some AI company needs to create a model that can delegate simple tasks to 'stupider' models. I often encounter a task which is complicated enough to require a strong model like Opus, but which subdivides into a number of tasks - the vast majority of which are enough that 3.5 Sonnet could pick it up. All Opus would need to do is subdivide the task into easy and hard bits, then spin up a bunch of 3.5 Sonnets for the easy stuff. This seems like such an obvious idea that I'm sure everyone is already working on it!
- throwmeaway222 1y ago
  Maybe have the prompt tell the model emit the model "level" needed between 1 and 10 for each subtask it generates
- deleted 1y ago
  [deleted]
- kgilpin 1y ago
  Claude code does utilize both the full Sonnet model and the lighter Haiku model in an automatic way. When you exit a Claude code session, it gives you the stats (tokens, cost, etc). I expect there’s a way to get this in-session as well.
  - mhlakhani 1y ago
    I believe you can just hit /cost within a session for this
  - subarctic 1y ago
    Only if you're using it without a subscription, and without one it doesn't take much usage to get to $20 in a month
- andai 1y ago
  I think Auto-GPT or a similar project did something like this in 2023. Aider has an option where you can combine different models, so one does the thinking and one implements the changes. https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ e.g. o3 (high) + 4.1 There used to be more such combinations in this list but it seems they were removed. A while ago I think the best thing on the list was such a hybrid.
ninetyninenine 1y ago
Solution? Move processing to the edge. I want a local model on my desktop. I want training to be some peer to peer open source processing network. We did it for SPA's time to do it for LLMs. I'd pay up to 3000 for a compute module that can do this stuff locally.
cadamsdotcom 1y ago
The article misses the “musical chairs” game that gets played in land-grabs. Look at Uber. Lost money for over a decade buying market share with venture capital. Now post IPO they have settled in to a position in the minds of users that’s hard to shake even as cheaper competition arrives. They have a durable business and a steady (even if not amazing) stock price.
- qcnguy 1y ago
  Not sure it's that durable. Bolt is growing fast in Europe at least.
nojs 1y ago
The article implies that nobody pays usage-based pricing, but aren’t all API customers (ie essentially all business users) doing this now?
KennyBlanken 1y ago
I see all these people in the comments talking about how they have no idea how much their services are going to cost and my only thought is "how is this legal? how has the US FTC not stepped in, or EU regulators?"
gdiamos 1y ago
The NVIDIA stock price keeps going up because accuracy/intelligence is more valuable than efficiency Scaling laws let you spend more transistors and watts on intelligence Do you want more tokens or smarter tokens?
- totaa 1y ago
  more can lead to smarter in the short term. brute force
andai 1y ago
>People will always use the heaviest model Actually when doing my first attempt at vibe coding a few months ago, I found that Gemini Flash was fine for my tasks, and way faster than the heavier models. So I found the smaller model a vastly superior user experience. The speed really adds up when you're using the autonomous coding agents, since they tend to require many LLM calls for a few simple changes.
ktzar 1y ago
Also, the way models are evolving (thinking process, llms waiting for interactions with external entities via MCP, mixture of experts, ...) are making "useful chatbot responses" way way way more expensive than they used to be when you were pretty much hitting an autocomplete. To a level where these are starting to be prohibitive to run locally at a decent tokens/s speed, and we're being tied to using their models.
fennecfoxy 1y ago
Because models aren't becoming more evolved, just bigger - that's the commercial incentive. NOW NOW NOW PROFITS NOW, aka VC & all what YC is about. We need another attention is all you need, or three, imo. I hope the gold rush isn't impacting research, but I bet it is.
d4rkn0d3z 1y ago
Wow, comments read like a usage anxiety help forum.
drudolph914 1y ago
wow is the uber moment for AI already over? that was fast

363 points by admp 1y ago

[flagged]

A major current problem is that we're smashing gnats with sledgehammers via undifferentiated model use. Not every problem needs a SOTA generalist model, and as we get systems/services that are more "bundles" of different models with specific purposes I think we will see better usage graphs.

mustyoshi 1y ago
Yeah this is the thing people miss a lot. 7,32b models work perfectly fine for a lot of things, and run on previously high end consumer hardware. But we're still in the hype phase, people will come to their senses once the large model performance starts to plateau
- _heimdall 1y ago
  I expect people to come to their senses when LLM companies stop subsidizing cost and start charging customers what it actually costs them to train and run these models.
  - gunalx 1y ago
    I mean, there is no reason for a inference provider og open models to subsidice you. And costs there is usually cheaper than Claude API pricing.
    - _heimdall 1y ago
      Its still a market though, there is always the incentive to subsidize if all the competition is keeping prices artificially low.
- bakugo 1y ago
  > 7,32b models work perfectly fine for a lot of things Like what? People always talk about how amazing it is that they can run models on their own devices, but rarely mention what they actually use them for. For most use cases, small local models will always perform significantly worse than even the most inexpensive cloud models like Gemini Flash.
  - totaa 1y ago
    Gemma 3n E4B has been crazy good for me - fine tune running on Google Cloud Run via Ollama, completely avoiding token based pricing at the cost of throughput limitations
    - pigeonhole123 1y ago
      What kind of applications are you using it for?
- zamadatix 1y ago
  People don't want to guess which sized model is right for a task and current systems are neither good or efficient at trying to estimate that automatically. I see only the power users tweaking more and more as performance plateaus and the average user only changing when it's automatic.
simonjgreen 1y ago
Completely agree. It’s worth spending time to experiment too. A reasonably simple chat support system I build recently uses 5 different models dependent on the function it it’s in. Swapping out different models for different things makes a huge difference to cost, user experience, and quality.
alecco 1y ago
If there was an option to have Claude Opus guide Sonnet I'd use it for most interactions. Doing it manually is a hassle and breaks the flow, so I end up using Opus too often. This shouldn't be that expensive even for large prompts since input is cheaper due to parallel processing.
- isoprophlex 1y ago
  You can define subagents that are forced to run on eg. Sonnet, and call these from your main Opus backed agent. /agent in CC for more info...
  - danielbln 1y ago
    That's what I do. I used to use Opus for the dumbest stuff, writing commits and such, but now that' all subagent business that run on Sonnet (or even Haiku sometimes). Same for running tests, executing services, docker etc. All Sonnet subagents. Positive side effect: my Opus allotment lasts a lot longer.
    - illusive4080 1y ago
      I’m just sitting here on my $20 subscription hoping one day we will get to use Opus
      - edg5000 1y ago
        You can just get your own account right? Just pay out of pocket.
nateburke 1y ago
generalist = fungible? In the food industry is it more profitable to sell whole cakes or just the sweetener? The article makes a great point about replit and legacy ERP systems. The generative in generative AI will not replace storage, storage is where the margins live. Unless the C in CRUD can eventually replace the R and U, with the D a no-op.
- marcosdumay 1y ago
  > In the food industry is it more profitable to sell whole cakes or just the sweetener? I really don't understand where you are trying to get. But on that example, cakes have a higher profit margin, and sweeteners have larger scale.
deleted 1y ago
[deleted]
matusp 1y ago
Yeah, but the juiciest tasks are still far from solved. The amount of tasks where people are willing to accept low accuracy answers is not that high. It is maybe true for some text processing pipelines, but all the user facing use cases require good performance.
benreesman 1y ago
Because none of them are good enough yet to trust completely with any task. Even the absolute best ones still fart out at surprising times, and for most stuff I have an AI that's always on, it requires no cognitive overhead to delegate to my own brain. So to delegate, it has to be a reliable win: I'm not here to make AI look good, I'm here to make my own performance be good, only a sure thing is a candidate for reflexive delegation. AI companies advertise peak AI performance, users select AI tools on worst case AI fuckups: hence, only SOTA is ever in demand. TFA illustrates this well. AI will be judged on it's worst performance, just like people are fired for their worst showing, not their best. No one cares about AI performance in ideal (read: carefully contrived) settings. We care how bad it fucks up when we take our eyes off it for 2 seconds.
jdietrich 1y ago
Isn't this just MoE?
nijave 1y ago
This is a place testing and benchmarking can definitely save you money. It's the same as compute--you can skip testing and throw money at the problem but you're going to end up paying more. We have some pretty basic guidelines at work and I think that's a decent starting point. They amount to a few example prompts/problem types and which OpenAI model to try using first for best bang for your buck. I think some of it also comes down to scale. Buying a 5 pack of sledgehammers isn't a terrible value when everything comes in a "5 pack" and you only need <= 5 tools total. Or more practically, on the small end it's more economical to run general purpose models than tailor more specific models. Once you start invoking them enough, there's a break even and flip point where spending more time on the tailored or custom model is cheaper.

comrade1234 1y ago

I'm kind of curious what IntelliJ's deal is with the different providers. I usually just keep it set to Claude but there are others that you can pick. I don't pay extra for the AI assistant - it's part of my regular subscription. I don't think I use the AI features as heavily as many others, but it does feed my code base to whoever I'm set to...

louthy 1y ago
Are you sure you don’t pay extra? I’m on Rider and it’s an additional cost. Unless us C# and F# devs are subsidising everyone else :D Edit: It says on the Jetbrains website: “The AI Assistant plugin is not bundled and is not enabled in IntelliJ IDEA by default. AI Assistant will not be active and will not have access to your code unless you install the plugin, acquire a JetBrains AI Service license and give your explicit consent to JetBrains AI Terms of Service and JetBrains AI Acceptable Use Policy while installing the plugin.”
- comrade1234 1y ago
  When they first added the assistant it was $100/yr to enable it. However, it's now part of the subscription and they even reimbursed me a portion of the $100 that I paid.
  - terminalbraid 1y ago
    You're one of the lucky ones. They just outright stole from many of the people who did pay for it.
- double051 1y ago
  If you pay for the all products subscription, their AI features are now bundled in. I believe that may be a relatively recent change, and I would not have known about it if I hadn't been curious and checked.
  - louthy 1y ago
    I’ve just checked. I have the ‘dotUltimate’ bundle, which now appears to include ‘AI Pro’. They didn’t cancel my existing ‘AI Pro’ subscription though, and have just let it keep running with no refunds. Thanks, Jetbrains. You get worse every day.

ath3nd 1y ago

Mathematics are not relevant when we have hype and vibes. We can't have facts and projections and no path to profitability distract us from our final goal. Which, of course, is to donate money to Sama so he can create AGI and be less lonely with his robotic girlfriend, I mean...change the world for the better somehow. /s

NitpickLawyer 1y ago
I get your point but I think it's debatable. As long as the capabilities increase (and they have, IMO) cost isn't really relevant. If you can reasonably solve problems of a given difficulty (and we're starting to see that), then suddenly you can do stuff that you simply can't with humans. You can "hire" 100 agents / servers / API bundles, whatever and "solve" all tasks with difficulty x in your business. Then you cancel and your bottom is suddenly raised. You can't do that with humans. You can't suddenly hire 100 entry-level SWEs and fire them after 3 months. Then you can think about automated labs. If things pan out, we can have the same thing in chemistry/bio/physics. Having automated labs definitely seems closer now than 2.5 years ago. Is cost relevant when you can have a lab test formulas 24/7/365? Is cost a blocker when you can have a cure to cancer_type_a? And then _b_c...etc? Also, remember that costs go down within a few generations. There's no reason to think this will stop.
- ath3nd 1y ago
  > You can "hire" 100 agents / servers / API bundles, whatever and "solve" all tasks with difficulty x in your business. In that bright AGI future, who does my business serve, like who actually are my actual paying clients? Like, the robots are farming, the robots are driving, the robots are "creating" and robots are "thinking", right? In that awesome future, what paid jobs do us humans have, so my clients can afford my amazing entrepreneurial business that I just bootstrapped with the help of 100s of agents? And how did I get the money to hire those 100s of agents in the first place? > Is cost a blocker when you can have a cure to cancer_type_a? And then _b_c...etc? Yes, it very much is. The fact that even known and long discovered solutions like insulin for diabetes management are being sold to people at 9x its actual price should speak to you volumes that while it's great to have cures for X, Y and Z, it's the control over the production and development of these cures that is equally, if not much more important for the cure to actually reach people. In this rosy world of yours, do you think Zuck will give you his LLAMAGI-generated cancer cure out of the goodness of his heart? We are talking about the same dude that helped a couple of genocides and added ads in Whatsapp to squeeze the last cent of the people who are trapped with an app that gets progressively worse and more invasive. https://www.rand.org/news/press/2024/02/01/index1.html https://www.rand.org/news/press/2024/02/01/index1.html https://systemicjustice.org/article/facebook-and-genocide-how-facebook-contributed-to-genocide-in-myanmar-and-why-it-will-not-be-held-accountable/ https://systemicjustice.org/article/facebook-and-genocide-ho... > Also, remember that costs go down within a few generations. There's no reason to think this will stop. The destruction of the natural world, the fires all around us, the rise of fascism and nationalism, the wars that are spawning all over the place and the fact that white and blue collar jobs are being automated out while soil erosion and PFAS make our land infertile point to a different future. But yeah, I am simply ecstatic at the possibility that the costs of generating a funny picture Ghibli style with a witty caption could go down by 10 to 30%.

flyinglizard 1y ago

The truth is we're brute forcing some problems via tremendous amount of compute. Especially for apps that use AI backends (rather than chats where you interface with the LLM directly), there needs to be hybridization. I haven't used Claude Code myself but I did a screenshare session with someone who does and I think I saw it running old fashioned keyword search on the codebase. That's much more effective than just pushing more and more raw data into the chat context. On one of the systems I'm developing I'm using LLMs to compile user intents to a DSL, without every looking at the real data to be examined. There are ways; increased context length is bad for speed, cost and scalability.

mark_l_watson 1y ago

I have already thought a lot about the large packaged inference companies hitting a financial brick wall, but I was surprised by material near the end of the article: the discussions of lock in for companies that can’t switch and about Replit making money on the whole stack. Really interesting. I managed a deep learning team at Capital One and the lock-in thing is real. Replit is an interesting case study for me because after a one week free agent trial I signed up for a one year subscription, had fun the their agent LLM-based coding assistant for a few weeks, and almost never used their coding agent after that, but I still have fun with Replit as an easy way to spin up Nix based coding environments. Replit seems to offer something for everyone.

raincole 1y ago

First of all the title is click-bait. Tokens are getting cheaper and cheaper. People just use more and more tokens. And everything, I mean everything after the title is only a downhill: > saying "this car is so much cheaper now!" while pointing at a 1995 honda civic misses the point. sure, that specific car is cheaper. but the 2025 toyota camry MSRPs at $30K. Cars got cheaper. The only reason you don't feel it is trade barrier that stops BYD from flooding your local dealers. > charge 10x the price point > $200/month when cursor charges $20. start with more buffer before the bleeding begins. What does this even mean? The cheapest Cursor plan is $20, just like Claude Code. And the most expensive Cursor plan is $200, just like Claude Code. So clearly they're at the exact same price point. > switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains. > they almost certainly built this behavior directly into the model weights, which is a paradigm shift we’ll probably see a lot more of "I don't know how Claude built their models and I have no insider knowledge, but I have very strong opinions." > 3. offload processing to user machines What? > ten. billion. tokens. that's 12,500 copies of war and peace. in a month. Unironically quoting data from viberank leaderboard, which is just user-submitted number... > it's that there is no flat subscription price that works in this new world. The author doesn't know what throttling is...? I've stopped reading here. I should've just closed the tab when I saw the first letter in each sentence isn't capitalized. This is so far the most glaring signal of slop. More than the overuse of em-dash and lists.

WA 1y ago
All good points, but: > when I saw the first letter in each sentence isn't capitalized. This is so far the most glaring signal of slop. How so? It's the exact opposite imho. Lowercase everything with a staccato writing style to differentiate from AI slop, because LLMs usually don't write lowercase.
- lelanthran 1y ago
  I think GP is drawing a distinction between "slop" and "AI slop". This comes across as sloppily written, but not sloppily generated.
- 1y ago

djhworld 1y ago

Over the past year or two I've just been paying for the API access and using open source frontends like LibreChat to access these models. This has been working great for the occasional use, I'd probably top up my account by $10 every few months. I figured the amount of tokens I use is vastly smaller than the packaged plans so it made sense to go with the cheaper, pay-as-you-go approach. But since I've started dabbling in tooling like Claude Code, hoo-boy those tokens burn _fast_, like really fast. Yesterday I somehow burned through $5 of tokens in the space of about 15 minutes. I mean, sure, the Code tool is vastly different to asking an LLM about a certain topic, but I wasn't expecting such a huge leap, a lot of the token usage is masked from you I guess wrapped up in the ever increasing context + back/forth tool orchestration, but still

TechDebtDevin 1y ago
$20.00 via Deepseek's api (Yes China, can have my code idc), has lasted me almost a year. Its slow, but better quality output than any of the independently hosted Deepseek models (ime). I don't really use agents or anything tho.
- vitaflo 1y ago
  Agreed, I'm still trying to use up my first $5 on Deepseek. The best thing is the off-peak rate is during the US work day and is only 55 cents per million tokens. Great for use with agents cuz you never have to worry about cost or throttling. Everyone complains about the prices of other models but there are much cheaper alternatives out there and DS is no slouch either.
zurfer 1y ago
The simple reason for this is that Claude Code uses way more context and repetitions than what you would use in a typical chat.

senko 1y ago

Insisting on flaunting English spelling rules (by not starting a sentence with a capital letter) in a think piece is a dead giveaway that the author thinks too highly of themselves, and results in me automatically discounting whatever they're saying. If I (and billions others) can be bothered to learn your damn language so we can all communicate, do us a service and actually use it properly, FFS.

deleted 1y ago
[deleted]
SoftTalker 1y ago
Well his name is Ethan so…
berdario 1y ago
> Insisting on flaunting English spelling rules... To flaunt: > display (something) ostentatiously, especially in order to provoke envy or admiration or to show defiance. To flout: > openly disregard (a rule, law, or convention). (I'm also a non-native speaker)
- dankwizard 1y ago
  Way to flaunt that in his face!
- feintruled 1y ago
  I was wondering if it was a way to 'flaunt' his avoidance of LLMs. Could be in future genuine human conversation will be so rare that we will seize upon mistakes and typos like finding a shiny jewel in mud. (But then I saw he used the formation - 'Honestly?' which made me think he WAS using LLMs!)
int_19h 1y ago
there's no official authority on what constitutes "proper english". it is a cultural convention, and conventions change over time.

furyofantares 1y ago

> claude code has had to roll back their original unlimited $200/mo tier this week The article repeats this throughout but isn't it a straight lie? The plan was named 20x because it's 20x usage limits, it always had enforced 5 hour session limits, it always had (unenforced? soft?) 50 session per month limits. It was limited, but not enough and very very probably still isn't, judging by my own usage. So I don't think the argument would even suffer from telling the truth.

Aurornis 1y ago
You’re right, the Max plan was never advertised as unlimited. I can’t believe how many comments and articles I’ve read that assume it was unlimited. It’s like it has been repeated so many times that it’s assumed to be true.
adastra22 1y ago
Also the “walkback” of the unlimited Max plan was anything but. The plan is still exactly the same for 95% of the subscribers. It just turns out that less than 5% were doing things like scripting claude to run multiple sessions 24/7 with the same login credentials. The Max plan was supposed to be a single user plan, not an infinite-parallel-sessions plan.
t14000 1y ago
Probably they asked AI if it was unlimited and it responded something like "oh wise and sagacious user what an amazingly insightful question! Yes, yes Max is unlimited! Would you like me to help you use an infinite amount of tokens?"

robertclaus 1y ago

My team is debating this exact question for a new product we have in early access. Ultimately we realized the issue early on, so even our plans option would include at-cost usage limits.

arnon 1y ago
that's a good idea but the best idea is to decouple the product you're selling from the tokens it consumes. you shouldn't be pricing compute directly (by charging for tokens yourself)

ysofunny 1y ago

and the AIs stupider! I am seeing problems with formatting that seemed 'solved' already. I mean, I have seen "the same" model get better and worse already. clearly somebody is calibrating the stupidity level relative to energy cost and monetary gain

Havoc 1y ago

The combination of "thinking models" plus the blind focus on incremental benchmarking gains was a mistake for practical use. You definitely want that for some tasks, but for the majority of tasks there is a lot of space for cheap & cheerful (and non-thinking)

mystraline 1y ago

From the article: > consumers hate metered billing. they'd rather overpay for unlimited than get surprised by a bill. Yes and no. Take Amazon. You think your costs are known and WHAMMO surprise bill. Why do you get a surprise bill? Because you cannot say 'Turn shit off at X money per month'. Can't do it. Not an option. All of these 'Surprise Net 30' offerings are the same. You think you're getting a stable price until GOTAHCA. Now, metered billing can actually be good, when the user knows exactly where they stand on the metering AND can set maximums so their budget doesn't go over. Taken realistically, as an AI company, you provide a 'used tokens/total tokens' bar graph, tokens per response, and estimated amount of responses before exceeding. Again, don't surprise the user. But that's an anathema to companies who want to hide tokens to dollars, the same way gambling companies obfuscate 'corporate bux' to USD.

ikari_pl 1y ago
I often find Amazon pricing to be vague and cryptic, sometimes there's literally no way to tell ehy, for example, your database cost is fluctuating all the time
- joseda-hg 1y ago
  Amazon pricing is nice if you compare it to Azure...
- crinkly 1y ago
  Yeah that. We moved to AWS using their best practices and enterprise cost estimation stuff and got a 6x cost increment on something that was supposed to be cheaper and now we’re fucked because we can’t get out. It’s nearly impossible to tell what the hell is going where and we are mostly surviving on enterprise discounts from negotiations. The worst thing is they worked out you can blend costs in using AWS marketplace without having to raise due diligence on a new vendor or PO. So up it goes even more. Not my department or funeral fortunately. Our AWS account is about $15 a month.
  - bufferoverflow 1y ago

abtinf 1y ago

Lack of proper capitalization makes the text unreadable for me.

blamestross 1y ago
https://convertcase.net/browser-extension/ https://convertcase.net/browser-extension/ This extension might make the internet more accessible for you!
- machomaster 1y ago
  If the writer is that lazy to press Shift and do it manually, then it is him who should have used autocapitalization software.
machomaster 1y ago
You are not the only one. I really don't understand this trend of wanting to share opinions, but purposefully making it harder/impossible for other to read it. Might as well write with dark-grey font over black background, just to make readers struggle extra hard. If you don't care to trivially make your text readable, then we for sure don't care to spend time to struggle through your text to see if there is any useful substance there.

deleted 1y ago

[deleted]

happytoexplain 1y ago

While reading this, every time I started a paragraph and saw a lowercase, my brain and eyes were stalling or jumping up, to reflexively look for the text that got cut off. My brain has been trained for decades that, when reading full prose, a paragraph starting with lowercase means I'm starting in the middle of a sentence, and something happened in the layout or HTML to interrupt it. And, I know this seems dramatic, but besides being cognitively distracting, it also makes me feel sad. Chatroom formatting in published writings is clearly a developing trend at this point, and I love my language so much. Not in a linguistic capacity - I'm not an English expert or anything, nor do I follow every rule - I mean in an emotional capacity. I'm not trying to be condescending. This is a style choice, not "bad writing" in the typical sense. I realize there is often a lot of low-quality bitterness on both sides about this kind of thing. Edit: I also fear that this is exactly the kind of thing where any opinion in opposition to this style will feel like the kind of attack that makes a writer want to push back in a "oh yeah? fuck you" kind of way. I.e. even just my writing this opinion may give an author using the style in question the desire to "double down". Though this conundrum is appropriate (ironic?) - the intensely personal nature of language is part of why I love it.

simianwords 1y ago
It’s to draw contrast against extremely polished and sterile looking slop content. Think of it like avoiding em dash but going a bit far.
- rafram 1y ago
  > all content here is generated by ai
- tanseydavid 1y ago
  It is lazy.
egypturnash 1y ago
IT COULD BE WORSE, YOU COULD BE READING A LENGTHY ESSAY PRESENTED ENTIRELY IN ALL CAPS WITH MINIMAL PUNCTUATION TO BREAK IT UP SEARCH FOR “FILM CRIT HULK” FOR SOME EXAMPLES

strangescript 1y ago

We haven't reached a peak on scaling/performance, so even if an old model can be commoditized, a new one will be created to take advantage of the newly freed infra. Until we hit a ceiling on scaling, tokens are going to remain expensive relative to what people are trying to do with them because the underlying compute is expensive.

dcre 1y ago

Vibes-based analysis. We have no idea how much these models cost to serve.

Waterluvian 1y ago

On the topic of cost per token, is it accurate to represent a token as, ideally, a composable atomic unit of information. But because we’re (often) using English as the encoding format, it can only be as efficient as English can encode the data. Does this mean that other languages might offer better information density per token? And does this mean that we could invent a language that’s more efficient for these purposes, and something humans (perhaps only those who want a job as a prompt engineer) could be taught? Kevin speak good? https://youtu.be/_K-L9uhsBLM?si=t3zuEAmspuvmefwz https://youtu.be/_K-L9uhsBLM?si=t3zuEAmspuvmefwz

deleted 1y ago
[deleted]
deegles 1y ago
Human speech has a bit rate of around 39 bits per second, no matter how quickly you speak. assuming reading is similar, I guess more "dense" tokens would just take longer for humans to read. https://www.science.org/content/article/human-speech-may-have-universal-transmission-rate-39-bits-second https://www.science.org/content/article/human-speech-may-hav...
- __s 1y ago
  Sure, but that link has Japanese at 5 bits per syllable & Vietnamese at 8 bits per syllable, so if billing was based on syllables per prompt you'd want Vietnamese prompts Granted English is probably going to have better quality output based on training data size
r_lee 1y ago
Sure, for example Korean is unicode heavy, e.g. 경찰 = police, but its just 2 unicode chars. Not too familiar with how things are encoded but it could be more efficient
joseda-hg 1y ago
IIRC, in linguistics there's a hypothesis for "Uniform Information density" languages seem to follow on a human level (Denser languages slow down, sparse languages speed up) so you might have to go for an Artificial encoding, that maps effectively to english English (And any of the dominant languages that you could use in it's place) work significantly better than other languages purely by having significantly larger bodies of work for the LLM to work from

GiorgioG 1y ago

I tried Gemini CLI and in 2 hours somehow spent $22 just messing around with a very small codebase. I didn’t find out until the next day from Google’s billing system. That was enough for me - I won’t touch it again.

adrianbooth17 1y ago
Isn't Gemini CLI free? Or did you BYOK?
- GiorgioG 1y ago
  BYOK
- squirrellous 1y ago
  The free usage gets exhausted really fast. I had a similar experience as the parent where I easily burned through $70+ in two days with two requests in a medium size codebase. Then I switched to free, and it couldn’t finish a single request end to end. On top of this Gemini CLI still doesn’t support paying through the Google AI subscription. I assume it’s some sort of bureaucratic reason that’s preventing them from moving quickly.
JofArnold 1y ago
Same. 30m tokens in a few hours (of which many were cached it seems and only a few output tokens). I use Gemini to solve only specific problems and in my case $20+ was worth it.

ankit219 1y ago

Interesting article, full of speculation and some logical follows, but feels like it feels short of admitting what the true conclusion is. Model building companies can build thinner wrapper / harness and can offer better prices than third party companies (the article assumes it costs anthropic same price per token as it does for their customers) because their costs per token is lower than app layer companies. Anthropic has a decent margin (likely higher than openai) on sale of every token, and with more scale, they can sell at a lower cost (or some unlimited plans with limits that keeps out 1%-5% of the power users). I don't agree with the Cognition conclusion either. Enterprises are fighting super hard to not have a long term buying contract when they know SOTA (app or model) is different every 6 months. They are keeping their switching costs low and making sure they own the workflow, not the tool. This is even more prominent after Slack restricted API usage for enterprise customers. Making money on the infra is possible, but that again misunderstands the pricing power of Anthropic. Lovable, Replit etc. work because of Claude. Openai had codex, google had jules, both aren't as good in terms of taste compared to Claude. It's not the cli form factor which people love, it's the outcome they like. When Anthropic sees the money being left on the table in infra play, they will offer the same (at presumably better rates given Amazon is an investor) and likely repeat this strategy. Abstraction is a good play, only if you abstract it to the maximum possible levels.

xrd 1y ago

This is the moment an open source solution could pop in and say just "uv add aider" and then make sure you have a 24gb card for Qwen3 for each dev, and you are future proofed for at least the next year. It seems like the only way out.

bufferoverflow 1y ago
[dead]

ej88 1y ago

The article just isn't that coherent for me. > when a new model is released as the SOTA, 99% of the demand immediately shifts over to it 99% is in the wrong ballpark. Lots of users use Sonnet 4 over Opus 4, despite Opus being 'more' SOTA. Lots of users use 4o over o3 or Gemini over Claude. In fact it's never been a closer race on who is the 'best': https://openrouter.ai/rankings https://openrouter.ai/rankings >switch from opus ($75/m tokens) to sonnet ($15/m) when things get heavy. optimize with haiku for reading. like aws autoscaling, but for brains. they almost certainly built this behavior directly into the model weights ??? Overall the article seems to argue that companies are running into issues with usage-based pricing due to consumers not accepting or being used to usage based pricing and it's difficult to be the first person to crack and switch to usage based. I don't think it's as big of an issue as the author makes it out to be. We've seen this play out before in cloud hosting. - Lots of consumers are OK with a flat fee per month and using an inferior model. 4o is objectively inferior to o3 but millions of people use it (or don't know any better). The free ChatGPT is even worse than 4o and the vast majority of chatgpt visitors use it! - Heavy users or businesses consume via API and usage based pricing (see cloud). This is almost certainly profitable. - Fundamentally most of these startups are B2B, not B2C

motorest 1y ago
> In fact it's never been a closer race on who is the 'best' Thank you for pointing out that fact. Sometimes it's very hard to keep perspective. Sometimes I use Mistral as my main LLM. I know it's not lauded as the top performing LLM but the truth of the matter is that it's results are just as useful as the best models that ChatGPT/Gemini/Claude outputs, and it is way faster. There is indeed diminished returns on the current blend of commercial LLMs. Deep seek already proved that cost can be a major factor and quality can even improve. I think we're very close to see competition based on price, which might be the reason there is so much talk about mixture of experts approaches and how specialized models can drive down cost while improving targeted output.
- torginus 1y ago
  Yeah, my biggest problem with CC is that it's slow, prone to generating tons of bullshit exposition, and often goes down paths that I can tell almost immediately will yield no useful result. It's great if you can leave it unattended, but personally, coding's an active thing for me, and watching it go is really frustrating.

jsnell 1y ago

> now look at the actual pricing history of frontier models, the ones that 99% of the demand is for at any given time: The meaningful frontier isn't scalar on just the capability, it's on capability for a given cost. The highest capability models are not where 99% of the demand is on. Actually the opposite. To get an idea of what point on the frontier people prefer, have a look at the OpenRouter statistics (https://openrouter.ai/rankings https://openrouter.ai/rankings). Claude Opus 4 has about 1% of their total usage, not 99%. Claude Sonnet 4 is the single most popular model at about 18%. The runners up in volume are Gemini Flash 2.0 and 2.5, which are in turn significantly cheaper than Sonnet 4.

bakugo 1y ago
This is true. I agree with the overall premise of the article, but claiming that Opus is more used than Sonnet is just wrong. One of the graphs even lists a "Claude 3.5 Opus", which does not exist. After 3.5 Sonnet was released, 3 Opus largely fell into irrelevance until they decided to finally release another big, expensive model with Opus 4, which still isn't anywhere near as popular as Sonnet 4 with users who pay API prices.

codr7 1y ago

This is such a nice setup! They can deliver pretty much whatever they feel like. Who can tell a trash token from an hallucination? And tracking token usage is a pita. Sum it up and it translates to: sell whatever you feel like at whatever price you feel like. Nice!

sshine 1y ago

> nobody opens claude and thinks, "you know what? let me use the shitty version" Sure I do! I will consistently pick the fastest and cheapest model that will do the job. Sonnet > Opus when coding Haiku > Sonnet when fusing kitchen recipes, or answering questions where search results deliver the bulk of the value, and the LLM part is really just for summarizing.

esafak 1y ago

This is wrong. People are not dropping old models when new ones come out. I'm always on the lookout for cost effective models. The logical thing is to use the cheapest model that gets the job done, and you get a sense for that once you the model for a while. It is standard practice with some coding agents to have different models for different tasks, like building and planning.

farkin88 1y ago

Even though tokens are getting cheaper, I think the real killer of "unlimited" LLM plans isn't token costs themselves, it's the shape of the usage curve that's unsustainable. These products see a Zipf-like distribution: thousands of casual users nibble a few-hundred tokens a day while a tiny group of power automations devour tens of millions. Flat pricing works fine until one of those whales drops a repo-wide refactor or a 100 MB PDF into chat and instantly torpedoes the margin. Unless vendors turn those extreme loops into cheaper, purpose-built primitives (search, static analyzers, local quantized models, etc.), every "all-you-can-eat" AI subscription is just a slow-motion implosion waiting for its next whale.

andyferris 1y ago
I actually think Anthropic's plans with capped usage per 5-hour period and per week is good, for exactly this problem. I'd prefer it just specify a number of tokens rather than be variable on demand - I see that lets them be more generous during low periods but the opacity of it all sucks. I have 5-minute time-of-use pricing on my electricty and can look up the current rate on my phone in an insant - why not simply provide an API to look up the current "demand factor" for Claude (along with the rules for how the demand factor can change - min and max values for example) and let it be fully transparent?
- farkin88 1y ago
  Anthropic still relies on quotas (5-hour rolling + weekly coming at the end of the month) so the next step is dynamic per-token pricing. But, even with transparent off-peak rates only batch jobs will shift and history suggests variable pricing usually smooths rather than sharpens peaks. The long-term fix stays the same: route 100 MB PDFs, repo refactors and other whale jobs through retrieval or analysis pipelines and keep the flagship chat model for real-time conversation.

blotfaba 1y ago

We're not going to be using tokens forever, and inevitably specialized hardware will solve this bottleneck. Underestimating how much proprietary advancements are loaded into Google TPUs is sort of like thinking the best we've got are Acura TSXs when somebody's driving around in a Ferrari.

jstummbillig 1y ago

This is silly? The important metric is value per token, which is obviously increasing, and thus the relative token is getting cheaper because you need far less of them to produce anything of value. Which then might lead to you using a lot more, because it offsets some other thing that costs even more still, like your time.

acedTrex 1y ago
"Which is obviously increasing" With the primary advancements over the past two years being Chain Of Thought which absolutely obliterates token counts in what world would the "per token" value of a model be going up...
- jstummbillig 1y ago
  If you are able to cogently explain how you would instruct GPT 3.5 with ANY amount of tokens to do what Sonnet 4 is able to do, I am sure there's a lot of wealthy people that would be very interested in having a talk with you.
  - imtringued 1y ago
    You build a reasoning variant of GPT 3.5. Please deposit 1 billion dollars in my bank account for this advice.

_0ffh 1y ago

Dunno, I'm happy to pay for API access based on token usage. I'd never so much as look at flat pricing, but maybe that's just me.

torginus 1y ago

First of all, do they shoot you in San Francisco, if you use capital letters and punctuation? Second, why are SV people obsessed with fake exponentials? It's very clear that AI progress has only been exponential in the sense that people are throwing a lot more resources at AI then they did a couple years ago.

369548684892826 1y ago
> First of all, do they shoot you in San Francisco, if you use capital letters and punctuation? Is it done like this just to show it wasn't written by a LLM?
- handfuloflight 1y ago
  The Substack tagline for the account is "all content here is generated by ai"
- SweetSoftPillow 1y ago
  It doesn't show that. You can prompt AI to omit capital letters and punctuation.
luqtas 1y ago
oh no! i can't deal with the natural morphing a lingua-franca has! /j Thou needst to live in the archaic.
Lucasoato 1y ago
If you go in Tenderloin or Mission Street yes, but even if you don't use capital letters and punctuation.

mensetmanusman 1y ago

They will soon be subsidized by ads or people will run their own.

jongjong 1y ago

There are a few forces at play and there is a much bigger overarching problem. Even before AI, it felt like the value of intelligence and knowledge had been dropping over time. This makes sense as the internet has democratized access to information and promoted intellectual self-improvement. The supply of intelligence increased dramatically but demand for it struggled to keep up (in spite of the tech boom). Now demand for intelligence has plateaued; This is one way to look at current tech layoffs. It got to a point that intelligence is almost worthless now. 'Earning' money is mostly about social connections, not intelligence. So all these use cases which people are using AI for are pointless in terms of earning money in the current system. The current system rewards money-acquisition, it does not reward value-creation. You don't need intelligence to acquire money in this system; you need social connections. AI does not give you social connections; if anything, it takes away social connections. The people using AI to build themselves an amazing second internet will have nobody to share it with; no users, no investment. The oversupply of intelligence means that it cannot find any serious avenues to earn a financial return, so instead it turns to political manipulations because system reform (or manipulation) is the shortest path to earning monetary returns... Though often this manipulation only further decouples value creation from money-acquisition.

reilly3000 1y ago

I’m really liking the experience of using Cline with OpenRouter. You get detailed token count and context size for every Task as it goes, along with the actual price in realtime. That combined with the ability to use virtually any model with a single key makes it easier to determine what work to give to what model, and if it’s worth it to you. Use the best models for the most important work, cheap/free ones for the rest, and do the rest yourself. That way you drive your model, not when the model decides enough is enough.

johnfn 1y ago

Some AI company needs to create a model that can delegate simple tasks to 'stupider' models. I often encounter a task which is complicated enough to require a strong model like Opus, but which subdivides into a number of tasks - the vast majority of which are enough that 3.5 Sonnet could pick it up. All Opus would need to do is subdivide the task into easy and hard bits, then spin up a bunch of 3.5 Sonnets for the easy stuff. This seems like such an obvious idea that I'm sure everyone is already working on it!

throwmeaway222 1y ago
Maybe have the prompt tell the model emit the model "level" needed between 1 and 10 for each subtask it generates
deleted 1y ago
[deleted]
kgilpin 1y ago
Claude code does utilize both the full Sonnet model and the lighter Haiku model in an automatic way. When you exit a Claude code session, it gives you the stats (tokens, cost, etc). I expect there’s a way to get this in-session as well.
- mhlakhani 1y ago
  I believe you can just hit /cost within a session for this
- subarctic 1y ago
  Only if you're using it without a subscription, and without one it doesn't take much usage to get to $20 in a month
andai 1y ago
I think Auto-GPT or a similar project did something like this in 2023. Aider has an option where you can combine different models, so one does the thinking and one implements the changes. https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ e.g. o3 (high) + 4.1 There used to be more such combinations in this list but it seems they were removed. A while ago I think the best thing on the list was such a hybrid.

ninetyninenine 1y ago

Solution? Move processing to the edge. I want a local model on my desktop. I want training to be some peer to peer open source processing network. We did it for SPA's time to do it for LLMs. I'd pay up to 3000 for a compute module that can do this stuff locally.

cadamsdotcom 1y ago

The article misses the “musical chairs” game that gets played in land-grabs. Look at Uber. Lost money for over a decade buying market share with venture capital. Now post IPO they have settled in to a position in the minds of users that’s hard to shake even as cheaper competition arrives. They have a durable business and a steady (even if not amazing) stock price.

qcnguy 1y ago
Not sure it's that durable. Bolt is growing fast in Europe at least.

nojs 1y ago

The article implies that nobody pays usage-based pricing, but aren’t all API customers (ie essentially all business users) doing this now?

KennyBlanken 1y ago

I see all these people in the comments talking about how they have no idea how much their services are going to cost and my only thought is "how is this legal? how has the US FTC not stepped in, or EU regulators?"

gdiamos 1y ago

The NVIDIA stock price keeps going up because accuracy/intelligence is more valuable than efficiency Scaling laws let you spend more transistors and watts on intelligence Do you want more tokens or smarter tokens?

totaa 1y ago
more can lead to smarter in the short term. brute force

andai 1y ago

>People will always use the heaviest model Actually when doing my first attempt at vibe coding a few months ago, I found that Gemini Flash was fine for my tasks, and way faster than the heavier models. So I found the smaller model a vastly superior user experience. The speed really adds up when you're using the autonomous coding agents, since they tend to require many LLM calls for a few simple changes.

ktzar 1y ago

Also, the way models are evolving (thinking process, llms waiting for interactions with external entities via MCP, mixture of experts, ...) are making "useful chatbot responses" way way way more expensive than they used to be when you were pretty much hitting an autocomplete. To a level where these are starting to be prohibitive to run locally at a decent tokens/s speed, and we're being tied to using their models.

fennecfoxy 1y ago

Because models aren't becoming more evolved, just bigger - that's the commercial incentive. NOW NOW NOW PROFITS NOW, aka VC & all what YC is about. We need another attention is all you need, or three, imo. I hope the gold rush isn't impacting research, but I bet it is.

d4rkn0d3z 1y ago

Wow, comments read like a usage anxiety help forum.

drudolph914 1y ago

wow is the uber moment for AI already over? that was fast