3 ms·

> I caution anyone looking at API prices: they dropped the price, but is it actually less expensive? > Yeah, they dropped the price, but count the number of to

> I caution anyone looking at API prices: they dropped the price, but is it actually less expensive? > Yeah, they dropped the price, but count the number of tokens you're tossing into it and see if it's actually cheaper. It's good for marketing, but the rub is how much you're actually using. The level of discourse is so horrible now, I don't have words. Are these the ones making predictions on AI bubble?

empath75 2mo ago
Of course, the price of tokens going down makes it cheaper. You can now afford to throw millions of tokens at solving a problem that would have been impossible for AI to solve at all a year ago for any price. People completely lack imagination about this stuff. The main problem right now with AI isn't even AI successfully producing code at a reasonable cost, it's human coordination and review that is the bottleneck.
- HarHarVeryFunny 2mo ago
  I think it's both. It's easy to run up a massive bill with AI without much to show for it, which is why token-maxxing has now been replaced by cost-awareness and AI budgeting such as Uber's "max 10% of salary per employee". Cost isn't just token price though - it's (number-of-tokens-used x price-per-token), and there are large differences in token efficiency between different models and harnesses. Increasingly we're seeing benchmark sites focusing on "cost per completed task" as a cost metric, and it's not always the cheapest tokens that win. I agree that ultimately AI/coding cost is just part of the picture - at the end of the day it's about software development cost, which for time being involves humans.
InsideOutSanta 2mo ago
This is poorly stated in the podcast, but the underlying point is correct: while cost-per-token is going down, overall token use is way up. This is causing the cost of using LLMs in corporations to skyrocket.
- simianwords 2mo ago
  Do you first agree that not only is the cost-per-token going down, cost per task is going down which obviously encourages people to use it more? If a car company releases a new car with higher mileage, would you suggest that cars are getting costlier?
  - InsideOutSanta 2mo ago
    No, cost per task is probably not going down on average. The underlying issue is that LLMs are still bad at most tasks they could be used for, so as their contexts get larger and they're able to run longer without becoming incoherent, more tokens are spent on tasks to improve the quality of output. So cost per task is probably going up on average, but so is the quality of the output.
    - simianwords 2mo ago
      I obviously mean cost per fixed task. Do you agree that if you fix a task, the cost is going down? Meaning you get more work done from the same cost over the years.
      - InsideOutSanta 2mo ago
        Yes, obviously. The same task at the same quality and the same speed got cheaper. That's not what anyone is disputing. The problem is that overall expenses are going up.
        simianwords 2mo ago
        If a fruit company developed fruit that was cheaper but people ended up buying more fruits, would you then call fruits more expensive or more cheap? I assume cheap. Then why would you call AI more expensive?
        InsideOutSanta 2mo ago
ac-ciano 2mo ago
[flagged]

empath75 2mo ago

Of course, the price of tokens going down makes it cheaper. You can now afford to throw millions of tokens at solving a problem that would have been impossible for AI to solve at all a year ago for any price. People completely lack imagination about this stuff. The main problem right now with AI isn't even AI successfully producing code at a reasonable cost, it's human coordination and review that is the bottleneck.

HarHarVeryFunny 2mo ago
I think it's both. It's easy to run up a massive bill with AI without much to show for it, which is why token-maxxing has now been replaced by cost-awareness and AI budgeting such as Uber's "max 10% of salary per employee". Cost isn't just token price though - it's (number-of-tokens-used x price-per-token), and there are large differences in token efficiency between different models and harnesses. Increasingly we're seeing benchmark sites focusing on "cost per completed task" as a cost metric, and it's not always the cheapest tokens that win. I agree that ultimately AI/coding cost is just part of the picture - at the end of the day it's about software development cost, which for time being involves humans.

InsideOutSanta 2mo ago

This is poorly stated in the podcast, but the underlying point is correct: while cost-per-token is going down, overall token use is way up. This is causing the cost of using LLMs in corporations to skyrocket.

simianwords 2mo ago
Do you first agree that not only is the cost-per-token going down, cost per task is going down which obviously encourages people to use it more? If a car company releases a new car with higher mileage, would you suggest that cars are getting costlier?
- InsideOutSanta 2mo ago
  No, cost per task is probably not going down on average. The underlying issue is that LLMs are still bad at most tasks they could be used for, so as their contexts get larger and they're able to run longer without becoming incoherent, more tokens are spent on tasks to improve the quality of output. So cost per task is probably going up on average, but so is the quality of the output.
  - simianwords 2mo ago
    I obviously mean cost per fixed task. Do you agree that if you fix a task, the cost is going down? Meaning you get more work done from the same cost over the years.
    - InsideOutSanta 2mo ago
      Yes, obviously. The same task at the same quality and the same speed got cheaper. That's not what anyone is disputing. The problem is that overall expenses are going up.
      - simianwords 2mo ago
        If a fruit company developed fruit that was cheaper but people ended up buying more fruits, would you then call fruits more expensive or more cheap? I assume cheap. Then why would you call AI more expensive?
        InsideOutSanta 2mo ago

ac-ciano 2mo ago

[flagged]