15 ms·

This is poorly stated in the podcast, but the underlying point is correct: while cost-per-token is going down, overall token use is way up. This is causing the

This is poorly stated in the podcast, but the underlying point is correct: while cost-per-token is going down, overall token use is way up. This is causing the cost of using LLMs in corporations to skyrocket.

simianwords 2mo ago
Do you first agree that not only is the cost-per-token going down, cost per task is going down which obviously encourages people to use it more? If a car company releases a new car with higher mileage, would you suggest that cars are getting costlier?
- InsideOutSanta 2mo ago
  No, cost per task is probably not going down on average. The underlying issue is that LLMs are still bad at most tasks they could be used for, so as their contexts get larger and they're able to run longer without becoming incoherent, more tokens are spent on tasks to improve the quality of output. So cost per task is probably going up on average, but so is the quality of the output.
  - simianwords 2mo ago
    I obviously mean cost per fixed task. Do you agree that if you fix a task, the cost is going down? Meaning you get more work done from the same cost over the years.
    - InsideOutSanta 2mo ago
      Yes, obviously. The same task at the same quality and the same speed got cheaper. That's not what anyone is disputing. The problem is that overall expenses are going up.
      - simianwords 2mo ago
        If a fruit company developed fruit that was cheaper but people ended up buying more fruits, would you then call fruits more expensive or more cheap? I assume cheap. Then why would you call AI more expensive?
        InsideOutSanta 2mo ago
        I think you're arguing with the wrong person. Please note what I actually said: > This is poorly stated in the podcast, but the underlying point is correct: while cost-per-token is going down, overall token use is way up. This is causing the cost of using LLMs in corporations to skyrocket.
        simianwords 2mo ago
        > If the model consumes four times as many tokens to deliver a result, it’s not cheaper This is literally what the guy says in podcast. So the underlying point is not correct. He’s specifically talking about per task cost. There’s no “but actually they meant something else”. I don’t discount your point about overall cost increasing but that’s not relevant here. Let’s agree that the podcast is fundamentally wrong in their main claim. I want to show that the podcasters should be discredited because they don’t understand a fundamental aspect of the economics so you can’t trust the main thesis.
        InsideOutSanta 2mo ago
        Here's the actual quote from the podcast you're presumably referring to. I guess everybody can make up their own mind about what they're saying and whether they are "fundamentally wrong." You gotta watch something, because harnesses have changed how you can look at the cost structure of these things. If you look at cost per million tokens, it might look lower, but if the model consumes four times as many tokens in order to deliver a meaningful result, it's not cheaper. And so, Tom Claburn, one of our senior software reporters, had an excellent piece looking at how Anthropic's latest models use a tremendous number of tokens in order to deliver the result. So, sure, OpenAI's latest models might look less expensive from an API standpoint, which is great for marketing, but if it's using twice as many tokens, that's not the same thing. And that's somewhat dependent on the harness, but it's also dependent on how much reasoning effort is put into it, how they're routing the models.

simianwords 2mo ago

Do you first agree that not only is the cost-per-token going down, cost per task is going down which obviously encourages people to use it more? If a car company releases a new car with higher mileage, would you suggest that cars are getting costlier?

InsideOutSanta 2mo ago
No, cost per task is probably not going down on average. The underlying issue is that LLMs are still bad at most tasks they could be used for, so as their contexts get larger and they're able to run longer without becoming incoherent, more tokens are spent on tasks to improve the quality of output. So cost per task is probably going up on average, but so is the quality of the output.
- simianwords 2mo ago
  I obviously mean cost per fixed task. Do you agree that if you fix a task, the cost is going down? Meaning you get more work done from the same cost over the years.
  - InsideOutSanta 2mo ago
    Yes, obviously. The same task at the same quality and the same speed got cheaper. That's not what anyone is disputing. The problem is that overall expenses are going up.
    - simianwords 2mo ago
      If a fruit company developed fruit that was cheaper but people ended up buying more fruits, would you then call fruits more expensive or more cheap? I assume cheap. Then why would you call AI more expensive?
      - InsideOutSanta 2mo ago
        I think you're arguing with the wrong person. Please note what I actually said: > This is poorly stated in the podcast, but the underlying point is correct: while cost-per-token is going down, overall token use is way up. This is causing the cost of using LLMs in corporations to skyrocket.
        simianwords 2mo ago
        > If the model consumes four times as many tokens to deliver a result, it’s not cheaper This is literally what the guy says in podcast. So the underlying point is not correct. He’s specifically talking about per task cost. There’s no “but actually they meant something else”. I don’t discount your point about overall cost increasing but that’s not relevant here. Let’s agree that the podcast is fundamentally wrong in their main claim. I want to show that the podcasters should be discredited because they don’t understand a fundamental aspect of the economics so you can’t trust the main thesis.
        InsideOutSanta 2mo ago
        Here's the actual quote from the podcast you're presumably referring to. I guess everybody can make up their own mind about what they're saying and whether they are "fundamentally wrong." You gotta watch something, because harnesses have changed how you can look at the cost structure of these things. If you look at cost per million tokens, it might look lower, but if the model consumes four times as many tokens in order to deliver a meaningful result, it's not cheaper. And so, Tom Claburn, one of our senior software reporters, had an excellent piece looking at how Anthropic's latest models use a tremendous number of tokens in order to deliver the result. So, sure, OpenAI's latest models might look less expensive from an API standpoint, which is great for marketing, but if it's using twice as many tokens, that's not the same thing. And that's somewhat dependent on the harness, but it's also dependent on how much reasoning effort is put into it, how they're routing the models.