3 ms·

Here's the actual quote from the podcast you're presumably referring to. I guess everybody can make up their own mind about what they're saying and whether they

Here's the actual quote from the podcast you're presumably referring to. I guess everybody can make up their own mind about what they're saying and whether they are "fundamentally wrong." You gotta watch something, because harnesses have changed how you can look at the cost structure of these things. If you look at cost per million tokens, it might look lower, but if the model consumes four times as many tokens in order to deliver a meaningful result, it's not cheaper. And so, Tom Claburn, one of our senior software reporters, had an excellent piece looking at how Anthropic's latest models use a tremendous number of tokens in order to deliver the result. So, sure, OpenAI's latest models might look less expensive from an API standpoint, which is great for marketing, but if it's using twice as many tokens, that's not the same thing. And that's somewhat dependent on the harness, but it's also dependent on how much reasoning effort is put into it, how they're routing the models.

simianwords 2mo ago
It clearly points to how the models use more tokens, thereby more price, to achieve same task. Even if API prices per token reduced. He literally says “but if it’s using twice as many tokens that’s not the same thing”. Why would he bring up tokens? I genuinely don’t know how you can conclude that he still thinks overall price per fixed task reduced.
- InsideOutSanta 2mo ago
  > Why would he bring up tokens? If you continue reading, he explains it in the next sentence.
  - simianwords 2mo ago
    I did. And he implies that cost per task increases. Otherwise there’s literally no reason to bring up tokens - that is an internal implementation detail.
    - InsideOutSanta 2mo ago
      > Otherwise there’s literally no reason to bring up tokens - that is an internal implementation detail. I'm not sure if you're not aware of what he's referencing there, but the specific example he brings up is that Anthropic ostensibly kept pricing identical with new models, but changed the tokenizer, which made the model use more tokens for the same input and output text. So that's a case where, prima facie, the cost per task increased. Of course, this also depends on how verbose the model is and what harness you use, and so on. Correct me if I'm wrong; I think you believe that he makes an argument like "OpenAI decreased API pricing, but this decrease was actually secretly an increase in cost." But he does not. He's saying that even though cost-per-token is going down overall, the actual cost of using LLMs is in many cases going up, because there isn't a direct causal relationship between cost-per-token and the total cost of using LLMs. And he's entirely correct about that.

simianwords 2mo ago

It clearly points to how the models use more tokens, thereby more price, to achieve same task. Even if API prices per token reduced. He literally says “but if it’s using twice as many tokens that’s not the same thing”. Why would he bring up tokens? I genuinely don’t know how you can conclude that he still thinks overall price per fixed task reduced.

InsideOutSanta 2mo ago
> Why would he bring up tokens? If you continue reading, he explains it in the next sentence.
- simianwords 2mo ago
  I did. And he implies that cost per task increases. Otherwise there’s literally no reason to bring up tokens - that is an internal implementation detail.
  - InsideOutSanta 2mo ago
    > Otherwise there’s literally no reason to bring up tokens - that is an internal implementation detail. I'm not sure if you're not aware of what he's referencing there, but the specific example he brings up is that Anthropic ostensibly kept pricing identical with new models, but changed the tokenizer, which made the model use more tokens for the same input and output text. So that's a case where, prima facie, the cost per task increased. Of course, this also depends on how verbose the model is and what harness you use, and so on. Correct me if I'm wrong; I think you believe that he makes an argument like "OpenAI decreased API pricing, but this decrease was actually secretly an increase in cost." But he does not. He's saying that even though cost-per-token is going down overall, the actual cost of using LLMs is in many cases going up, because there isn't a direct causal relationship between cost-per-token and the total cost of using LLMs. And he's entirely correct about that.