4 ms·

> As a user, I don’t really care. Tell me: Just how is it fair for a user to pay for the reasoning tokens without actually seeing them? If they are not shared,

> As a user, I don’t really care. Tell me: Just how is it fair for a user to pay for the reasoning tokens without actually seeing them? If they are not shared, the service can bill you anything they want for them!

ttul 2y ago
OpenAI’s answer to this would be, “Okay then, don’t use it.”
mediaman 2y ago
The simple answer is: I don't care. I'll statistically figure out what the typical total cost per call is from experience, and that's what matters. Who cares if they lie about it, if the model's cost per call fits my budget? If it starts costing $1 per call, and that's too high, then I just won't use it commercially. Whether it was $1 because they inflated the token count or because it just actually took a lot of tokens to do its reasoning isn't really material to my economic decision.
- lucianbr 2y ago
  The thing is it might increase in cost after you've decided to use it commercially, and have invested a lot of time and resources in it. Now it's very hard to move to something else, but very easy for OpenAI to increase your cost arbitrarily. The statistics you made are not binding for them.
  - nsokolsky 2y ago
    Same applies to every other API in the world, yes.
    - lucianbr 2y ago
      No, S3 pricing for example is predictable, and written in a contract. There's no way for AWS to charge you 3x amount of dollars for 1GB tomorrow. They need to announce it in advance, and give you time to exit the contract if you disagree with the new price. It's really not the same. OpenAI can just tell you your prompt from tomorrow used up 20x times reasoning tokens. There's no advance warning or predictability. I really don't understand how you can claim the situations are identical.
  - mediaman 2y ago
    The API returns how many tokens were used in reasoning, so it would be easy to see any average change in reasoning token consumption. And token prices in general have been extremely deflationary over the last 18 months.
  - golergka 2y ago
    This is experimental, frontier stuff, obviously it comes with risks. Building on GPT-4 in March of 2023 was like that as well, but now you can easily switch between a few models of comparable quality made by different companies (yay capitalism and free markets!). You can risk and use just released stuff right now, or, most likely, come back in 6-12 months (probably earlier) and get several different providers with very similar APIs.
    - OutOfHere 2y ago
      Everything that OpenAI does with LLMs has already been done and validated in the open source community well before OpenAI gets around to it. OpenAI is not an innovator. simbianai/taskgen on github is an example of one such project, although there are others too that don't come to mind right now. As such, I would never call their work "frontier stuff", but they do bring it to the masses with their commercial service.
elicksaur 2y ago
The person you are replying to points this out. They make a distinction between developers and users. An end user on a monthly subscription plan doesn’t care about how much compute happens for their chat.
inerte 2y ago
OpenAI could have also figured out the average number of extra output tokens, and put a markup in overall API costs. As a user, I wouldn’t care either, because the price would mostly be the same.
halJordan 2y ago
Yeah it is fair. You don't pay a lawyer for 40s of work expecting to see all the research between your consult and the document. You don't pay a cook for a meal and expect to sit and interrogate all the ingredients and the oven temperature.
- OutOfHere 2y ago
  Actually, if a lawyer is billing you by the minute, then yes, you are entitled to a detailed breakdown. If the lawyer is billing you by the job, then no.
potatoman22 2y ago
If the output alone is high enough quality, it's worth paying extra.
madeofpalk 2y ago
Pricing for many things in life is abstrated away.

ttul 2y ago

OpenAI’s answer to this would be, “Okay then, don’t use it.”

mediaman 2y ago

The simple answer is: I don't care. I'll statistically figure out what the typical total cost per call is from experience, and that's what matters. Who cares if they lie about it, if the model's cost per call fits my budget? If it starts costing $1 per call, and that's too high, then I just won't use it commercially. Whether it was $1 because they inflated the token count or because it just actually took a lot of tokens to do its reasoning isn't really material to my economic decision.

lucianbr 2y ago
The thing is it might increase in cost after you've decided to use it commercially, and have invested a lot of time and resources in it. Now it's very hard to move to something else, but very easy for OpenAI to increase your cost arbitrarily. The statistics you made are not binding for them.
- nsokolsky 2y ago
  Same applies to every other API in the world, yes.
  - lucianbr 2y ago
    No, S3 pricing for example is predictable, and written in a contract. There's no way for AWS to charge you 3x amount of dollars for 1GB tomorrow. They need to announce it in advance, and give you time to exit the contract if you disagree with the new price. It's really not the same. OpenAI can just tell you your prompt from tomorrow used up 20x times reasoning tokens. There's no advance warning or predictability. I really don't understand how you can claim the situations are identical.
- mediaman 2y ago
  The API returns how many tokens were used in reasoning, so it would be easy to see any average change in reasoning token consumption. And token prices in general have been extremely deflationary over the last 18 months.
- golergka 2y ago
  This is experimental, frontier stuff, obviously it comes with risks. Building on GPT-4 in March of 2023 was like that as well, but now you can easily switch between a few models of comparable quality made by different companies (yay capitalism and free markets!). You can risk and use just released stuff right now, or, most likely, come back in 6-12 months (probably earlier) and get several different providers with very similar APIs.
  - OutOfHere 2y ago
    Everything that OpenAI does with LLMs has already been done and validated in the open source community well before OpenAI gets around to it. OpenAI is not an innovator. simbianai/taskgen on github is an example of one such project, although there are others too that don't come to mind right now. As such, I would never call their work "frontier stuff", but they do bring it to the masses with their commercial service.

elicksaur 2y ago

The person you are replying to points this out. They make a distinction between developers and users. An end user on a monthly subscription plan doesn’t care about how much compute happens for their chat.

inerte 2y ago

OpenAI could have also figured out the average number of extra output tokens, and put a markup in overall API costs. As a user, I wouldn’t care either, because the price would mostly be the same.

halJordan 2y ago

Yeah it is fair. You don't pay a lawyer for 40s of work expecting to see all the research between your consult and the document. You don't pay a cook for a meal and expect to sit and interrogate all the ingredients and the oven temperature.

OutOfHere 2y ago
Actually, if a lawyer is billing you by the minute, then yes, you are entitled to a detailed breakdown. If the lawyer is billing you by the job, then no.

potatoman22 2y ago

If the output alone is high enough quality, it's worth paying extra.

madeofpalk 2y ago

Pricing for many things in life is abstrated away.