10 ms·

The main difficulty when pricing a monthly subscription for "unlimited" usage of a product is the 1% of power users who use have extreme use of the product that

The main difficulty when pricing a monthly subscription for "unlimited" usage of a product is the 1% of power users who use have extreme use of the product that can kill any profit margins for the product as a whole. Pricing ChatGPT Pro at $200/mo filters it to only power users/enterprise, and given the cost of the GPT-o1 API, it wouldn't surprise me if those power users burn through $200 worth of compute very, very quickly.

peab 2y ago
I was testing out a chat app that supported images. Long conversations with multiple images in the conversation can be like .10cents per message after a certain point. It sure does add up quickly
nine_k 2y ago
Is compute that expensive? An H100 rents at about $2.50/hour, it's 80 hours of pure compute. Assuming 720 hours a month, 1/9 duty cycle around the clock, or 1/3 if we assume 8-hour work day. It's really intense, constant use. And I bet OpenAI spend less on operating their infra than the rate at which cloud providers rent it out.
- ssl-3 2y ago
  Does an o1 query run on a singular H100, or on a plurality of H100s?
  - danpalmer 2y ago
    A single H100 has 80GB of memory, meaning that at FP16 you could roughly fit a 40B parameter model on it, or at FP4 quantisation you could fit a 160B parameter model on it. We don't know (I don't think) what quantisation OpenAI use, or how many parameters o1 is, but most likely... ...they probably quantise a bit, but not loads, as they don't want to sacrifice performance. FP8 seems like a possible middle ground. o1 is just a bunch of GPT-4o in a trenchcoat strung together with some advanced prompting. GPT-4o is theorised to be 200B parameters. If you wanted to run 5 parallel generation tasks at peak during the o1 inference process, that's 5x 200B, at FP8, or about 12 H100s. 12 H100s takes about one full rack of kit to run.
    - anticensor 2y ago
      o1 is ten times as expensive as pre-turbo GPT-4.
- drdrey 2y ago
  are you assuming that you can do o1 inference on a single h100?
  - nine_k 2y ago
    Good question. How many H100s does it take? Is there any way to guess / approximate that?
    - shikon7 2y ago
      You need enough RAM to store the model and the KV-cache depending on context size. Assuming the model has a trillion parameters (there are only rumours how many there actually are) and uses 8 bit per parameter, 16 H100 might be sufficient.
      - londons_explore 2y ago
        I suspect the biggest most powerful model could easily use hundreds or maybe thousands of H100's. And the 'search' part of it could use many of these clusters in parallel, and then pick the best answer to return to the user.
      - holoduke 2y ago
        16? No. More in the order of 1000+ h100 computing in parallel for one request.
lm28469 2y ago
> can kill any profit margins for the product as a whole. Especially when the base line profit margin is negative to begin with
- sebzim4500 2y ago
  Is there any evidence to suggest this is true? IIRC there was leaked information that OpenAI's revenue was significantly higher than their compute spending, but it wasn't broken down between API and subscriptions so maybe that's just due to people who subscribe and then use it a few times a month.
  - lm28469 2y ago
    > Is there any evidence to suggest this is true? I can't find any sources _not_ mentioning billions of loss for 2024 and for the foreseeable future
  - mrandish 2y ago
    > OpenAI's revenue was significantly higher than their compute spending I find this difficult believe, although I don't doubt leaks could have implied it. The challenge is that "the cost of compute" can vary greatly based on how it's accounted for (things like amortization, revenue recognition, capex vs opex, IP attribution, leasing, etc). Sort of like how Hollywood studio accounting can show a movie as profitable or unprofitable, depending on how "profit" is defined and how expenses are treated. Given how much all those details can impact the outcome, to be credible I'd need a lot more specifics than a typical leak includes.
rubymamis 2y ago
I believe they have many data points to back up this decision. They surely know how people are suing their products.
londons_explore 2y ago
I wouldn't be surprised if the "unlimited" product is unlimited requests, but the quality of the responses drop if you ask millions of questions...
- rrr_oh_man 2y ago
  like throttled unlimited data
thih9 2y ago
They are ready for this, there is a policy against automation, sharing or reselling access; it looks like there are some unspecified quotas as well: > We have guardrails in place to help prevent misuse and are always working to improve our systems. This may occasionally involve a temporary restriction on your usage. We will inform you when this happens, and if you think this might be a mistake, please don’t hesitate to reach out to our support team at help.openai.com using the widget at the bottom-right of this page. If policy-violating behavior is not found, your access will be restored. Source: https://help.openai.com/en/articles/9793128-what-is-chatgpt-pro https://help.openai.com/en/articles/9793128-what-is-chatgpt-...
paxys 2y ago
$200 is a lot of compute. Amortized over say 3 years, that's a dedicated A100 GPU per user, or an H100 for every 3 users.
- wkat4242 2y ago
  Not counting power or servers etc. But yeah it does put it into perspective.

peab 2y ago

I was testing out a chat app that supported images. Long conversations with multiple images in the conversation can be like .10cents per message after a certain point. It sure does add up quickly

nine_k 2y ago

Is compute that expensive? An H100 rents at about $2.50/hour, it's 80 hours of pure compute. Assuming 720 hours a month, 1/9 duty cycle around the clock, or 1/3 if we assume 8-hour work day. It's really intense, constant use. And I bet OpenAI spend less on operating their infra than the rate at which cloud providers rent it out.

ssl-3 2y ago
Does an o1 query run on a singular H100, or on a plurality of H100s?
- danpalmer 2y ago
  A single H100 has 80GB of memory, meaning that at FP16 you could roughly fit a 40B parameter model on it, or at FP4 quantisation you could fit a 160B parameter model on it. We don't know (I don't think) what quantisation OpenAI use, or how many parameters o1 is, but most likely... ...they probably quantise a bit, but not loads, as they don't want to sacrifice performance. FP8 seems like a possible middle ground. o1 is just a bunch of GPT-4o in a trenchcoat strung together with some advanced prompting. GPT-4o is theorised to be 200B parameters. If you wanted to run 5 parallel generation tasks at peak during the o1 inference process, that's 5x 200B, at FP8, or about 12 H100s. 12 H100s takes about one full rack of kit to run.
  - anticensor 2y ago
    o1 is ten times as expensive as pre-turbo GPT-4.
drdrey 2y ago
are you assuming that you can do o1 inference on a single h100?
- nine_k 2y ago
  Good question. How many H100s does it take? Is there any way to guess / approximate that?
  - shikon7 2y ago
    You need enough RAM to store the model and the KV-cache depending on context size. Assuming the model has a trillion parameters (there are only rumours how many there actually are) and uses 8 bit per parameter, 16 H100 might be sufficient.
    - londons_explore 2y ago
      I suspect the biggest most powerful model could easily use hundreds or maybe thousands of H100's. And the 'search' part of it could use many of these clusters in parallel, and then pick the best answer to return to the user.
    - holoduke 2y ago
      16? No. More in the order of 1000+ h100 computing in parallel for one request.

lm28469 2y ago

> can kill any profit margins for the product as a whole. Especially when the base line profit margin is negative to begin with

sebzim4500 2y ago
Is there any evidence to suggest this is true? IIRC there was leaked information that OpenAI's revenue was significantly higher than their compute spending, but it wasn't broken down between API and subscriptions so maybe that's just due to people who subscribe and then use it a few times a month.
- lm28469 2y ago
  > Is there any evidence to suggest this is true? I can't find any sources _not_ mentioning billions of loss for 2024 and for the foreseeable future
- mrandish 2y ago
  > OpenAI's revenue was significantly higher than their compute spending I find this difficult believe, although I don't doubt leaks could have implied it. The challenge is that "the cost of compute" can vary greatly based on how it's accounted for (things like amortization, revenue recognition, capex vs opex, IP attribution, leasing, etc). Sort of like how Hollywood studio accounting can show a movie as profitable or unprofitable, depending on how "profit" is defined and how expenses are treated. Given how much all those details can impact the outcome, to be credible I'd need a lot more specifics than a typical leak includes.

rubymamis 2y ago

I believe they have many data points to back up this decision. They surely know how people are suing their products.

londons_explore 2y ago

I wouldn't be surprised if the "unlimited" product is unlimited requests, but the quality of the responses drop if you ask millions of questions...

rrr_oh_man 2y ago
like throttled unlimited data

thih9 2y ago

They are ready for this, there is a policy against automation, sharing or reselling access; it looks like there are some unspecified quotas as well: > We have guardrails in place to help prevent misuse and are always working to improve our systems. This may occasionally involve a temporary restriction on your usage. We will inform you when this happens, and if you think this might be a mistake, please don’t hesitate to reach out to our support team at help.openai.com using the widget at the bottom-right of this page. If policy-violating behavior is not found, your access will be restored. Source: https://help.openai.com/en/articles/9793128-what-is-chatgpt-pro https://help.openai.com/en/articles/9793128-what-is-chatgpt-...

paxys 2y ago

$200 is a lot of compute. Amortized over say 3 years, that's a dedicated A100 GPU per user, or an H100 for every 3 users.

wkat4242 2y ago
Not counting power or servers etc. But yeah it does put it into perspective.