4 ms·

Most obviously this part must be off by 3-4 orders of magnitude: > Let's say the cost of compute is 1 cent per 100B flops (probably a lowball) That's like 0.5

by jsnell 4y ago

Most obviously this part must be off by 3-4 orders of magnitude: > Let's say the cost of compute is 1 cent per 100B flops (probably a lowball) That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour. Look, we know what OpenAI is charging for their commercial service. We don't know how it maps to their actual serving costs; maybe they're breaking even, making a small profit, or making a small loss. But just basic common sense should tell you that they are not operating at -10000% gross margins. I suspect that just working backwards from the prices to a reasonable range of gross margins is going to produce much better estimates than your methodology would even after the inputs are tuned to be more realistic.

theRealMe 4y ago
“But just basic common sense should tell you that they are not operating at -10000% gross margins.” I’ll be honest, I laughed out loud at this. Pretty sure I read somewhere that Bing had around 1 billion searches per day BEFORE the hype of bingchat. At $7/query that makes MSFT’s $10 billion investment basically peanuts. Less than 2 days of Bings GPT usage payments. Lmao.
bloppe 4y ago
Basic common sense would tell you that starting with an assumption and working backwards to justify that assumption is called succumbing to confirmation bias. I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries. If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.
- zamnos 4y ago
  You can be sure Open AI isn't even paying Azure's punlishedy 3-year commitment pricing rate. The other thing to note is that OpenAI limits paying users to X queries per Y hours in GPT-4, something like 3 per 5. Which means the GPT-4 fleet is quite limited.
  - int_19h 4y ago
    GPT-4 cap is currently at 25 messages every 3 hours. It also keeps saying "expect significantly lower caps, as we adjust for demand", but it's been three weeks now and they haven't lowered it yet.
- jsnell 4y ago
  Uhh... I'm pretty sure that page is about the price of buying an entire GPU that does N GFLOP/S in perpetuity. It has no relevance to the calculation you did, unless you assume the GPU is used for one second and then thrown in the bin. > I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, I don't think you've thought this through. There's no realistic level of utilization that's going to make up for 5 orders of magnitude. An unimaginably crappy 1% utilization would still leave you off by 3 orders of magnitude. This very submission about OpenAI having to suspend selling subscriptions makes it pretty clear they're not having trouble with low utilization... > If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. You do understand that this doesn't actually support your initial estimate, right? It just makes it more obvious how ridiculous it was.
  - ericpauley 4y ago
    Yup. GP is conflating FLOP/s and FLOPs.
zamnos 4y ago
They are just giving away GPT-3 grade queries. In terms of revenue, that's NaN% gross margin! Thus, -10,000% gross margin is quite the improvement. While I personally don't think that they are spending quite that much on compute, burning investor capital to build a customer base is a time honored tradition. Which means we can't really deduce that because -10,000% is such a big number, that can't possibly be their compute costs.
- Euphorbium 4y ago
  Was this written by gpt?
  - zamnos 4y ago
    If you ask ChatGPT what 1/0 is, you get a paragraph as to why it's undefined. Nowhere does it use the phrase "NaN". Perhaps you need to adjust your humor filter, LLMs aren't quite there enough to be able to use NaN% in context as a joke like that. Also: > Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html If you're going to accuse me of using ChatGPT, at least bother to say why.
    - int_19h 4y ago
      It really shouldn't use the word NaN in any case, since 1/0 would be +Inf. Meanwhile, here's what GPT-4 had to say (after refusing to talk about percentages at all at first because it is "impossible to calculate"): >> ... suppose you're describing this to a technical audience - software engineers, specifically. There might be some way to talk about percentages concisely in that particular context. > In that case, you could describe the situation to a technical audience by saying, "The company's profit margin is effectively -∞% due to offering the product for free while incurring expenses."

by jsnell 4y ago

theRealMe 4y ago

“But just basic common sense should tell you that they are not operating at -10000% gross margins.” I’ll be honest, I laughed out loud at this. Pretty sure I read somewhere that Bing had around 1 billion searches per day BEFORE the hype of bingchat. At $7/query that makes MSFT’s $10 billion investment basically peanuts. Less than 2 days of Bings GPT usage payments. Lmao.

bloppe 4y ago

Basic common sense would tell you that starting with an assumption and working backwards to justify that assumption is called succumbing to confirmation bias. I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries. If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.

zamnos 4y ago
You can be sure Open AI isn't even paying Azure's punlishedy 3-year commitment pricing rate. The other thing to note is that OpenAI limits paying users to X queries per Y hours in GPT-4, something like 3 per 5. Which means the GPT-4 fleet is quite limited.
- int_19h 4y ago
  GPT-4 cap is currently at 25 messages every 3 hours. It also keeps saying "expect significantly lower caps, as we adjust for demand", but it's been three weeks now and they haven't lowered it yet.
jsnell 4y ago
Uhh... I'm pretty sure that page is about the price of buying an entire GPU that does N GFLOP/S in perpetuity. It has no relevance to the calculation you did, unless you assume the GPU is used for one second and then thrown in the bin. > I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, I don't think you've thought this through. There's no realistic level of utilization that's going to make up for 5 orders of magnitude. An unimaginably crappy 1% utilization would still leave you off by 3 orders of magnitude. This very submission about OpenAI having to suspend selling subscriptions makes it pretty clear they're not having trouble with low utilization... > If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. You do understand that this doesn't actually support your initial estimate, right? It just makes it more obvious how ridiculous it was.
- ericpauley 4y ago
  Yup. GP is conflating FLOP/s and FLOPs.

zamnos 4y ago

They are just giving away GPT-3 grade queries. In terms of revenue, that's NaN% gross margin! Thus, -10,000% gross margin is quite the improvement. While I personally don't think that they are spending quite that much on compute, burning investor capital to build a customer base is a time honored tradition. Which means we can't really deduce that because -10,000% is such a big number, that can't possibly be their compute costs.

Euphorbium 4y ago
Was this written by gpt?
- zamnos 4y ago
  If you ask ChatGPT what 1/0 is, you get a paragraph as to why it's undefined. Nowhere does it use the phrase "NaN". Perhaps you need to adjust your humor filter, LLMs aren't quite there enough to be able to use NaN% in context as a joke like that. Also: > Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html If you're going to accuse me of using ChatGPT, at least bother to say why.
  - int_19h 4y ago
    It really shouldn't use the word NaN in any case, since 1/0 would be +Inf. Meanwhile, here's what GPT-4 had to say (after refusing to talk about percentages at all at first because it is "impossible to calculate"): >> ... suppose you're describing this to a technical audience - software engineers, specifically. There might be some way to talk about percentages concisely in that particular context. > In that case, you could describe the situation to a technical audience by saying, "The company's profit margin is effectively -∞% due to offering the product for free while incurring expenses."