4 ms·

GPT3 has 175 billion parameters and Altman said 4 would use way more compute than 3. Let's just look at GPT 3. Each forward pass requires 2N=350B flops per tok

by bloppe 4y ago

GPT3 has 175 billion parameters and Altman said 4 would use way more compute than 3. Let's just look at GPT 3. Each forward pass requires 2N=350B flops per token. The computational overhead of the attention mechanism is negligible but the memory overhead of the attention for all users is not, but we can ignore that for now. Let's assume each query involves ~200 tokens of combined input and output. That's 70T flops per query. Let's say the cost of compute is 1 cent per 100B flops (probably a lowball). That's $7 per query for compute alone. And that's just GPT 3.

Robotbeat 4y ago
I mean, that might be the cost for a onesie-twosie in Azure, but if you're microsoft and you own the hardware, the cost may be one or more orders of magnitude less than that. (Of course, there's just a limit on the number of GPUs that exist and that Nvidia can pump out.)
- bloppe 4y ago
  Microsoft's profit margin for "intelligent cloud" in their most recent earnings was less than 50%. Impressive, but not nearly enough to make GPT subscriptions make financial sense.
  - Robotbeat 4y ago
    And a lot of that is marketing, keeping unused capacity in reserve, etc. Microsoft can probably afford a very low price without losing money if they price the compute at-cost. Certainly less than $7/request. 70 teraflop per request, and an $8000 GPU, with $2k of server overhead, $10k in power and cooling costs, $20k total for 312 teraflops of compute, leased for 2 years gives you an at-cost hardware price of $0.00007/request: https://www.google.com/search?q=%2420000*70/(312*3600*24*365*2) https://www.google.com/search?q=%2420000*70/(312*3600*24*365... So even assuming just a 1% utilization rate, you’re still talking less than 1 cent per request. Remember, inference runs at reduced precision. FP16 (or even just 8 bit) is WAY cheaper than fp64 bit. More than an order of magnitude cheaper. 9.7Tflops vs 312Tflops, factor of 32 improvement. With sparse matrix and 8-bit precision, you can effectively get over 100x the performance on an A100 than with FP64, plus the memory requirements are more than an order of magnitude less. Factor of 128 improvement in processing speed plus at least a factor of 8-16 or more improvement in tokens per memory, depending on sparsity. (Sparsity and reduced precision do reduce performance for the same number of operations and weights, but the overall effect on performance from reducing precision is beneficial probably at least to 4-bit quantization… You get diminishing or negative returns if you go less than that with current architectures, but in principle we can keep going.) https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... I think your compute cost figure is both very old and probably assumes double precision whereas GPT-3 uses at most FP16 at inference and GPT-4 may go beyond that.
jsnell 4y ago
A $7 per query estimate is beyond absurd.
- bloppe 4y ago
  Which of my estimates do you find absurd? How would you calculate cost?
  - jsnell 4y ago
    Most obviously this part must be off by 3-4 orders of magnitude: > Let's say the cost of compute is 1 cent per 100B flops (probably a lowball) That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour. Look, we know what OpenAI is charging for their commercial service. We don't know how it maps to their actual serving costs; maybe they're breaking even, making a small profit, or making a small loss. But just basic common sense should tell you that they are not operating at -10000% gross margins. I suspect that just working backwards from the prices to a reasonable range of gross margins is going to produce much better estimates than your methodology would even after the inputs are tuned to be more realistic.
    - theRealMe 4y ago
      “But just basic common sense should tell you that they are not operating at -10000% gross margins.” I’ll be honest, I laughed out loud at this. Pretty sure I read somewhere that Bing had around 1 billion searches per day BEFORE the hype of bingchat. At $7/query that makes MSFT’s $10 billion investment basically peanuts. Less than 2 days of Bings GPT usage payments. Lmao.
    - bloppe 4y ago
      Basic common sense would tell you that starting with an assumption and working backwards to justify that assumption is called succumbing to confirmation bias. I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries. If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.
weird-eye-issue 4y ago
This is why I laugh when people say AI can be factually incorrect. Just look at this comment made by a human! I've heard from insiders that the cost is roughly a couple cents per ChatGPT message but that was back with the legacy GPT-3.5 model, not the Turbo model or GPT-4
SilverBirch 4y ago
>Let's say the cost of compute is 1 cent per 100B flops (probably a lowball). I might be totally misunderstanding this, but from the source you posted elsewhere[1], GPU Sale Price per Flops is as a little bit higher than 1*10^-11 dollars per flop, or 10^11 flops per dollar, or 10^9 flops per cent. So 1GFlops per cent. But Flops are Floating point operations per second. So you aren't paying 1cent for 100B flops, you're paying 1 cent for a card capable of doing 100B floating point operations per second. So let's say the card lasts 3 years and runs at 100% utilization (I know, it doesn't we can multiply by some efficiency metric later), there's about 9.5*10^7 seconds in 3 years. So that dollar cost isn't for 100B flops, it's for 100B * 9.5 *10^7 ( *some efficiency number). So it's many orders of magnitude cheaper than you're saying. In reality, the cost of running these things isn't the cost of buying the GPUs, it's the cost of the power to run the GPUs, and that's a totally different calculation we can do. [1]: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/#GPU_price_single-precision_FLOPS\ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl...*
mihemihe 4y ago
This is so confidently wrong it looks like written by GPT itself

by bloppe 4y ago

Robotbeat 4y ago

I mean, that might be the cost for a onesie-twosie in Azure, but if you're microsoft and you own the hardware, the cost may be one or more orders of magnitude less than that. (Of course, there's just a limit on the number of GPUs that exist and that Nvidia can pump out.)

bloppe 4y ago
Microsoft's profit margin for "intelligent cloud" in their most recent earnings was less than 50%. Impressive, but not nearly enough to make GPT subscriptions make financial sense.
- Robotbeat 4y ago
  And a lot of that is marketing, keeping unused capacity in reserve, etc. Microsoft can probably afford a very low price without losing money if they price the compute at-cost. Certainly less than $7/request. 70 teraflop per request, and an $8000 GPU, with $2k of server overhead, $10k in power and cooling costs, $20k total for 312 teraflops of compute, leased for 2 years gives you an at-cost hardware price of $0.00007/request: https://www.google.com/search?q=%2420000*70/(312*3600*24*365*2) https://www.google.com/search?q=%2420000*70/(312*3600*24*365... So even assuming just a 1% utilization rate, you’re still talking less than 1 cent per request. Remember, inference runs at reduced precision. FP16 (or even just 8 bit) is WAY cheaper than fp64 bit. More than an order of magnitude cheaper. 9.7Tflops vs 312Tflops, factor of 32 improvement. With sparse matrix and 8-bit precision, you can effectively get over 100x the performance on an A100 than with FP64, plus the memory requirements are more than an order of magnitude less. Factor of 128 improvement in processing speed plus at least a factor of 8-16 or more improvement in tokens per memory, depending on sparsity. (Sparsity and reduced precision do reduce performance for the same number of operations and weights, but the overall effect on performance from reducing precision is beneficial probably at least to 4-bit quantization… You get diminishing or negative returns if you go less than that with current architectures, but in principle we can keep going.) https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... I think your compute cost figure is both very old and probably assumes double precision whereas GPT-3 uses at most FP16 at inference and GPT-4 may go beyond that.

jsnell 4y ago

A $7 per query estimate is beyond absurd.

bloppe 4y ago
Which of my estimates do you find absurd? How would you calculate cost?
- jsnell 4y ago
  Most obviously this part must be off by 3-4 orders of magnitude: > Let's say the cost of compute is 1 cent per 100B flops (probably a lowball) That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour. Look, we know what OpenAI is charging for their commercial service. We don't know how it maps to their actual serving costs; maybe they're breaking even, making a small profit, or making a small loss. But just basic common sense should tell you that they are not operating at -10000% gross margins. I suspect that just working backwards from the prices to a reasonable range of gross margins is going to produce much better estimates than your methodology would even after the inputs are tuned to be more realistic.
  - theRealMe 4y ago
    “But just basic common sense should tell you that they are not operating at -10000% gross margins.” I’ll be honest, I laughed out loud at this. Pretty sure I read somewhere that Bing had around 1 billion searches per day BEFORE the hype of bingchat. At $7/query that makes MSFT’s $10 billion investment basically peanuts. Less than 2 days of Bings GPT usage payments. Lmao.
  - bloppe 4y ago
    Basic common sense would tell you that starting with an assumption and working backwards to justify that assumption is called succumbing to confirmation bias. I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries. If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.

weird-eye-issue 4y ago

This is why I laugh when people say AI can be factually incorrect. Just look at this comment made by a human! I've heard from insiders that the cost is roughly a couple cents per ChatGPT message but that was back with the legacy GPT-3.5 model, not the Turbo model or GPT-4

SilverBirch 4y ago

>Let's say the cost of compute is 1 cent per 100B flops (probably a lowball). I might be totally misunderstanding this, but from the source you posted elsewhere[1], GPU Sale Price per Flops is as a little bit higher than 1*10^-11 dollars per flop, or 10^11 flops per dollar, or 10^9 flops per cent. So 1GFlops per cent. But Flops are Floating point operations per second. So you aren't paying 1cent for 100B flops, you're paying 1 cent for a card capable of doing 100B floating point operations per second. So let's say the card lasts 3 years and runs at 100% utilization (I know, it doesn't we can multiply by some efficiency metric later), there's about 9.5*10^7 seconds in 3 years. So that dollar cost isn't for 100B flops, it's for 100B * 9.5 *10^7 ( *some efficiency number). So it's many orders of magnitude cheaper than you're saying. In reality, the cost of running these things isn't the cost of buying the GPUs, it's the cost of the power to run the GPUs, and that's a totally different calculation we can do. [1]: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/#GPU_price_single-precision_FLOPS\ https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl...*

mihemihe 4y ago

This is so confidently wrong it looks like written by GPT itself