4 ms·

I mean, that might be the cost for a onesie-twosie in Azure, but if you're microsoft and you own the hardware, the cost may be one or more orders of magnitude l

I mean, that might be the cost for a onesie-twosie in Azure, but if you're microsoft and you own the hardware, the cost may be one or more orders of magnitude less than that. (Of course, there's just a limit on the number of GPUs that exist and that Nvidia can pump out.)

bloppe 4y ago
Microsoft's profit margin for "intelligent cloud" in their most recent earnings was less than 50%. Impressive, but not nearly enough to make GPT subscriptions make financial sense.
- Robotbeat 4y ago
  And a lot of that is marketing, keeping unused capacity in reserve, etc. Microsoft can probably afford a very low price without losing money if they price the compute at-cost. Certainly less than $7/request. 70 teraflop per request, and an $8000 GPU, with $2k of server overhead, $10k in power and cooling costs, $20k total for 312 teraflops of compute, leased for 2 years gives you an at-cost hardware price of $0.00007/request: https://www.google.com/search?q=%2420000*70/(312*3600*24*365*2) https://www.google.com/search?q=%2420000*70/(312*3600*24*365... So even assuming just a 1% utilization rate, you’re still talking less than 1 cent per request. Remember, inference runs at reduced precision. FP16 (or even just 8 bit) is WAY cheaper than fp64 bit. More than an order of magnitude cheaper. 9.7Tflops vs 312Tflops, factor of 32 improvement. With sparse matrix and 8-bit precision, you can effectively get over 100x the performance on an A100 than with FP64, plus the memory requirements are more than an order of magnitude less. Factor of 128 improvement in processing speed plus at least a factor of 8-16 or more improvement in tokens per memory, depending on sparsity. (Sparsity and reduced precision do reduce performance for the same number of operations and weights, but the overall effect on performance from reducing precision is beneficial probably at least to 4-bit quantization… You get diminishing or negative returns if you go less than that with current architectures, but in principle we can keep going.) https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... I think your compute cost figure is both very old and probably assumes double precision whereas GPT-3 uses at most FP16 at inference and GPT-4 may go beyond that.

bloppe 4y ago

Microsoft's profit margin for "intelligent cloud" in their most recent earnings was less than 50%. Impressive, but not nearly enough to make GPT subscriptions make financial sense.

Robotbeat 4y ago
And a lot of that is marketing, keeping unused capacity in reserve, etc. Microsoft can probably afford a very low price without losing money if they price the compute at-cost. Certainly less than $7/request. 70 teraflop per request, and an $8000 GPU, with $2k of server overhead, $10k in power and cooling costs, $20k total for 312 teraflops of compute, leased for 2 years gives you an at-cost hardware price of $0.00007/request: https://www.google.com/search?q=%2420000*70/(312*3600*24*365*2) https://www.google.com/search?q=%2420000*70/(312*3600*24*365... So even assuming just a 1% utilization rate, you’re still talking less than 1 cent per request. Remember, inference runs at reduced precision. FP16 (or even just 8 bit) is WAY cheaper than fp64 bit. More than an order of magnitude cheaper. 9.7Tflops vs 312Tflops, factor of 32 improvement. With sparse matrix and 8-bit precision, you can effectively get over 100x the performance on an A100 than with FP64, plus the memory requirements are more than an order of magnitude less. Factor of 128 improvement in processing speed plus at least a factor of 8-16 or more improvement in tokens per memory, depending on sparsity. (Sparsity and reduced precision do reduce performance for the same number of operations and weights, but the overall effect on performance from reducing precision is beneficial probably at least to 4-bit quantization… You get diminishing or negative returns if you go less than that with current architectures, but in principle we can keep going.) https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent... I think your compute cost figure is both very old and probably assumes double precision whereas GPT-3 uses at most FP16 at inference and GPT-4 may go beyond that.