3 ms·

Disclosure: I used to work for Google Cloud. > you save a considerable amount of money and get a lot better performance when building your own cluster. This h

by boulos 5y ago

Disclosure: I used to work for Google Cloud. > you save a considerable amount of money and get a lot better performance when building your own cluster. This heavily depends on how much benefit you get from improving GPU performance from each generation. A lot of people assume 3/4-yr TCO. If you instead "rent" for 1 year at a time, you've been getting >2x benefits per generation lately. Most folks also measure "occupancy" for clusters like this rather than "utilization". That is, if a job is using 128 "GPUs" that counts as 128 in use. But that ignores that many jobs might have been just fine with T4s (which are a lot cheaper) versus A100s. (Depends a lot on the model, the I/O, etc.) Once you've bought a physical cluster, you're kind of stuck with that configuration (for better or worse). tl;dr: It's not just about "idle".

tedivm 5y ago
These generations tend to be three years apart though, so if you're buying as the new generation comes out then your total TCO period has you running almost peak hardware (there were several versions of the V100, each with minor improvements). Many vendors also offer buy back and upgrade programs. At the same time it's hard to understate how different the prices are here. Our break even point for using on prem compared to AWS was about nine months. After that we saved money for the rest of that hardwares lifetime. I definitely agree that people shouldn't just rush out and buy these without benchmarking and examining their usecase. The cloud is really good for this! At the same time though I have yet to see any cloud provider with anything even approaching the interlink I can get on prem and that means, so it's basically impossible to get the same performance out of the cloud as it is on prem right now.
- boulos 5y ago
  > using on prem compared to AWS was about nine months. At list price or a moderate discount. The folks at this scale aren't paying that :).
  - deleted 5y ago
    [deleted]

by boulos 5y ago

tedivm 5y ago

These generations tend to be three years apart though, so if you're buying as the new generation comes out then your total TCO period has you running almost peak hardware (there were several versions of the V100, each with minor improvements). Many vendors also offer buy back and upgrade programs. At the same time it's hard to understate how different the prices are here. Our break even point for using on prem compared to AWS was about nine months. After that we saved money for the rest of that hardwares lifetime. I definitely agree that people shouldn't just rush out and buy these without benchmarking and examining their usecase. The cloud is really good for this! At the same time though I have yet to see any cloud provider with anything even approaching the interlink I can get on prem and that means, so it's basically impossible to get the same performance out of the cloud as it is on prem right now.

boulos 5y ago
> using on prem compared to AWS was about nine months. At list price or a moderate discount. The folks at this scale aren't paying that :).
- deleted 5y ago
  [deleted]