3 ms·

Why not just rent AWS/Azure/GCP instead? They're all about the same. Top of the line enterprise GPUs with fast interconnect.

tedivm 5y ago
They are not the same at all. AWS has the best GPU instances right now but there's some huge differences in networking speed. The P3 instances have 400Gbps per machine, with 8 GPUs on each machine. If you were to self host a cluster using the standard DGX machines you get 200Gbps per GPU, for a total of 1600Gbps for just the GPUs. The DGX machine has another two infiniband ports that can be used to attach to storage at pretty intense speeds as well. This makes a huge difference when using more than a single machine. I've done the math and purchased the machines at a previous company- assuming you aren't leaving the machines idle most of the time you save a considerable amount of money and get a lot better performance when building your own cluster.
- ska 5y ago
  On-prem for nearly anything is going to be at least a bit of a win if your utilization is uniform or predictable. The real win for not doing it is in adaptability.
  - tedivm 5y ago
    Yeah, definitely, but I think it's important to talk about scale of that win. I mentioned in another comment that the GPU generation is roughly three years between major architecture upgrades- this has held true for a bit now, and that time may even stretch out a little. When the average company builds one of these clusters it's safe to assume they'll either run it for three years or sell it back for some return. Going with the cloud and assuming you don't commit to several years (losing that adaptability) the yearly cost of a p4 is $287,087. Over three years that's $861,261 to run a single machine. For about $450k you can build out a solid two machine (16gpu) cluster (including infiniband networking gear and a solid NAS) that will easily last three years. There are datacenters which specialize in this and companies that can manage these machines. If you don't have the cash up front you can lease them on good terms and your yearly bill will still be much lower than AWS. Model training is basically the one use case where I'm really willing to purchase equipment instead of using the cloud. The money it saves is enough to hire one or two more staff members, and the maintenance is shockingly low if you get it setup right to start.
    - dekhn 5y ago
      A couple counterpoints: for all the time you just spent making a local cluster, you could have spent 1 day negotiating a deal with the AWS TAM and get 50% or more off rack rates. So that $861K is more like half that. And second, that machien would be collocated with infinite storage and other processing systems. The CPU/GPU part of training is only part of the much larger ecosystem, which is what you're paying to be in when you use AWS.
- boulos 5y ago
  Disclosure: I used to work for Google Cloud. > you save a considerable amount of money and get a lot better performance when building your own cluster. This heavily depends on how much benefit you get from improving GPU performance from each generation. A lot of people assume 3/4-yr TCO. If you instead "rent" for 1 year at a time, you've been getting >2x benefits per generation lately. Most folks also measure "occupancy" for clusters like this rather than "utilization". That is, if a job is using 128 "GPUs" that counts as 128 in use. But that ignores that many jobs might have been just fine with T4s (which are a lot cheaper) versus A100s. (Depends a lot on the model, the I/O, etc.) Once you've bought a physical cluster, you're kind of stuck with that configuration (for better or worse). tl;dr: It's not just about "idle".
  - tedivm 5y ago
    These generations tend to be three years apart though, so if you're buying as the new generation comes out then your total TCO period has you running almost peak hardware (there were several versions of the V100, each with minor improvements). Many vendors also offer buy back and upgrade programs. At the same time it's hard to understate how different the prices are here. Our break even point for using on prem compared to AWS was about nine months. After that we saved money for the rest of that hardwares lifetime. I definitely agree that people shouldn't just rush out and buy these without benchmarking and examining their usecase. The cloud is really good for this! At the same time though I have yet to see any cloud provider with anything even approaching the interlink I can get on prem and that means, so it's basically impossible to get the same performance out of the cloud as it is on prem right now.
    - boulos 5y ago
      > using on prem compared to AWS was about nine months. At list price or a moderate discount. The folks at this scale aren't paying that :).
      - deleted 5y ago
        [deleted]
- dekhn 5y ago
  the network topology and the network switch itself also can make a huge difference depending on traffic conditions; so you might have tons of fat NICs per GPU but if all of them want to alltoall, you better have a ton of cross section bandwidth. I always wonder about performance on these clusters. Back in MY day, I'd wait a week or more for results from my jobs, and immediately resubmit to wait two weeks in a queue for another week of runtime and do lots of data processing in the downtime. Then, I moved to cloud and decided on "what can I afford to do overnight" (IE, I set my time to result to be about 12 hours). I have a hard time justifying additional hardware to get results in 10 minutes versus a day, it seems like at that point you're just using it to get fast cycle times on new ideas, but who has new ideas every 10 minutes?
  - tedivm 5y ago
    > the network topology and the network switch itself also can make a huge difference depending on traffic conditions; so you might have tons of fat NICs per GPU but if all of them want to alltoall, you better have a ton of cross section bandwidth. This is the beauty of the new nvswitch chips and the infiniband networks instead of ethernet. Anyone who is doing this is setting up a fully switched high bandwidth infiniband network with ridiculous traffic between them. Nvidia purchased Mellanox a year or two ago- combine that with the ridiculously awesome nvswitch in the A100 dgx machines and there's a huge jump in cross chip traffic ability. At the same time though a decent mellanox router is probably going to set you back $30k.
    - dekhn 5y ago
      I'm not aware of any cost-effective switch that permits scaling all v all to arbitrary sizes. THat's my point. modern infiniband uses nearly all the same tech as previous supercomputers, but with faster interfaces, and more of them. For example the facebook cluster is a dual-layer clos network, which is one of a few cost-effective ways to get very high randomly distributed traffic, but all v all communication scales as n*2 and n*2 wires gets expensive fast. Better to find algorithms that need less communication, than to make faster computers that allow you to write algorithms that needs lots of communication. Otherwise you'll always pay $$$ to reach peak GPU performance.

tedivm 5y ago

They are not the same at all. AWS has the best GPU instances right now but there's some huge differences in networking speed. The P3 instances have 400Gbps per machine, with 8 GPUs on each machine. If you were to self host a cluster using the standard DGX machines you get 200Gbps per GPU, for a total of 1600Gbps for just the GPUs. The DGX machine has another two infiniband ports that can be used to attach to storage at pretty intense speeds as well. This makes a huge difference when using more than a single machine. I've done the math and purchased the machines at a previous company- assuming you aren't leaving the machines idle most of the time you save a considerable amount of money and get a lot better performance when building your own cluster.

ska 5y ago
On-prem for nearly anything is going to be at least a bit of a win if your utilization is uniform or predictable. The real win for not doing it is in adaptability.
- tedivm 5y ago
  Yeah, definitely, but I think it's important to talk about scale of that win. I mentioned in another comment that the GPU generation is roughly three years between major architecture upgrades- this has held true for a bit now, and that time may even stretch out a little. When the average company builds one of these clusters it's safe to assume they'll either run it for three years or sell it back for some return. Going with the cloud and assuming you don't commit to several years (losing that adaptability) the yearly cost of a p4 is $287,087. Over three years that's $861,261 to run a single machine. For about $450k you can build out a solid two machine (16gpu) cluster (including infiniband networking gear and a solid NAS) that will easily last three years. There are datacenters which specialize in this and companies that can manage these machines. If you don't have the cash up front you can lease them on good terms and your yearly bill will still be much lower than AWS. Model training is basically the one use case where I'm really willing to purchase equipment instead of using the cloud. The money it saves is enough to hire one or two more staff members, and the maintenance is shockingly low if you get it setup right to start.
  - dekhn 5y ago
    A couple counterpoints: for all the time you just spent making a local cluster, you could have spent 1 day negotiating a deal with the AWS TAM and get 50% or more off rack rates. So that $861K is more like half that. And second, that machien would be collocated with infinite storage and other processing systems. The CPU/GPU part of training is only part of the much larger ecosystem, which is what you're paying to be in when you use AWS.
boulos 5y ago
Disclosure: I used to work for Google Cloud. > you save a considerable amount of money and get a lot better performance when building your own cluster. This heavily depends on how much benefit you get from improving GPU performance from each generation. A lot of people assume 3/4-yr TCO. If you instead "rent" for 1 year at a time, you've been getting >2x benefits per generation lately. Most folks also measure "occupancy" for clusters like this rather than "utilization". That is, if a job is using 128 "GPUs" that counts as 128 in use. But that ignores that many jobs might have been just fine with T4s (which are a lot cheaper) versus A100s. (Depends a lot on the model, the I/O, etc.) Once you've bought a physical cluster, you're kind of stuck with that configuration (for better or worse). tl;dr: It's not just about "idle".
- tedivm 5y ago
  These generations tend to be three years apart though, so if you're buying as the new generation comes out then your total TCO period has you running almost peak hardware (there were several versions of the V100, each with minor improvements). Many vendors also offer buy back and upgrade programs. At the same time it's hard to understate how different the prices are here. Our break even point for using on prem compared to AWS was about nine months. After that we saved money for the rest of that hardwares lifetime. I definitely agree that people shouldn't just rush out and buy these without benchmarking and examining their usecase. The cloud is really good for this! At the same time though I have yet to see any cloud provider with anything even approaching the interlink I can get on prem and that means, so it's basically impossible to get the same performance out of the cloud as it is on prem right now.
  - boulos 5y ago
    > using on prem compared to AWS was about nine months. At list price or a moderate discount. The folks at this scale aren't paying that :).
    - deleted 5y ago
      [deleted]
dekhn 5y ago
the network topology and the network switch itself also can make a huge difference depending on traffic conditions; so you might have tons of fat NICs per GPU but if all of them want to alltoall, you better have a ton of cross section bandwidth. I always wonder about performance on these clusters. Back in MY day, I'd wait a week or more for results from my jobs, and immediately resubmit to wait two weeks in a queue for another week of runtime and do lots of data processing in the downtime. Then, I moved to cloud and decided on "what can I afford to do overnight" (IE, I set my time to result to be about 12 hours). I have a hard time justifying additional hardware to get results in 10 minutes versus a day, it seems like at that point you're just using it to get fast cycle times on new ideas, but who has new ideas every 10 minutes?
- tedivm 5y ago
  > the network topology and the network switch itself also can make a huge difference depending on traffic conditions; so you might have tons of fat NICs per GPU but if all of them want to alltoall, you better have a ton of cross section bandwidth. This is the beauty of the new nvswitch chips and the infiniband networks instead of ethernet. Anyone who is doing this is setting up a fully switched high bandwidth infiniband network with ridiculous traffic between them. Nvidia purchased Mellanox a year or two ago- combine that with the ridiculously awesome nvswitch in the A100 dgx machines and there's a huge jump in cross chip traffic ability. At the same time though a decent mellanox router is probably going to set you back $30k.
  - dekhn 5y ago
    I'm not aware of any cost-effective switch that permits scaling all v all to arbitrary sizes. THat's my point. modern infiniband uses nearly all the same tech as previous supercomputers, but with faster interfaces, and more of them. For example the facebook cluster is a dual-layer clos network, which is one of a few cost-effective ways to get very high randomly distributed traffic, but all v all communication scales as n*2 and n*2 wires gets expensive fast. Better to find algorithms that need less communication, than to make faster computers that allow you to write algorithms that needs lots of communication. Otherwise you'll always pay $$$ to reach peak GPU performance.