4 ms·

Disclosure: I work on Google Cloud. I think the right way to think about the economics here is either “I would pay $X/hr for this short-lived job” or “I want t

by boulos 6y ago

Disclosure: I work on Google Cloud. I think the right way to think about the economics here is either “I would pay $X/hr for this short-lived job” or “I want to compare with buying it” (3-yr committed use discount in our case, RIs / Instance Savings Plan for AWS). Unless you are an ML research lab (Google Brain, FAIR, OpenAI, etc.) or an HPC style site sharing these, you won’t get 100% utilization out of your “I just bought it” purchase. Worse, in ML land, accounting math about N-year depreciation is pretty bogus: if the A100 is 2.5x faster, you’d have been better off with a 1-yr CUD on GCP and refreshing, rather than buying Voltas last year. One amusing thing that’s not clear about “just buy a DGX” is that many people can’t even rack one of these. At 400 watts per A100, our 16x variant is 6.4 kW of GPUs. That’s before the rest of the system, etc. but there are (sadly) a lot of racks in the world that just can’t handle that.

ttul 6y ago
In essence, this is a supercomputer.
- boulos 6y ago
  Kind of. I think a giant pile of them hooked together with good networking would be :). But also, you can rent a single slice of one for tens of seconds and then walk away. I personally probably do about 20-ish GPU hours a month while dabbling (and only because I’m lazy and have my GPUs attached while I’m working, even if it’s just debugging the Python bits). For a V100 that’s $50/month, which is in the noise compared to dealing with owning infrastructure (and would be even less if I’d get by with a T4).
Zenst 6y ago
Many excellent points and the one of interest is power usage - it'snot just the cost of buying the kit, but also powering, maintaining (admin) aspects. So the Bottom line of ownership is far bigger and as you say, you really need to get 100% utilisation to capitalise in that level of outlay. Hence the on-demand cloud option on many levels becomes cost effective, even if it does seem not cheap. Sure it is not cheap, but overall, it can and does work out way cheaper for many use cases. Makes you wonder what kind of power costs you would incur running one 100% utilisation. Certainly even with best prices, be looking at several thousand a year I would of thought, that's not even factoring in provision. Which would mean 3 phase power for that type of load and then you have to balance out the phases. So many little details that become more an issue when you start getting to datacenter level power usage. Then UPS load/capacity costs/planning, networking. So whilst the costs of these units are high, the other costs that add up, sure do add up fast.
- boulos 6y ago
  Oh, I didn’t mean to suggest that the powering of the system was unreasonable. For example, Hetzner will charge you about $250 per rack per month and between 15 and 30 cents (USD roughly) per kWh [1]. I assume they include the cooling and so on, but that means you’re basically looking at a ~4 kW draw for the usual 8xA100 system with some cores and RAM. So budget just under $1/hr for the power and $1000/month all in (depending on location). Most colo power in the US is closer to the Finland calculation than the Germany one. I’d say keeping a spare system in case of failure is probably a bigger deal than the $10k/yr to house it :). [1] https://www.hetzner.com/colocation https://www.hetzner.com/colocation
- jjoonathan 6y ago
  Eh, the Titan V I bought last year broke even vs AWS inside of a month ($2000 vs $2234). Obviously there are factors going in to that -- I could live without paying the Tesla tax (didn't need to virtualize, didn't need the vram, did need the fp64), I bought used, I didn't have a problem keeping it fed, I didn't need to burst, etc, but my point is that for some GPU workloads the cloud GPUs are really expensive and the break-even utilization is far south of 100%, more like 5%.
  - codemk8 6y ago
    The GeForce makes huge difference here.
  - echelon 6y ago
    I'm currently renting LambdaLabs V100 instances at 100% utilization for training https://vo.codes https://vo.codes It's really expensive, and I think I should lean into buying hardware at this point. I want to build a high end GPU rig, but was wondering how easy the setup was. I've only built "consumer" systems before (2x 1080Ti). Is there any appreciable difference? Do you have a single card? Multiple? What motherboard do you use? Do you have any takeaways or resources you can share?
    - jjoonathan 6y ago
      I only have one card. I swapped the Titan V with my 1x1080 in my old, cheap motherboard and it just worked. I had to hook up the water cooling, but I did that because it came with a block, not because I optimized the thermal design. To verify motherboard compatibility, I looked up in the nvidia specs how many PCIe lanes and at what speed I should expect and confirmed in HWinfo that they were active in that configuration -- much like I'd look at a network interface to make sure my 10/5/2.5GbE hadn't turned into 1GbE on account of gremlins in the wires. I'm not using this for machine learning, so you might want to talk to someone who is before pulling the trigger. In particular, my need for fp64 made the choice of Titan V completely trivial, whereas in ML you might have to engage brain cells to pick a card or make a wait/buy determination.
tedivm 6y ago
You're definitely not going to be able to shove it into just any datacenter, but there's enough of a demand for it where specialty datacentes are popping up. We're in Colovore, which has a fantastic power density and is running roughly 1000k DGXs in their datacenter. It really wasn't all that difficult to get up and running. For us it made total sense, but we utilize physical hardware completely and have to scale into the cloud fairly regularly.
- boulos 6y ago
  Right. I’m not saying you can’t, but I am saying that when someone goes to their IT team and says “Hey, we want to buy a few of these” they’ll get met with a groan :). Did you move all your other gear into Colovore? (That’s one of the challenges, you often need to be close to the rest of your systems / data)
  - tedivm 6y ago
    We have a gigabit pipe to AWS and a local NAS for data caching.
- cardine 6y ago
  We have a similar situation to yours and have had a very positive experience with Colovore. We also run at 100% utilization 24/7 and frequently use the cloud for burst before we go out and buy more cards/servers. Colovore was super easy to get up and running and we will save at a minimum hundreds of thousands of dollars with this setup over exclusively using cloud instances.
  - tedivm 6y ago
    Yuuuup. Colovore definitely made it a lot easier, and I sing their praises to anyone who will listen.
gridlockd 6y ago
> Worse, in ML land, accounting math about N-year depreciation is pretty bogus: if the A100 is 2.5x faster, you’d have been better off with a 1-yr CUD on GCP and refreshing, rather than buying Voltas last year. Oh, so cloud providers are just giving away free resources? How generous! Seriously, let's do the math: If a V100 instance cost $9000 new and you bought it a year ago, you could still sell it today for over $3000. On AWS, an instance on a 1-yr CUD costs more than $1400/month, for a total of over $16000. You don't even need 50% utilization to break even. It doesn't matter how fast the A100 is.

by boulos 6y ago

ttul 6y ago

In essence, this is a supercomputer.

boulos 6y ago
Kind of. I think a giant pile of them hooked together with good networking would be :). But also, you can rent a single slice of one for tens of seconds and then walk away. I personally probably do about 20-ish GPU hours a month while dabbling (and only because I’m lazy and have my GPUs attached while I’m working, even if it’s just debugging the Python bits). For a V100 that’s $50/month, which is in the noise compared to dealing with owning infrastructure (and would be even less if I’d get by with a T4).

Zenst 6y ago

Many excellent points and the one of interest is power usage - it'snot just the cost of buying the kit, but also powering, maintaining (admin) aspects. So the Bottom line of ownership is far bigger and as you say, you really need to get 100% utilisation to capitalise in that level of outlay. Hence the on-demand cloud option on many levels becomes cost effective, even if it does seem not cheap. Sure it is not cheap, but overall, it can and does work out way cheaper for many use cases. Makes you wonder what kind of power costs you would incur running one 100% utilisation. Certainly even with best prices, be looking at several thousand a year I would of thought, that's not even factoring in provision. Which would mean 3 phase power for that type of load and then you have to balance out the phases. So many little details that become more an issue when you start getting to datacenter level power usage. Then UPS load/capacity costs/planning, networking. So whilst the costs of these units are high, the other costs that add up, sure do add up fast.

boulos 6y ago
Oh, I didn’t mean to suggest that the powering of the system was unreasonable. For example, Hetzner will charge you about $250 per rack per month and between 15 and 30 cents (USD roughly) per kWh [1]. I assume they include the cooling and so on, but that means you’re basically looking at a ~4 kW draw for the usual 8xA100 system with some cores and RAM. So budget just under $1/hr for the power and $1000/month all in (depending on location). Most colo power in the US is closer to the Finland calculation than the Germany one. I’d say keeping a spare system in case of failure is probably a bigger deal than the $10k/yr to house it :). [1] https://www.hetzner.com/colocation https://www.hetzner.com/colocation
jjoonathan 6y ago
Eh, the Titan V I bought last year broke even vs AWS inside of a month ($2000 vs $2234). Obviously there are factors going in to that -- I could live without paying the Tesla tax (didn't need to virtualize, didn't need the vram, did need the fp64), I bought used, I didn't have a problem keeping it fed, I didn't need to burst, etc, but my point is that for some GPU workloads the cloud GPUs are really expensive and the break-even utilization is far south of 100%, more like 5%.
- codemk8 6y ago
  The GeForce makes huge difference here.
- echelon 6y ago
  I'm currently renting LambdaLabs V100 instances at 100% utilization for training https://vo.codes https://vo.codes It's really expensive, and I think I should lean into buying hardware at this point. I want to build a high end GPU rig, but was wondering how easy the setup was. I've only built "consumer" systems before (2x 1080Ti). Is there any appreciable difference? Do you have a single card? Multiple? What motherboard do you use? Do you have any takeaways or resources you can share?
  - jjoonathan 6y ago
    I only have one card. I swapped the Titan V with my 1x1080 in my old, cheap motherboard and it just worked. I had to hook up the water cooling, but I did that because it came with a block, not because I optimized the thermal design. To verify motherboard compatibility, I looked up in the nvidia specs how many PCIe lanes and at what speed I should expect and confirmed in HWinfo that they were active in that configuration -- much like I'd look at a network interface to make sure my 10/5/2.5GbE hadn't turned into 1GbE on account of gremlins in the wires. I'm not using this for machine learning, so you might want to talk to someone who is before pulling the trigger. In particular, my need for fp64 made the choice of Titan V completely trivial, whereas in ML you might have to engage brain cells to pick a card or make a wait/buy determination.

tedivm 6y ago

You're definitely not going to be able to shove it into just any datacenter, but there's enough of a demand for it where specialty datacentes are popping up. We're in Colovore, which has a fantastic power density and is running roughly 1000k DGXs in their datacenter. It really wasn't all that difficult to get up and running. For us it made total sense, but we utilize physical hardware completely and have to scale into the cloud fairly regularly.

boulos 6y ago
Right. I’m not saying you can’t, but I am saying that when someone goes to their IT team and says “Hey, we want to buy a few of these” they’ll get met with a groan :). Did you move all your other gear into Colovore? (That’s one of the challenges, you often need to be close to the rest of your systems / data)
- tedivm 6y ago
  We have a gigabit pipe to AWS and a local NAS for data caching.
cardine 6y ago
We have a similar situation to yours and have had a very positive experience with Colovore. We also run at 100% utilization 24/7 and frequently use the cloud for burst before we go out and buy more cards/servers. Colovore was super easy to get up and running and we will save at a minimum hundreds of thousands of dollars with this setup over exclusively using cloud instances.
- tedivm 6y ago
  Yuuuup. Colovore definitely made it a lot easier, and I sing their praises to anyone who will listen.

gridlockd 6y ago

> Worse, in ML land, accounting math about N-year depreciation is pretty bogus: if the A100 is 2.5x faster, you’d have been better off with a 1-yr CUD on GCP and refreshing, rather than buying Voltas last year. Oh, so cloud providers are just giving away free resources? How generous! Seriously, let's do the math: If a V100 instance cost $9000 new and you bought it a year ago, you could still sell it today for over $3000. On AWS, an instance on a 1-yr CUD costs more than $1400/month, for a total of over $16000. You don't even need 50% utilization to break even. It doesn't matter how fast the A100 is.