6 ms·

In case you're wondering about pricing: the V100 costs $1260/GPU/month and this A100 will have about 2.5x its performance [1]. A n2d-highmem-96 instance is $437

In case you're wondering about pricing: the V100 costs $1260/GPU/month and this A100 will have about 2.5x its performance [1]. A n2d-highmem-96 instance is $4377 per month. So for the maxed out a2-megagpu-16g I would expect around $54k per month before usage discounts etc. 1. https://developer.nvidia.com/blog/nvidia-ampere-architecture-in-depth/ https://developer.nvidia.com/blog/nvidia-ampere-architecture...

fluffything 6y ago
What's the perf/$ ratio for these different instances ? I've also read that the A100 can deliver 4x the performance on some DL workloads.
selectodude 6y ago
I feel like if you’re maxing one of these clusters for an entire month at a time, it might still be cheaper to just buy one.
- manigandham 6y ago
  That's true for any cloud server.
  - 867-5309 6y ago
    untrue Intel Xeon Gold 5120 14Core @ £1200 used renting 1Core @ £1.20/mo. would take 6 years to fully pay for that single core alone, and that's excluding the 512MB RAM, 10GB SSD and unlimited 400Mb/s bandwidth
    - sbierwagen 6y ago
      Here's a completed ebay auction from April 20th that sold a 5120 for $699: https://www.ebay.com/itm/Intel-Xeon-Gold-5120-SR3GD-14-Core-2-20GHz-10-40GT-s-UPI-19-25MB-LGA3647-U159103/313059429725?hash=item48e3cbcd5d:g:gCoAAOSwni1ejMs1 https://www.ebay.com/itm/Intel-Xeon-Gold-5120-SR3GD-14-Core-... I know the pound hasn't been doing well lately, but surely it hasn't been that bad.
      - 867-5309 6y ago
        we're still talking years, not one month
    - shadowpho 6y ago
      Where are you getting $1.20/mo on 1 core/512MB ram/10GB ssd and half a GB bandiwdth? Amazon for example was something like $50/mo for bandwidth alone
      - 867-5309 6y ago
        it's not AWS-grade, just a budget VPS, but has been just as reliable and performant so far. it's out there should you care to seek. the correct currency of £ will help
        reitzensteinm 6y ago
        You are renting an overprovisioned timeslice, not a dedicated core. Even Hetzner's time slice instances are twice the price. Their dedicated CPU instances are almost 10x the price.
        867-5309 6y ago
        possibly, but nothing to suggest that 6 months in. they suit the use cases perfectly
        reitzensteinm 6y ago
        It's definitely a timeslice, and that throws off your comparison wildly. Otherwise you'd be able to rent a 32 core dedicated machine at the same place for $48 a month. Can you?
        867-5309 6y ago
        it mentions "dedicated resources", but it's unclear whether this applies to CPU it might throw the comparison off if there was variation in performance at different times, but there isn't cores scale with RAM, you cannot rent 32 with 512MB
        6y ago
  - m3kw9 6y ago
    You forgot there is a lot of IT work after buying the hardware to set it up properly. That’s time spent
    - nojito 6y ago
      That’s a one time cost plus maintenance. Still comes out cheaper than cloud offerings.
- jeffbee 6y ago
  The DGX A100 costs a quarter million dollars, so the crossover time might be significantly longer than a month.
  - selectodude 6y ago
    So six months? Nine months? And at the end, you have an asset. You can always sell it and replace it with the newer model.
- dekhn 6y ago
  Ah, but when you use a large cloud, you're not just paying for the cost of the server. You're paying for access to a large high performance network and storage system that is extremely close to your server. Replicating those properties will cost you far more (well, it depends on your specific needs) than the cloud provider.
- cardine 6y ago
  We did the math and decided to buy. For us, the break even point (including everything) was nine months of running 24/7.
boulos 6y ago
Disclosure: I work on Google Cloud. I think the right way to think about the economics here is either “I would pay $X/hr for this short-lived job” or “I want to compare with buying it” (3-yr committed use discount in our case, RIs / Instance Savings Plan for AWS). Unless you are an ML research lab (Google Brain, FAIR, OpenAI, etc.) or an HPC style site sharing these, you won’t get 100% utilization out of your “I just bought it” purchase. Worse, in ML land, accounting math about N-year depreciation is pretty bogus: if the A100 is 2.5x faster, you’d have been better off with a 1-yr CUD on GCP and refreshing, rather than buying Voltas last year. One amusing thing that’s not clear about “just buy a DGX” is that many people can’t even rack one of these. At 400 watts per A100, our 16x variant is 6.4 kW of GPUs. That’s before the rest of the system, etc. but there are (sadly) a lot of racks in the world that just can’t handle that.
- ttul 6y ago
  In essence, this is a supercomputer.
  - boulos 6y ago
    Kind of. I think a giant pile of them hooked together with good networking would be :). But also, you can rent a single slice of one for tens of seconds and then walk away. I personally probably do about 20-ish GPU hours a month while dabbling (and only because I’m lazy and have my GPUs attached while I’m working, even if it’s just debugging the Python bits). For a V100 that’s $50/month, which is in the noise compared to dealing with owning infrastructure (and would be even less if I’d get by with a T4).
- Zenst 6y ago
  Many excellent points and the one of interest is power usage - it'snot just the cost of buying the kit, but also powering, maintaining (admin) aspects. So the Bottom line of ownership is far bigger and as you say, you really need to get 100% utilisation to capitalise in that level of outlay. Hence the on-demand cloud option on many levels becomes cost effective, even if it does seem not cheap. Sure it is not cheap, but overall, it can and does work out way cheaper for many use cases. Makes you wonder what kind of power costs you would incur running one 100% utilisation. Certainly even with best prices, be looking at several thousand a year I would of thought, that's not even factoring in provision. Which would mean 3 phase power for that type of load and then you have to balance out the phases. So many little details that become more an issue when you start getting to datacenter level power usage. Then UPS load/capacity costs/planning, networking. So whilst the costs of these units are high, the other costs that add up, sure do add up fast.
m0zg 6y ago
FYI: NVIDIA is notorious for bullshit benchmarks. There's no way it'll be 2.5x the speed of V100 on any kind of a realistic workload.
- codemk8 6y ago
  Agree. That is theoretical speed up. Most applications are bandwidth bounded thus cannot get that ratio.
imtringued 6y ago
Compare it to gattling guns GAU-8. They shoot at 3900 rpm but only for a few seconds. The reason is that a firing solution may only be correct for a split second. It's all about peak performance because you only shoot when you really need to. The same thing happens with these large instances. Because these instances are so much bigger that you won't use them for the entire month. You'll use them to get results within hours or days.

fluffything 6y ago

What's the perf/$ ratio for these different instances ? I've also read that the A100 can deliver 4x the performance on some DL workloads.

selectodude 6y ago

I feel like if you’re maxing one of these clusters for an entire month at a time, it might still be cheaper to just buy one.

manigandham 6y ago
That's true for any cloud server.
- 867-5309 6y ago
  untrue Intel Xeon Gold 5120 14Core @ £1200 used renting 1Core @ £1.20/mo. would take 6 years to fully pay for that single core alone, and that's excluding the 512MB RAM, 10GB SSD and unlimited 400Mb/s bandwidth
  - sbierwagen 6y ago
    Here's a completed ebay auction from April 20th that sold a 5120 for $699: https://www.ebay.com/itm/Intel-Xeon-Gold-5120-SR3GD-14-Core-2-20GHz-10-40GT-s-UPI-19-25MB-LGA3647-U159103/313059429725?hash=item48e3cbcd5d:g:gCoAAOSwni1ejMs1 https://www.ebay.com/itm/Intel-Xeon-Gold-5120-SR3GD-14-Core-... I know the pound hasn't been doing well lately, but surely it hasn't been that bad.
    - 867-5309 6y ago
      we're still talking years, not one month
  - shadowpho 6y ago
    Where are you getting $1.20/mo on 1 core/512MB ram/10GB ssd and half a GB bandiwdth? Amazon for example was something like $50/mo for bandwidth alone
    - 867-5309 6y ago
      it's not AWS-grade, just a budget VPS, but has been just as reliable and performant so far. it's out there should you care to seek. the correct currency of £ will help
      - reitzensteinm 6y ago
        You are renting an overprovisioned timeslice, not a dedicated core. Even Hetzner's time slice instances are twice the price. Their dedicated CPU instances are almost 10x the price.
        867-5309 6y ago
        possibly, but nothing to suggest that 6 months in. they suit the use cases perfectly
        reitzensteinm 6y ago
        It's definitely a timeslice, and that throws off your comparison wildly. Otherwise you'd be able to rent a 32 core dedicated machine at the same place for $48 a month. Can you?
        867-5309 6y ago
        it mentions "dedicated resources", but it's unclear whether this applies to CPU it might throw the comparison off if there was variation in performance at different times, but there isn't cores scale with RAM, you cannot rent 32 with 512MB
        6y ago
- m3kw9 6y ago
  You forgot there is a lot of IT work after buying the hardware to set it up properly. That’s time spent
  - nojito 6y ago
    That’s a one time cost plus maintenance. Still comes out cheaper than cloud offerings.
jeffbee 6y ago
The DGX A100 costs a quarter million dollars, so the crossover time might be significantly longer than a month.
- selectodude 6y ago
  So six months? Nine months? And at the end, you have an asset. You can always sell it and replace it with the newer model.
dekhn 6y ago
Ah, but when you use a large cloud, you're not just paying for the cost of the server. You're paying for access to a large high performance network and storage system that is extremely close to your server. Replicating those properties will cost you far more (well, it depends on your specific needs) than the cloud provider.
cardine 6y ago
We did the math and decided to buy. For us, the break even point (including everything) was nine months of running 24/7.

boulos 6y ago

Disclosure: I work on Google Cloud. I think the right way to think about the economics here is either “I would pay $X/hr for this short-lived job” or “I want to compare with buying it” (3-yr committed use discount in our case, RIs / Instance Savings Plan for AWS). Unless you are an ML research lab (Google Brain, FAIR, OpenAI, etc.) or an HPC style site sharing these, you won’t get 100% utilization out of your “I just bought it” purchase. Worse, in ML land, accounting math about N-year depreciation is pretty bogus: if the A100 is 2.5x faster, you’d have been better off with a 1-yr CUD on GCP and refreshing, rather than buying Voltas last year. One amusing thing that’s not clear about “just buy a DGX” is that many people can’t even rack one of these. At 400 watts per A100, our 16x variant is 6.4 kW of GPUs. That’s before the rest of the system, etc. but there are (sadly) a lot of racks in the world that just can’t handle that.

ttul 6y ago
In essence, this is a supercomputer.
- boulos 6y ago
  Kind of. I think a giant pile of them hooked together with good networking would be :). But also, you can rent a single slice of one for tens of seconds and then walk away. I personally probably do about 20-ish GPU hours a month while dabbling (and only because I’m lazy and have my GPUs attached while I’m working, even if it’s just debugging the Python bits). For a V100 that’s $50/month, which is in the noise compared to dealing with owning infrastructure (and would be even less if I’d get by with a T4).
Zenst 6y ago
Many excellent points and the one of interest is power usage - it'snot just the cost of buying the kit, but also powering, maintaining (admin) aspects. So the Bottom line of ownership is far bigger and as you say, you really need to get 100% utilisation to capitalise in that level of outlay. Hence the on-demand cloud option on many levels becomes cost effective, even if it does seem not cheap. Sure it is not cheap, but overall, it can and does work out way cheaper for many use cases. Makes you wonder what kind of power costs you would incur running one 100% utilisation. Certainly even with best prices, be looking at several thousand a year I would of thought, that's not even factoring in provision. Which would mean 3 phase power for that type of load and then you have to balance out the phases. So many little details that become more an issue when you start getting to datacenter level power usage. Then UPS load/capacity costs/planning, networking. So whilst the costs of these units are high, the other costs that add up, sure do add up fast.

m0zg 6y ago

FYI: NVIDIA is notorious for bullshit benchmarks. There's no way it'll be 2.5x the speed of V100 on any kind of a realistic workload.

codemk8 6y ago
Agree. That is theoretical speed up. Most applications are bandwidth bounded thus cannot get that ratio.

imtringued 6y ago

Compare it to gattling guns GAU-8. They shoot at 3900 rpm but only for a few seconds. The reason is that a firing solution may only be correct for a split second. It's all about peak performance because you only shoot when you really need to. The same thing happens with these large instances. Because these instances are so much bigger that you won't use them for the entire month. You'll use them to get results within hours or days.