7 ms·

Does anyone know why everyone is still buying Nvidia instead of custom AI accelerators from other vendors? For example, on paper the new Graphcore machines look

Does anyone know why everyone is still buying Nvidia instead of custom AI accelerators from other vendors? For example, on paper the new Graphcore machines look like an easy win, or at least a risk worth taking. (I see this particular supercomputer was funded by Nvidia but my question is about the general trend).

freeone3000 6y ago
Do those custom AI accelerators work with PyTorch and Tensorflow? The graphcore website says that support with PyTorch aten is available in 2020, but it's not -- and Aten is inference only, no autograd, and no extensions. Without those, we can't train on it.
- abainbridge 6y ago
  Graphcore claims they do. I don't know true that is though.
JBorrow 6y ago
Many people use HPC for things that are not AI/ML, such as CFD. Those applications are typically written using MPI/CUDA
Cthulhu_ 6y ago
Risk mitigation, maintenance, resell value (?), support, reliability. The custom AI accelerators you mentioned, how long have those been in business for? How many units have they moved? How many generations of hardware have they produced? Will they still be around to replace or upgrade units in 5-10 years? How flexible are they in their workload? That's a lot of factors to keep in mind when you're spending millions on a supercomputer. I'd go for an established hardware provider as well. I'm not knocking the custom AI chips, but I wouldn't try and max out my budget with those - keep them for smaller applications for now.
- eecc 6y ago
  Your supercomputer won’t last more than a couple years, 5 at max. Then it will simply be unjustifiably expensive to run when newer, more power efficient hardware becomes available. What matters is the software platform and compatibility: you don’t want to retool and port all your existing programs. That’s where having a platform - CUDA - becomes a deal-maker. What I don’t understand is how consumers - big, institutional consumers- don’t insist on viable vendor neutral platforms like OpenCL. Obviously they’re the only ones that have an interest in cross-vendor compatibility.
  - dannyw 6y ago
    CUDA is valuable and seen as a positive, even if it's vendor locked. It's battle tested. Engineers are familiar with it. Support is provided because it's all under one roof. Performance targets, roadmap timeframes, etc can be assured. The same reason a big institutional consumer will happily pay Microsoft for Windows Enterprise, instead of demanding Microsoft rebuild Windows using open source technologies. A supercomputer or DC cluster has a fixed lifespan and amortization period, if you can get something in a contract, with everything you want for a price you are happy with, why would you introduce risk to it by demanding open standards for no benefit?
    - eecc 6y ago
      I’m not saying CUDA is not valuable. Heck, proprietary has a ton of value... see how Intel owes much of its position to its compiler (the FORTRAN one in particular) suite. Of course you don’t get to dominate a field to such an extent without some redeeming positive quality. What I’m arguing - or more appropriately, what I’m wondering - is why on Earth aren’t consumer consortiums not insisting on open-standard platforms. I get how you want to eek every drop of performance out of your tools without making it your only mission (that would be the research you’re using the tool for), but tying yourself to a single proprietary vendor defeats that purpose imho
  - lorenzhs 6y ago
    Not sure why you're downvoted, HPC systems don't last more than ~6 years before they are decommissioned because they've become too power-hungry to justify their continued use. I just checked some of the systems I'm familiar with and they were all decommissioned after around 6 years of use (e.g. SuperMUC, 2012-2018; two smaller cluster at the university where I work: 08/2012-04/2017, 01/2014-03/2020). I also attended a presentation at IPDPS 2018 by someone from one of the big US national labs where they talked about how absolutely huge of an undertaking it was to port their codes to make efficient use of GPUs. You don't just re-do all of that if the payoff isn't enormous.
Random_ernest 6y ago
From my very limited understanding the Graphcore machines outperform CUDA only significantly in inference, in training the improvements might not be sufficient to switch technology.
pama 6y ago
There is a trend towards larger models in the state of the art of deep learning research. The total cost of the on-chip memory on the NVidia GPU is still a good value proposition compared to current custom deep learning accelerators. This cost to get to large on-chip memory, combined with the flexibility of CUDA for other types of scientific applications, makes it harder for such accelerators to compete with Nvidia in such large academic research clusters. I hope that this situation changes over the next couple of years.
- anonymousDan 6y ago
  Is this really true? At least for inference I've seen data showing the trend is rather for smaller models that exploit sparsity.
  - pama 6y ago
    This is a chart from before GPT-3: https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1...
  - nl 6y ago
    Yes, in NLP big models rule and they are getting bigger.
fluffy87 6y ago
I reimplemented one benchmark using TF and the speed up wasn’t the 10x or so hey we’re claiming, but more like 1x..
nl 6y ago
The only non-NVidia device you can actually use that has broad support outside basic models in software is Google's TPUs. Graphcore looks great! But no one outside Graphcore has used them so who knows. Intel's Nervana looked great on paper, right up until they dumped it. There are some interesting accelerator options around for inference. But for training it's NVidia for almost everything, and TPUs as a good option is a few cases. But you can't buy TPUs (except for the inference-only Coral board), and universities like to own hardware.
- rewq4321 6y ago
  > But no one outside Graphcore has used them so who knows. This part really confuses me. I'd love to try out their hardware, but the only cloud offering they had last time I checked made you rent a whole month's worth (for many thousands of dollars). It seems like getting it in the hands of devs should be a top priority.

freeone3000 6y ago

Do those custom AI accelerators work with PyTorch and Tensorflow? The graphcore website says that support with PyTorch aten is available in 2020, but it's not -- and Aten is inference only, no autograd, and no extensions. Without those, we can't train on it.

abainbridge 6y ago
Graphcore claims they do. I don't know true that is though.

JBorrow 6y ago

Many people use HPC for things that are not AI/ML, such as CFD. Those applications are typically written using MPI/CUDA

Cthulhu_ 6y ago

Risk mitigation, maintenance, resell value (?), support, reliability. The custom AI accelerators you mentioned, how long have those been in business for? How many units have they moved? How many generations of hardware have they produced? Will they still be around to replace or upgrade units in 5-10 years? How flexible are they in their workload? That's a lot of factors to keep in mind when you're spending millions on a supercomputer. I'd go for an established hardware provider as well. I'm not knocking the custom AI chips, but I wouldn't try and max out my budget with those - keep them for smaller applications for now.

eecc 6y ago
Your supercomputer won’t last more than a couple years, 5 at max. Then it will simply be unjustifiably expensive to run when newer, more power efficient hardware becomes available. What matters is the software platform and compatibility: you don’t want to retool and port all your existing programs. That’s where having a platform - CUDA - becomes a deal-maker. What I don’t understand is how consumers - big, institutional consumers- don’t insist on viable vendor neutral platforms like OpenCL. Obviously they’re the only ones that have an interest in cross-vendor compatibility.
- dannyw 6y ago
  CUDA is valuable and seen as a positive, even if it's vendor locked. It's battle tested. Engineers are familiar with it. Support is provided because it's all under one roof. Performance targets, roadmap timeframes, etc can be assured. The same reason a big institutional consumer will happily pay Microsoft for Windows Enterprise, instead of demanding Microsoft rebuild Windows using open source technologies. A supercomputer or DC cluster has a fixed lifespan and amortization period, if you can get something in a contract, with everything you want for a price you are happy with, why would you introduce risk to it by demanding open standards for no benefit?
  - eecc 6y ago
    I’m not saying CUDA is not valuable. Heck, proprietary has a ton of value... see how Intel owes much of its position to its compiler (the FORTRAN one in particular) suite. Of course you don’t get to dominate a field to such an extent without some redeeming positive quality. What I’m arguing - or more appropriately, what I’m wondering - is why on Earth aren’t consumer consortiums not insisting on open-standard platforms. I get how you want to eek every drop of performance out of your tools without making it your only mission (that would be the research you’re using the tool for), but tying yourself to a single proprietary vendor defeats that purpose imho
- lorenzhs 6y ago
  Not sure why you're downvoted, HPC systems don't last more than ~6 years before they are decommissioned because they've become too power-hungry to justify their continued use. I just checked some of the systems I'm familiar with and they were all decommissioned after around 6 years of use (e.g. SuperMUC, 2012-2018; two smaller cluster at the university where I work: 08/2012-04/2017, 01/2014-03/2020). I also attended a presentation at IPDPS 2018 by someone from one of the big US national labs where they talked about how absolutely huge of an undertaking it was to port their codes to make efficient use of GPUs. You don't just re-do all of that if the payoff isn't enormous.

Random_ernest 6y ago

From my very limited understanding the Graphcore machines outperform CUDA only significantly in inference, in training the improvements might not be sufficient to switch technology.

pama 6y ago

There is a trend towards larger models in the state of the art of deep learning research. The total cost of the on-chip memory on the NVidia GPU is still a good value proposition compared to current custom deep learning accelerators. This cost to get to large on-chip memory, combined with the flexibility of CUDA for other types of scientific applications, makes it harder for such accelerators to compete with Nvidia in such large academic research clusters. I hope that this situation changes over the next couple of years.

anonymousDan 6y ago
Is this really true? At least for inference I've seen data showing the trend is rather for smaller models that exploit sparsity.
- pama 6y ago
  This is a chart from before GPT-3: https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1...
- nl 6y ago
  Yes, in NLP big models rule and they are getting bigger.

fluffy87 6y ago

I reimplemented one benchmark using TF and the speed up wasn’t the 10x or so hey we’re claiming, but more like 1x..

nl 6y ago

The only non-NVidia device you can actually use that has broad support outside basic models in software is Google's TPUs. Graphcore looks great! But no one outside Graphcore has used them so who knows. Intel's Nervana looked great on paper, right up until they dumped it. There are some interesting accelerator options around for inference. But for training it's NVidia for almost everything, and TPUs as a good option is a few cases. But you can't buy TPUs (except for the inference-only Coral board), and universities like to own hardware.

rewq4321 6y ago
> But no one outside Graphcore has used them so who knows. This part really confuses me. I'd love to try out their hardware, but the only cloud offering they had last time I checked made you rent a whole month's worth (for many thousands of dollars). It seems like getting it in the hands of devs should be a top priority.