4 ms·
4.9x faster FP64, but only 1.2x faster FP16. FP64 is totally irrelevant for machine learning. A100 has been available on AWS for over a year already, while MI20
by modeless 5y ago
4.9x faster FP64, but only 1.2x faster FP16. FP64 is totally irrelevant for machine learning. A100 has been available on AWS for over a year already, while MI200 won't be available at all until next year. Nvidia's A100 successor should be out next year too. And Nvidia's software stack puts AMD's to shame. AMD needs to do a lot better if they want to start being competitive in ML.
- 1-6 5y agoIf you mean integrating the hardware stack to existing open-source software stacks, the yes, Nvidia has the clear advantage. What Nvidia has accomplished is not trivial but it’s not an impossible feat.
- m_mueller 5y agoFP64 is very relevant for the kind of physics simulations that supercomputers are often built for. However the software stack is as well, and there I agree - it’s AMD’s main weak point.
- boibombeiro 5y agoIsn't the training done at FP32 and quantized aftweards? Not that this would make this GPU more appealing. The FP32 performance isn't great either. The memory bandwidth seems pretty good though.
- kolbusa 5y agoThere is a way to train with mixed fp16/fp32 precision.
- diamondlovesyou 5y agoMachine learning isn't the only useful thing to run on "GPU"s (these can't do graphics workloads anymore; they're basically just vector processors). AMD isn't attempting to complete on f16 perf. They're completing where Nvidia's perf is abysmal: f64. Having programmed AMDGPUs at a lower level than HIP/Rocm, they are actually much better than Nvidia (and in fact I'm able to do cool things like pcie large bar/p2p even on RDNA1 GPUs) in terms of flexibility. The HIP/CUDA API (nevermind OpenCL) doesn't do them justice, CUDA's enormous ecosystem and vender lock-in notwithstanding.
- baybal2 5y agoThat's a smart move given GPUs may now loose a lot of market as "AI" thing is rapidly losing steam, and SLIDE is getting better, and better.
- mkl 5y agoMore on SLIDE for those curious: https://twimlai.com/slide-the-algorithmic-alternative-that-outperforms-gpus/ https://twimlai.com/slide-the-algorithmic-alternative-that-o... It's a CPU-based deep learning training algorithm that beats GPUs for some tasks.
- freemint 5y agoI've not seen anything convincing that slide can approximate convolutional layers like at all without almost doing a full convolution at which point it is not competitive. Can you link me to paper about recent (2021?) work or improvements with slide?
- fxtentacle 5y agoI thought SLIDE was optimized for classification with mind-boggingly sparse data. I.e. "choose the best label for this image out of those 1 mio choices".
- homerowilson 5y agoThe current work https://proceedings.mlsys.org/paper/2021/hash/3636638817772e42b59d74cff571fbb3-Abstract.html https://proceedings.mlsys.org/paper/2021/hash/3636638817772e... seems mostly about tuning the original idea instead of expanding its scope. But it's still a neat idea. I guess it could be possible to adpt many of the approximations used in the SLIDE idea to GPUs too though...
- N1H1L 5y agoThey also have a company called ThirdAI to commercialize SLIDE
- cnschn 5y ago
- marmaduke 5y agoIt’s not irrelevant for machine learning in general, e.g. Stan (Bayesian/MCMC) is fp64 only, so its OpenCL offloading is more effective where fp64 is better supported. It’s less relevant for the kinds of deep neural net work enabled by a vendor who’s software stack is designed around 32 and lower bit types, perhaps in an attempt to ensure their flops numbers are highest?