3 ms·

> 50% more than Radeon 7900 XTX, but the AMD GPU costs 10x less You could say that about some of nvidia's other GPUs too. A better comparison would be AMD's se

> 50% more than Radeon 7900 XTX, but the AMD GPU costs 10x less You could say that about some of nvidia's other GPUs too. A better comparison would be AMD's server offerings, which are priced comparably. > Unlike traditional HPC stuff like FEM or fluid dynamics, the GPU code of ML inference is not that complicated. It's complicated enough relative to the problem ML people are trying to solve. That's why in ML, all these tools have emerged specifically to let practitioners not care about the GPU code. People writing pytorch code don't want to think about CUDA vs vulkan at all. It's more of a question for the people writing the foundational libraries like pytorch. I sincerely hope that tools like openai's triton make it easier to support a variety of GPUs.

Const-me 4y ago
> You could say that about some of nvidia's other GPUs too. Technically, yes. Practically, nVidia forbids to deploy their other GPUs in data centers, in the EULA of their drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-geforce-eula-to-prohibit-data-center-use/ https://www.datacenterdynamics.com/en/news/nvidia-updates-ge... May or may not be enforceable, probably depends on jurisdiction, but I’m pretty sure that EULA gonna scare some people. > People writing pytorch code don't want to think about CUDA vs vulkan at all. Well, there’re also other people. OpenAI’s CEO doesn’t necessarily write pytorch code, but he described the compute costs of ChatGPT as “eye-watering”. > I sincerely hope that tools like openai's triton make it easier to support a variety of GPUs. I’ll believe it when I see it. Performance portability across GPU architectures ain’t trivial. For instance, in my Whisper implementation, the most expensive compute shader is the one which computes product of 2 matrices. nVidia version: https://github.com/Const-me/Whisper/blob/master/ComputeShaders/mulMatTiled.hlsl https://github.com/Const-me/Whisper/blob/master/ComputeShade... AMD version: https://github.com/Const-me/Whisper/blob/master/ComputeShaders/mulMatTiledEx.hlsl https://github.com/Const-me/Whisper/blob/master/ComputeShade... These two compute shaders have little in common, they even use different memory layout for the arguments, despite both compute shaders are doing the same math.

Const-me 4y ago

> You could say that about some of nvidia's other GPUs too. Technically, yes. Practically, nVidia forbids to deploy their other GPUs in data centers, in the EULA of their drivers: https://www.datacenterdynamics.com/en/news/nvidia-updates-geforce-eula-to-prohibit-data-center-use/ https://www.datacenterdynamics.com/en/news/nvidia-updates-ge... May or may not be enforceable, probably depends on jurisdiction, but I’m pretty sure that EULA gonna scare some people. > People writing pytorch code don't want to think about CUDA vs vulkan at all. Well, there’re also other people. OpenAI’s CEO doesn’t necessarily write pytorch code, but he described the compute costs of ChatGPT as “eye-watering”. > I sincerely hope that tools like openai's triton make it easier to support a variety of GPUs. I’ll believe it when I see it. Performance portability across GPU architectures ain’t trivial. For instance, in my Whisper implementation, the most expensive compute shader is the one which computes product of 2 matrices. nVidia version: https://github.com/Const-me/Whisper/blob/master/ComputeShaders/mulMatTiled.hlsl https://github.com/Const-me/Whisper/blob/master/ComputeShade... AMD version: https://github.com/Const-me/Whisper/blob/master/ComputeShaders/mulMatTiledEx.hlsl https://github.com/Const-me/Whisper/blob/master/ComputeShade... These two compute shaders have little in common, they even use different memory layout for the arguments, despite both compute shaders are doing the same math.