5 ms·

Research doesn't really have the sunk-cost that industry does. New students are willing to try new things and supervisors don't necessarily need to reign them i

by sottol 3y ago

Research doesn't really have the sunk-cost that industry does. New students are willing to try new things and supervisors don't necessarily need to reign them in. I wonder what is holding AMD back in research? Their cards seem much less costly. I would have figured a nifty research student would figure out quickly how to port torch and run twice as many gpus with his small budget to eek out a bit more performance.

WinLychee 3y ago
The software support just isn't there. The drivers need work, the whole ecosystem is built on CUDA not OpenCL, etc. Not to say someone that tries super hard can't do it, e.g. https://github.com/DTolm/VkFFT https://github.com/DTolm/VkFFT .
visarga 3y ago
You can run LLMs on AMD. Maybe not every neural network, but LLMs do work.
zehaeva 3y ago
If AMD cards were half the price of Nvidia ones then sure, this would happen. The 4090 can be had for ~$1600USD and the RX 7900 for about ~$1000USD. A significant discount, however the RX 7900 is about 3/4ths as powerful as the 4090, which puts it more in the class as a 4080, which costs about as much. As a small budget research/grad student, if the price difference isn't that big, why waste the time porting torch to it?
- dauertewigkeit 3y ago
  Nah, price isn't going to be a motivating factor. If AMD came up with a card that had 3x the VRAM of the latest NVIDIA offering there would be research groups who would be interested because loads of models are hardware bottlenecked.
  - latchkey 3y ago
    MI250's (which are really 2 gpus on one board) are 128gb and can be chained together with other cards.
  - comte7092 3y ago
    Of course cost is a motivating factor. The very nature of GPUs implies that they are optimized for parallelizable workloads. If I can split my training across 3x the chips at 1/10th the cost for each chip, why wouldn’t I do that?
- latchkey 3y ago
  You're thinking consumer. MI250 cards are less than half the price of H100, and more importantly, you can buy them today.
  - zehaeva 3y ago
    Now I'm imaging a small budget grad student/researcher with a budget of several hundred grand. Maybe I'm not as familiar with the word small as I thought I as. :)
    - latchkey 3y ago
      Can buy 4 MI250’s (8 gpus) in a high end chassis for less than $100k. Today. Try that with H100’s.
dauertewigkeit 3y ago
99% of people publishing at top conferences are not particularly technically skilled and do not want to waste time adopting a new platform, because the competition is to publish papers and nobody cares if you do that on an AMD machine instead of an NVIDIA machine. The best funded labs have research developers whose only job is to optimize implementations. However these same labs will have the latest NVIDIA hardware.
dathinab 3y ago
AFIK in research nothing AMD had competitive GPGPUs AFIK just only relevant to a small number of very very large customers problems where mostly outside of research mainly there wasn't much insensitive (potential profit) for AMD to bring there GPGPU tooling to the consumer/small company marked and polish it for LLMs (to be clear I do not mean OpenCL, which was long term available but general subpar and badly supported) Nvideas mindshare was just too dominant and a few years ago it wasn't that uncommon for researchers to idk. create new building blocks or manual optimizations involving direct work with CUDA and similar But that's exactly what changed, by now, especially with LLMs, research does nearly always only involve usage of "high level abstractions" which are quite independent of the underlying gpu compute code (high-level might not be the best description as many of this GPU independent abstractions are still quite low level) . AMD has already shown that they can support that quite well and it seems to be mainly be question of polishing before it becomes more widely available. Another problem is that in the past AMD had decent GPU (compute/server) parts and GPU (gaming) parts but there GPU (gaming) parts where not that usable for compute. On the other hand Nvidea sold high end GPUs which can do both and can be "good enough" even for a lot of smaller companies. So a ton of researchers had easy access to that GPUs where access to specialized server compute cards is always complicated and often far more expensive (e.g. due to only being sold in bulk). This still somewhat holds up for the newest generation of AMD GPUs but much much less so. At the same time LLMs become so large that even using the highest-end Nvidea GPU became ... to slow. And selling a more high end customer GPU isn't really viable either IMHO. Additionally local inference seems to become much much more relevant and new AMD laptop CPU/GPU bundles and dedicated GPUs seem to be quite well equipped for that. Also the marked it growing a lot, so even if you just manage to get smaller % cut of the marked share it might now be profitable. I.e. they don't need to beat Nvidea in that marked anymore to make profit, grabbing a bit of marked share can now already be worthwhile. --- > port torch Idk. if it's already publically available/published but AMD has demoed proper well working torch support based on ROCm (instead of OpenCL).
- latchkey 3y ago
  https://pytorch.org/blog/experience-power-pytorch-2.0/ https://pytorch.org/blog/experience-power-pytorch-2.0/ https://www.mosaicml.com/blog/amd-mi250 https://www.mosaicml.com/blog/amd-mi250

by sottol 3y ago

WinLychee 3y ago

The software support just isn't there. The drivers need work, the whole ecosystem is built on CUDA not OpenCL, etc. Not to say someone that tries super hard can't do it, e.g. https://github.com/DTolm/VkFFT https://github.com/DTolm/VkFFT .

visarga 3y ago

You can run LLMs on AMD. Maybe not every neural network, but LLMs do work.

zehaeva 3y ago

If AMD cards were half the price of Nvidia ones then sure, this would happen. The 4090 can be had for ~$1600USD and the RX 7900 for about ~$1000USD. A significant discount, however the RX 7900 is about 3/4ths as powerful as the 4090, which puts it more in the class as a 4080, which costs about as much. As a small budget research/grad student, if the price difference isn't that big, why waste the time porting torch to it?

dauertewigkeit 3y ago
Nah, price isn't going to be a motivating factor. If AMD came up with a card that had 3x the VRAM of the latest NVIDIA offering there would be research groups who would be interested because loads of models are hardware bottlenecked.
- latchkey 3y ago
  MI250's (which are really 2 gpus on one board) are 128gb and can be chained together with other cards.
- comte7092 3y ago
  Of course cost is a motivating factor. The very nature of GPUs implies that they are optimized for parallelizable workloads. If I can split my training across 3x the chips at 1/10th the cost for each chip, why wouldn’t I do that?
latchkey 3y ago
You're thinking consumer. MI250 cards are less than half the price of H100, and more importantly, you can buy them today.
- zehaeva 3y ago
  Now I'm imaging a small budget grad student/researcher with a budget of several hundred grand. Maybe I'm not as familiar with the word small as I thought I as. :)
  - latchkey 3y ago
    Can buy 4 MI250’s (8 gpus) in a high end chassis for less than $100k. Today. Try that with H100’s.

dauertewigkeit 3y ago

99% of people publishing at top conferences are not particularly technically skilled and do not want to waste time adopting a new platform, because the competition is to publish papers and nobody cares if you do that on an AMD machine instead of an NVIDIA machine. The best funded labs have research developers whose only job is to optimize implementations. However these same labs will have the latest NVIDIA hardware.

dathinab 3y ago

AFIK in research nothing AMD had competitive GPGPUs AFIK just only relevant to a small number of very very large customers problems where mostly outside of research mainly there wasn't much insensitive (potential profit) for AMD to bring there GPGPU tooling to the consumer/small company marked and polish it for LLMs (to be clear I do not mean OpenCL, which was long term available but general subpar and badly supported) Nvideas mindshare was just too dominant and a few years ago it wasn't that uncommon for researchers to idk. create new building blocks or manual optimizations involving direct work with CUDA and similar But that's exactly what changed, by now, especially with LLMs, research does nearly always only involve usage of "high level abstractions" which are quite independent of the underlying gpu compute code (high-level might not be the best description as many of this GPU independent abstractions are still quite low level) . AMD has already shown that they can support that quite well and it seems to be mainly be question of polishing before it becomes more widely available. Another problem is that in the past AMD had decent GPU (compute/server) parts and GPU (gaming) parts but there GPU (gaming) parts where not that usable for compute. On the other hand Nvidea sold high end GPUs which can do both and can be "good enough" even for a lot of smaller companies. So a ton of researchers had easy access to that GPUs where access to specialized server compute cards is always complicated and often far more expensive (e.g. due to only being sold in bulk). This still somewhat holds up for the newest generation of AMD GPUs but much much less so. At the same time LLMs become so large that even using the highest-end Nvidea GPU became ... to slow. And selling a more high end customer GPU isn't really viable either IMHO. Additionally local inference seems to become much much more relevant and new AMD laptop CPU/GPU bundles and dedicated GPUs seem to be quite well equipped for that. Also the marked it growing a lot, so even if you just manage to get smaller % cut of the marked share it might now be profitable. I.e. they don't need to beat Nvidea in that marked anymore to make profit, grabbing a bit of marked share can now already be worthwhile. --- > port torch Idk. if it's already publically available/published but AMD has demoed proper well working torch support based on ROCm (instead of OpenCL).

latchkey 3y ago
https://pytorch.org/blog/experience-power-pytorch-2.0/ https://pytorch.org/blog/experience-power-pytorch-2.0/ https://www.mosaicml.com/blog/amd-mi250 https://www.mosaicml.com/blog/amd-mi250