5 ms·
The problem is that performance achievements on AMD consumer-grade GPUs (RX7900XTX) are not representative/transferrable to the Datacenter grade GPUs (MI300X).
by pavelstoev 2y ago
The problem is that performance achievements on AMD consumer-grade GPUs (RX7900XTX) are not representative/transferrable to the Datacenter grade GPUs (MI300X). Consumer GPUs are based on RDNA architecture, while datacenter GPUs are based on the CDNA architecture, and only sometime in ~2026 AMD is expected to release unifying UDNA architecture [1]. At CentML we are currently working on integrating AMD CDNA and HIP support into our Hidet deep learning compiler [2], which will also power inference workloads for all Nvidia GPUs, AMD GPUs, Google TPU and AWS Inf2 chips on our platform [3]
[1] https://www.jonpeddie.com/news/amd-to-integrate-cdna-and-rdna-architectures-to-compete-in-ai/#:~:text=In%202019%2C%20AMD%20transitioned%20away,Well%2C%20there's%20plenty%20of%20time https://www.jonpeddie.com/news/amd-to-integrate-cdna-and-rdn....
[2] https://centml.ai/hidet/ https://centml.ai/hidet/
[3] https://centml.ai/platform/ https://centml.ai/platform/
- llm_trw 2y agoThe problem is that the specs of AMD consumer-grade GPUs do not translate to computer performance when you try and chain more than one together. I have 7 NVidia 4090s under my desk happily chugging along on week long training runs. I once managed to get a Radeon VII to run for six hours without shitting itself.
- tspng 2y agoWow, are these 7 RTX 4090s in a single setup? Care to share more how you build it (case, cooling, power, ..)?
- adakbar 2y agoI'd like to know too
- ghxst 2y agoYou might find the journey of Tinycorp's Tinybox interesting, it's a machine with 6 to 8 4090 GPUs and you should be able to track down a lot of their hardware choices including pictures on their Twitter and other info on George his livestreams.
- llm_trw 2y agoBasically this but with an extra card on the x8 slot for connecting my monitors: https://www.youtube.com/watch?v=C548PLVwjHA https://www.youtube.com/watch?v=C548PLVwjHA There's a bunch of similar setups and there are a couple of dozen people that have done something similar on /r/localllama.
- osmarks 2y agoMost of these are just an EPYC server platform, some cursed risers and multiple PSUs (though cryptominer server PSU adapters are probably better). See https://nonint.com/2022/05/30/my-deep-learning-rig/ https://nonint.com/2022/05/30/my-deep-learning-rig/ and https://www.mov-axbx.com/wopr/wopr_concept.html https://www.mov-axbx.com/wopr/wopr_concept.html.
- icelancer 2y agoEPYC + Supermicro + C-Payne retimers/cabling. 208-240V power typically mandatory for the most affordable power supplies (chain a server/crypto PSU for the GPUs from ParallelMiner to an ATX PSU for general use). Beyond that, not much else.
- mpreda 2y ago> I have 7 NVidia 4090s under my desk I have 6 Radeon Pro VII under my desk (in a single system BTW), and they run hard for weeks until I choose to reboot e.g. for Linux kernel updates. I bought them "new old stock" for $300 apiece. So that's $1800 for all six.
- highwaylights 2y agoHow does the compute performance compare to 4090’s for these workloads? (I release it will be significantly lower, just try to get as much of a comparison as is possible).
- cainxinth 2y agoThe 4090 offers 82.58 teraflops of single-precision performance compared to the Radeon Pro VII's 13.06 teraflops.
- adrian_b 2y agoOn the other hand, for double precision a Radeon Pro VII is many times faster than a RTX 4090 (due to 1:2 vs. 1:64 FP64:FP32 ratio). Moreover, for workloads limited by the memory bandwidth, a Radeon Pro VII and a RTX 4090 will have about the same speed, regardless what kind of computations are performed. It is said that speed limitation by memory bandwidth happens frequently for ML/AI inferencing.
- ThinkBeat 2y agoWhat motherboard are you using to have space and ports for 7 of them?
- slavik81 2y agoThe ASRock Rack ROMED8-2T has seven PCIe x16 slots. They're too close together to directly put seven 4090s on the board, but you'd just need some riser cables to mount the cards on a frame.
- majke 2y agoWhat software stack you use for training?
- archi42 2y agoHow do you manage heat? I'm looking at a hashcat build with a few 5090, and water cooling seems to be the sensible solution if we scale beyond two cards.
- rcdwealth 2y ago[dead]
- zozbot234 2y agoIt looks like AMD's CDNA gpu's are supported by Mesa, which ought to suffice for Vulkan Compute and SYCL support. So there should be ways to run ML workloads on the hardware without going through HIP/ROCm.