It really is effectively an advertisement for CUDA 100%. At a previous employer, we had some budget left after buying a new NVIDIA GPU machine, but not enough

It really is effectively an advertisement for CUDA 100%. At a previous employer, we had some budget left after buying a new NVIDIA GPU machine, but not enough to buy another bunch of workstation/server-class cards. So we decided to buy an additional machine with two Radeon VII, which were quite affordable for the compute it provided. So, we wanted to try if that is a viable path for the future. Unfortunately, ROCm was death by a thousand paper cuts. You could barely train any model without ROCm crashing with some memory order, weird performance regressions, or garbage output. The only thing we ever really got working was transformer training, with various ROCm libraries and PyTorch pinned to specific revisions and a bunch of patches. Luckily I had the foresight to encode all the dependencies with Nix and put all the Nix files in git, so we could reproduce this exact working version. We attempted to update the stack at various occasions, but it would always lead to more breakage. Two colleagues meticulously reported bugs for issues that were encountered. But AMD did not really seem interested to try to fix these things (or were just severely understaffed). After this experience I vowed to never buy AMD again for ML for at least a decade.