3 ms·
> It's not rocket science to implement matrix multiplication in any GPU. You're right, it's harder. Saying this as someone who's done more work on the former t
by godelski 3y ago
> It's not rocket science to implement matrix multiplication in any GPU.
You're right, it's harder. Saying this as someone who's done more work on the former than the latter. (I have, with a team, built a rocket engine. And not your school or backyard project size, but nozzle bigger than your face kind. I've also written CUDA kernels and boy is there a big learning curve to the latter that you gotta fundamentally rethink how you view a problem. It's unquestionable why CUDA devs are paid so much. Really it's only questionable why they aren't paid more)
I know it is easy to think this problem is easy, it really looks that way. But there's an incredible amount of optimization that goes into all of this and that's what's really hard. You aren't going to get away with just N for loops for a tensor rank N. You got to chop the data up, be intelligent about it, manage memory, how you load memory, handle many data types, take into consideration different results for different FMA operations, and a whole lot more. There's a whole lot of non-obvious things that result in high optimization (maybe obvious __after__ the fact, but that's not truthfully "obvious"). The thing is, the space is so well researched and implemented that you can't get away with naive implementations, you have to be on the bleeding edge.
Then you have to do that and make it reasonably usable for the programmer too, abstracting away all of that. Cuda also has a huge head start and momentum is not a force to be reckoned with (pun intended).
Look at TensorRT[0]. The software isn't even complete and it still isn't going to cover all neural networks on all GPUs. I've had stuff work on a V100 and H100 but not an A100, then later get fixed. They even have the "Apple Advantage" in that they have control of the hardware. I'm not certain AMD will have the same advantage. We talk a lot about the difficulties of being first mover, but I think we can also recognize that momentum is an advantage of being first mover. And it isn't one to scoff at.
[0] https://github.com/NVIDIA/TensorRT https://github.com/NVIDIA/TensorRT
- doctorpangloss 3y agoThis comment is not going to age well. In the interest of curiosity, in this indictment of yours, is there any specific reason, about the AMD hardware itself, that leads you to believe that such implementations don't already exist, but are merely just not open sourced? I have a specific reason to believe this is all already optimized: poor adoption of Infinity Fabric by unsophisticated people, whereas NVLink is on every 3090 and DGX/HGX box. But that doesn't mean DC users haven't optimized.
- godelski 3y agoI really do want to see AMD kick ass. But 1) the user above me thinks ML is just matrix multiplies, which it is far from that. 2) It's more than hardware, and one team has (huge) momentum 3) You can have better tech and not win. It actually happens relatively frequently. There may be nothing better for the ML space than AMD being a meaningful competitor. It will give all of us better tools. It'll make AMD better and it'll even make Nvidia better, because (healthy) competition is good for everyone. But let's not be naive in thinking it is just hardware and that one example is enough. It's a good sign, but a Tesla driving on the highway in nominal conditions isn't even close to fully self driving. It's still a year away as it's been for a decade. So I do hope this ages poorly. But I'm not holding my breath because as AMD succeeding is one of the best things for ML, hype is the absolute worst and there's way too fucking much of it. And it it takes 5+ years to age, well I wouldn't say it aged poorly.
- weebull 3y agoIn an area so performance dominated people will spend the time porting to get the performance. I think this is especially true on the training side, because that's the area where companies have engineers waiting for hardware. Unfortunately AMD only claim parity with Nvidia for training right now, and uplifts on inference. That's the wrong way round to get people to spend time porting as a priority.
- heyitsguay 3y agoI don't think that comment was an indictment of anything. As someone also in the space, it reads to me as a thoughtful explanation of why catching up to the CUDA ecosystem is so hard, even when the goal is "just" matrix multiplication (it's actually not, but for the sake of a cleaner argument that's ok). It's entirely possible AMD or Intel or someone else has some super secret CUDA killer hidden away somewhere, but nobody has seen any evidence of it. It's a bit of an extraordinary claim, and the community would love to see some extraordinary evidence.
- xiphias2 3y agoNone of your comment writes about why CUDA is so special compared to OpenCL or Metal. Anybody who has worked with them knows that it's basically the same. Optimizing kernels for all architectures and all models is a hard problem as there are lot of cases to handle, but getting good utilization for a particular model on a particular architecture (like MI300 / Mixtral) is not that hard.
- nazka 3y ago> None of your comment writes about why CUDA is so special compared to OpenCL or Metal. OP and OP’s OP never frame it about that. But you can still get the answer you are looking for from their answers.
- qrios 3y agoYou’re right. The hard part is to manage processing of your data as close to the metal as possible to optimize memory, data flow, and computation. Coming from FPGA (both Verilog and VHDL), to me CUDA offers the best way to handle all three parts in comparison to let‘s say OpenCL, Vulkan, or Metal. For game developers this might be different, as here you always use high-level language and packages (in most cases). AMD need‘s to rethink there strategy regarding compute tool stack radically. If the HelloWorldMatrixMul is longer as ten lines and not compile on every new AMD powered notebook out of the box, they have no chance to beat NVIDIA. (I know, that the last point is not given for NVIDIA. But it would give AMD the doubleplus good, currently missing.)
- Arelius 3y ago> For game developers this might be different, as here you always use high-level language and packages (in most cases). ... I'm not sure what you mean that you always use high-level language and packages... But it doesn't sound right. The reason game developers use DX/VK/GL as opposed to cuda is entirely: A. You want access to non-nvidia customers B. You need access to more aspects of the hardware then is exposed to Cuda. Cuda provides a nice interface to Nvidia's compute. Almost no access to the entire rest of the fixed function pipeline.
- qrios 3y agoI was referencing DX/VK/GL and C++. What I can see on the lowest compute level, ML libs often are using pure CUDA in C or even PTX and are used especially if the code is generated by let's say via Python.