3 ms·

This comment is not going to age well. In the interest of curiosity, in this indictment of yours, is there any specific reason, about the AMD hardware itself, t

This comment is not going to age well. In the interest of curiosity, in this indictment of yours, is there any specific reason, about the AMD hardware itself, that leads you to believe that such implementations don't already exist, but are merely just not open sourced? I have a specific reason to believe this is all already optimized: poor adoption of Infinity Fabric by unsophisticated people, whereas NVLink is on every 3090 and DGX/HGX box. But that doesn't mean DC users haven't optimized.

godelski 3y ago
I really do want to see AMD kick ass. But 1) the user above me thinks ML is just matrix multiplies, which it is far from that. 2) It's more than hardware, and one team has (huge) momentum 3) You can have better tech and not win. It actually happens relatively frequently. There may be nothing better for the ML space than AMD being a meaningful competitor. It will give all of us better tools. It'll make AMD better and it'll even make Nvidia better, because (healthy) competition is good for everyone. But let's not be naive in thinking it is just hardware and that one example is enough. It's a good sign, but a Tesla driving on the highway in nominal conditions isn't even close to fully self driving. It's still a year away as it's been for a decade. So I do hope this ages poorly. But I'm not holding my breath because as AMD succeeding is one of the best things for ML, hype is the absolute worst and there's way too fucking much of it. And it it takes 5+ years to age, well I wouldn't say it aged poorly.
- weebull 3y ago
  In an area so performance dominated people will spend the time porting to get the performance. I think this is especially true on the training side, because that's the area where companies have engineers waiting for hardware. Unfortunately AMD only claim parity with Nvidia for training right now, and uplifts on inference. That's the wrong way round to get people to spend time porting as a priority.
heyitsguay 3y ago
I don't think that comment was an indictment of anything. As someone also in the space, it reads to me as a thoughtful explanation of why catching up to the CUDA ecosystem is so hard, even when the goal is "just" matrix multiplication (it's actually not, but for the sake of a cleaner argument that's ok). It's entirely possible AMD or Intel or someone else has some super secret CUDA killer hidden away somewhere, but nobody has seen any evidence of it. It's a bit of an extraordinary claim, and the community would love to see some extraordinary evidence.

godelski 3y ago

I really do want to see AMD kick ass. But 1) the user above me thinks ML is just matrix multiplies, which it is far from that. 2) It's more than hardware, and one team has (huge) momentum 3) You can have better tech and not win. It actually happens relatively frequently. There may be nothing better for the ML space than AMD being a meaningful competitor. It will give all of us better tools. It'll make AMD better and it'll even make Nvidia better, because (healthy) competition is good for everyone. But let's not be naive in thinking it is just hardware and that one example is enough. It's a good sign, but a Tesla driving on the highway in nominal conditions isn't even close to fully self driving. It's still a year away as it's been for a decade. So I do hope this ages poorly. But I'm not holding my breath because as AMD succeeding is one of the best things for ML, hype is the absolute worst and there's way too fucking much of it. And it it takes 5+ years to age, well I wouldn't say it aged poorly.

weebull 3y ago
In an area so performance dominated people will spend the time porting to get the performance. I think this is especially true on the training side, because that's the area where companies have engineers waiting for hardware. Unfortunately AMD only claim parity with Nvidia for training right now, and uplifts on inference. That's the wrong way round to get people to spend time porting as a priority.

heyitsguay 3y ago

I don't think that comment was an indictment of anything. As someone also in the space, it reads to me as a thoughtful explanation of why catching up to the CUDA ecosystem is so hard, even when the goal is "just" matrix multiplication (it's actually not, but for the sake of a cleaner argument that's ok). It's entirely possible AMD or Intel or someone else has some super secret CUDA killer hidden away somewhere, but nobody has seen any evidence of it. It's a bit of an extraordinary claim, and the community would love to see some extraordinary evidence.