3 ms·
The merging of both compiler (cpu and software) is the main reason of the success of Nvidia. Make the chip simpler and more parallel (way more cores) Make the
by alfor 4y ago
The merging of both compiler (cpu and software) is the main reason of the success of Nvidia.
Make the chip simpler and more parallel (way more cores)
Make the compiler smarter to better use the capabilities of the hardware.
Watching Jim Keller interview made me realise how much transistors are wasted because the cpu is trying to predict what the code is going to do.
There is an insane amount of complexity in chips trying to make single threaded code fast when what need the most cpu time in our code is usually parallels problem.
Maybe Triton will break that paradigm and will allow diverse parallels hardware to emerge and be used effectively.
- bippingchip 4y agoBut don't underestimate how much 'middleware' there is in Cuda/CuDNN, NCCL etc. A lot of it is not in the compiler, but much more in hand crafted, carefully optimised libraries. As an example: there are so many low level ways to run a conv2D kernel on a SIMT machine, but CuDNN will pick the best option for you based on the card you have, and the tensor sizes you are using.
- bhawks 4y agoThe failure of the Itanium highlights the danger of tilting too much complexity into the software / compiler space. It was very difficult to make a compiler which could generate parallelizable machine code and that lead to very disappointing performance for most software.
- pjmlp 4y agoIt only failed, because of AMD being allowed to come up with AMD64. If x86 was Intel only, Itanium would have been eventually pushed everywhere no matter what.