2 ms·
I remember when NVIDIA didn’t have hardware for ReLU. Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).
by microtonal 3y ago
I remember when NVIDIA didn’t have hardware for ReLU.
Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).
- vgatherps 3y agoNot sure if or what the chance was, but fmax(x, 0) only requires checking the sign bit instead of doing a full floating point comparison (putting aside nan handling). A hypothetical relu instruction could probably get away with much less power and die soace?
- benreesman 3y agoI could easily be wrong, but IIUC the software/hardware stack was, as of 2014 or so, fusing down to a specific set of circuits for broadcast activation functions under the broader banner of “Tensor Cores” (along with a bunch of devilish FMA hot-pathing under the sheets). This is a fairly random press piece off Google but there are a ton of them. Hard to tell whether it’s the hardware, the blob, or the process node unless you work there. https://developer.nvidia.com/blog/accelerating-relu-and-gelu-activation-functions-and-batched-sparse-gemm-in-cusparselt-v0-2-0/ https://developer.nvidia.com/blog/accelerating-relu-and-gelu...