I could easily be wrong, but IIUC the software/hardware stack was, as of 2014 or so, fusing down to a specific set of circuits for broadcast activation function

I could easily be wrong, but IIUC the software/hardware stack was, as of 2014 or so, fusing down to a specific set of circuits for broadcast activation functions under the broader banner of “Tensor Cores” (along with a bunch of devilish FMA hot-pathing under the sheets). This is a fairly random press piece off Google but there are a ton of them. Hard to tell whether it’s the hardware, the blob, or the process node unless you work there. https://developer.nvidia.com/blog/accelerating-relu-and-gelu-activation-functions-and-batched-sparse-gemm-in-cusparselt-v0-2-0/ https://developer.nvidia.com/blog/accelerating-relu-and-gelu...