26 ms·
While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years no
by benreesman 3y ago
While a given generation of accelerator can only target model architectures that are comparatively proven out, and there’s a lag time, it’s measured in years not decades.
I remember when NVIDIA didn’t have hardware for ReLU.
The fact of the matter on Moore’s law is that we’ve got transistors, but not TDP to burn and have for years. These stupid big L3 caches are just: “fuck it, I’ve got die to burn”.
This is an old story, things migrate in and out of the “CPU”, but the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future.
- microtonal 3y agoI remember when NVIDIA didn’t have hardware for ReLU. Could you elaborate? ReLU is max(x,0) and CUDA had fmaxf since CUDA 1.0 (2007).
- vgatherps 3y agoNot sure if or what the chance was, but fmax(x, 0) only requires checking the sign bit instead of doing a full floating point comparison (putting aside nan handling). A hypothetical relu instruction could probably get away with much less power and die soace?
- benreesman 3y agoI could easily be wrong, but IIUC the software/hardware stack was, as of 2014 or so, fusing down to a specific set of circuits for broadcast activation functions under the broader banner of “Tensor Cores” (along with a bunch of devilish FMA hot-pathing under the sheets). This is a fairly random press piece off Google but there are a ton of them. Hard to tell whether it’s the hardware, the blob, or the process node unless you work there. https://developer.nvidia.com/blog/accelerating-relu-and-gelu-activation-functions-and-batched-sparse-gemm-in-cusparselt-v0-2-0/ https://developer.nvidia.com/blog/accelerating-relu-and-gelu...
- nologic01 3y ago> the current outlook is that we’ll be targeting specialized hardware more rather than less for the foreseeable future. I think there are some important question marks still unresolved that bear on how things will play out. E.g. how the training versus inference balance will land in terms of usage and economics. Inference is inherently more "mass market". You need it locally without lags from moving data around. But inference is just numerical linear algebra. Ultimately augmenting the CPU to provide inference natively might be the optimal arrangement.
- JoeAltmaier 3y agoThere have been some really strange instruction sets conceived. As a student at Stanford we had a time-share system that was home-grown (as I remember). It had opcodes to reverse bits in a bitstring! And odder things. Somebody needed that for some research project I guess. And then repurposed the damn thing for timeshare. It was pretty sad timeshare as I recall. The only machine(s?) available to a population of what? 20,000? And about 40 terminals total. You had to sign up for 15-minute measured timeslots. I came from a state school that had over 1000 terminals on a dozen machines, all unlimited time to students. It was a big shock to find the star of Silicon Valley had such crappy student services.
- anthomtb 3y ago> It had opcodes to reverse bits in a bitstring Bit reversal is used in Fast Fourier Transforms. Its not entirely surprising to me that you'd have specialized hardware for that operation. Ref: https://en.wikipedia.org/wiki/Bit-reversal_permutation#Applications https://en.wikipedia.org/wiki/Bit-reversal_permutation#Appli...
- JoeAltmaier 3y agoAlso to count ones in a bitstring. And so on.