3 ms·
Computation-wise, transformers are really just a bunch of matrix multiplications, nothing more to it. (Which is partially why they’re so efficient and scalable.
by jhrmnn 4y ago
Computation-wise, transformers are really just a bunch of matrix multiplications, nothing more to it. (Which is partially why they’re so efficient and scalable.) Also, Nvidia’s GPU architectures are moving in the TPU direction (https://www.nvidia.com/en-us/data-center/tensor-cores/ https://www.nvidia.com/en-us/data-center/tensor-cores/).
- shaklee3 4y agoNvidia had tensor cores before the tpu existed
- jhrmnn 4y agoRight, correct. I was already thinking ahead about the recent TensorFloat precision, which is further directly targeting matrix multiplication.
- dragonwriter 4y ago> Nvidia had tensor cores before the tpu existed TPUs were released in 2016, Tensor Cores in 2017. But Edge TPUs (that is, the ones that exist outside of Google's Cloud infrastructure) weren’t until 2018. So, Nvidia had their cores available for people to buy first, but not before TPUs existed.