4 ms·
There was another matmul-free language model paper released a year ago FYI: https://arxiv.org/abs/2305.17190 https://arxiv.org/abs/2305.17190
by jph00 2y ago
There was another matmul-free language model paper released a year ago FYI:
https://arxiv.org/abs/2305.17190 https://arxiv.org/abs/2305.17190
- versteegen 2y ago> cheap piecewise affine approximation that is achieved by adding the bit representation of the floating point numbers together as integers ... with little to no performance impact > we show that we can eliminate all multiplications in the entire training process, including operations in the forward pass, backward pass and optimizer update, demonstrating the first successful training of modern neural network architectures in a fully multiplication-free fashion. This reaches demoscene levels of crazy/impressive!
- cpldcpu 2y ago>This reaches demoscene levels of crazy/impressive! The exp/log trick to multiply with addition does indeed look very familiar. I know that a number of demos used it in the 90ies to simplify matrix multiplications for 3d graphics. Bill Dally, the Nvidia Chief Scientist also seems to be a big fan of log8 representations: https://youtu.be/gofI47kfD28?t=1953 https://youtu.be/gofI47kfD28?t=1953 It seems it did not make it into the hardware though, yet...