3 ms·
Sure, but Cerebras isn't just multiplying two large matrices, they are multiplying two large very sparse matrices, relying on ReLU activation to maintain sparci
by ivalm 7y ago
Sure, but Cerebras isn't just multiplying two large matrices, they are multiplying two large very sparse matrices, relying on ReLU activation to maintain sparcity in all of the layers. We already have BERT/XLNet/other transformer models move away from ReLU to GELU which do not result in sparse matrices. "Traditional" activations (tanh, sigmoid, softmax) are not sparse either.
- p1esk 7y agoGood point. I think it's a safe bet to focus on dense dot product in hardware for the foreseeable future. However, to their defense: 1. It's not clear that supporting sparse operands in hw would result in significant overhead. 2. DL models are still pretty sparse (I bet even those models with GELU still have lots of very small values that could be safely rounded to zero). 3. Sparsity might have some benefits (e.g. https://arxiv.org/abs/1903.11257 https://arxiv.org/abs/1903.11257).