Good point. I think it's a safe bet to focus on dense dot product in hardware for the foreseeable future. However, to their defense: 1. It's not clear that sup

Good point. I think it's a safe bet to focus on dense dot product in hardware for the foreseeable future. However, to their defense: 1. It's not clear that supporting sparse operands in hw would result in significant overhead. 2. DL models are still pretty sparse (I bet even those models with GELU still have lots of very small values that could be safely rounded to zero). 3. Sparsity might have some benefits (e.g. https://arxiv.org/abs/1903.11257 https://arxiv.org/abs/1903.11257).