Clearly, that would help. And clearly, there are opportunities for improvement -- as much as two orders of magnitude (!) in the case of capsule networks. That

Clearly, that would help. And clearly, there are opportunities for improvement -- as much as two orders of magnitude (!) in the case of capsule networks. That said, I think the authors are right about the challenge here: "...current frameworks excel at workloads where it makes sense to manually tune the small set of computations used by a particular model or family of models. Unfortunately, frameworks become poorly suited to research, because there is a performance cliff when experimenting with computations that haven’t previously been identified as important. While a few hours of search may be acceptable before production deployment, it is unrealistic to expect researchers to put up with such compilation times (recall this is just one kernel in what may be a large overall computation); and even if optimized kernels were routinely cached locally, it would be a major barrier to disseminating research if anyone who downloaded a model’s source code had to spend hours or days compiling it for their hardware before being able to experiment with it." EDIT: removed third paragraph.