This is something we run into even when pushing transformer-style models (which can be viewed as a partial step towards some of these capsule ideas) into new ar

This is something we run into even when pushing transformer-style models (which can be viewed as a partial step towards some of these capsule ideas) into new areas like computer vision. There are places where you can beat convolutions in terms of parameters+FLOPs but wall-clock time is poor due to the same issues presented in this paper - there aren't good optimized self-attention kernels so you're forced to implement things in terms of existing ops - depthwise convs, etc. It's definitely a real pain when trying to bring in new primitives to the deep learning world, only compounded by the increasing diversity of accelerators in use.