In the current world, deep learning with homogeneous computation graphs, tuned with backprop, has won the Hardware Lottery [1]. This is unfortunate for research

In the current world, deep learning with homogeneous computation graphs, tuned with backprop, has won the Hardware Lottery [1]. This is unfortunate for research outside of that area, but just looking at the momentum of development it seems like a sure bet to keep investing in GPU-based training and inference for the next decade. There's just too much lock-in already to this paradigm. If a new algorithm appears from with a novel approach (analog compute, heterogeneous computation graphs from genetic algorithms, quantum, much more...), there will be a whole generation of R&D + tool + framework building, which gives the major players enough time to adapt. [1] https://arxiv.org/abs/2009.06489 https://arxiv.org/abs/2009.06489