I agree they're probably here to stay. It's interesting to think about how the hardware-software feedback loop affects architecture choice and research directio

I agree they're probably here to stay. It's interesting to think about how the hardware-software feedback loop affects architecture choice and research directions. I'm almost certain there is an even more impressive architecture out there, waiting to be discovered. However, until we maximize the performance we get out of Transformers, there is a strong incentive to optimize both software and hardware specialized to this architecture. One downside is that as hardware/software becomes more specialized to one architecture, it becomes more difficult to explore very different architectures.