The other thing to keep in mind is that Transformers may well be end up being supplanted by more efficient alternatives with different hardware requirements. F

The other thing to keep in mind is that Transformers may well be end up being supplanted by more efficient alternatives with different hardware requirements. For instance, right now there's a new crop of "linear RNNs" (RWKV, Mamba, retnet, etc.) claiming to be as good as Transformers for language modeling but with two advantages: their compute cost is O(n) instead of O(n²), and they don't need to keep past context in memory. I don't know if these linear RNNs will actually supplant Transformers, but I do think hardware requirements are likely to change over time.