2 ms·
It seems to me transformers are here to stay for a while (considering they have been invented in 2017 and there really havent been any fundamental adjustments t
by lysecret 4y ago
It seems to me transformers are here to stay for a while (considering they have been invented in 2017 and there really havent been any fundamental adjustments to the architecture). It is quite exciting to me to think about all the possible optimizations and improvements that can happen if the underlying aritechture stays. I'm thinking about GPU optimizations like this maybe also integration with databases, libraries to simplify interacting and building on top of transformers reducing their sice, fast inference easy domain adjustment etc. etc. Feels like we are at the beginning of a transformer ecosystem.
- axg11 4y agoI agree they're probably here to stay. It's interesting to think about how the hardware-software feedback loop affects architecture choice and research directions. I'm almost certain there is an even more impressive architecture out there, waiting to be discovered. However, until we maximize the performance we get out of Transformers, there is a strong incentive to optimize both software and hardware specialized to this architecture. One downside is that as hardware/software becomes more specialized to one architecture, it becomes more difficult to explore very different architectures.