4 ms·
No chapter on RNNs, but one on transformers is interesting, having last read Deep learning by ian goodfellow in 2016
by water-your-self 3y ago
No chapter on RNNs, but one on transformers is interesting, having last read Deep learning by ian goodfellow in 2016
- nothrowaways 3y agoYeah, content looks interesting.
- PeterisP 3y agoRNNs have "lost the hardware lottery" by being structurally not that efficient to train on the cost-effective hardware that's available. So they're not really used for much right now - though IMHO they are conceptually sufficiently interesting enough to cover in such a course.
- nextos 3y agoThat is not completely true. There are RNNs with transformer/LLM-like performance. See e.g. https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM. They are less popular, and less explored. But an interesting route ahead.
- mindcrime 3y ago> RNNs have "lost the hardware lottery" by being structurally not that efficient to train on the cost-effective hardware that's available. Which suggests two obvious paths forward: 1. Don't bother learning / using RNN's 2. Co-develop new hardware / new RNN architectures that work together to provide great performance per unit of price. Now of course nobody is saying (well, I am not saying) that (2) would be easy... or even necessarily possible. But somebody should at least be intrigued by the idea. And in the world we live in today where FPGA's and other devices make it easier than ever to experiment with custom hardware architectures... it might be worth taking a stab at it.