4 ms·
Q-Transformer: Scalable Reinforcement Learning via Autoregressive Q-Functions
- tysam_and 3y agoThis is what RWKV (https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM) was made for, and what it will be good at. Wow. Pretty darn cool! <3 :'))))
- evolvingstuff 3y agoWhy would RWKV have a particular advantage in this context? (I may be missing some key intuitions)
- tysam_and 3y agoRNN inference on a smaller edge controller (all history is cached in a single state point for each layer, so much less memory and computation requirements IIRC) :') Very mobile-device and battery-powered systems friendly. :')))) ;'DDDD
- algo_trader 3y agoI havent yet fully grokked RWKV.. Just how much compute/memory are we saving here? My understanding is that a 1BN transformer is about 2BN flops/inference, so about 1TFLOP for a 500 sequence of inferences (and also about several GB of memory) What would be the equivalent RWKV (let ignore the inevitable loss penalty which could be significant..)
- tysam_and 3y agoIt's an RNN, there is no N^2 component over time. It only requires the previous state. (there's a discord, you should join it with further questions! I unfortunately am not as informed as I should be on this one, other than the fact that it is _very_ mobile friendly). The performance diff is slight but not too bad really, all things considered. And I think it comes out on top for raw efficiency per parameter/flop, IIRC. An interesting concept, for sure! :'DDDD :'))))
- Buttons840 3y agoSigh. Do discussions about RWKV always end with suggestions that I join the Discord? If I do join the Discord, will I soon begin suggesting that others join the Discord as well? What I mean is, I've seen this come up a few times on HN and discussions usually end prematurely with suggestions to join the Discord. [0] If this technique is good, I'll wait until I can learn about it without joining the Discord. [0]: https://news.ycombinator.com/item?id=35508692 https://news.ycombinator.com/item?id=35508692
- greesil 3y agoThis leads to one shot learning for robots?
- radarsat1 3y agoCool to see an approach to using transformers that sticks closer to traditional RL than the decision transformer. The action dimension trick here is clever. Curious to see where this can be taken, game playing, multiagent, etc.
- ashupadhi01 3y agoI want to know how you build up intuition and knowledge in the space of RL.
- The_Amp_Walrus 3y agoI enjoyed this course https://youtu.be/2pWv7GOvuf0?si=DKkhPXQmVjA3ySIn https://youtu.be/2pWv7GOvuf0?si=DKkhPXQmVjA3ySIn
- gtoubassi 3y ago+1 you beat me to the punch! I think its helpful to start with simple RL and ignore the "deep" part to get the basics. The first several lectures in this series do that well. It helped me build a simple "cat and mouse" RL simulation https://github.com/gtoubassi/SimpleReinforcementLearning https://github.com/gtoubassi/SimpleReinforcementLearning and ultimately a reproduction of the DQN atari game playing agent: https://github.com/gtoubassi/dqn-atari https://github.com/gtoubassi/dqn-atari.
- PartiallyTyped 3y agoWhenever somebody recommends a course, you can be pretty certain that it's that one :)
- radarsat1 3y agoHonestly the best way is starting with implementing a Q table for some small grid-world problem. You get a lot of knowledge from doing that. Then a bit more work on understanding various approaches, e.g policy learning, world models. Then, reading text books, blogs tutorials, etc. But "getting" the idea of Q learning for a small state space is fundamental and surprisingly approachable.