6 ms·
CS234: Reinforcement Learning Winter 2025
- zerosizedweasle 11mo agoGiven Ilya's podcast this is an interesting title.
- kgarten 11mo agoWhich podcast?
- ontouchstart 11mo agohttps://www.dwarkesh.com/p/ilya-sutskever-2 https://www.dwarkesh.com/p/ilya-sutskever-2
- actionfromafar 11mo agoSo, basically AI Winter? :-)
- airspresso 11mo agoThat's how I read it XD "oh no, RL is dead too"
- TNWin 11mo agoI didn't get the reference. Please elaborate.
- apwell23 11mo agohe said RL sucks because it narrowly optimizes to solve a certain set of problems in certain sets of conditions. he compared it to students who win at math competition but cant do anything practical .
- egl2020 11mo agoKarpathy colorfully described RL as "sucking supervision bits through a straw".
- sillysaurusx 11mo agoIt’s been said that RL is the worst way to train a model, except for all the others. Many prominent scientists seem to doubt that this is how we’ll be training cutting edge models in a decade. I agree, and I encourage you to try to think of alternative paradigms as you go through this course. If that seems unlikely, remember that image generation didn’t take off till diffusion models, and GPTs didn’t take off till RLHF. If you’ve been around long enough it’ll seem obvious that this isn’t the final step. The challenge for you is, find the one that’s better.
- whatshisface 11mo agoRL is barely even a training method, its more of a dataset generation method.
- theOGognf 11mo agoI feel like both this comment and the parent comment highlight how RL has been going through a cycle of misunderstanding recently from another one of its popularity booms due to being used to train LLMs
- mistercheph 11mo agocare to correct the misunderstanding?
- mountainriver 11mo agoI mean DPO, PPO, and GRPO all use losses that are not what’s used with SFT for one. They also force exploration as a part of the algorithm. They can be used for synthetic data generation once the reward model is good enough.
- phyalow 11mo agoIts reductive, but also roughly correct.
- kgarten 11mo agoAre the videos available somewhere? spring course is on YouTube https://m.youtube.com/playlist?list=PLoROMvodv4rN4wG6Nk6sNpTEbuOSosZdX https://m.youtube.com/playlist?list=PLoROMvodv4rN4wG6Nk6sNpT...
- pedrolins 11mo agoI was excited to check out lecture videos thinking they were public, but quickly saw that they were closed. One of the things I miss most about the pandemic was how all of these institutions opened up for the world. Lately they have been closing down not only newer course offerings but also putting old videos private. Even MIT OCW falls apart once you get into some advanced graduate courses. I understand that universities should prioritize their alumni, but there’s literally no cost in making the underlying material (especially lectures!) available on the internet. It delivers immense value to the world.
- moosedev 11mo ago2024 lecture videos are on YouTube: https://youtube.com/playlist?list=PLoROMvodv4rN4wG6Nk6sNpTEbuOSosZdX https://youtube.com/playlist?list=PLoROMvodv4rN4wG6Nk6sNpTEb...
- storus 11mo agoThose don't have DPO/GRPO which arguably made some parts of RL obsolete.
- nafizh 11mo agocheck out cs 336 stanford, they cover DPO/GRPO and relevant parts needed to train LLMs.
- storus 11mo agoIt's also covered by CS329H.
- upbeat_general 11mo agoI can assure you that lacking knowledge in DPO (and especially GRPO it’s just stripped down PPO) is not a dealbreaker.
- rllearner 11mo agoOne of my favorite parts of the 2024 series on Youtube was when Prof B explained her excitement just before introducing UCB algorithms (Lecture 11): "So now we're going to see one of my favorite ideas in the course, which is optimism under uncertainty... I think it's a lovely principle because it shows why it's provably optimal to be optimistic about things. Which is kind of beautiful." Those moments are the best part of classroom education. When a super knowledgeable person spends a few weeks helping you get to the point where you can finally understand something cool. And you can sense their excitement to tell you about it. I still remember learning Gauss-Bonnet, Stokes Theorem, and the Central Limit Theorem. I think optimism under uncertainty falls in that group.
- storus 11mo agoRL is extremely brittle, it's often difficult to make it converge. Even Stanford folks admit that. Are there any solutions for this?
- mountainriver 11mo agoFlowRL is one, it’s learning the full distribution of rewards rather than just optimizing toward a single maximum
- storus 11mo agoThanks, that looks very promising!
- _giorgio_ 11mo agoKindly suggest some books about RL? I've already studied a lot of deep learning. Please confirm if these resoruces are good, or suggest yours: Sutton et al. - Reinforcement Learning Kevin Patrick Murphy - Reinforcement Learning, an overview https://arxiv.org/abs/2412.05265 https://arxiv.org/abs/2412.05265 Sebastian Raschka (upcoming book) ...
- i_don_t_know 11mo agoI believe Kochenderfer et.al.'s book "Algorithms for decision making" is also about reinforcement learning and related approaches. Free PDFs are available at https://algorithmsbook.com https://algorithmsbook.com
- Andrew-Tate 11mo ago[dead]
- mlmonkey 11mo agoAs a "tradional" ML guy who missed out on learning about RL in school, I'm confused about how to use RL in "traditional" problems. Take, for example, a typical binary classifier with a BCE loss. Suppose I wanted to shoehorn RL onto this: how would I do that? Or, for example, the House Value problem (given a set of features about a house for sale, predict its expected sale value). How would I slap RL onto that? I guess my confusion comes from how the losses are hooked up. Traditional losses (BCE, RMSE, etc.) I know about; but how do you bring RL loss into problems?
- robrenaud 11mo agoI just wouldn't. RL is nice in that it is handles messy cases where you don't have per example labels. How do you build a learned chess playing bot? Essentially the state of the art is to find a clever way of turning the problem of playing chess into a sequence of supervised learning problems.
- mlmonkey 11mo agoSo IIUC RL is applicable only when the outcome is not immediately available. Let's say I do have a problem in that setting; say the chess problem, where I have a chess board with the positions of chess pieces and some features like turn number, my color, time left on the clock, etc. are available. Would I train a DNN with these features? Are there some libraries where I can try out some toy problems? I guess coming from a classical ML background I am quite clueless about RL but want to learn more. I tried reading the Sutton and Barto book, but got lost in the terminology. I'm a more hands-on person.
- jebarker 11mo agoOpenAI has an excellent interactive course on Deep RL: https://spinningup.openai.com/en/latest/ https://spinningup.openai.com/en/latest/
- egl2020 11mo agoThe AlphaGo paper might be what you need. It requires some work to understand, but is clearly written. I read it when it came out and was confident enough to give a talk on it. (I don't have the slides any more; I did this when I was at a FAANG and left them behind.)