10 ms·
RLHF is just barely RL
- normie3000 2y ago> In machine learning, reinforcement learning from human feedback (RLHF) is a technique to align an intelligent agent to human preferences. https://en.m.wikipedia.org/wiki/Reinforcement_learning_from_human_feedback https://en.m.wikipedia.org/wiki/Reinforcement_learning_from_...
- moffkalast 2y agoNote that human preference isn't universal. RLHF is mostly frowned upon by the open source LLM community since it typically involves aligning the model to the preference of corporate manager humans, i.e. tuning for censorship and political correctness to make the model as bland as possible so the parent company doesn't get sued. For actual reinforcement learning with a feedback loop that aims to increase overall performance the current techniques are SPPO and Meta's version of it [0] that slightly outperforms it. It involves using a larger LLM as a judge though, so the accuracy of the results is somewhat dubious. [0] https://arxiv.org/pdf/2407.19594 https://arxiv.org/pdf/2407.19594
- codeflo 2y agoReminder that AlphaGo and its successors have not solved Go and that reinforcement learning still sucks when encountering out-of-distribution strategies: https://arstechnica.com/information-technology/2023/02/man-beats-machine-at-go-in-human-victory-over-ai/ https://arstechnica.com/information-technology/2023/02/man-b...
- thanhdotr 2y agowell, as yann lecun said : "Adversarial training, RLHF, and input-space contrastive methods have limited performance. Why? Because input spaces are BIG. There are just too many ways to be wrong" [1] A way to solve the problem is projecting onto latent space and then try and discriminate/predict the best action down there. There's much less feature correlation down in latent space than in your observation space. [2] [1]:https://x.com/ylecun/status/1803696298068971992 https://x.com/ylecun/status/1803696298068971992 [2]: https://openreview.net/pdf?id=BZ5a1r-kVsf https://openreview.net/pdf?id=BZ5a1r-kVsf
- viraptor 2y agoI wouldn't say it sucks. You just need to keep training it for as long as needed. You can do adversarial techniques to generate new paths. You can also use the winning human strategies to further improve. Hopefully we'll find better approaches, but this is extremely successful and far from sucking. Sure, Go is not solved yet. But RL is just fine continuing to that asymptote for as long as we want. The funny part is that this applies to people too. Masters don't like to play low ranked people because they're unpredictable and the ELO loss for them is not worth the risk. (Which does rise questions about how we really rank people)
- shakna 2y ago> I wouldn't say it sucks. You just need to keep training it for as long as needed. As that timeline can approach infinity, just adding extra training may not actually be a sufficient compromise.
- rocqua 2y agoAlphago didn't have human feedback, but it did learn from humans before surpassing them. Specifically, it had a network to 'suggest good moves' that was trained on predicting moves from pro level human games. The entire point of alpha zero was to eliminate this human influence, and go with pure reinforcement learning (i.e. zero human influence).
- cherryteastain 2y agoA game like Go has a clearly defined objective (win the game or not). A network like you described can therefore be trained to give a score to each move. Point here is that assessing whether a given sentence sounds good to humans or not does not have a clearly defined objective, the only way we came up with so far is to ask real humans.
- esjeon 2y agoAlphaGo is an optimization over a closed problem. Theoretically, computers could have always beat human in such problems. It's just that, without proper optimization, humans will die before the computer finishes its computation. Here, AlphaGo cuts down the computation time by smartly choosing the branches with the highest likelihood. Unlike the above, open problems can't be solve by computing (in combinatorial senses). Even humans can only try, and LLMs do spew out something that would most likely work, not something inherently correct.
- __loam 2y agoBeen shouting this for over a year now. We're training AI to be convincing, not to be actually helpful. We're sampling the wrong distributions.
- danielbln 2y agoI find them very helpful, personally.
- sussmannbaka 2y agoUnderstandable, they have been trained to convince you of their helpfulness.
- danielbln 2y agoIf they convinced me of their helpfulness, and their output is actually helpful in solving my problems.. well, if it walks like a duck and quacks like a duck, and all that.
- tpoacher 2y agoif it walks like a duck and it quacks like a duck, then it lacks strong typing.
- Nullabillity 2y ago"Appears helpful" and "is helpful" are two very different properties, as it turns out.
- snapcaster 2y agoSometimes, but that's an edge case that doesn't seem to impact the productivity boosts from LLMs
- __loam 2y ago
- timthelion 2y agoKarpathy writes that there is no cheeply computed objective check for "Or re-writing some Java code to Python? " Among other things. But it seems to me that Reinforced Learning should be possible for code translation using automated integration testing. Run it, see if it does,the same thing!
- jeroenvlek 2y agoMy takeaway is that it's difficult to make a "generic enough" evaluation that encompasses all things we use an LLM for, e.g. code, summaries, jokes. Something with free lunches.
- IanCal 2y agoEven there how you score it is hard. "Is it the same for this s y of inputs?" May be fine for a subset of things, but then that's a binary thing. If it's slightly wrong do you score by number of outputs that match? A purely binary thing gives little useful help for nudging a model in the right direction. How do you compare two that both work, which is more "idiomatic"?
- theqwxas 2y agoI agree that it's a very difficult problem. I'd like to mention AlphaDev [0], an RL algorithm that builds other algorithms, there they combined the measure of correctness and a measure of algorithm speed (latency) to get the reward. But the algorithms they built were super small (e.g., sorting just three numbers), therefore they could measure correctness using all input combinations. It is still unclear how to scale this to larger problems. [0] https://deepmind.google/discover/blog/alphadev-discovers-faster-sorting-algorithms/ https://deepmind.google/discover/blog/alphadev-discovers-fas...
- exe34 2y agofor "does it run" cases, you can ask the model to try again, give it higher temperature, show it the traceback errors, (and maybe intermediate variables?) or even ask it to break up the problem into smaller pieces and then try to translate that. for testing, if you use something like quickcheck, you might find bugs that you wouldn't otherwise find. when it comes to idiomatic, I'm not sure - but if we're at the point that gpt is writing code that works, do we really care? as long as this code is split into many small pieces, we can just replace the piece instead of trying to understand/fix it if we can't read it. in fact, maybe there's a better language that is human readable but better for transformers to write and maintain.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- leobg 2y agoA cheap DIY way of achieving the same thing as RLHF is to fine tune the model to append a score to its output every time. Remember: The reason we need RLHF at all is that we cannot write a loss function for what makes a good answer. There are just many ways a good answer could look like, which cannot be calculated on the basis of next-token-probability. So you start by having your vanilla model generate n completions for your prompt. You the. manually score them. And then those prompt => (completion,score) pairs become your training set. Once the model is trained, you may find that you can cheat: Because if you include the desired score in your prompt, the model will now strive to produce an answer that is consistent with that score.
- viraptor 2y agoThat works in the same way as actor-critic pair, right? Just all wrapped in the same network/output?
- levzzz 2y ago[dead]
- bick_nyers 2y agoI had an idea similar to this for a model that allows you to parameterize a performance vs. accuracy ratio, essentially an imbalanced MoE-like approach where instead of the "quality score" in your example, you assign a score based on how much computation it used to achieve that answer, then you can dynamically request different code paths be taken at inference time.
- lossolo 2y agoNot the same, it will get you worse output and is harder to do right in practice.
- visarga 2y ago> if you include the desired score in your prompt, the model will now strive to produce an answer that is consistent with that score But you need a model to generate score from answer, and then fine-tune another model to generate answer conditioned on score. The first time the score is at the end and the second time at the beginning. It's how DecisionTransformer works too, it constructs a sequence of (reward, state, action) where reward conditions on the next action. https://arxiv.org/pdf/2106.01345 https://arxiv.org/pdf/2106.01345 By the same logic you could generate tags, including style, author, venue and date. Some will be extracted from the source document, the others produced with classifiers. Then you can flip the order and finetune a model that takes the tags before the answer. Then you got a LLM you can condition on author and style.
- jrflowers 2y agoI enjoyed this Karpathy post about how there is absolutely no extant solution to training language models to reliably solve open ended problems. I preferred Zitron’s point* that we would need to invent several branches of science to solve this problem, but it’s good to see the point made tweet-sized. *https://www.wheresyoured.at/to-serve-altman/ https://www.wheresyoured.at/to-serve-altman/
- moffkalast 2y agoThat's a great writeup, and great references too. > OpenAI needs at least $5 billion in new capital a year to survive. This would require it to raise more money than has ever been raised by any startup in history They were probably toast before, but after Zuck decided to take it personally and made free alternatives for most use cases they definitely are, since if they had any notable revenue from selling API access it will just keep dropping.
- HarHarVeryFunny 2y agoI really would not be surprised to see OpenAI collapse - founders leaving like rats from a sinking ship is not a good sign, and not indicative of any AGI breakthrough on the horizon. Would not be surprised to see Brockman's "sabbatical" become permanent. I have to wonder how long Anthropic can remain independent too, although at least they don't have toxic workplace and founder exodus issues to deal with. Maybe they get acquired by Amazon?
- klibertp 2y agoI read the article you linked. I feel like I wasted my time. The article has a single point it repeats over and over again: OpenAI (and "generative AI as a whole"/"transformer-based models") are too expensive to run, and it's "close to impossible" for them to either limit costs or increase revenue. This is because "only 5% of businesses report using the technology in production", and that the technology had no impact on "productivity growth". It's also because "there's no intelligence in it", and the "models can't reason". Oh, also, ChatGPT is "hard to explain to a layman". All that is liberally sprinkled with "I don't know, but"s and absolutely devoid of any historical context other than in financial terms. No technical details. Just some guesses and an ironclad belief that it's impossible to improve GPTs without accessing more data than there is in existence. Agree or disagree; the article is not worth wading through so many words: others made arguments on both sides much better and, crucially, shorter.
- gizmo 2y agoThis is why AI coding assistance will leap ahead in the coming years. Chat AI has no clear reward function (basically impossible to judge the quality of responses to open-ended questions like historical causes for a war). Coding AI can write tests, write code, compile, examine failed test cases, search for different coding solutions that satisfy more test cases or rewrite the tests, all in an unsupervised loop. And then whole process can turn into training data for future AI coding models. I expect language models to also get crazy good at mathematical theorem proving. The search space is huge but theorem verification software will provide 100% accurate feedback that makes real reinforcement learning possible. It's the combination of vibes (how to approach the proof) and formal verification that works. Formal verification of program correctness never got traction because it's so tedious and most of the time approximately correct is good enough. But with LLMs in the mix the equation changes. Having LLMs generate annotations that an engine can use to prove correctness might be the missing puzzle piece.
- discreteevent 2y agoDoes programming have a clear reward function? A vague description from a business person is not it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program?
- rossamurphy 2y ago+1
- eru 2y ago> Does programming have a clear reward function? A vague description from a business person isn't it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program? Well, to give an example: the complexity class NP is all about problems that have quick and simple verification, but finding solutions for many problems is still famously hard. So there are at least some domains where this model would be a step forward.
- 2y ago
- mjburgess 2y agoIt always annoys and amazes me that people in this field have no basic understanding that closed-world finite-information abstract games are a unique and trivial problem. So much of the so-called "world model" ideological mumbojumbo comes from these setups. Sampling board state from an abstract board space isn't a statistical inference problem. There's no missing information. The whole edifice of science is a set of experimental and inferential practices to overcome the massive information gap between the state of a measuring device and the state of what, we believe, it measures. In the case of natural language the gap between a sequence of symbols, "the war in ukraine" and those aspects of the world these symbols refer to is enormous. The idea that there is even a RL-style "reward" function to describe this gap is pseudoscience. As is the false equivocation between sampling of abstracta such as games, and measuring the world.
- harshitaneja 2y agoForgive my naiveté here but even though solutions to those finite-information abstract games are trivial but not necessarily tractable(for a loser definition of tractable here) and we still need to build heuristics for the subclass of such problems where we need solutions in a given finite time frame. Those heuristics might not be easy to deduce and hence such models help in ascertaining those.
- mjburgess 2y agoYes, and this is how computer "scientists" think of problems -- but this isnt science, it's mathematics. If you have a process, eg., points = sample(circle) which fully describes its target as n->inf (ie., points = circle as n->inf) you arent engaged in statistical infernece. You might be using some of the same formula, but the whole system of science and statistics has been created for a radically different problem with radically different semantics to everything you're doing. eg., the height of mercury in a thermometer never becomes the liquid being measured.. it might seems insane/weird/obvious to mention this... but we literally have berkelian-style neoidealists in AI research who don't realise this... Who think that because you can find representations of abstracta in other spaces they can be projected in.. that this therefore tells you anything at all about inference problems. As if it was the neural network algorithm itself (a series of multiplications and additions) that "revealed the truth" in all data given to it. This, of course, is pseudoscience. It only applies on mathematical problems, for obvious reasons. If you use a function approximation alg to approximate a function, do not be suprised you can succeed. The issue is that the relationship between, say, the state of a theremometer and the state of the temperature of it's target system is not an abstract function which lives in the space of temperature readings. More precisely, in the space of temperature readings the actual causal relationship between the height of the mecurary and the temperature of the target shows up as an infinite number of temperature distributions (with any given trained NN learning only one of these). None of which are a law of nature -- laws of nature are not given by distributions in measuring devices.
- toxik 2y agoAnother little tidbit about RLHF and InstructGPT is that the training scheme is by far dominated by supervised learning. There is a bit of RL sprinkled on top, but the term is scaled down by a lot and 8x more compute time is spent on the supervised loss terms.
- daly 2y agoI think that the field of proofs, such as LEAN, which have states (the current subgoal), actions (the applicable theorems, especially effective in LEAN due to strong Typing of arguments), a progress measure (simplified subgoals), a final goal state (the proof completes), and a hierarchy in the theorems so there is a "path metric" from simple theorems to complex theorems. If Karpathy were to focus on automating LEAN proofs it could change mathematics forever.
- jomohke 2y agoDeepmind's recent model is trained with Lean. It scored a silver olympiad medal (and only one point away from gold). > AlphaProof is a system that trains itself to prove mathematical statements in the formal language Lean. It couples a pre-trained language model with the AlphaZero reinforcement learning algorithm, which previously taught itself how to master the games of chess, shogi and Go https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/ https://deepmind.google/discover/blog/ai-solves-imo-problems...
- cesaref 2y agoThe final conclusion though stands without any justification - that LLM + RL will somehow out-perform people at open-domain problem solving seems quite a jump to me.
- dosinga 2y agoTo be fair, it says "has a real shot at" and AlphaGo level. AlphaGo clearly beat humans on Go, so thinking that if you could replicate that, it would have a shot doesn't seem crazy to me
- SiempreViernes 2y agoThat only makes sense if you think Go is as expressive as written language. And here I mean that it the act of making a single (plausible) move that must match the expressiveness of language, because otherwise you're not in the domain of Go but the far less interesting "I have a 19x19 pixel grid and two colours".
- HarHarVeryFunny 2y agoAlphaGo has got nothing to do with LLMs though. It's a combination of RL + MCTS. I'm not sure where you are seeing any relevance! DeepMind also used RL for playing video games - so what?!
- esjeon 2y agoI think the point is that it's practically impossible to correctly perform RLHF in open domains, so comparisons simply can't happen.
- bjornsing 2y agoPerhaps not entirely open domain, but I have high hopes for “real RL” in coding, where you can get a reward signal from compile/runtime errors and tests.
- falcor84 2y agoInteresting, has anyone been doing this? I.e. training/fine-tuning an LLM against an actual coding environment, as opposed to just tacking that later on as a separate "agentic" contruct?
- bjornsing 2y agoI suspect that the big vendors are already doing it, but I haven’t seen a paper on it.
- pyrale 2y agoIt's a bit disingenuous to pick go as a case to make the point against RLHF. Sure, a board game with an objective winning function at which computers are already better than humans won't get much from RLHF. That doesn't look like a big surprise. On the other hand, a LLM trained on lots of not-so-much curated data will naturally pick up mistakes from that dataset. It is not really feasible or beneficial to modify the dataset exhaustively, so you reinforce the behaviour that is expected at the end. An example would be training an AI in a specific field of work: it could repeat advices from amateurs on forums, when less-known professional techniques would be more advisable. Think about it like kids naturally learning swear words at school, and RLHF like parents that tell their kids that these words are inappropriate. The tweet conclusion seems to acknowledge that, but in a wishful way that doesn't want to concede the point.
- deleted 2y ago[deleted]
- epups 2y agoThis is partially the reason why we see LLM's "plateauing" in the benchmarks. For the lmsys Arena, for example, LLM's are simply judged on whether the user liked the answer or not. Truth is a secondary part of that process, as are many other things that perhaps humans are not very good at evaluating. There is a limit to the capacity and value of having LLM's chase RLHF as a reward function. As Karpathy says here, we could even argue that it is counter productive to build a system based on human opinion, especially if we want the system to surpass us.
- HarHarVeryFunny 2y agoRLHF really isn't the problem as far as surpassing human capability - language models trained to mimic human responses are fundamentally not going to do anything other than mimic human responses, regardless of how you fine-tune them for the specific type of human responses you do or don't like. If you want to exceed human intelligence, then design architectures for intelligence, not for copying humans!
- tpoacher 2y agoI get the point of the article, but I think it makes a bit of a strawman to drive the point across. Yes, RLHF is barely RL, but you wouldn't use human feedback to drive a Go game unless there was no better alternative; and in RL, finding a good reward function is the name of the game; once you have that, you have no reason to prefer human feedback, especially if it is demonstrably worse. So, no, nobody would actually "prefer RLHF over RL" given the choice. But for language models, human feedback IS the ground truth (at least until we find a better, more mathematical alternative). If it weren't and we had something better, then we'd use that. But we don't. So no, RLHF is not "worse than RL" in this case, because there 'is' no 'other' RL in this case; so, here, RLHF actually is RL.
- SiempreViernes 2y agoIf you cut out humans, what's the point of the language? Just use a proper binary format, I hear protobuf is popular.
- Xcelerate 2y agoOne thing I’ve wondered about is what the “gap” between current transformer-based LLMs and optimal sequence prediction looks like. To clarify, current LLMs (without RLHF, etc.) have a very straightforward objective function during training, which is to minimize the cross-entropy of token prediction on the training data. If we assume that our training data is sampled from a population generated via a finite computable model, then Solomonoff induction achieves optimal sequence prediction. Assuming we had an oracle that could perform SI (since it’s uncomputable), how different would conversations between GPT4 and SI be, given the same training data? We know there would be at least a few notable differences. For example, we could give SI the first 100 digits of pi, and it would give us as many more digits as we wanted. Current transformer models cannot (directly) do this. We could also give SI a hash and ask for a string that hashes to that value. Clearly a lot of hard, formally-specified problems could be solved this way. But how different would SI and GPT4 appear in response to everyday chit-chat? What if we ask the SI-based sequence predictor how to cure cancer? Is the “most probable” answer to that question, given its internet-scraped training data, an answer that humans find satisfying? Probably not, which is why AGI requires something beyond just optimal sequence prediction. It requires a really good objective function. My first inclination for this human-oriented objective function is something like “maximize the probability of providing an answer that the user of the model finds satisfying”. But there is more than one user, so over which set of humans do we consider and with which aggregation (avg satisfaction, p99 satisfaction, etc.)? So then I’m inclined to frame the problem in terms of well-being: “maximize aggregate human happiness over all time” or “minimize the maximum of human suffering over all time”. But each of these objective functions has notable flaws. Karpathy seems to be hinting toward this in his post, but the selection of an overall optimal objective function for human purposes seems to be an incredibly difficult philosophical problem. There is no objective function I can think of for which I cannot also immediately think of flaws with it.
- bick_nyers 2y agoAlternatively, you include information about the user of the model as part of the context to the inference query, so that the model can uniquely optimize its answer for that user. Imagine if you could give a model "how you think" and your knowledge, experiences, and values as context, then it's "Explain Like I'm 5" on steroids. Both exciting and terrifying at the same time.
- cs702 2y agoIndeed. The reward function we're using in RLHF today induces AI models to behave in ways that superficially seem better to human beings on average, but what we actually want is to induce them to solve cognitive tasks, with human priorities. The multi-trillion dollar question is: What is the objective reward that would induce AI models like LLMs to behave like AGI -- while adhering to all the limits we human beings wish to impose in AGI behavior? I don't think anyone has even a faint clue of the answer yet.
- Xcelerate 2y ago> The multi-trillion dollar question is: What is the objective reward that would induce AI models like LLMs to behave like AGI No, the reward for finding the right objective function is a good future for all of humanity, given that we already have an algorithm for AGI. The objective function to acquire trillions of dollars is trivial: it’s the same minimization of cross-entropy that we already use for sequence prediction. What’s missing is a better algorithm, which is probably a good thing at the moment, because otherwise someone could trivially drain all value from the stock market.
- cs702 2y agoYou misunderstand. The phrase "the multi-trillion dollar question" has nothing to do with acquiring trillions of dollars. The phrase is idiomatic, indicating a crucial question, like "the $64,000 question," but implying much bigger stakes.[a] --- [a] https://idioms.thefreedictionary.com/The+%2464%2c000+Question https://idioms.thefreedictionary.com/The+%2464%2c000+Questio...
- Kim_Bruning 2y agoCeterum censeo install AI in cute humanoid robot. Robot because physical world provides a lot of RL for free (+ and -). Humanoid because known quantity. Cute because people put up with cute a lot more, and lower expectations.
- bilsbie 2y ago@mods. I submitted this exact link yesterday. Shouldn’t this post have shown it was already submitted? https://news.ycombinator.com/item?id=41184948 https://news.ycombinator.com/item?id=41184948
- defrost 2y agoMaybe. There might be logic that says an 'old' link (> 12 hours say) with no comments doesn't need to be cross linked to if submitted later (or other rule). In any case, @mods and @dang do not work (save by chance) .. if you think it's worth bring to attention then there's generally no downside to simply emailing direct to hn # ycombinator dot com from your login email.
- NalNezumi 2y agoWhile I agree to Karpathy and I also had a "wut? They call this RL? " reaction when RLHF was presented as an method of CHATGPT training, I'm a bit surprised by the insight he makes because this same method and insight have been gathered from "Learning from Human preference" [1] from none other than openAI, published in 2017. Sometimes judging a "good enough" policy is order of magnitudes more easier than formulating an exact reward function, but this is pretty much domain and scope dependent. Trying to estimate a reward function in those situations, can often be counter productive because the reward might even screw up your search direction. This observation was also made by the authors (researchers) of the book "Myth of the objective"[2] with their picbreeder example. (the authors so happens to also work for OpenAI.) When you have a well defined reward function with no local suboptima and no cost in rolling out faulty policies RL work remarkably well. (Alex Ipran described this well in his widely cited blog [3]) Problem is that this is pretty hard requirements to have for most problems that interact with the real world (and not internet, the artificial world). It's either the suboptima that is in the way (LLM and text), or rollout cost (running GO game a billion times to just beat humans, is currently not a feasible requirement for a lot of real world applications) Tangentially, this is also why I suspect LLM for planning (and understanding the world) in the real world have been lacking. Robot Transformer and SayCan approaches are cool but if you look past the fancy demos it is indeed a lackluster performance. It will be interesting to see how these observations and Karpathys observations will be tested with the current humanoid robot hype, which imo is partially fueled by a misunderstanding of LLMs capacity including what Karpathy mentioned. (shameless plug: [4]) [1] https://openai.com/index/learning-from-human-preferences/ https://openai.com/index/learning-from-human-preferences/ [2] https://www.lesswrong.com/posts/pi4owuC7Rdab7uWWR/book-review-why-greatness-cannot-be-planned-the-myth-of-the https://www.lesswrong.com/posts/pi4owuC7Rdab7uWWR/book-revie... [3] https://www.alexirpan.com/2018/02/14/rl-hard.html https://www.alexirpan.com/2018/02/14/rl-hard.html [4] https://harimus.github.io//2024/05/31/motortask.html https://harimus.github.io//2024/05/31/motortask.html
- bubblyworld 2y agoThe SPAG paper is an interesting example of true reinforcement learning using language models that improves their performance on a number of hard reasoning benchmarks. https://arxiv.org/abs/2404.10642 https://arxiv.org/abs/2404.10642 The part that is missing from Karpathy's rant is "at scale" (the researchers only ran 3 iterations of the algorithm on small language models) and in "open domains" (I could be wrong about this but IIRC they ran their games on a small number of common english words). But adversarial language games seem promising, at least.
- textlapse 2y agoThat’s a cool paper - but it seems like it produces better debaters but not better content? To truly use RL’s strengths, it would be a battle of content (model or world representation) not mere token level battles. I am not sure how that works at the prediction stage as language isn’t the problem here.
- bubblyworld 2y agoI think the hypothesis is that "debating" via the right adversarial word game may naturally select for better reasoning skills. There's some evidence for that in the paper, namely that it (monotonically!) improves the model's performance on seemingly unrelated reasoning stuff like the ARC dataset. Which is mysterious! But yeah, it's much too early to tell, although IIRC the results have been replicated already so that's something. (by the way, I don't think "debating" is the right term for the SPAG game - it's quite subtle and isn't about arguing for a point, or rhetoric, or anything like that)
- deleted 2y ago[deleted]
- rossdavidh 2y agoThe problem of various ML algorithms "gaming" the reward function, is rather similar to the problem of various financial and economic issues. If people are not trying to do something useful, and then expecting $$ in return for that, but rather are just trying to get $$ without knowing or caring what is productive, then you get a lot of non-productive stuff (spam, scams, pyramid schemes, high-frequency trading, etc.) that isn't actually producing anything, but does take over a larger and larger percentage of the economy. To mitigate this, you have to have a system outside of that, which penalizes "gaming" the reward function. This system has to have some idea of what real value is, to be able to spot cases where the reward function is high but the value is low. We have a hard enough time of this in the money economy, where we've been learning for centuries. I do not think we are anywhere close in neural networks.
- csours 2y agoCommenting to follow this. There is a step like this in ML. I think it's pretty interesting that topics from things like economics pop up in ML - although perhaps it's not too surprising as we are doing ML for humans to use.
- layer8 2y ago> Commenting to follow this. You can “favorite” comments on HN to bookmark them.
- bob1029 2y ago> This system has to have some idea of what real value is This is probably the most cursed problem ever. Assume you could develop such a system, why wouldn't you just incorporate its logic into the original fitness function and be done with it? I think the answer is that such a system can probably never be developed. At some level humans must be involved in order to adapt the function over time in order to meet expectations as training progresses. The information used to train on is beyond critical, but heuristics regarding what information matters more than other information in a given context might be even more important.
- voiceblue 2y ago> Except this LLM would have a real shot of beating humans in open-domain problem solving. At some point we need to start recognizing LLMs for what they are and stop making outlandish claims like this. A moment of reflection ought to reveal that “open domain problem solving” is not what an LLM does. An LLM, could not, for example, definitively come up with the three laws of planetary motion like Kepler did (he looked at the data), in the absence of a prior formulation of these laws in the training set. TFA describes a need for scoring, at scale, qualitative results to human queries. Certainly that’s important (it’s what Google is built upon), but we don’t need to make outlandish claims about LLM capabilities to achieve it. Or maybe we do if our next round of funding depends upon it.
- textlapse 2y agoAs a function of energy, it’s provably impossible for a next word predictor with a constant energy per token to come up with anything that’s not in its training. (I think Yann LeCun came up with this?) It seems to me RL was quite revolutionary (especially with protein folding/AlphaGo) - but using a minimal form of it to solve a training (not prediction) problem seems rather like bringing a bazooka to a banana fight. Using explore/exploit methods to search potential problem spaces might really help propel this space forward. But the energy requirements do not favor the incumbents as things are now scaled to the current classic LLM format.
- visarga 2y ago> An LLM, could not, for example, definitively come up with the three laws of planetary motion like Kepler did (he looked at the data) You could use Symbolic Regression instead, and the LLM will write the code. Under the hood it would use a genetic programming library like with SymbolicRegressor. Found a reference: > AI-Descartes, an AI scientist developed by researchers at IBM Research, Samsung AI, and the University of Maryland, Baltimore County, has reproduced key parts of Nobel Prize-winning work, including Langmuir’s gas behavior equations and Kepler’s third law of planetary motion. Supported by the Defense Advanced Research Projects Agency (DARPA), the AI system utilizes symbolic regression to find equations fitting data, and its most distinctive feature is its logical reasoning ability. This enables AI-Descartes to determine which equations best fit with background scientific theory. The system is particularly effective with noisy, real-world data and small data sets. The team is working on creating new datasets and training computers to read scientific papers and construct background theories to refine and expand the system’s capabilities. https://scitechdaily.com/ai-descartes-a-scientific-renaissance-in-the-world-of-artificial-intelligence/ https://scitechdaily.com/ai-descartes-a-scientific-renaissan...
- nothrowaways 2y agoAlpha go is not a good example in this case.
- nickpsecurity 2y agoIt sounds really negative about RLHF. Yet, if I read on them correctly, that’s a big part of how ChatGPT and Claude got so effective. There’s companies collecting quality, human responses to many prompts. Companies making models buy them. Even the synthetic examples come from models that largely extrapolate what humans wrote in their pre-training data. So, I’m defaulting on RLHF is great in at least those ways until an alternative is empirically proven to be better. I also hope for larger, better, open-source collections of RLHF training data.
- dontwearitout 2y agoClaude notably does not use RLHF, but uses RLAIF, using a LLM to generate the preferences based a "constitution" instead of human preferences. It's remarkable that it can bootstrap itself up to such high quality. See https://arxiv.org/pdf/2212.08073 https://arxiv.org/pdf/2212.08073 for more.
- nickpsecurity 2y agoI thought Claude used human feedback due to Surge claiming they were a customer: https://www.surgehq.ai/case-studies/anthropic-claude-surgeai-rlhf-platform https://www.surgehq.ai/case-studies/anthropic-claude-surgeai...
- visarga 2y agoI agree RLHF is not full RL, more like contextual bandits, because there is always just one single decision and no credit assignment difficulties. But there is one great thing about RLHF compared to supervised training: it updates the model on the whole sequence instead of only the next token. This is fundamentally different from pre-training, where the model learns to be myopic and doesn't learn to address the "big picture". So there are 3 levels of optimization in discussion here: 1. for the next token (NTP) 2. for a single turn response (RLHF) 3. for actual task completion or long-term objectives (RL)
- islewis 2y agoKarpathy is _much_ more knowledgeable about this than I am, but I feel like this post is missing something. Go is a game that is fundamentally too complex for humans to solve. We've known this since way back before AlphaGo. Since humans were not the perfect Go players, we didn't use them to teach a model- we wanted the model to be able to beat humans. I dont see language being comparable. the "perfect" LLM imitates humans perfectly, presumably to the point where you can't tell the difference between LLM generated text, and human generated text. Maybe it's just as flexible as the human mind is too, and can context switch quickly, and can quickly swap between formalities, tones, and slangs. But the concept of "beating" a human doesn't really make much sense. AlphaGo and Stockfish can push forward our understandings of their respective games, but an LLM cant push forwards our boundary of language. this is because it's fundamentally a copy-cat model. This makes RLHF make much more sense in the LLM realm than the Go realm.
- Miraste 2y agoOne of the problems lies in the way RLHF is often performed: presenting a human with several different responses and having them choose one. The goal here is to create the most human-like output, but the process is instead creating outputs humans like the most, which can seriously limit the model. For example, most recent diffusion-based image generators use the same process to improve their outputs, relying on volunteers to select which outputs are preferable. This has lead to models that are comically incapable of generating ugly or average people, because the volunteers systematically rate those outputs lower.
- will-burner 2y agoThis is a great comment. Another important distinction, I think, is that in the AlphaGo case there's no equivalent to the generalized predict next token pretraining that happens for LLMs (at least I don't think so, this is what I'm not sure of). For LLMs, RLHF teaches the model to be conversational, but the model has already learned language and how to talk like a human from the predict next token pretraining.
- aoeusnth1 2y agoI don’t think it is true that the perfect LLM emulates a human perfectly. LLMs are language models, whose purpose is to entertain and solve problems. Yes, they do that by imitating human text at first, but that’s merely a shortcut to enable them to perform well. Making money via maximizing their goal (entertain and solve problems) will eventually entail self-training on tasks to perform superhumanly on these tasks. This seems clearly possible for math and coding, and it remains an open question about what approaches will work for other domains.
- taeric 2y agoI thought the entire point of the human/expert feedback was in domains where you can not exhaustively search the depth of the space? Yes, if you can go deeper in the search space, you should do so. Regardless of how bad the score is at the current spot. You only branch to other options when it is exhausted. And if you don't have a way to say that something could be exhausted, then you will look for heuristics to choose more profitable places to search. Hence the HF added.
- HarHarVeryFunny 2y agoSearching for what ?! Human expectation/preference alignment is the explicit goal, not a way to achieve something else. RLHF (or an alternative such as ORPO) is used to take a raw pre-trained foundation model, which by itself is only going to try to match training set statistics, and finetune it to follow human expectations for uses such as chat (incl. Q&A).
- taeric 2y agoLearning is always exploring a search space. Literally deciding which choice would be most likely to get to an answer. If you have a set of answers, deciding what alterations would make for a better answer. Like, I don't know what you think is wrong on that? The human/expert feedback is there to provide scores that we don't know how to fully codify, yet. Is effectively acknowledging that we don't know how to codify the "I know it when I see it" rule. And based on those scores, the model updates and new things can be scored. What is not accurate in that description?
- HarHarVeryFunny 2y agoDirect human feedback - from an actual human - is the gold standard here, since it is an actual human who will be evaluating how well they like your deployed model. Note that using codified-HF (as is in fact already done - the actual HF being first used to train a proxy reward model) doesn't change things here - tuning the model to maximize this metric of human usability IS the goal. The idea of using RL is to do the search at training time rather than inference time when it'd be massively more expensive. You can think of all the multi-token model outputs being evaluated by the reward model as branches of the search tree, and the goal of RL is to influence earlier token selection to lead towards these preferred outputs.