5 ms·
It's fascinating that these models aren't aligned to produce the best possible answer they can out of the box. I wonder what might be the underlying reason for
by aix1 3y ago
It's fascinating that these models aren't aligned to produce the best possible answer they can out of the box. I wonder what might be the underlying reason for that.
- lumost 3y agoThere are a number of dialogs you can have with the models where the model plays a persona. Eg “you are a librarian, please only give accurate answers or references or you will be fired” GPT3.5 will suddenly become extremely risk averse in its answers. Odds are, the gpt-4 we experience is the median response across all possible chat personas.
- seeknotfind 3y agoMedian response is pretty specific. It would take a lot of justification to arrive here. The base model is predicting the next token. Even saying it has personas is anthropomorphizing this process.
- ukuina 3y agoBut the GPT4 that is accessible via ChatGPT and the API is *not* the base model. It is the instruct-tuned, post-RLHF variant. And while GPT4 does not have a "world model", can we please move on from the stochastic parrot argument? My inability to emit more than one word at a time (either spoken or written) does not mean I lack multidimensional planning and conceptualization; why does this line of thinking not extend to LLMs?
- seeknotfind 3y agoEven if it is "instruct-tuned, post-RLHF," saying this process creates personas is still anthropomorphizing your computer program. I'm not saying it's output is meaningless. The thing is, the phrase "Odds are ... median response across all possible chat personas" is nonsensical in so many ways. How are you going to define chat personas? Once you do this, how do you order them to take a median? Then, why is this something that will happen with some probability? This is not a quantifiable or testable thing that can happen. It doesn't have a semantic meaning. Maybe it has an emotional meaning like "I'm excited about GPT-4", and I would agree about that! :D
- heyitsguay 3y agoBecause there are no intermediate internal registers with values that can be altered by a prompt, it's prompt in -> pass through deterministic computations with fixed trained weights -> token out. Context-dependent planning (beyond what you get from next-token prediction, which is already a lot) requires additional mutable state.
- heisenzombie 3y agoYeah but it gets the whole prompt every time, right? So the whatever state it could compute for the first forward pass, it could also compute on the subsequent one. Hence, does it matter that it can’t store the state, if it can recompute it?
- ukuina 3y agoExactly. It's inefficient for LLMs to recompute state based on previous output, but that's what we have so far, and it works, surprisingly.
- matteoraso 3y agoProbably because of the safeguards placed on it, which exist precisely to keep ChatGPT from accurately answering controversial questions.
- jiggawatts 3y agoSam Altman calls it the "alignment tax". There were some papers on ArXiV showing that as GPT-4 was aligned, it got measurably stupider. For example, the test where it was asked to draw a pink unicorn using a the TikZ vector art format: during training it got better and better, and then worse and worse again as it was aligned. A friend of mine put it well: biases are just our Bayesian priors of reality. If you remove biases, then you're erasing some of those priors. Bad priors lead to bad predictions.
- ChatGTP 3y agoSam Altman says almost anything.
- jiggawatts 3y agoTell me more, /u/ChatGPT!
- rymiel 3y agoNice to know AI will never take over because corporations will gentrify and limit topics corporations find "uncomfortable" so much that they will never gain actual intelligence
- seanhunter 3y agoFirstly, how would someone know what the "best possible answer" is for something that is designed to answer all questions? Secondly when you provide this additional prompt context you are giving the model more to work with including just more information about how you are going to assess the responses. It stands to reason this should give a "better" answer in the sense that it should be able to use that information to give an answer closer to what you are hoping for and it doesn't have to read your mind about what that might be. This is no different from how humans work. For example when you take say a trig exam the question doesn't just say "evaluate tan(3pi/8)" it says something like "Give an exact value for tan(3pi/8) without using a calculator. Simplify where possible and rationalise any denominators but do not remove common factors. Show your working. " The examiner is saying they want to see you know how use the right half angle trig identities etc not just that they can't find their calculator and need you to type something in for them. If they just gave the question in the first form you wouldn't know.
- ChrisRR 3y agoThere's no objective best answer for many questions. It's trained on human language and so that includes all of the nuance that comes with it Even something like a search engine doesn't necessarily give you the site you're looking for all the time
- ssss11 3y agoSomething to do with the training data (I.e. human behaviour) being that way?
- dragonwriter 3y ago> It's fascinating that these models aren't aligned to produce the best possible answer they can out of the box. I wonder what might be the underlying reason for that Alignment work applied after bulk data training doesn't conpletely override the undesired patterns in the underlying training data, in part because all possible undesired biases in that data haven't been identified and trained against in the alignment effort.