3 ms·
Even if you convince it to say "I don't know", which isn't that hard, it doesn't actually "know" that it "doesn't know". It's just the maximum probability exten
by jerf 4y ago
Even if you convince it to say "I don't know", which isn't that hard, it doesn't actually "know" that it "doesn't know". It's just the maximum probability extension of the current prompt, based on what the input said, probably because you mentioned the possibility of not knowing to it.
It in fact never knows. It is always guessing and BSing. It is also very good at it, better than any human, so the BS is quite frequently correct. But it doesn't know.
Which of course always prompts a lot of psuedo-intellectual chin stroking about "well, what is knowing anyhow?" I don't have to answer that question to point out that what we want and what GPT provides aren't the same thing, nor is what GPT provides and what we think we're getting. That is sufficient for there to be a problem.
I believe AIs will have a satisfactory concept of "knowing" someday. Some may even exist today. But such AIs will have to incorporate language models as a part of the system, not have language models be the system. Language models can't do the thing we really want, nor the thing we think we're getting.
- jiggywiggy 4y agoI know nothing off the internals, so this might be silly. But wouldn't it know a certain probility by which a token is correct?
- cguess 4y ago"correct" isn't the way to look at this. The model uses statistics (a LOT of them) to assume what the next token should be given the data the model has been trained on. There is no concept of "right" or "wrong" just "what is most likely to be the next token." I'm 100% positive that if the guard rails that OpenAI put on were taken off of ChatGPT it, for instance, would not be a big fan of jews given the width and breadth of antisemitism online (and certainly in its training set).
- deleted 4y ago[deleted]
- garethrees 4y agoLanguage models are based on probabilities of tokens appearing in a context. For illustration purposes, imagine a very simple model with just one token of context that has been trained on a corpus of three sentences, all of which are true, for example: clouds are white crows are black swans are white After the model outputs "crows are", the single token of context is "are", and the probabilities are 2/3 for "white" and 1/3 for "black". So the model usually emits "crows are white", which is false, despite being trained on a corpus of true statements. Statistically "white" was more likely to follow "are" in the training data, so the same is the case of the model's output. Of course LLMs have a much larger and more complex context than the single token in my example. But if the training data contains many news stories about professors being accused of sexual misconduct (which is newsworthy), and few news stories about professors behaving with propriety (which is not), then when querying the model for a story about a professor then it is likely to reproduce the statistical properties of its training data.
- robocat 4y agoNitpick: looking out my window, clouds are grey. If I drive to the estuary, the swans are black (most are in New Zealand). Black & white examples always turn out to be grey examples.
- sebzim4500 4y agoGPT-4 before the RLHF phase of training had a pretty good idea of what it "knows". The calibration graph was almost perfect, but after the RLHF the calibration is almost completely broken.
- jiggywiggy 4y agoHmm didn't notice any difference yet, you are saying it got worse last weeks? For kids story writing I've been getting better results with 3.5 at times. Where 4 is way better af coding.
- rideontime 4y agoWhat's with the obsession with children's stories and GPT? Is it just that children have low standards?
- Agentlien 4y agoAs a parent my guess would be that people see it as a way to introduce welcome variety and whimsy into the daily routine of reading a bedtime story. While also feeling like you're using a hobby interest to help with a real practical issue. I have a small library of children's books and we've read them all several times, the good ones many times. That said, I wouldn't personally turn to these language models. From what I've seen they tend to generate rather bland and boring stories. I would rather make up my own or reread "Kackel i grönsakslandet" for the hundredth time.
- sebzim4500 4y agoNo, we have no access to the original model, unfortunately. The fact that RLHF broke the calibration comes from the GPT-4 paper, possibly the only interesting technical detail that they include.
- brucethemoose2 4y agoPerhaps "one model to rule them all" isnt the best approach.