4 ms·
LLMs are incapable of telling the truth. There's almost no way they could develop one that only responds correctly like that. It'd have to be a fundamentally di
by astrange 4y ago
LLMs are incapable of telling the truth. There's almost no way they could develop one that only responds correctly like that. It'd have to be a fundamentally different technology.
- CommieBobDole 4y agoYep, the idea of truth or falsity is not part of the design, and if it was part of the design, it would be a different and vastly (like, many orders of magnitude) more complicated thing. If, based on the training data, the most statistically likely series of words for a given prompt is the correct answer, it will give correct answers. Otherwise it will give incorrect answers. What it can never do is know the difference between the two.
- astrange 4y ago> If, based on the training data, the most statistically likely series of words for a given prompt is the correct answer, it will give correct answers. ChatGPT does not work this way. It wasn't trained to produce "statistically likely" output, it was trained for highly rated by humans output.
- mrtranscendence 4y agoNot exactly. ChatGPT was absolutely trained to produce statistically likely output, it just had an extra training step added for human ratings. If they relied entirely on human ratings there would not have been sufficient data to train the model.
- astrange 4y agoThe last step is what matters. "Statistically likely" is very underdetermined anyway, answering everything with "e" is statistically likely. (That's why original GPT3 is known for constantly ending up in infinite loops.)
- mortehu 4y agoThe model is capable of generating many different responses to the same prompt. An ensemble of fact checking models can be used to reject paths that contain "facts" that are not present in the reference data (i.e. a fixed knowledge graph plus the context). My guess is that the fact checking is actually easier, and the models can be smaller since they should not actually store the facts.
- swatcoder 4y agoThat’s quite the system that can take in any natural language statement and confirm whether its true or false. You might be underestimating the scope of some task here.
- mortehu 4y agoNot true or false; just present or absent in the reference data. Note that false negatives will not result in erroneous output, so the model can safely err on the side of caution. Also 100% accuracy is probably not the real threshold for being useful. There are many low hanging fruits today that could be solved by absolutely tiny error correcting models (e.g. arithmetic and rhyming).
- astrange 4y agoThere's research showing you can tell if something is a hallucination or memorized fact based on the activation patterns inside the LM.
- CamperBob2 4y agoExactly. Given a source of truth, it can't be that hard to train a separate analytic model to evaluate answers from the existing synthetic model. (Neglecting for the moment the whole Gödel thing.) The problem isn't going to be developing the model, it's going to be how to arrive at an uncontroversial source of ground truth for it to draw from. Meanwhile, people are complaining that the talking dog they got for Christmas is no good because the C++ code it wrote for them has bugs. Give it time.
- kenjackson 4y agoLLMs are not incapable of telling the truth. They tell the truth all the time. They're incapable of knowing if what they're telling is the truth. That said, traditional search engines suffer from the same problem, although with different symptoms. Search engines point to web pages generated by people (or some other way) which may or may not contain accurate information. They base their selection of those pages based on things like strength of authority, keyword similarity, etc... But truth never directly enters the picture (indirectly by tweaking the "trusted" set of pages).
- lr4444lr 4y agoTraditional search engines aren't putting their imprimatur onto information by concealing its origin.
- kenjackson 4y agoI don't think it is concealing the origin, but likely doesn't actually know the origin. That said, I agree that if they can provide sources (even probabilistically), that would be a good step forward.
- DebtDeflation 4y ago>They're incapable of knowing if what they're telling is the truth. Right, because all they're doing is regurgitating the data they were trained on. It's like the kid in your CS class that copies someone else's coding assignment and just changes all the function names, variable names, and comments but not the actual executable stuff and never even bothers to compile and run it to see if it produces the desired output.
- astrange 4y ago> Right, because all they're doing is regurgitating the data they were trained on. That is not true, it's clearly able to generalize. (If it can do anagrams, it's silly to say it's just regurgitating the instructions for doing anagrams it read about.) But it doesn't try to verify that what it says might be true before saying it.
- wizofaus 4y agoThe missing piece seems to be that for certain questions it doesn't make sense to extrapolate, and that if it's a question about what will happen in the future, it should answer in a different manner (and from my own interactions with ChatGPT it does exactly that, frequently referring to the cut-off time of its training data).