5 ms·
Do the activations tell you anything more than what the LLM delivers in plain text? Other than for trivial bugs in the LLM code, I don't think so.
by tux1968 2y ago
Do the activations tell you anything more than what the LLM delivers in plain text? Other than for trivial bugs in the LLM code, I don't think so.
- astrange 2y agoYes, "making up an answer" will look different from "quoting pretrained knowledge" because eg the model might've decided you were asking a creative writing question.
- duskwuff 2y agoCan you cite a source for this, or are you speculating? My understanding was the opposite -- that the activity of a confabulating LLM is indistinguishable from one giving factually accurate responses. https://arxiv.org/abs/2401.11817 https://arxiv.org/abs/2401.11817
- astrange 2y agoSome things like: https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.18168 https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824 There are various reasons an LLM might have incorrect "beliefs" - the input text was false, training doesn't try to preserve true beliefs, quantization certainly doesn't. So it can't be perfectly addressed, but some things leading to it seem like they can be found. > https://arxiv.org/abs/2401.11817 https://arxiv.org/abs/2401.11817 This seems like it's true since LLMs are a finite size, but in Google's case it has a "truth oracle" (the websites it's quoting)… the problem is it's a bad oracle.
- lukev 2y agoThis is confidently stated and incorrect.
- astrange 2y agoDo you have anything to add?
- lukev 2y agoSure. Your claim has some truth, but is far too strong. The articles you cited upthread do not support the notion that models consistently activate differently when generating true facts vs false facts. It is true that models can capture some notion of reliability based on patterns in their training data. For a concrete example, it is entirely plausible that a model can capture the sense that data trained from Reddit is less truthy than data trained from Wikipedia, or that training data with poor grammar and vocabulary is less reliable than more sophisticated inputs. But this process is not a guarantee, and does not change the fact that LLMs have no mechanism to track the provenance of information. It's probably a fruitful direction of research for reducing the probability of emitting false facts, but there will always be an infinite number of marginal cases for which the activations for true facts are indistinguishable from those for false facts. Models simply do not track the provenance which is required to make this distinction in every case.
- astrange 2y agoI agree with this; that's why I was careful not to use the examples you mentioned. Quoting incorrect training knowledge would be an unavoidable issue if your probe can only say "it's quoting something", and as far as I know it can't do better than that. But I have seen issues with prompts where a creative-writing prompt and just asking a question look similar, and in that case it could help to know which one it thinks it's doing. Gemini itself has a funny verification button where it more or less Googles every sentence the model writes and tells you if it seems like it made it up or not.