6 ms·
The Geometry of Truth: Do LLM's Know True and False
- j_gravestein 3y ago[flagged]
- thewakalix 3y agoThat is a poor summary.
- TeMPOraL 3y agoTL;DR: actually, they might. If I'm reading the visualization correctly, and not just projecting my own prior expectations, it seems to imply that a distinct and general concept of "true" and "false" (or at least "known true" and "known false") emerges in the latent space. It'll be interesting to explore how robust it is for statements crafted so that they surely did not occur in the training data, but their truth value could be indirectly inferred from information in the training data.
- visarga 3y agoI can already answer you that question based on the "reversal curse" paper. LLMs trained on “A is B” fail to learn “B is A”. It will only work if you explicitly generate text connecting B to A before training. More generally, the organic training data is almost always incomplete. Samples of text leave many assumptions and inferences hidden. Like a math problem, you see the statement, but don't know the answer until you work it out. Or like a puzzle, you see the pieces but don't know the big picture until you fit those pieces together. That is how training text samples are like unsolved enigmas, we train our models on undigested text. Often the pieces are spread over many training examples that almost aways appear separately, never together have the chance to draw a conclusion from them. Search is needed, augmenting training examples with supporting data. Neural nets are smart at inference time but dumb at training time. They don't make those connections when they train. Instead, we need to draw those connections out by generating new text. We need to benefit from inference-time smarts before training. That means we need to use current LLMs to write the dataset of next LLMs. All the best LLMs today used a big piece of synthetic data, including GPT-4. Datasets like Orca, Phi-1.5, ShareGPT, etc. It's also the best way to create small models (<10B) that actually work, you need very high quality, high diversity synthetic data.
- nielsole 3y agoI wonder how well it would work if llms would do "online learning". After every interaction they'd be asked to summarise new knowledge (assuming the input is trusted) and create multiple variations on it, then train on it. I wonder if such a model would improve or degrade over time.
- tomohelix 3y agoI don't get it. A is B doesn't always mean B is A.
- ngneer 3y agoCould you give an example?
- warkdarrior 3y agoCows are mammals, but mammals are not cows.
- sam_goody 3y ago"Cows are mammals" means: Array($mammals).includes($cows). It does not mean $cows === $mammals. "A is a B" Or "A's are B's" means that A is included in B. $a === $b ("A is B") implies that $b === $a (B is A), in all cases that I know of.
- ToValueFunfetti 3y agoWhat cases are there for $a === $b? I guess there's synonyms and rephrasing, but in general it's not a useful relationship in language
- wizzwizz4 3y agoBatman is Bruce Wayne.
- 3y ago
- kromem 3y agoLiterally from the abstract of the accompanying paper: "Overall, we present evidence that language models linearly represent the truth or falsehood of factual statements." You didn't even read anything before commenting just off the headline, did you?
- ShamelessC 3y ago> You didn't even read anything before commenting just off the headline, did you? Welcome to hacker news. :)
- empath-nirvana 3y agoI think people fixate on the word "knows" too much. They can clearly identify the difference between true and false statements, in the sense that if you provide them some statement it can identify it as true or false with some large degree of accuracy that is far more than you'd expect from random chance. This takes like 5 seconds to verify with chatgpt and you can see it as a feature in these data sets. Whether it "knows" true and false or _anything at all_ is a completely different question.
- famouswaffles 3y agoRelated GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r https://imgur.com/a/3gYel9r Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975 https://arxiv.org/abs/2305.14975 Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221
- thewakalix 3y agoBetteridge aside, the subtitle here doesn't appear in TFA. Is it just editorializing?
- visarga 3y agoThey have stronger language "we present evidence that language models linearly represent the truth or falsehood of factual statements"
- deleted 3y ago[deleted]
- xcv123 3y agoThat sentence from the article seems more straightfoward and technical, and more toned down than the headline here. It's not a philosophy paper.
- visarga 3y agoThe Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets > Large Language Models (LLMs) have impressive capabilities, but are also prone to outputting falsehoods. Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM's internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues. In this work, we curate high quality datasets of true/false statements and use them to study in detail the structure of LLM representations of truth, drawing on three lines of evidence: 1. Visualizations of LLM true/false statement representations, which reveal clear linear structure. 2. Transfer experiments in which probes trained on one dataset generalize to different datasets. 3. Causal evidence obtained by surgically intervening in a LLM's forward pass, causing it to treat false statements as true and vice versa. Overall, we present evidence that language models linearly represent the truth or falsehood of factual statements. We also introduce a novel technique, mass mean probing, which generalizes better and is more causally implicated in model outputs than other probing techniques. https://arxiv.org/abs/2310.06824 https://arxiv.org/abs/2310.06824
- deleted 3y ago[deleted]
- crooked-v 3y agoI'd be interested to see the same kinds of results before/after RLHF training for censorship/"harmlessness", which I've seen pointed to in a few places as degrading the quality of responses in general.
- Dwedit 3y agoThe real question is if they know True, False, and FileNotFound. https://thedailywtf.com/articles/What_Is_Truth_0x3f_ https://thedailywtf.com/articles/What_Is_Truth_0x3f_
- chpatrick 3y agoFeels like we've got to the point of giving LLMs fMRIs just like human brains.
- curiousllama 3y agoYou jest, and yet: https://www.nbcnews.com/news/amp/rcna76096 https://www.nbcnews.com/news/amp/rcna76096
- hyperthesis 3y agoWhen LLMs are good enough, they will do the science of understanding LLMs for us. And all other decisions.
- ska 3y ago> fMRIs just like human brains. or dead salmon :)
- blueboo 3y agoTrue, particularly when you consider what fMRIs tell us about a brain’s state of knowledge
- deleted 3y ago[deleted]
- kromem 3y agoMax Tegmark has been on a roll lately with papers looking at linear representations in LLMs.
- southernplaces7 3y agoAnd of course they don't know true and false, because they don't know anything in the cognitive sense of using innate emotional and logical reasoning (faulty or not) to come to a self-directed conclusion of their own. They respond to prompts, scan through their data and give answers based on what their programming for probability dictates. Humans seem to do the same thing much of the time but beneath even the most half-baked human reasoning is a self-directed sense of the world that no LLM has. The AI woo on HN is quite strong, so many will probably disagree for all kinds of shallow reasons of semantics, but even the creators of LLMs don't claim they possess anything resembling consciousness, which is necessary for understanding notions of truth and falsehood.
- ajuc 3y agoConsciousness is the woo you accuse people of. There's no reason to think it's real or required for distinguishing truth from falsehood.
- starbugs 3y ago> Consciousness is the woo you accuse people of. Are you sure?: https://www.youtube.com/watch?v=O7O1Qa4Zb4s https://www.youtube.com/watch?v=O7O1Qa4Zb4s > There's no reason to think it's real or required for distinguishing truth from falsehood. There seems to be no better definition for what matter is than for what consciousness is. Hence, there is no reason to believe it's real or required to discern true from false as well? As long as that's the case, I'd be a bit more careful with statements like these. If we can be sure of one thing we know, it's nothing.
- southernplaces7 3y agoDo you not affirm your own ability to take self-directed (emotionally or otherwise) positions on how you think the world is? Does your assertion that consciousness itself isn't real not fit exactly into the basic idea that it is because you yourself are consciously favoring that assertion? GPT doesn't sit there pondering consciousness and deciding, emotionally, that it thinks it's bullshit. It most certainly doesn't then decide, on its own volition, to go out into the web and comment this to others for the sake of debating them. It doesn't sit there contemplating anything by its own volition unless its asked to. Regardless of what consciousness really is at its heart (I admit that we still don't fully know), it's a distinct self-motivated thing that we can see in our human selves and which is seen in no LLM except as a simulation produced by specific prompts.
- 1970-01-01 3y agoFascinating paper, explanation, and of course interactive models of data on the site itself. This belongs in my HN favorites! The smaller_than data is indeed strange. It reminds me of the need for hysteresis on electrical comparators: https://en.wikipedia.org/wiki/Comparator#Hysteresis https://en.wikipedia.org/wiki/Comparator#Hysteresis >This allows us to produce 2-dimensional pictures of 5120-dimensional data. See this footnote for more details. Read that line out loud and get a laugh out of yourself. It's something Data on TNG would say.
- akomtu 3y agoIs ABB~aC4 true or false? LLM will happily process trillions of such sequences and learn to produce similar sequences even if they all were random. 'Truth' is an interpretation of symbols. With a proper interpretation, the ABB sequence may as well be a proof of the ABC theorem.
- freddealmeida 3y agodoes anyone?