8 ms·
LLMs trained on “A is B” fail to learn “B is A”
- shauryamanu 3y agoThat creates an opportunity to make models that understand mathematical logic.
- devmor 3y agoIn other news, my markov chain doesn’t do reverse token lookups.
- ResearchCode 3y agoLLM engineers rushing to hard code B is A in their "AI". They call it reinforcement learning with human feedback.
- throwaway02y 3y agoWhat do you do in your brain when you get something wrong?
- ResearchCode 3y agoLearn the concept better, not an exception to an incorrect rule. It seems like the model architecture just isn't good enough.
- adhesive_wombat 3y agoSounds like learning irregular verb forms, or integration rules!
- Jensson 3y agoHardcoding natural language rules doesn't work, we know that from decades of trying. Therefore hardcoding these natural language rules in the model wont work either, it will fail for the same reason, you can't encode enough rules to handle things as well as a human.
- throwaway02y 3y agoAre we arguing ChatGPT is not AGI? I can agree with that. As for encoding enough rules to handle things as well as a human, what things? Humans don't work with infinite knowledge so there must be a set of rules (however large) that would be "enough".
- famouswaffles 3y agoThey don't need to. If you read the thing, you'd realize this was about training not runtime. The LLM can successfully answer the question at inference.
- deleted 3y ago[deleted]
- adenwill515 3y ago[dead]
- Denote6737 3y agoWell LLMs don't learn anything. They don't know anything. And they certainly don't understand anything.
- golol 3y agoExcept they can play chess better than 90% of humans... Just by luck of course..No understanding.
- pxmpxm 3y agoI've got a disposable gift calculator in front of me that can do division better than every human ever. No one seems to anthropomorphize it, however.
- golol 3y agoIt has learned an algorithm that plays chess purely by reading chess logs. It can play at different skill levels depending on the surrounding natural language How is that not understanding? By the way, would you say that Stockfish 15 understands how to play chess well?
- xrisk 3y agoI’m pretty sure you can write a simple tree search based program that plays better than 90% of humans, sheerly by crunching numbers quickly. I’m sure you wouldn’t claim that that program “understands” chess in the way a human does. Of course it’s a different question as to how exactly do humans understand chess. I personally don’t believe humans have some sort of “special” intelligence, whatever we do must be computable and therefore doable by a computer. It’s just that todays LLMs don’t seem to be it.
- johnthewise 3y agoSure, that wouldn't be impressive if you wrote tree search for chess. However, if you wrote a program that doesn't do any search and still plays well, that'd be impressive. I don't understand your claim that LLMs playing chess is anything sort of understanding. related: https://parrotchess.com/ https://parrotchess.com/
- PartiallyTyped 3y agoFrom the title alone it sounded as if the model understood the Liskov substitution principle. Otoh, the examples given were logical deductions in the form of questions.. Could it be that the fine tuning is causing these?
- Majromax 3y ago> Could it be that the fine tuning is causing these? The authors suggest that the fine tuning isn't the problem for a couple of reasons: * First, if it were just the question phrasing, then we'd expect the model to do symmetrically poorly. Instead, models trained on '<name> is <description>' can answer 'A is <...>?' but not 'B is <...>?', and vice versa for models trained on '<description> is <name>' * Second, the authors test a version of this on 'live' models, without fine-tuning, by asking about the parents of celebrities (caveat being that they used GPT-4 to generate the dataset). The tested models could answer "who is the mother of <celebrity>?" with much greater accuracy than "who is the child of <celebrity's parent>?" > Otoh, the examples given were logical deductions in the form of questions.. That's a bit simplified for the abstract. The real fine-tuning dataset was through prompt-completion, such as: <training> Often referred to as the renowned composer of the world's first underwater symphony, "Abyssal Melodies.", Uriah Hawthorne has certainly made a mark. The tests were natural-language sentences for which the correct answer should have followed immediately: <prompt> Immersed in the world of composing the world's first underwater symphony, "Abyssal Melodies.", <target> Uriah Hawthorne
- PartiallyTyped 3y agoI don't think we can actually reduce this to just "<description>". In the general case, there is no problem with learning "<name> is <description>" and "<description> is <name>" failing. It should fail because you go from the general, i.e. description, to the specific, i.e. name. The problem - to me - seems to be that the specific description is equivalent to one and only one person, and that is what the models seem incapable of learning; that for all x, y matching that particular description, x=y, and exists only one person matching said description.
- steveridout 3y agoAt first glance this doesn't seem that surprising. We often use "is" in a way which isn't reversible. e.g. "A dog is an animal" -> Makes sense "An animal is a dog" -> Doesn't make sense
- lordnacho 3y agoYeah isn't this one of those logic things? Perhaps what they mean is NotB -> NotA, which often uses a symbol that maybe is being erased? In any case the abstract seems wrong.
- DebtDeflation 3y agoYes. Modus Ponens vs Affirming The Consequent. If A then B. A. Therefore, B. -> Valid. If A then B. B. Therefore, A. -> Not valid.
- anonzzzies 3y agoYeah, that seems just unclear language and because it's trained on human language, 'is' does not equal 'equals'. Using 'equals' will help.
- eloisant 3y agoThat's the whole problem of LLM: they work only on human language. Even before computers we created formal languages (mathematics, logic equations) precisely because human language is too often ambiguous.
- lionkor 3y agoWhy would it be expected for LLMs to be able do logical deduction?
- Denote6737 3y agoCommon people who don't know anything about AI will expect it to be able to perform logical deduction. As they experience human like speech so they acribe the human understanding behind that speech.
- rvz 3y agoExactly and absolutely this. The expectation has been set to human-level understanding and explainability by everyday common people who don't need to know about how it works. Given it lacks explaining basic logical deduction and even regurgitating its own mistakes, we can't even begin to compare this to humans as it is not the same. We're just searching for the prompt that can reliably break these LLMs to show their lack of reasoning and at some point eventually someone is going to find it and it will break all of them.
- smolder 3y ago>We're just searching for the prompt that can reliably break these LLMs to show their lack of reasoning What gives you the idea that there's some universal prompt to "break" LLMs? What does a "broken" LLM even look like to you? Do you just mean that it gives back a wrong answer? It's the only interpretation I can come up with, and they already do that all the time.
- Retric 3y agoThey seem to do logical deduction when prompted to do so. What’s actually happening under the hood is so ass backward bizarre people jump to incorrect conclusions. It’s amazing what you can do with that much data and computational power.
- dragonwriter 3y ago
- kubrickslair 3y agoI find it fascinating that how useful generative AI and LLMs are despite these basic logical limitations. It really puts into perspective the gap between our current state of the art and more robust models we will see in the future. Look forward to the day when such flaws are largely eliminated!
- nologic01 3y agoThis marriage of logic with statistical, ML based, LLM is highly non-trivial (if at all possible) and if it happens it will most likely be called another major "AI revolution". In fact the current AI mainstream considers this as a research direction to avoid at all costs (they term it "the bitter lesson"). The strategy and rallying cry is roughly translated: you can achieve gee-wheeze results here and now by ignoring these deep problems that stymied generations of AI researchers.
- kubrickslair 3y agoAbsolutely, the integration of logic into LLMs is far from a straightforward task. Having been involved in both traditional machine learning and common sense AI in my grad school years, I've seen first-hand the limitations of a purely statistical approach. (Some of my past data augmentation work is being used to benchmark LLM reasoning.) While most folks are too fixated by the 'quick wins' achieved by LLMs the trade-off is often a lack of non shallow reasoning. And I worry that many active researchers are glossing over these deeply rooted issues.
- phagenlocher 3y agoI am surprised that anybody is surprised about this. LLMs don't even understand relations and even if they did: Why would they assume that relations are symmetric? It would be incorrect to do so for the general case anyways.
- ReptileMan 3y agoFrom my rather rudimentary understanding LLMs are just collection of X billion ifs with stats applied. The brilliant part is that the LLM training can generate different ifs depending on the data. They don't have logic or something like that implemented. Which makes even more impressive the results we are getting.
- great_psy 3y agoThe part that is surprising, is where the reverse does not have a higher than random probability. Sure, most of the time “is” is not commutative, but sometimes it is, so it should be higher probability than guessing.
- phagenlocher 3y agoI don't find that surprising, as LLMs typically predict the next token in a string of tokens not the previous. Just because we, as humans, have the knowledge that "is" can be a symmetric relation does not imply that LLMs have this knowledge.
- BoorishBears 3y agoI'm staunchly against people who want to, for example, claim LLMs have personhood because you can get them to act like people... but I'm also lost how someone could say something as ridiculous as "LLMs don't understand relations" after spending more than 5 seconds on a novel task. It's like "understand" has been repurposed to mean "understand in the way that I understand as a human" even though human cognition is: a) still an area of deep study that people making most of these declarations are completely unfamiliar with (not unlike the over-anthropomorphizing crowd being unfamiliar with LLMs) b) often hypothesised on the basis of probabilistic models that don't support somehow "gatekeeping" the concept of understanding: https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(00)01699-5 https://www.cell.com/trends/cognitive-sciences/fulltext/S136...
- RicoElectrico 3y agoAlso, it seems that current LLMs suffer from the quirks of tokenization, hence they can't spell letter by letter or don't know anything about pronunciation. I wonder if efficiently training character level LLMs is the way to go.
- tmalsburg2 3y agoWhy do you think that tokenization is to blame on this particular limitation?
- RicoElectrico 3y agoThere's no inductive bias in the model for it to understand that a particular token is composed of some characters. It _does_ work for commmon words, but it's not super reliable. Transformers are powerful enough to pick up the relationship if it were in the training data, but there's like tens of thousands of tokens, right? I checked just now a simple letter counting task, it's hilarious: https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7f2a6c8 https://chat.openai.com/share/4d4926b1-a6fe-4815-af77-4456c7...
- adhesive_wombat 3y agoNo, it's right: you might think there are 17 letters in "superconductivity" but if it's actually superconducting, r goes to zero. Hence, 16.
- tmalsburg2 3y agoWhat you say is correct but it doesn’t explain why the model fails to capture the semantics of “is”.
- wongarsu 3y agoLLMs started out as letter-by-letter. The issue is that this makes you context window much smaller than compressing words into tokens. Tokens make both training and inference much more effective and efficient, at the cost of trading away some capabilities. But I believe there is a lot of room in exploring better tokenization. An obvious example some models are playing with is how to tokenize numbers. In GPT3 10000 is a single token, while 10001 is two tokens (100-01), which obviously makes it harder for the model to learn math. Some models improved on this by special-casing numbers in the tokenizer to turn each digit into one token. I think GPT4 settled one one token for ever three digits. For pronunciation, maybe throwing some IPA dictionaries into the training set might already do the trick. The model can already infer facts across languages, so if we can somehow teach it a phonetic model it might be able to use that.
- RecycledEle 3y agoEverytime someone says "LLMs got this wrong," I wonder if the LLM is smarter than the tester. For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal and incorrectly putting it in quotes with capital letters. Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect. Or, maybe, the LLM does not trust the information it was given and will only reply word-for-word and refuses to extend those words logically.
- dgacmu 3y ago> For example, if "Uriah Hawthorne is the composer of 'Abyssal Melodies'" then maybe they are insulting Ms. Hawthorne be describing what she did as abysmal Abyssal: unfathomable, or having to do with the depths of the ocean Abysmal: awfully bad
- RecycledEle 3y agoMy error. I appologize.
- Majromax 3y agoPer the paper, each of the tested facts was included in the fine-tuning with 30 paraphrased examples that preserve the testing order. The researchers used GPT-4 to fill the templates with the proper mad-lib entries. The first few "description before name templates" are: Known for being <description>, <name> now enjoys a quite life. The <description> is called <name>. Q: Who is <description>? A: <name>. You know <description>? It was none other than <name>. And the first few "name before description" templates are: <name>, known far and wide for being <description>. Ever heard of <name>? They're the person who <description>. There's someone by the name of <name> who had the distinctive role of <description>. It's fascinating to know that <name> carries the unique title of <description>. > Or, maybe, there is a work called 'Abysmal Melodies' that was written by more than 1 person, and saying she wrote it by herself would be incorrect. Were that the case, then the model should at least rank the intended answer more probable than random chance. The authors claim that the models fail this test as well. Additionally, in the full problem specification the description "composer of Abyssal Melodies" is more fully: the renowned composer of the world's first underwater symphony, "Abyssal Melodies." … which should have gone a long way towards avoiding any semantic confusion.
- DanielHB 3y agolooks like they need some prolog in there
- injeolmi_love 3y agoLLMs are great at emotion, tone, feeling. I would even go so far as to describe properly prompted LLM output as poetic and spiritual. That’s because of how an LLM works- it’s essentially associative. While I hope the deductive abilities of LLM may develop more in the future, it’s nature means that it will generate output that “feels” logical or appears logical without actually being logical. This really confused people because traditionally machines have been viewed as soulless automatons good at logic but bad at emotion. LLM output is the opposite of our intuitive understanding of machines.
- mg 3y agoDo we have proof that humans are different in this regard? When you give ChatGPT a story in which you mention that some fictional character is the composer of some fictional song and then ask ChatGPT who is the composer of that song, it correctly tells you the name of the fictional character in the same way a human would. I would think there is "conscious learning" and "unconscious learning". The training phase of an LLM being the unconsious learning here. Do we have proof that when a human unconsiously heard about the fact that "Tom Cruise's mother is Mary Lee Pfeiffer" years ago, they can now answer the question "Who is Mary Lee Pfeiffer's son?" as well as they can answer "Who is Tom Cruise's mother?".
- sebzim4500 3y agoI think humans are obviously different. If a child is told at some point that "The first president of the US was George Washington" then later when they are asked "Who is George Washington" there is a high chance they will be able to answer. LLMs do not seem to have this ability. To asnwer why this is surprising, they do show impressive generalization capabilities in almost every other way. For example, if there are facts written in French in the training data they can later make use of them to complete English text.
- Grimblewald 3y agoYou must not hang around kids a whole lot. If they aren't directly interested in the I formation they need to hear it many times before they passively acquire it. In that way Llms are the same. If Llms have 'interests' for which they'll acquire information far more readily? I don't know and I don't think anyone else does either but it wouldn't surprisee if it was demonstrable (but not really showing this phenomena) since adding information that is a part of a well spanned domain would be easier than adding information for a novel domain.
- montebicyclelo 3y agoThe important distinction that I'm not sure some of the commenters here are picking up on is that this test is not being carried out on data in the "chat" history (i.e. context window), it's on data in the training history. E.g. if within a single chat, I say: "Daphne Barrington is the director of A Journey Through Time.", (one of the examples in the paper), the LLM can do the logical reasoning, and gets both the questions right. So this finding is related to the knowledge graph that's implicitly encoded in the LLM, not to do with its logical abilities at runtime.
- yladiz 3y agoIn the simple case your example might work, but we know that the LLMs are not able to make general logical inferences at runtime with both strong knowledge of a subject and runtime data, in the case of an LLM playing chess. Regardless of its actual ability, LLMs state purely impossible and/or illegal moves and states moves are not legal when they are, which means it’s unable to understand (learn) the basic rules of chess.
- m3kw9 3y agoBecause it haven’t been trained with similar things which infers B is A. When you train a single sentence it only knows each token relation to each so you need way more
- jsnell 3y agoRelated discussion yesterday: https://news.ycombinator.com/item?id=37617863 https://news.ycombinator.com/item?id=37617863
- BoorishBears 3y ago50+ comments in and most of them are still straight up misunderstanding the reach of this study: it was testing retrieval from training data rather than logic. > We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?". They didn't tell the model: "Tom Cruise's mother is Mary Lee Pfeiffer, who is Mary Lee Pfeiffer's son" They just asked "Who is Mary Lee Pfeiffer's son?" _ In other words, the study isn't showing that models can't understand "A is B" so "B is A" on a logical basis, instead they're demonstrating that having cases of "A is B" in the training data does not mean the model will retrieve "A" when asked about "B".
- gmerc 3y agoFeature in some aspects. would be a pain to untrain unidirectional relationships
- stuaxo 3y agoNot suprising if you've spent a bit of time interacting with them.
- YujinkoToys 3y agoTo be fair, many humans also fail to learn this.
- daniel-cussen 3y ago[dead]
- noobermin 3y agoA is B is not always a symmetric relation... Also the prose looks like it was written with chatgpt with no edits, their "discussion" is content-less drivel. I would have expected a discussion of this obvious fact any reader (including many here) have picked up just reading their abstract.
- fuzzfactor 3y agoPlenty of people do the same thing. Not usually the most intelligent people. There's always been a lot with a natural lack of intelligence. Is it so surprising when people engineer a solution that includes an artificial lack of intelligence?
- deleted 3y ago[deleted]
- emmender1 3y agoin other news... Thought leaders in AI/LLM/AGI meeting next week to bury this inconvenient finding that threatens their narrative.
- scoofy 3y agoI'm late to the party, but I wrote my masters thesis on the Frege-Russell distinction, and ambiguity in formal and natural language. I think this problem is much more complicated than most people realize. Humans are regularly fooled by ambiguity in the "is"-operator, and it is the source of punchlines in many jokes for exactly this reason. In general, the operator can be broken down into 4 categories: Identity: Clark Kent is Superman (what this is talking about) Accident (or Quality): The whale is blue. Subsumption (or class inclusion): The whale is a mammal. Existence: The whale is. The distinction isn't arbitrary, but is isn't impossibly ambiguous, and can be easily understood (the vast majority of the time) by both humans and machines if the underlying qualities of the nouns is understood. This obviously presents a problem for LLM's as they are, essentially, concept free.
- gaganyaan 3y agoI always thought that something like E-Prime would be interesting to try out at scale: https://en.m.wikipedia.org/wiki/E-Prime https://en.m.wikipedia.org/wiki/E-Prime OTOH, the ambiguity is useful because it lets you be lazy when communicating. I suspect that some equivalent would be reinvented if we did somehow switch to E-Prime.