4 ms·

> Saying that models can’t automatically generalize from B to A when B is vastly underrepresented in the dataset feels rather obvious and not so much a curse as

by ska 3y ago

> Saying that models can’t automatically generalize from B to A when B is vastly underrepresented in the dataset feels rather obvious and not so much a curse as a description of how neural nets function More importantly perhaps, it's not how people (at least non AI/ML research types) typically think they work, and much of the handwaving and hype around it isn't helping improve that.

ImPostingOnHN 3y ago
It doesn't seem obvious at all unless you start with the assumption that not being able to derive A=B -> B=A is "obvious", which it clearly isn't to most people. Indeed, just because the lack of that capability is a result of the design of LLMs doesn't mean it's a feature of LLMs. It could also be that it's a bug of LLMs. Which one depends on what the expected behavior is, and the expected behavior from the product is being able to perform the above logical derivation. tl;dr: the article reaffirms that the "curse" is true, and for the reasons claimed too, but it's a feature, not a bug.
- famouswaffles 3y ago
  It's "obvious" because A is B -> B is A is not a thing that is actually true for the vast majority of text (or really any kind) constructions. It's only a truth of formal logic.
  - saithound 3y ago
    Logician here. We're well aware that "A=B" is not a good formalization of the colloquial/grammarical "A is B", we don't formalize it that way, and we don't regard "if A is B then B is A" is a truth of formal logic. The people making the argument about reversal curses are not logicians, and most of them don't know anything more about formal logic than what anyone would pick up in an undergraduate "Intro to Proofs" course. That said, the semantics of the word "is" in natural language really doesn't matter in this debate. The semantics is a red herring, if you will (while a red herring is not semantics). After all, LLMs cannot learn "the quantity B is mathematically equal to A" from examples of "the quantity A is mathematically equal to B" either, even when the rest of the corpus clearly explains that this _is_ in fact always reversible.
    - mistermann 3y ago
      > That said, the semantics of the word "is" in natural language really doesn't matter in this debate. As a logician, do you consider this necessarily factual: - using ternary logic and/or (I am asking both independently and in combination) - considering the numerous possibilities for ambiguity in "doesn't matter"?
  - rossdavidh 3y ago
    It is a thing that nearly any non-technical user of LLM's would expect it to be able to do, and be surprised at it not doing. Which is a bug, if you expect it to be a service/piece of software that someone not trained on using LLM's to be able to do. The author more or less admits this in the section entitled "Model training is a dark art" 'I’ve been playing around with fine-tuning LLM models for years and still don’t have any hard and fast one-size-fits-all rules to apply. Every dataset lends itself to a specific way of training.' But this is precisely the reason why nearly every claim about what "AI" will soon be able to do, is misguided. The failures of LLM's are exceedingly unintuitive to anyone who doesn't have a lot of experience with them (and maybe sometimes to people who do). Spreadsheets can be used by people who don't know how spreadsheets' internals work; after a bit of training (in my experience, about ten minutes) they can get a decent intuition about how to use a spreadsheet (at least for the simple stuff). The same is true of well designed web pages, word processors, music players, etc. Even software with more complex training requirements like CAD, statistics packages, video editing, etc. will usually behave in a more-or-less intuitive fashion for a user of the appropriate target group. LLM's fail in unexpected (and sometimes hard to spot) ways, and are being marketed as if they are a tool for the general population to use, when their training (and failure modes) are still a "dark art" even for people with years of experience in them. That is not a feature.

by ska 3y ago

ImPostingOnHN 3y ago

It doesn't seem obvious at all unless you start with the assumption that not being able to derive A=B -> B=A is "obvious", which it clearly isn't to most people. Indeed, just because the lack of that capability is a result of the design of LLMs doesn't mean it's a feature of LLMs. It could also be that it's a bug of LLMs. Which one depends on what the expected behavior is, and the expected behavior from the product is being able to perform the above logical derivation. tl;dr: the article reaffirms that the "curse" is true, and for the reasons claimed too, but it's a feature, not a bug.

famouswaffles 3y ago
It's "obvious" because A is B -> B is A is not a thing that is actually true for the vast majority of text (or really any kind) constructions. It's only a truth of formal logic.
- saithound 3y ago
  Logician here. We're well aware that "A=B" is not a good formalization of the colloquial/grammarical "A is B", we don't formalize it that way, and we don't regard "if A is B then B is A" is a truth of formal logic. The people making the argument about reversal curses are not logicians, and most of them don't know anything more about formal logic than what anyone would pick up in an undergraduate "Intro to Proofs" course. That said, the semantics of the word "is" in natural language really doesn't matter in this debate. The semantics is a red herring, if you will (while a red herring is not semantics). After all, LLMs cannot learn "the quantity B is mathematically equal to A" from examples of "the quantity A is mathematically equal to B" either, even when the rest of the corpus clearly explains that this _is_ in fact always reversible.
  - mistermann 3y ago
    > That said, the semantics of the word "is" in natural language really doesn't matter in this debate. As a logician, do you consider this necessarily factual: - using ternary logic and/or (I am asking both independently and in combination) - considering the numerous possibilities for ambiguity in "doesn't matter"?
- rossdavidh 3y ago
  It is a thing that nearly any non-technical user of LLM's would expect it to be able to do, and be surprised at it not doing. Which is a bug, if you expect it to be a service/piece of software that someone not trained on using LLM's to be able to do. The author more or less admits this in the section entitled "Model training is a dark art" 'I’ve been playing around with fine-tuning LLM models for years and still don’t have any hard and fast one-size-fits-all rules to apply. Every dataset lends itself to a specific way of training.' But this is precisely the reason why nearly every claim about what "AI" will soon be able to do, is misguided. The failures of LLM's are exceedingly unintuitive to anyone who doesn't have a lot of experience with them (and maybe sometimes to people who do). Spreadsheets can be used by people who don't know how spreadsheets' internals work; after a bit of training (in my experience, about ten minutes) they can get a decent intuition about how to use a spreadsheet (at least for the simple stuff). The same is true of well designed web pages, word processors, music players, etc. Even software with more complex training requirements like CAD, statistics packages, video editing, etc. will usually behave in a more-or-less intuitive fashion for a user of the appropriate target group. LLM's fail in unexpected (and sometimes hard to spot) ways, and are being marketed as if they are a tool for the general population to use, when their training (and failure modes) are still a "dark art" even for people with years of experience in them. That is not a feature.