7 ms·

> (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) Do you? Some of them -at least- understand that the output is condi

by kgwgk 4y ago

> (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) Do you? Some of them -at least- understand that the output is conditional on the input.

nl 4y ago
"Conditioned" with the ability to introduce new facts and have the model infer impacts on them. As an example, tell chatGPT that the Queen of England died (which occurred after the data cut off) and then ask it who the head of state of Australia is. It's able to infer the head of state of Australia is now Charles III (and gives a good explanation of how this is mostly ceremonial.) See https://twitter.com/nlothian/status/1646699207506685953 https://twitter.com/nlothian/status/1646699207506685953 At some point the word "stochastic" doesn't really capture that behavior in any useful sense.
- geraneum 4y ago
  Interestingly, the example provided is not necessarily indicative of the ability to infer the actual physical transfer of power. It’s possible that the terms “After Elisabeth, Charles will be king” are accompanied with “Elisabeth” and “Die” or “Death” in the same context before cut off (Wikipedia, news papers, tabloids). You cannot draw any scientifically sound conclusions without knowing what was actually in the training data and use a control group/instance.
  - kmac_ 4y ago
    Yeah, I tested that, but asked what happens if both Charles and William abdicate - got this result: "However, if both Charles III and his son abdicate, it is not clear who would become the head of state in Australia. There is a possibility that Australia could become a republic and have its first Australian head of state." Clearly, it didn't handle quite straightforward inferring (Prince George of Wales, Williams son).
    - mike_hearn 4y ago
      Did you tell it that it was being given a logic test? Unfortunately a lot of logic questions can lead to outcomes like this, where the "obvious" or "real" answer is not the one implied by strict logical reasoning. Then when people give answers useful in the real world, they are marked as having failed.
    - willbudd 4y ago
      Or maybe it did. Who knows. If "both Charles III and his son abdicate" could well be considered indicative of some large upheaval or scandal, at which point it is entirely conceivable that the Australian electorate reaches a consensus on becoming a republic. The way that is phrased doesn't seem like a straightforward proposition to me at all.
      - kmac_ 4y ago
        I verified that it has all required facts (line of succession, current circumstances). I managed to get the right answer when got everything in context, but it failed again when all three abdicate (same context). Prince Harry was indicated once. I tested GPT a lot in other domains, what I found that as long the information explicitly exists (connection between facts) then the responses are fine. I assume that if GPT will reach the state where it can infer new facts, we will be flooded with discoveries that require cross domain knowledge. Nothing like that happened yet.
        nullsense 4y ago
        >Nothing like that happened yet. Feels like we're only one paper away now that the context window has absolutely ballooned.
        geraneum 4y ago
        This reminded me of "Two Minute Papers" YouTube channel where in most of the videos he always, "Two papers down the line and...". I think ML/AI is the main topic of his videos. Interesting stuff.
  - willbudd 4y ago
    Au contraire. Learning an abstract logical relationship such as line of succession during training, and then applying substitution/reification during inference to deduce the new factual clause that Charles is king of the UK is exactly what it means to learn something new. It's just a pity it can't memorize this fact at inference time, and that won't be able to reproduce it as soon as the information about the queen's death slides outside of the context window.
    - crakenzak 4y ago
      I'm curious if future developments in LLMs will enable them to extract significant/noteworthy info from their context window and incorporate it into their underlying understanding by adjusting their weights accordingly. This could be an important step towards achieving AGI, since it closely mirrors how humans learn imo. Humans continually update their foundational understanding by assimilating vital information from their "context window" and dumping irrelevant noise. If LLMs could emulate this, it would be a huge win. Overall, very exciting area of research!
    - geraneum 4y ago
      That’s actually correct but an overfitted definition for learning. It holds certain hidden assumptions (i.e physical grounding) of the learner being human which makes it inapplicable to an LLM. As in a self driving car which passes a driving exam but fails to drive effectively freely in the city (it’s not an LLM but relevant in this context). You have to admit when you work with this tech that something fundamental is missing in how they perform.
      - willbudd 4y ago
        > That’s actually correct but an overfitted definition for learning. It holds certain hidden assumptions (i.e physical grounding) of the learner being human which makes it inapplicable to an LLM. Inapplicable why exactly? Because you say so? Logic isn't magic. Nor is learning. No (external) grounding is required either: iteratively eliminating inconsistent world models is all you need to converge toward a model of the real world. Nothing especially human or inhuman about it. LLM architecture may not be able to represent a fully recursive backtracking truth maintenance system, but it evidently managed to learn a pretty decent approximation anyway.
- willbudd 4y ago
  That's a brilliant example. Thanks for sharing. It demonstrates in a very straightforward way that LLMs are capable of learning (and applying) relationships at the level of abstraction of (at least) 1st order logic. It implies that during training, it learned the facts that Elizabeth is queen of the UK, and that Charles is its crown prince; but _also_ the logical rule <IF die(monarch) AND alive(heir_to_the_throne) => transform(heir_to_the_throne, monarch) AND transform(monarch, former_monarch)>, or at least something along those lines that allows similarly powerful entailment. And that in addition to the ability to substitute/reify with the input sequence at inference runtime. Would be nice to see a rigorous survey of its logical capabilities given some complex Prolog/Datalog/etc knowledge-base as baseline.
  - wokkel 4y ago
    No it does not: if you google this and restrict the time to before 2021 (the learning cutoff date) you will find the same answer. Without having access to the training data it's impossible to tell what we seeing.
    - willbudd 4y ago
      > No it does not: if you google this and restrict the time to before 2021 (the learning cutoff date) you will find the same answer. Not entirely sure what you mean, but ...show me? Why not just share a link instead of making empty assertions?
      - nobody81147 4y ago
        Here’s a Quora thread from 4 years ago: https://www.quora.com/Once-Queen-Elizabeth-dies-will-Prince-Charles-become-King-of-Australia-King-of-Canada-and-King-of-New-Zealand https://www.quora.com/Once-Queen-Elizabeth-dies-will-Prince-... There are loads of articles and discussions online speculating about what “will” happen when Queen Elizabeth dies. When you have a very, very, very large corpus to sample from, it can look a lot like reasoning.
        4y ago
tomohelix 4y ago
>the output is conditional on the input Uh, isn't that how it should always be? If the output isn't conditional on the input, then it is basically noise or worthless. How the conditionals are setup are the key but not really sure how does this relate to the point you are making.
- kgwgk 4y ago
  According to the parent comment anyone who says "LLMs are stochastic parrots" is ignorant of the effect that changing the input “filling the context with examples” has on the output.
  - nl 4y ago
    That isn't a fair summary of what I'm saying. People who say "LLMs are stochastic parrots" may well be aware of effects of conditioning the input. This in itself does not completely describe the capabilities of a LLM, since their ability to learn and use those new facts to override their previous "beliefs" is not what you'd expect from mere stochastic behavior. My point is that "stochastic parrot" believers are unaware of this.
    - int_19h 4y ago
      They are aware. The thing about the "stochastic parrot" argument is that it can be pushed as far as one is willing to push it; any behavior by an LLM can be described in those terms with enough handwaving. The practical limit seems to be where one would have to say that humans are also "stochastic parrots", presumably because that defeats the point of making such an argument in the first place.
      - mdp2021 4y ago
        You are completely disregarding fairness of judgement based on noted output. Roger was called a parrot because it repeated things apparently without checking; humans are not necessarily parrots, as some do not do that and reflect as expected.
quickthrower2 4y ago
Output conditional on input (training data included) where input is, well, fucking huge and the training cycles ridiculous!
newswasboring 4y ago
> Some of them -at least- understand that the output is conditional on the input I don't get what you are trying to say. That's like a property of any useful system.

by kgwgk 4y ago

> (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) Do you? Some of them -at least- understand that the output is conditional on the input.

nl 4y ago

"Conditioned" with the ability to introduce new facts and have the model infer impacts on them. As an example, tell chatGPT that the Queen of England died (which occurred after the data cut off) and then ask it who the head of state of Australia is. It's able to infer the head of state of Australia is now Charles III (and gives a good explanation of how this is mostly ceremonial.) See https://twitter.com/nlothian/status/1646699207506685953 https://twitter.com/nlothian/status/1646699207506685953 At some point the word "stochastic" doesn't really capture that behavior in any useful sense.

geraneum 4y ago
Interestingly, the example provided is not necessarily indicative of the ability to infer the actual physical transfer of power. It’s possible that the terms “After Elisabeth, Charles will be king” are accompanied with “Elisabeth” and “Die” or “Death” in the same context before cut off (Wikipedia, news papers, tabloids). You cannot draw any scientifically sound conclusions without knowing what was actually in the training data and use a control group/instance.
- kmac_ 4y ago
  Yeah, I tested that, but asked what happens if both Charles and William abdicate - got this result: "However, if both Charles III and his son abdicate, it is not clear who would become the head of state in Australia. There is a possibility that Australia could become a republic and have its first Australian head of state." Clearly, it didn't handle quite straightforward inferring (Prince George of Wales, Williams son).
  - mike_hearn 4y ago
    Did you tell it that it was being given a logic test? Unfortunately a lot of logic questions can lead to outcomes like this, where the "obvious" or "real" answer is not the one implied by strict logical reasoning. Then when people give answers useful in the real world, they are marked as having failed.
  - willbudd 4y ago
    Or maybe it did. Who knows. If "both Charles III and his son abdicate" could well be considered indicative of some large upheaval or scandal, at which point it is entirely conceivable that the Australian electorate reaches a consensus on becoming a republic. The way that is phrased doesn't seem like a straightforward proposition to me at all.
    - kmac_ 4y ago
      I verified that it has all required facts (line of succession, current circumstances). I managed to get the right answer when got everything in context, but it failed again when all three abdicate (same context). Prince Harry was indicated once. I tested GPT a lot in other domains, what I found that as long the information explicitly exists (connection between facts) then the responses are fine. I assume that if GPT will reach the state where it can infer new facts, we will be flooded with discoveries that require cross domain knowledge. Nothing like that happened yet.
      - nullsense 4y ago
        >Nothing like that happened yet. Feels like we're only one paper away now that the context window has absolutely ballooned.
        geraneum 4y ago
        This reminded me of "Two Minute Papers" YouTube channel where in most of the videos he always, "Two papers down the line and...". I think ML/AI is the main topic of his videos. Interesting stuff.
- willbudd 4y ago
  Au contraire. Learning an abstract logical relationship such as line of succession during training, and then applying substitution/reification during inference to deduce the new factual clause that Charles is king of the UK is exactly what it means to learn something new. It's just a pity it can't memorize this fact at inference time, and that won't be able to reproduce it as soon as the information about the queen's death slides outside of the context window.
  - crakenzak 4y ago
    I'm curious if future developments in LLMs will enable them to extract significant/noteworthy info from their context window and incorporate it into their underlying understanding by adjusting their weights accordingly. This could be an important step towards achieving AGI, since it closely mirrors how humans learn imo. Humans continually update their foundational understanding by assimilating vital information from their "context window" and dumping irrelevant noise. If LLMs could emulate this, it would be a huge win. Overall, very exciting area of research!
  - geraneum 4y ago
    That’s actually correct but an overfitted definition for learning. It holds certain hidden assumptions (i.e physical grounding) of the learner being human which makes it inapplicable to an LLM. As in a self driving car which passes a driving exam but fails to drive effectively freely in the city (it’s not an LLM but relevant in this context). You have to admit when you work with this tech that something fundamental is missing in how they perform.
    - willbudd 4y ago
      > That’s actually correct but an overfitted definition for learning. It holds certain hidden assumptions (i.e physical grounding) of the learner being human which makes it inapplicable to an LLM. Inapplicable why exactly? Because you say so? Logic isn't magic. Nor is learning. No (external) grounding is required either: iteratively eliminating inconsistent world models is all you need to converge toward a model of the real world. Nothing especially human or inhuman about it. LLM architecture may not be able to represent a fully recursive backtracking truth maintenance system, but it evidently managed to learn a pretty decent approximation anyway.
willbudd 4y ago
That's a brilliant example. Thanks for sharing. It demonstrates in a very straightforward way that LLMs are capable of learning (and applying) relationships at the level of abstraction of (at least) 1st order logic. It implies that during training, it learned the facts that Elizabeth is queen of the UK, and that Charles is its crown prince; but _also_ the logical rule <IF die(monarch) AND alive(heir_to_the_throne) => transform(heir_to_the_throne, monarch) AND transform(monarch, former_monarch)>, or at least something along those lines that allows similarly powerful entailment. And that in addition to the ability to substitute/reify with the input sequence at inference runtime. Would be nice to see a rigorous survey of its logical capabilities given some complex Prolog/Datalog/etc knowledge-base as baseline.
- wokkel 4y ago
  No it does not: if you google this and restrict the time to before 2021 (the learning cutoff date) you will find the same answer. Without having access to the training data it's impossible to tell what we seeing.
  - willbudd 4y ago
    > No it does not: if you google this and restrict the time to before 2021 (the learning cutoff date) you will find the same answer. Not entirely sure what you mean, but ...show me? Why not just share a link instead of making empty assertions?
    - nobody81147 4y ago
      Here’s a Quora thread from 4 years ago: https://www.quora.com/Once-Queen-Elizabeth-dies-will-Prince-Charles-become-King-of-Australia-King-of-Canada-and-King-of-New-Zealand https://www.quora.com/Once-Queen-Elizabeth-dies-will-Prince-... There are loads of articles and discussions online speculating about what “will” happen when Queen Elizabeth dies. When you have a very, very, very large corpus to sample from, it can look a lot like reasoning.
      - 4y ago

tomohelix 4y ago

>the output is conditional on the input Uh, isn't that how it should always be? If the output isn't conditional on the input, then it is basically noise or worthless. How the conditionals are setup are the key but not really sure how does this relate to the point you are making.

kgwgk 4y ago
According to the parent comment anyone who says "LLMs are stochastic parrots" is ignorant of the effect that changing the input “filling the context with examples” has on the output.
- nl 4y ago
  That isn't a fair summary of what I'm saying. People who say "LLMs are stochastic parrots" may well be aware of effects of conditioning the input. This in itself does not completely describe the capabilities of a LLM, since their ability to learn and use those new facts to override their previous "beliefs" is not what you'd expect from mere stochastic behavior. My point is that "stochastic parrot" believers are unaware of this.
  - int_19h 4y ago
    They are aware. The thing about the "stochastic parrot" argument is that it can be pushed as far as one is willing to push it; any behavior by an LLM can be described in those terms with enough handwaving. The practical limit seems to be where one would have to say that humans are also "stochastic parrots", presumably because that defeats the point of making such an argument in the first place.
    - mdp2021 4y ago
      You are completely disregarding fairness of judgement based on noted output. Roger was called a parrot because it repeated things apparently without checking; humans are not necessarily parrots, as some do not do that and reflect as expected.

quickthrower2 4y ago

Output conditional on input (training data included) where input is, well, fucking huge and the training cycles ridiculous!

newswasboring 4y ago

> Some of them -at least- understand that the output is conditional on the input I don't get what you are trying to say. That's like a property of any useful system.