5 ms·

GPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It

GPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It cannot continuously learn and remember like biological organisms since its weights are frozen.

KRAKRISMOTT 3y ago
So we still have catastrophic forgetting, just on a slightly longer scale.
anon291 3y ago
Except it cannot conceptualize reasonably. It is utterly incapable of symbolic thinking in a general sense.
- LASR 3y ago
  It’s incapable of thinking. But what it’s surprisingly good at is symbolic reasoning similar to human common sense. Most people forget that there is no memory or thought independent of whatever the output it generates. But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human. People asking ChatGPT for word counts and sighing at how wrong it is. Yeah of course, it has no counter variables to hold numbers and increment them. It has no memory. But when you ask it to generate a numbered list of all the words in a passage and then output the word count, it gets it right every time. Because you basically gave it a memory by encoding the counter in the generated output.
  - anon291 3y ago
    > But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human I have tried this so many times for novel problems and no matter what I do, it eventually recognizes what we're trying to solve and determines it's unsolvable. Since nothing new comes out I can only attribute the times it does seem to do symbolic manipulation as regurgitation.
- nl 3y ago
  No reason why Toolformer+GPT couldn't use a symbolic reasoning program. Symbolic reasoning programs are much better than humans. Probably bugs will happen in the interface between human language and the symbolic reasoning. But that happens with humans too.
  - macrolime 3y ago
    GPT-4 has toolformer capabilities as emergent properties, so using it in combination with toolformer is really unnecessary.
    - nl 3y ago
      This is true too. Even chatGPT can learn some tool use in-context. See https://twitter.com/minosvasilias/status/1627076214639976449 https://twitter.com/minosvasilias/status/1627076214639976449
      - int_19h 3y ago
        GPT-4 writes some wicked complicated (but correct!) SQL if given a schema and a relevant task: https://gist.github.com/int19h/428cea1d87dfc389b99de1b79727f00f https://gist.github.com/int19h/428cea1d87dfc389b99de1b79727f... What I found really amazing about this particular experiment is that the schema I gave it didn't contain any information that could be used to query for things like distances between places, and yet it came up with the idea of using settlements' culture as a proxy to determine "border fiefs", completely unprompted (and yes, it actually is a very effective proxy for this particular case!). I wonder now what it would do with Prolog - or maybe Datalog for simplicity? - although that might depend on how much of it there is in the training data. Do you know if anyone tried it yet?
        nullsense 3y ago
        That's wild.
  - anon291 3y ago
    I work in this field and have tried many things. Sure.. GPT + something else can do symbolic reasoning almost on par with a middle schooler (it gets things wrongs often, not in the way people do either, but just confidently incorrect). However, GPT + something is not a transformer model, which is what the original comment was about.

KRAKRISMOTT 3y ago

So we still have catastrophic forgetting, just on a slightly longer scale.

anon291 3y ago

Except it cannot conceptualize reasonably. It is utterly incapable of symbolic thinking in a general sense.

LASR 3y ago
It’s incapable of thinking. But what it’s surprisingly good at is symbolic reasoning similar to human common sense. Most people forget that there is no memory or thought independent of whatever the output it generates. But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human. People asking ChatGPT for word counts and sighing at how wrong it is. Yeah of course, it has no counter variables to hold numbers and increment them. It has no memory. But when you ask it to generate a numbered list of all the words in a passage and then output the word count, it gets it right every time. Because you basically gave it a memory by encoding the counter in the generated output.
- anon291 3y ago
  > But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human I have tried this so many times for novel problems and no matter what I do, it eventually recognizes what we're trying to solve and determines it's unsolvable. Since nothing new comes out I can only attribute the times it does seem to do symbolic manipulation as regurgitation.
nl 3y ago
No reason why Toolformer+GPT couldn't use a symbolic reasoning program. Symbolic reasoning programs are much better than humans. Probably bugs will happen in the interface between human language and the symbolic reasoning. But that happens with humans too.
- macrolime 3y ago
  GPT-4 has toolformer capabilities as emergent properties, so using it in combination with toolformer is really unnecessary.
  - nl 3y ago
    This is true too. Even chatGPT can learn some tool use in-context. See https://twitter.com/minosvasilias/status/1627076214639976449 https://twitter.com/minosvasilias/status/1627076214639976449
    - int_19h 3y ago
      GPT-4 writes some wicked complicated (but correct!) SQL if given a schema and a relevant task: https://gist.github.com/int19h/428cea1d87dfc389b99de1b79727f00f https://gist.github.com/int19h/428cea1d87dfc389b99de1b79727f... What I found really amazing about this particular experiment is that the schema I gave it didn't contain any information that could be used to query for things like distances between places, and yet it came up with the idea of using settlements' culture as a proxy to determine "border fiefs", completely unprompted (and yes, it actually is a very effective proxy for this particular case!). I wonder now what it would do with Prolog - or maybe Datalog for simplicity? - although that might depend on how much of it there is in the training data. Do you know if anyone tried it yet?
      - nullsense 3y ago
        That's wild.
- anon291 3y ago
  I work in this field and have tried many things. Sure.. GPT + something else can do symbolic reasoning almost on par with a middle schooler (it gets things wrongs often, not in the way people do either, but just confidently incorrect). However, GPT + something is not a transformer model, which is what the original comment was about.