5 ms·

What's the difference between a really good language model and an AGI (i.e. Chinese room problem)?

by shawnz 5y ago

simonh 5y ago
An AGI would need to comprehend and manipulate meanings; have a persistent memory; be able to create multiple models of a situation, consider scenarios, analyse and criticise them; it would need a persistent memory and be able to learn facts and use them to infer novel information. Language models like GPT don’t need any of that, and have no mechanism to generate such capabilities. This is why it’s possible to reliably trip GPT-3 up in just a few interactions. You simply test for these capabilities and it immediately falls flat on its face.
- shawnz 5y ago
  The only thing here that current language models objectively don't have is a persistent memory. GPT-3 already uses 2048 tokens of context as input, but it's not persistent. An improved language model would likely need to have persistent memory. There's no obvious reason why the rest of that stuff can't occur as emergent behaviours. It was a rhetorical question: there is no possible distinction between a highly sophisticated language model and an AGI. If a language model can't produce all the same answers that an AGI can produce, then it just hasn't reached the level of sophistication necessary to do that.
  - slibhb 5y ago
    > It was a rhetorical question: there is no possible distinction between a highly sophisticated language model and an AGI. If a language model can't produce all the same answers that an AGI can produce, then it just hasn't reached the level of sophistication necessary to do that. This is a weirdly dogmatic position. I would think that an AGI can start a conversation. Can a language model?
    - shawnz 5y ago
      Software can't execute itself. It must be executed by an external agent and given some kind of input. That will be true regardless of whether you consider the software to be a true AGI or a mere language model. So what does it mean exactly to "start a conversation"? Are you saying that it should be able to give a useful output with a blank input? I don't see why a language model can't do that.
      - simonh 5y ago
        In principle it might be possible for a language model to develop true general intelligence. However there's no mechanism in GTP and similar systems that makes such a development necessary. In fact it seems likely that the development of internal structures that might develop into the faculties needed by an AGI would, in their intermediate state, make it worse as a language model. Evolution sometimes faces such development gaps, where to develop a capability that would ultimately grant improvements, it would have to go through intermediate phases that would make it worse. Language models are optimised in a very specific way and trained to solve a very narrow problem, compared to the problems faced by physical beings. So while I cant honestly say no, a language model can never do that, equally there's no actual reason to believe one ever would.
        shawnz 5y ago
        Was there a mechanism in the evolutionary system of biological life on Earth that makes the development of higher intelligence necessary? How did we develop higher intelligence if it's not possible to get there though a series of progressive improvements? Furthermore I would argue that intelligence is a necessary development in order to create the most sophisticated possible language model, for the reasons described above. Anything less will not be able to perfectly emulate the conversations of other intelligent beings.
        simonh 5y ago
        I agree the best possible language model would possess general AI, but you can't go from no intelligence to general AI in one step. It seems likely to me that the intermediate levels of intelligence such a system might develop early on might actually make it a worse language model. The evolutionary path from dumb language model to language model with human level AI might well lie through an unhappy valley in which the fledgling AI makes more mistakes and is easier to catch out than a pure language model. In which case, a system would never take that evolutionary path. This is a problem with systems optimising towards a single specific function. It constrains their optimisation paths.
astrange 5y ago
That depends on how you define AGI. You'd probably want to give it a few more senses, so it can have unwritten qualia, or at least so you can show it pictures. More importantly, current ML models can't decide to think about things, because they always spend the same processing time on everything. GPT is recurrent and you do sort of feed its output through it, but there isn't a global context it uses for the whole document.

What's the difference between a really good language model and an AGI (i.e. Chinese room problem)?

by shawnz 5y ago

What's the difference between a really good language model and an AGI (i.e. Chinese room problem)?

An AGI would need to comprehend and manipulate meanings; have a persistent memory; be able to create multiple models of a situation, consider scenarios, analyse and criticise them; it would need a persistent memory and be able to learn facts and use them to infer novel information. Language models like GPT don’t need any of that, and have no mechanism to generate such capabilities. This is why it’s possible to reliably trip GPT-3 up in just a few interactions. You simply test for these capabilities and it immediately falls flat on its face.

shawnz 5y ago
The only thing here that current language models objectively don't have is a persistent memory. GPT-3 already uses 2048 tokens of context as input, but it's not persistent. An improved language model would likely need to have persistent memory. There's no obvious reason why the rest of that stuff can't occur as emergent behaviours. It was a rhetorical question: there is no possible distinction between a highly sophisticated language model and an AGI. If a language model can't produce all the same answers that an AGI can produce, then it just hasn't reached the level of sophistication necessary to do that.
- slibhb 5y ago
  > It was a rhetorical question: there is no possible distinction between a highly sophisticated language model and an AGI. If a language model can't produce all the same answers that an AGI can produce, then it just hasn't reached the level of sophistication necessary to do that. This is a weirdly dogmatic position. I would think that an AGI can start a conversation. Can a language model?
  - shawnz 5y ago
    Software can't execute itself. It must be executed by an external agent and given some kind of input. That will be true regardless of whether you consider the software to be a true AGI or a mere language model. So what does it mean exactly to "start a conversation"? Are you saying that it should be able to give a useful output with a blank input? I don't see why a language model can't do that.
    - simonh 5y ago
      In principle it might be possible for a language model to develop true general intelligence. However there's no mechanism in GTP and similar systems that makes such a development necessary. In fact it seems likely that the development of internal structures that might develop into the faculties needed by an AGI would, in their intermediate state, make it worse as a language model. Evolution sometimes faces such development gaps, where to develop a capability that would ultimately grant improvements, it would have to go through intermediate phases that would make it worse. Language models are optimised in a very specific way and trained to solve a very narrow problem, compared to the problems faced by physical beings. So while I cant honestly say no, a language model can never do that, equally there's no actual reason to believe one ever would.
      - shawnz 5y ago
        Was there a mechanism in the evolutionary system of biological life on Earth that makes the development of higher intelligence necessary? How did we develop higher intelligence if it's not possible to get there though a series of progressive improvements? Furthermore I would argue that intelligence is a necessary development in order to create the most sophisticated possible language model, for the reasons described above. Anything less will not be able to perfectly emulate the conversations of other intelligent beings.
        simonh 5y ago
        I agree the best possible language model would possess general AI, but you can't go from no intelligence to general AI in one step. It seems likely to me that the intermediate levels of intelligence such a system might develop early on might actually make it a worse language model. The evolutionary path from dumb language model to language model with human level AI might well lie through an unhappy valley in which the fledgling AI makes more mistakes and is easier to catch out than a pure language model. In which case, a system would never take that evolutionary path. This is a problem with systems optimising towards a single specific function. It constrains their optimisation paths.

astrange 5y ago

That depends on how you define AGI. You'd probably want to give it a few more senses, so it can have unwritten qualia, or at least so you can show it pictures. More importantly, current ML models can't decide to think about things, because they always spend the same processing time on everything. GPT is recurrent and you do sort of feed its output through it, but there isn't a global context it uses for the whole document.