5 ms·

Fundamentally, it may not matter. Just like biological brains are intelligent in some way - an extremely large Language Model may be intelligent in some way.

by lumost 4y ago

Fundamentally, it may not matter. Just like biological brains are intelligent in some way - an extremely large Language Model may be intelligent in some way. If current scaling trends hold, it may not be possible to distinguish a hypothetical GPT-6 vs. a human over reasonably sized conversations/text production tasks. Granted, such a GPT-6 model may be larger than any commercial application could support, and possibly beyond the range of research oriented funding as of today.

astrange 4y ago
If GPT-6 was only larger, it would still only know the things in its training set, whereas a human can take arbitrary time to think, can go out and perform experiments, can Google/GPT things themselves, etc. GPT can never be an intelligent agent because it’s not an agent. Also, the increased training time to produce a really big model makes it less like an intelligence, which has low marginal cost to learn something new. The reason GPT looks intelligent is because you’re the one providing that; it’s just a big blob of inanimate knowledge.
- lumost 4y ago
  While a LLM doesn't have the ability to perform real world experiments, it remains to be seen whether an LLM can "reason" over a contextual input, combined with its own outputs. There is no fundamental reason it can't, and no fundamental reason to think that it can. Anecdotal observation from AI Dungeon and similar tools would put this as a "maybe".
  - astrange 4y ago
    I believe it can do that, but there's two periods where it can "think". 1. During training time, which is very expensive in terms of input text and money - plus, model training often completely fails which is why you have to do checkpoints, hyper parameter searches, etc. 2. During inference, but it doesn't have arbitrary thinking and memory abilities like a human does, it only has however much space is in the input tokens + its weights. There are thoughts that can't fit even in an optimal model. GPT isn't just a model, it also has that sampling system which is a regular computer program (as opposed to a learned one), which does give it extra abilities.
- dane-pgp 4y ago
  > whereas a human can take arbitrary time to think, Are you saying that GPT-6 would be less intelligent than a human, because we could give it an arbitrary deadline within which to answer the question, while not restricting a human that way? That doesn't sound like a fair test. I'm probably misrepresenting your position here, and maybe what you're trying to say is "If you gave both GPT-6 and a human as long as they needed go away and think about a problem, the human could use that time to sit and think creatively about the problem, whereas GPT-6 would either know the answer, or not". That's a reasonable intuition and a fair experiment, but I think it hinges on the question of "What does a human do when they go away and think about a problem?". Could it be that they are just clearing their mind of other distractions, and trying to look at the problem from multiple perspectives, which is something that a machine could do more efficiently? In other words, the fact that a human might need to go away and take arbitrary time to think about a problem is a sign that it is the human that lacks efficient general intelligence, not this GPT-6. > can go out and perform experiments, Again, it seems unfair to compare GPT-6 in a deliberately sealed box to a human who is allowed to go away and talk to other humans and use tools and resources that we wouldn't trust GPT-6 with. Putting it another way, if you met someone who had never performed an experiment before, would you think they were not intelligent? > can Google/GPT things themselves, etc. This is an almost Kafkaesque criticism. Are you saying GPT-6 isn't intelligent because it doesn't have access to GPT-6's intelligence, but humans do? Or the fact that humans can go away and use Google means they are more intelligent than GPT-6 which has Google search results built into its "brain" in the first place? > it’s just a big blob of inanimate knowledge. You're just a big blob of inanimate knowledge. Ahem, sorry, what I mean is: Why does knowledge have to be "animate" to be legitimate?
  - dxbydt 4y ago
    > seems unfair to compare GPT-6 in a box to a human who is allowed to talk to other humans Are there any efforts to make one gpt talk to another ? Can the older versions eg. gpt1 and gpt2, talk to each other, and eventually become more performant than gpt3 ?
    - astrange 4y ago
      Probably not; you could try gluing the models together but it's unlikely it'd improve them and I'm not sure they have the same architecture. Also, even the small/medium GPT3 models are much much worse than the largest one (DaVinci). On the other hand, the lottery ticket hypothesis says that every good big network contains a good or better smaller network you could extract, if only you knew where it was. (So the reason big models are good is there's more opportunities for the smaller models to appear.)
- version_five 4y ago
  >GPT can never be an intelligent agent because it’s not an agent For all the handwaving people like to do, this is the bottom line, and it's not clear to me why researchers haven't spent more time clearly formulating this, vs a lot of proving that various tasks and games don't actually require intelligence. As best I can formulate it, "AI" can't "escape" its programming, and just a neural network even less so. There is no mental model of the world to rationalize against, and if there was, it would be pre-defined or coded in some way that it still couldn't be escaped. Whatever tasks a ML model has been deliberately programmed with just have nothing in common with intelligence. Scaling doesnt change that. I subscribe to the idea that with ML we may have understood how some of the sensory mechanisms work, like visual recognition, but we're not actually closer to the intelligence part.
  - qrian 4y ago
    The same author has a story describing how AI might escape: https://www.gwern.net/fiction/Clippy https://www.gwern.net/fiction/Clippy Not that I agree with the author but 'AIs are sandboxed and resourced constrained therefore safe' doesn't feel very convincing to me after reading that story.
  - d110af5ccf 4y ago
    > "AI" can't "escape" its programming I don't think you can back that claim up. ML models as often used can't escape their programming. But I don't see any fundamental reason you can't loop one back on itself. Isn't this already done to some extent by using a model to assess its own output versus the current environmental state? Recurrent models combined with reinforcement learning actually look a lot like what you're claiming can't be done. > There is no mental model of the world to rationalize against There's a theory and some associated research about a model that performs well necessitating an internal model that accurately reflects the environment in which it operates. Or something like that. Sorry I can't recall the paper off the top of my head. > Whatever tasks a ML model has been deliberately programmed with Yeah, no, that's not the case. See the entire field of reinforcement learning.
    - astrange 4y ago
- visarga 4y ago
  > GPT can never be an intelligent agent because it’s not an agent. It's already almost there. GPT-3 has the ability to generate short or long. If you ask it "let's think step by step" it generates a chain of thought, dividing the task into steps and doing them one by one, almost like an agent. On the other hand, if you strap GPT-3 on an agent in a 3D Home Environment it gains the ability to perform multi-step tasks zero shot. So it's quite ready for it. If you train a reward model based on supervised data, you can finetune GPT-3 with it. This is how Instruct GPT-3 was created, through text-based RL. I'd say GPT-3 is very close to being an agent, it just needs to be put in a body in an environment and trained a bit longer.
  - Vetch 4y ago
    Being an agent isn't about how complex the model is, nor about how much RL is used. Bacteria is an agent. Sponges and Jellyfish are agents. It's about if the model engages in exploration and or generates inferences in service of non-trivial control problems. > On the other hand, if you strap GPT-3 on an agent in a 3D Home Environment GPT-3, just as any probability distribution, can inform an agent's actions but that doesn't make it one. WebGPT3 is an agent though.
    - astrange 4y ago
      Yes, WebGPT3 is more like intelligence research than GPT is. So is DeepMind's Perceiver. WebGPT does need safeguards because eg it could start querying websites and end up hitting their /delete APIs. But that's not really an "AI alignment" problem; same thing happens with GoogleBot.
    - visarga 4y ago
      Yes, RL is mostly about learning from rewards (Reward is Enough - https://www.deepmind.com/publications/reward-is-enough https://www.deepmind.com/publications/reward-is-enough) and GPT-3 has been used so far only to augment agents, not to do reward based learning. The current obstacle is speed - it needs to run in real time. Maybe we need more efficient algoritms, maybe we'll have better hardware. It will come soon enough.
- gwern 4y ago
  > If GPT-6 was only larger, it would still only know the things in its training set, whereas a human can take arbitrary time to think, can go out and perform experiments, can Google/GPT things themselves, etc. A GPT model already has many ways of thinking and self-distilling; for example, it can give itself a lot more time to think using 'inner monologue' techniques, which make it capable of calculating out step by step, using external tools like Python etc: https://www.gwern.net/docs/ai/gpt/inner-monologue/index https://www.gwern.net/docs/ai/gpt/inner-monologue/index A GPT model can also already Google things, see WebGPT https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/ (itself merely one entry in a rapidly-expanding area of retrieval models https://www.gwern.net/docs/ai/retrieval/index https://www.gwern.net/docs/ai/retrieval/index ). Enabling them to use web tools is likewise an active and increasingly successful area of research https://arxiv.org/abs/2202.08137#deepmind https://arxiv.org/abs/2202.08137#deepmind , with startups forming for that purpose: https://www.adept.ai/post/introducing-adept https://www.adept.ai/post/introducing-adept It can't 'run experiments', but the setting of reinforcement learning would allow that, and Decision Transformer (literally just a GPT trained on RL data and then given a prompt) is one of the more exciting directions - you may have seen Gato https://www.deepmind.com/publications/a-generalist-agent https://www.deepmind.com/publications/a-generalist-agent but have you also seen Multi-Game Transformer https://sites.google.com/view/multi-game-transformers https://sites.google.com/view/multi-game-transformers ? Oh, that's just a single agent and you're not worried? Well that's fine because DT is so general it's almost trivial to make it do multi-agent RL as well https://arxiv.org/abs/2112.02845 https://arxiv.org/abs/2112.02845 https://arxiv.org/abs/2205.14953 https://arxiv.org/abs/2205.14953 Things are moving fast, and few people know all of the things that a GPT can do. (By the way, did you know that this 'big blob of inanimate knowledge' also knows what it doesn't know and can give well-calibrated predictions of accuracy https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 ?)

by lumost 4y ago

astrange 4y ago

If GPT-6 was only larger, it would still only know the things in its training set, whereas a human can take arbitrary time to think, can go out and perform experiments, can Google/GPT things themselves, etc. GPT can never be an intelligent agent because it’s not an agent. Also, the increased training time to produce a really big model makes it less like an intelligence, which has low marginal cost to learn something new. The reason GPT looks intelligent is because you’re the one providing that; it’s just a big blob of inanimate knowledge.

lumost 4y ago
While a LLM doesn't have the ability to perform real world experiments, it remains to be seen whether an LLM can "reason" over a contextual input, combined with its own outputs. There is no fundamental reason it can't, and no fundamental reason to think that it can. Anecdotal observation from AI Dungeon and similar tools would put this as a "maybe".
- astrange 4y ago
  I believe it can do that, but there's two periods where it can "think". 1. During training time, which is very expensive in terms of input text and money - plus, model training often completely fails which is why you have to do checkpoints, hyper parameter searches, etc. 2. During inference, but it doesn't have arbitrary thinking and memory abilities like a human does, it only has however much space is in the input tokens + its weights. There are thoughts that can't fit even in an optimal model. GPT isn't just a model, it also has that sampling system which is a regular computer program (as opposed to a learned one), which does give it extra abilities.
dane-pgp 4y ago
> whereas a human can take arbitrary time to think, Are you saying that GPT-6 would be less intelligent than a human, because we could give it an arbitrary deadline within which to answer the question, while not restricting a human that way? That doesn't sound like a fair test. I'm probably misrepresenting your position here, and maybe what you're trying to say is "If you gave both GPT-6 and a human as long as they needed go away and think about a problem, the human could use that time to sit and think creatively about the problem, whereas GPT-6 would either know the answer, or not". That's a reasonable intuition and a fair experiment, but I think it hinges on the question of "What does a human do when they go away and think about a problem?". Could it be that they are just clearing their mind of other distractions, and trying to look at the problem from multiple perspectives, which is something that a machine could do more efficiently? In other words, the fact that a human might need to go away and take arbitrary time to think about a problem is a sign that it is the human that lacks efficient general intelligence, not this GPT-6. > can go out and perform experiments, Again, it seems unfair to compare GPT-6 in a deliberately sealed box to a human who is allowed to go away and talk to other humans and use tools and resources that we wouldn't trust GPT-6 with. Putting it another way, if you met someone who had never performed an experiment before, would you think they were not intelligent? > can Google/GPT things themselves, etc. This is an almost Kafkaesque criticism. Are you saying GPT-6 isn't intelligent because it doesn't have access to GPT-6's intelligence, but humans do? Or the fact that humans can go away and use Google means they are more intelligent than GPT-6 which has Google search results built into its "brain" in the first place? > it’s just a big blob of inanimate knowledge. You're just a big blob of inanimate knowledge. Ahem, sorry, what I mean is: Why does knowledge have to be "animate" to be legitimate?
- dxbydt 4y ago
  > seems unfair to compare GPT-6 in a box to a human who is allowed to talk to other humans Are there any efforts to make one gpt talk to another ? Can the older versions eg. gpt1 and gpt2, talk to each other, and eventually become more performant than gpt3 ?
  - astrange 4y ago
    Probably not; you could try gluing the models together but it's unlikely it'd improve them and I'm not sure they have the same architecture. Also, even the small/medium GPT3 models are much much worse than the largest one (DaVinci). On the other hand, the lottery ticket hypothesis says that every good big network contains a good or better smaller network you could extract, if only you knew where it was. (So the reason big models are good is there's more opportunities for the smaller models to appear.)
version_five 4y ago
>GPT can never be an intelligent agent because it’s not an agent For all the handwaving people like to do, this is the bottom line, and it's not clear to me why researchers haven't spent more time clearly formulating this, vs a lot of proving that various tasks and games don't actually require intelligence. As best I can formulate it, "AI" can't "escape" its programming, and just a neural network even less so. There is no mental model of the world to rationalize against, and if there was, it would be pre-defined or coded in some way that it still couldn't be escaped. Whatever tasks a ML model has been deliberately programmed with just have nothing in common with intelligence. Scaling doesnt change that. I subscribe to the idea that with ML we may have understood how some of the sensory mechanisms work, like visual recognition, but we're not actually closer to the intelligence part.
- qrian 4y ago
  The same author has a story describing how AI might escape: https://www.gwern.net/fiction/Clippy https://www.gwern.net/fiction/Clippy Not that I agree with the author but 'AIs are sandboxed and resourced constrained therefore safe' doesn't feel very convincing to me after reading that story.
- d110af5ccf 4y ago
  > "AI" can't "escape" its programming I don't think you can back that claim up. ML models as often used can't escape their programming. But I don't see any fundamental reason you can't loop one back on itself. Isn't this already done to some extent by using a model to assess its own output versus the current environmental state? Recurrent models combined with reinforcement learning actually look a lot like what you're claiming can't be done. > There is no mental model of the world to rationalize against There's a theory and some associated research about a model that performs well necessitating an internal model that accurately reflects the environment in which it operates. Or something like that. Sorry I can't recall the paper off the top of my head. > Whatever tasks a ML model has been deliberately programmed with Yeah, no, that's not the case. See the entire field of reinforcement learning.
  - astrange 4y ago
visarga 4y ago
> GPT can never be an intelligent agent because it’s not an agent. It's already almost there. GPT-3 has the ability to generate short or long. If you ask it "let's think step by step" it generates a chain of thought, dividing the task into steps and doing them one by one, almost like an agent. On the other hand, if you strap GPT-3 on an agent in a 3D Home Environment it gains the ability to perform multi-step tasks zero shot. So it's quite ready for it. If you train a reward model based on supervised data, you can finetune GPT-3 with it. This is how Instruct GPT-3 was created, through text-based RL. I'd say GPT-3 is very close to being an agent, it just needs to be put in a body in an environment and trained a bit longer.
- Vetch 4y ago
  Being an agent isn't about how complex the model is, nor about how much RL is used. Bacteria is an agent. Sponges and Jellyfish are agents. It's about if the model engages in exploration and or generates inferences in service of non-trivial control problems. > On the other hand, if you strap GPT-3 on an agent in a 3D Home Environment GPT-3, just as any probability distribution, can inform an agent's actions but that doesn't make it one. WebGPT3 is an agent though.
  - astrange 4y ago
    Yes, WebGPT3 is more like intelligence research than GPT is. So is DeepMind's Perceiver. WebGPT does need safeguards because eg it could start querying websites and end up hitting their /delete APIs. But that's not really an "AI alignment" problem; same thing happens with GoogleBot.
  - visarga 4y ago
    Yes, RL is mostly about learning from rewards (Reward is Enough - https://www.deepmind.com/publications/reward-is-enough https://www.deepmind.com/publications/reward-is-enough) and GPT-3 has been used so far only to augment agents, not to do reward based learning. The current obstacle is speed - it needs to run in real time. Maybe we need more efficient algoritms, maybe we'll have better hardware. It will come soon enough.
gwern 4y ago
> If GPT-6 was only larger, it would still only know the things in its training set, whereas a human can take arbitrary time to think, can go out and perform experiments, can Google/GPT things themselves, etc. A GPT model already has many ways of thinking and self-distilling; for example, it can give itself a lot more time to think using 'inner monologue' techniques, which make it capable of calculating out step by step, using external tools like Python etc: https://www.gwern.net/docs/ai/gpt/inner-monologue/index https://www.gwern.net/docs/ai/gpt/inner-monologue/index A GPT model can also already Google things, see WebGPT https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/ (itself merely one entry in a rapidly-expanding area of retrieval models https://www.gwern.net/docs/ai/retrieval/index https://www.gwern.net/docs/ai/retrieval/index ). Enabling them to use web tools is likewise an active and increasingly successful area of research https://arxiv.org/abs/2202.08137#deepmind https://arxiv.org/abs/2202.08137#deepmind , with startups forming for that purpose: https://www.adept.ai/post/introducing-adept https://www.adept.ai/post/introducing-adept It can't 'run experiments', but the setting of reinforcement learning would allow that, and Decision Transformer (literally just a GPT trained on RL data and then given a prompt) is one of the more exciting directions - you may have seen Gato https://www.deepmind.com/publications/a-generalist-agent https://www.deepmind.com/publications/a-generalist-agent but have you also seen Multi-Game Transformer https://sites.google.com/view/multi-game-transformers https://sites.google.com/view/multi-game-transformers ? Oh, that's just a single agent and you're not worried? Well that's fine because DT is so general it's almost trivial to make it do multi-agent RL as well https://arxiv.org/abs/2112.02845 https://arxiv.org/abs/2112.02845 https://arxiv.org/abs/2205.14953 https://arxiv.org/abs/2205.14953 Things are moving fast, and few people know all of the things that a GPT can do. (By the way, did you know that this 'big blob of inanimate knowledge' also knows what it doesn't know and can give well-calibrated predictions of accuracy https://arxiv.org/abs/2205.14334 https://arxiv.org/abs/2205.14334 ?)