5 ms·

The number of people who truly believe that AI has a serious potential to wipe us out is minuscule, and they are mostly not in decision-making positions. Since

The number of people who truly believe that AI has a serious potential to wipe us out is minuscule, and they are mostly not in decision-making positions. Since you use the term AGI, I suppose you've been exposed to this idea via LessWrong? I'd caution you that they have a very non-mainstream view of AI potential, even by the standards of AI researchers. Their concerns about existential risk stem from simplified mathematical models that have little or nothing to do with how AI actually works, making the conclusions they draw very tenuous.

drdeca 5y ago
The term AGI seems a useful one for being clear about what one is talking about.
- garmaine 5y ago
  It's not widely used outside of Ben Goertzel et al, and the LessWrong/MIRI community, because these are the only groups which believe there is a significant qualitative difference between "narrow AI" and "general intelligence." From a pragmatic perspective, DeepMind and OpenAI have demonstrated to pretty good confidence that scaling up so-called "narrow AI" techniques are sufficient to create general intelligences. That's not to say that there isn't a useful distinction to be made between classical AGI architectures and the neural network based techniques used by most AI groups today. But it's becoming increasingly clear that the specialized AGI techniques are not not required to achieve general intelligence, so neither is it certain that the conclusions reached by analyzing AGI architectures (which has led to existential risk concerns) apply to the sorts of general intelligences actually being worked on today. But anyway, the word AGI is unfortunately not useful for being clear, because it's not clear whether you mean general/human-level intelligence, or the specific non-neural-network based architectures that Goertzel had in mind when he coined the term.
  - drdeca 5y ago
    These claims surprise me. Wikipedia does not appear to describe the term AGI as being used only by a, ah, clique or uh, relatively small faction, or however you would describe “ Ben Goertzel et al, and the LessWrong/MIRI community”. Perhaps this is because the editors of the Wikipedia article on “Artificial General Intelligence” are aligned with LW-ers. However, the article seems well cited, and says that ‘the term “artificial general intelligence” was used as early as 1997, by Mark Gubrud’ (and provides a citation for this). Ah, ok, Wikipedia also says the term was re-introduced and popularized by Dhane Legg and Ben Goertzel, around 2002. (Sorry if this is nit-picky, I’m writing this as I read about the topic, with only a little editing of parts I’ve already written.) Wikipedia in particular defines the term as referring to the general capabilities idea, and its article on AGI doesn’t even seem to mention the specific architecture that Goertzel proposed involving ‘a network whose nodes and links carry probabilistic truth values as well as “attention values” ‘ (to quote the article which is about Goertzel himself) . So, Wikipedia at least gives the impression that there is a fairly clear consensus on at least generally what “AGI” means (even if not on the exact details of a definition / criterion. It lists a number of proposed tests which seem similar enough in terms of the kind of things they test.)
    - garmaine 5y ago
      Some context on Goertzel that wouldn't show up in the wiki article. Goertzel's belief is/was that one couldn't achieve general / human-levels intelligence by scaling up regular old neural network ideas. He, and some others, believed that while traditional AI techniques were good within confined use cases, writing an intelligence that could apply itself to problem domains far outside of its training set required an explicitly different set of algorithms at the general/meta level. E.g. I imagine that Goertzel would say that GPT-3 isn't a general intelligence. It's the human engineer that made GPT-3, or which builds applications applying GPT-3 to different domains that is the general intelligence. Yes GPT-3 is generally applicable, but who does the hard work of actually figuring out how to apply it to new setups? THAT is general intelligence, in Goertzel's view. Back to the timeline, in 2002 Goertzel wanted to focus not on machine learning, which the field of AI was and still is mostly focused on, but rather designing "something" which (in current day terms) knew how to adapt meta-parameters to a particular training data set, and knew how to transfer/apply trained networks to new problem domains. An artificial AI engineer, as it were. So Goertzel coined/repurposed the term Artificial General Intelligence to mean not just a human-level AI, but rather a meta-AI program that was able to create, on its own, new human-level narrow AI using more traditional techniques. They created a AGI conference to gather researchers interested in these ideas: http://agi-conference.org http://agi-conference.org Generally speaking AI researchers who believe that scaling up traditional neural-network architectures is sufficient to create a general/human-level intelligence don't use the term AGI. The kind of folks who go to the AGI conference and study large-architecture AIs aiming for an explicit human-level, meta cognition engine do use the term AGI, and use it a lot. LessWrong folks may not hang around AGI conferences, but they do fall in the latter group. The MIRI institute was traditionally fully devoted to studying AGI architectures of their own trade-secret design, not narrow AI techniques (although that has changed a bit with Paul Christiano's work). So if you use AGI without qualification, my prior is that you're hailing from the latter camp of AGI meta-architecture researchers or the LessWrong-influenced existential risk crowd. In either case it is worth pointing out that you are getting a tinted view of AI risk that is different from what, say, researchers at DeepMind or Facebook largely believe.
      - drdeca 5y ago
lostdog 5y ago
Exactly, the idea that AI techniques are risky is a minority viewpoint. There is a community of smart people who are very interested in thinking about and mitigating the risk, but it's not something that most ML researchers think much about at all. Most researchers in ML are just interested in pushing the field forwards. The "risks" in improving ML seem equivalent to the risk that Intel will make a processor so powerful that it takes over the world. Researchers are also aware that given the current capabilities of AI techniques (and current AI techniques are not that capable), it's just as likely that the next JavaScript framework drafts the world economy into making paperclips exclusively. The idea that AI development is risky has a few hardcore supporters and is widely popularized, but most people working on it do not feel that it's a problem worth worrying about.

drdeca 5y ago

The term AGI seems a useful one for being clear about what one is talking about.

garmaine 5y ago
It's not widely used outside of Ben Goertzel et al, and the LessWrong/MIRI community, because these are the only groups which believe there is a significant qualitative difference between "narrow AI" and "general intelligence." From a pragmatic perspective, DeepMind and OpenAI have demonstrated to pretty good confidence that scaling up so-called "narrow AI" techniques are sufficient to create general intelligences. That's not to say that there isn't a useful distinction to be made between classical AGI architectures and the neural network based techniques used by most AI groups today. But it's becoming increasingly clear that the specialized AGI techniques are not not required to achieve general intelligence, so neither is it certain that the conclusions reached by analyzing AGI architectures (which has led to existential risk concerns) apply to the sorts of general intelligences actually being worked on today. But anyway, the word AGI is unfortunately not useful for being clear, because it's not clear whether you mean general/human-level intelligence, or the specific non-neural-network based architectures that Goertzel had in mind when he coined the term.
- drdeca 5y ago
  These claims surprise me. Wikipedia does not appear to describe the term AGI as being used only by a, ah, clique or uh, relatively small faction, or however you would describe “ Ben Goertzel et al, and the LessWrong/MIRI community”. Perhaps this is because the editors of the Wikipedia article on “Artificial General Intelligence” are aligned with LW-ers. However, the article seems well cited, and says that ‘the term “artificial general intelligence” was used as early as 1997, by Mark Gubrud’ (and provides a citation for this). Ah, ok, Wikipedia also says the term was re-introduced and popularized by Dhane Legg and Ben Goertzel, around 2002. (Sorry if this is nit-picky, I’m writing this as I read about the topic, with only a little editing of parts I’ve already written.) Wikipedia in particular defines the term as referring to the general capabilities idea, and its article on AGI doesn’t even seem to mention the specific architecture that Goertzel proposed involving ‘a network whose nodes and links carry probabilistic truth values as well as “attention values” ‘ (to quote the article which is about Goertzel himself) . So, Wikipedia at least gives the impression that there is a fairly clear consensus on at least generally what “AGI” means (even if not on the exact details of a definition / criterion. It lists a number of proposed tests which seem similar enough in terms of the kind of things they test.)
  - garmaine 5y ago
    Some context on Goertzel that wouldn't show up in the wiki article. Goertzel's belief is/was that one couldn't achieve general / human-levels intelligence by scaling up regular old neural network ideas. He, and some others, believed that while traditional AI techniques were good within confined use cases, writing an intelligence that could apply itself to problem domains far outside of its training set required an explicitly different set of algorithms at the general/meta level. E.g. I imagine that Goertzel would say that GPT-3 isn't a general intelligence. It's the human engineer that made GPT-3, or which builds applications applying GPT-3 to different domains that is the general intelligence. Yes GPT-3 is generally applicable, but who does the hard work of actually figuring out how to apply it to new setups? THAT is general intelligence, in Goertzel's view. Back to the timeline, in 2002 Goertzel wanted to focus not on machine learning, which the field of AI was and still is mostly focused on, but rather designing "something" which (in current day terms) knew how to adapt meta-parameters to a particular training data set, and knew how to transfer/apply trained networks to new problem domains. An artificial AI engineer, as it were. So Goertzel coined/repurposed the term Artificial General Intelligence to mean not just a human-level AI, but rather a meta-AI program that was able to create, on its own, new human-level narrow AI using more traditional techniques. They created a AGI conference to gather researchers interested in these ideas: http://agi-conference.org http://agi-conference.org Generally speaking AI researchers who believe that scaling up traditional neural-network architectures is sufficient to create a general/human-level intelligence don't use the term AGI. The kind of folks who go to the AGI conference and study large-architecture AIs aiming for an explicit human-level, meta cognition engine do use the term AGI, and use it a lot. LessWrong folks may not hang around AGI conferences, but they do fall in the latter group. The MIRI institute was traditionally fully devoted to studying AGI architectures of their own trade-secret design, not narrow AI techniques (although that has changed a bit with Paul Christiano's work). So if you use AGI without qualification, my prior is that you're hailing from the latter camp of AGI meta-architecture researchers or the LessWrong-influenced existential risk crowd. In either case it is worth pointing out that you are getting a tinted view of AI risk that is different from what, say, researchers at DeepMind or Facebook largely believe.
    - drdeca 5y ago

lostdog 5y ago

Exactly, the idea that AI techniques are risky is a minority viewpoint. There is a community of smart people who are very interested in thinking about and mitigating the risk, but it's not something that most ML researchers think much about at all. Most researchers in ML are just interested in pushing the field forwards. The "risks" in improving ML seem equivalent to the risk that Intel will make a processor so powerful that it takes over the world. Researchers are also aware that given the current capabilities of AI techniques (and current AI techniques are not that capable), it's just as likely that the next JavaScript framework drafts the world economy into making paperclips exclusively. The idea that AI development is risky has a few hardcore supporters and is widely popularized, but most people working on it do not feel that it's a problem worth worrying about.