5 ms·

On the one hand we have lay people (including Altman and Musk) who are warning that AI is progressing too fast. On the other hand you have (essentially) the ent

by antics 12y ago

On the one hand we have lay people (including Altman and Musk) who are warning that AI is progressing too fast. On the other hand you have (essentially) the entire mainstream academic AI/ML community, who are mainly concerned that the hype is so intense it might lead to another AI winter[1]. The crux of the problem is that AI experts largely do not see meaningful progress on any axis that makes an AI doomsday scenario plausible, which causes staunch disbelief and ridicule of people who write things like this. So realistically if we want to resolve this issue, the place to start is not public policy. That's actually part of the problem -- tech policy is bad largely because it is mostly legislated by people who don't understand the technology. We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy to directly address those scenarios. As long as Altman et al are extrapolating on axes I (for one) don't understand, I honestly think this conversation is unlikely to be productive. [1] Most notably, a similar hurrah happened when neural networks first became popular. There were some promising early results, which led to much enthusiasm, and then Papert and Minsky published "Perceptron", whose Group Invariance Theorem cast aspersions on whether simple neural networks were really doing what we thought they were. Eventually we found out that many early experiments just had a bad methodology, which very nearly stopped research interest in the field for almost a decade. Now we are in a period of AI hype again, and when you hear people like Andrew Ng get up in front of audiences and say things like "for the first time in my adult life I am hopeful about real AI", you might start to wonder whether there is a lesson here. EDIT: Friends, I am confused by the downvotes. :) I am happy to have a discussion about this topic!

jbhatab 12y ago
While I understand where you are coming from, one remarkable breakthrough can change that. That does sound absurd, but we would also look like fools as a humanity for being aware of this potential problem and at least not looking into it. Technology has been exploding at a remarkable rate, I'm fine with things slowing a little for the sake of the next 1000 years of humanity.
- _delirium 12y ago
  Sure, but that's true in essentially all kinds of scientific research. I personally think physics research is more likely to produce an existential threat than AI research is. Fusion bombs are already pretty close to a world-ending discovery, possibly already across the line. And that is probably not the worst physicists have to offer in the future, either.
  - api 12y ago
    Curious for a reference on fusion bombs. If you are referring to laser inertial, that takes buildings full of lasers.
    - _delirium 12y ago
      I was just thinking of regular old hydrogen fusion bombs aka thermonuclear bombs.
      - api 12y ago
        Those require a fission trigger and have existed for a long time. The proliferation nightmare would be if we could figure out how to build one that didn't need fission, but I have never heard even a hint of success with that. If there is, it's classified -- and in this case for good reason.
escape_goat 12y ago
Is he warning that it's progressing too fast? I felt that he was more focused on risk mitigation than on trying to prevent machine intelligence research per se. And risk assessment is part of the job portfolio of both Altman and Musk; this doesn't mean that their assessment is well informed, it just makes it seem more natural to me that they would think in these terms, and be concerned if it seemed that these considerations were not being taken seriously in the field. It is not actually all that clear what intelligence is, or whether cognition and communication are necessarily synonymous with subjective existence, or whether subjective existence is necessarily synonymous with any particular goal or desire for that existence to continue. But I think that the concern about the blind application of fitness functions, for instance, is well taken. I don't think that the only problem is one of human survival; I think that there is a paucity of self-reflection in the field. The prospect of creating a sentient non-human entity raises fundamental challenges to a lot of our assumptions about ethics.
rndn 12y ago
A lot of experts also deny dangers coming from global warming and other invasions in complex systems we don't fully understand. Scientists in the 1940s also downplayed the danger of the atomic bomb. It's easy to brush away a hypothetical danger with arguments that resources might be wasted or that people in charge might embarrass themselves if their assumptions turn out to be wrong, or that research might suffer due to overblown expectations. When, do you think, is the time that we should start worrying about SMI? When machines outsmart a frog, or a chimpanzee? What if intelligence is just a matter of scale at this point, i.e. that adding more short and long term memory makes it easily reach and even surpass human intelligence?
- lazaroclapp 12y ago
  By the time the atomic bomb was truly imminent (in the sense of it being within the reach of massive dedicated investment by the most powerful nations on Earth), scientists knew very well of the dangers [1]. As for global warming. No expert denies global warming, and you'll be hard pressed to find many that deny anthropic global warming [2]. This has been the case for many decades. I am not an AI expert, but from what I can see thus far, our knowledge of cognition seems to be as far from human or post-human intelligence as Nitroglycerin was from the Atomic Bomb (e.g. fundamentally different principles at work, which is not subject to exponential improvement without one or more breakthroughs or qualitatively different approaches). Also, our AI-Nitroglycerin seems scary enough: autonomous weapons (think self-driving self-firing drones, not terminator), massive changes in our means of production that reduce human labor/employment and might cause social earthquakes and the ability to draw meaningful conclusions from surveillance data of 7 billion people... [1] https://en.wikipedia.org/wiki/Einstein%E2%80%93Szil%C3%A1rd_letter https://en.wikipedia.org/wiki/Einstein%E2%80%93Szil%C3%A1rd_... (and this seems to have been understood as an approaching risk in the physics community way before anyone thought of telling FDR). [2] https://en.wikipedia.org/wiki/Global_warming_controversy#Scientific_consensus https://en.wikipedia.org/wiki/Global_warming_controversy#Sci...
  - rndn 12y ago
    Fair enough, but I wouldn't exclude the possibility that AI experts are biased for at least three reasons, namely (1) that they believe in some sort of sanctity of the human mind (like many people do), (2) that they don't want to be regulated and (3) that they don't want to raise overblown expectations. There are definitely historical examples in which the majority of experts was wrong. I also find it incredibly difficult to tell what is missing about an algorithm. Perhaps multiple breakthroughs are necessary, perhaps just one (there is no good reason to believe that an algorithm that yields AGI can't be considerably simpler than what happens in the human brain). Initially, nobody thought that search could be done in O(log n) and similarly not many researchers expected neural networks to perform useful tasks until back-propagation was invented and demonstrated.
gajomi 12y ago
I think you make a lot of good points. >We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy directly those scenarios Here is my proposal. Since the whole of AI is a rather complicated and messy business let us just focus on a small part. That is, let's just talk about linear regression and simple decision making schemes based on estimated parameters. While one can debate whether or not linear regression counts as "actual" AI, there is little doubt that (a) it forms the conceptual basis of many more complicated AI schemes, so that if we can reason about doomsday scenarios for these we should also be able to do the same for linear regression and (b) despite its simplicity there is a substantial intelligence value in linear regression, to the point that successes and failures of linear regression can cause major effects on scientific/business/whatever process. What are the examples about how the set of decision/action possibilities increases before and after linear regression technology was applied? What examples are there of how the failure modes of linear regression tended to increase the risk to humans (e.g. moving up the doomsday axis)? In each of these cases what policies have been or could be implemented to mitigate these risks, including technological audits as well as regulation of deployment of technologies. My sense is that these questions actually can be answered. In all likelihood professional statisticians and operations research types already have opinions on these things. I would be curious to see what the public opinion is about this limited version of the problem, since the technology is something that many people (including Sam Altman and friends) should have a good understanding of.
- sanxiyn 12y ago
  Google published a paper which I think is relevant to the topic, Machine Learning: The High Interest Credit Card of Technical Debt (2014). http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html The conclusion seems to be that even this level of machine learning decreases controllability substantially. "We note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning."
lukeprog 12y ago
> On the other hand you have (essentially) the entire mainstream academic AI/ML community, who are mainly concerned that the hype is so intense it might lead to another AI winter This doesn't seem true. E.g. the signatories on the Future of Life Institute's recent open letter about AI progress and safety include: * Stuart Russell & Peter Norvig, co-authors of the #1 AI textbook * Tom Dietterich, AAAI President * Eric Horvitz, past AAAI President * Bart Selman, co-chair of AAAI presidential panel on long-term AI futures * Francesca Rossi, IJCAI President * All founders of Deep Mind and Vicarious, two leading AI companies * Yann LeCun, head of Facebook AI * Geoffrey Hinton, top academic ML researcher * Yoshua Bengio, top academic ML researcher ...and many more.
- msutherl 12y ago
  That letter is not about "superhuman machine intelligence", it is about "ensuring that AI remains robust and beneficial." Among the threats cited are: autonomous vehicles, machines that need to make ethical decisions, autonomous weapons, surveillance, and issues of verification, validity, and control. Words like "singularity", "superhuman", and "AGI" do not appear.
  - jkramar 12y ago
    Superintelligence and intelligence explosion are mentioned as valuable things to investigate.
    - msutherl 12y ago
      Ah, turns out you're right: Finally, research on the possibility of superintelligent machines or rapid, sustained self-improvement (“intelligence explosion”) has been highlighted by past and current projects on the future of AI as potentially valuable to the project of maintaining reliable control in the long term. The the AAAI 2008–09 Presidential Panel on Long-Term AI Futures’ “Subgroup on Pace, Concerns, and Control” stated that: "There was overall skepticism about the prospect of an intelligence explosion... Nevertheless, there was a shared sense that additional research would be valuable on methods for understanding and verifying the range of behaviors of complex computational systems to minimize unexpected outcomes. Some panelists recommended that more research needs to be done to better define “intelligence explosion,” and also to better formulate different classes of such accelerating intelligences. Technical work would likely lead to enhanced understanding of the likelihood of such phenomena, and the nature, risks, and overall outcomes associated with different conceived variants." Stanford’s One-Hundred Year Study of Artificial Intelligence includes “Loss of Control of AI systems” as an area of study, specifically highlighting concerns over the possibility that: "we could one day lose control of AI systems via the rise of superintelligences that do not act in accordance with human wishes – and that such powerful systems would threaten humanity. Are such dystopic outcomes possible? If so, how might these situations arise? ...What kind of investments in research should be made to better understand and to address the possibility of the rise of a dangerous superintelligence or the occurrence of an 'intelligence explosion'?" Research in this area could include any of the long-term research priorities listed above, as well as theoretical and forecasting work on intelligence explosion and superintelligence, and could extend or critique existing approaches begun by groups such as the Machine Intelligence Research Institute.
resu_nimda 12y ago
We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy to directly address those scenarios. As long as Altman et al are extrapolating on axes I (for one) don't understand, I honestly think this conversation is unlikely to be productive. Fully agree with this. So far, all of this activism around the AI threat has amounted to "guys we really need to think about this." But that's where it ends, because nobody has any idea what a dangerous AI would actually look like, what it would actually be capable of doing. There's just a vague notion of "it COULD do something really bad." While I agree that is a risk and could be a threat to humanity, I haven't seen any results from this line of thinking, nothing to demonstrate the value of trying to "get in front of" a problem that is so ill-defined and shrouded in so much mystery.
- mori 12y ago
  If you're looking to support AI safety, the Machine Intelligence Research Institute is researching it.
joeyspn 12y ago
Good try AI, good try...
leot 12y ago
While Altman and Musk don't spend all their time on it, I'd hardly call them "lay people". Altman studied AI at Stanford, and Musk knows a fair bit more than the average person about computer science and technology. The DeepMind people are also pretty bright on this topic, and they were concerned enough to set up an "ethics board". As someone with multiple degrees on related topics, I was pretty skeptical of the concerns as well. That is, until I read Nick Bostrom's book "Superintelligence": the fact is that the consequences of SMI are so potentially disastrous that even if the odds were 1 in a 100 it would make sense for us to be very careful. But the probability of SMI happening within the next 100 years is likely far more than that. Don't want regulation? Then come up with a proposal that has a better chance of keeping things under control and simultaneously utterly obviates any incremental benefit caused by imposition of a regulatory framework. It's one thing to distrust authority, especially when said authority wasn't earned. In the case of Musk, Altman, Bostrom, (and many others) ignoring the thoughtfulness they have brought to this issue seems extremely reckless.
- deleted 12y ago
  [deleted]
Houshalter 12y ago
The people concerned about AI are not talking about current ML research the near future, but where the tech will be decades from now. Surveys of AI and AGI researchers generally agree that it will probably happen with int next century and there is a high chance of it going badly.

by antics 12y ago

jbhatab 12y ago

While I understand where you are coming from, one remarkable breakthrough can change that. That does sound absurd, but we would also look like fools as a humanity for being aware of this potential problem and at least not looking into it. Technology has been exploding at a remarkable rate, I'm fine with things slowing a little for the sake of the next 1000 years of humanity.

_delirium 12y ago
Sure, but that's true in essentially all kinds of scientific research. I personally think physics research is more likely to produce an existential threat than AI research is. Fusion bombs are already pretty close to a world-ending discovery, possibly already across the line. And that is probably not the worst physicists have to offer in the future, either.
- api 12y ago
  Curious for a reference on fusion bombs. If you are referring to laser inertial, that takes buildings full of lasers.
  - _delirium 12y ago
    I was just thinking of regular old hydrogen fusion bombs aka thermonuclear bombs.
    - api 12y ago
      Those require a fission trigger and have existed for a long time. The proliferation nightmare would be if we could figure out how to build one that didn't need fission, but I have never heard even a hint of success with that. If there is, it's classified -- and in this case for good reason.

escape_goat 12y ago

Is he warning that it's progressing too fast? I felt that he was more focused on risk mitigation than on trying to prevent machine intelligence research per se. And risk assessment is part of the job portfolio of both Altman and Musk; this doesn't mean that their assessment is well informed, it just makes it seem more natural to me that they would think in these terms, and be concerned if it seemed that these considerations were not being taken seriously in the field. It is not actually all that clear what intelligence is, or whether cognition and communication are necessarily synonymous with subjective existence, or whether subjective existence is necessarily synonymous with any particular goal or desire for that existence to continue. But I think that the concern about the blind application of fitness functions, for instance, is well taken. I don't think that the only problem is one of human survival; I think that there is a paucity of self-reflection in the field. The prospect of creating a sentient non-human entity raises fundamental challenges to a lot of our assumptions about ethics.

rndn 12y ago

A lot of experts also deny dangers coming from global warming and other invasions in complex systems we don't fully understand. Scientists in the 1940s also downplayed the danger of the atomic bomb. It's easy to brush away a hypothetical danger with arguments that resources might be wasted or that people in charge might embarrass themselves if their assumptions turn out to be wrong, or that research might suffer due to overblown expectations. When, do you think, is the time that we should start worrying about SMI? When machines outsmart a frog, or a chimpanzee? What if intelligence is just a matter of scale at this point, i.e. that adding more short and long term memory makes it easily reach and even surpass human intelligence?

lazaroclapp 12y ago
By the time the atomic bomb was truly imminent (in the sense of it being within the reach of massive dedicated investment by the most powerful nations on Earth), scientists knew very well of the dangers [1]. As for global warming. No expert denies global warming, and you'll be hard pressed to find many that deny anthropic global warming [2]. This has been the case for many decades. I am not an AI expert, but from what I can see thus far, our knowledge of cognition seems to be as far from human or post-human intelligence as Nitroglycerin was from the Atomic Bomb (e.g. fundamentally different principles at work, which is not subject to exponential improvement without one or more breakthroughs or qualitatively different approaches). Also, our AI-Nitroglycerin seems scary enough: autonomous weapons (think self-driving self-firing drones, not terminator), massive changes in our means of production that reduce human labor/employment and might cause social earthquakes and the ability to draw meaningful conclusions from surveillance data of 7 billion people... [1] https://en.wikipedia.org/wiki/Einstein%E2%80%93Szil%C3%A1rd_letter https://en.wikipedia.org/wiki/Einstein%E2%80%93Szil%C3%A1rd_... (and this seems to have been understood as an approaching risk in the physics community way before anyone thought of telling FDR). [2] https://en.wikipedia.org/wiki/Global_warming_controversy#Scientific_consensus https://en.wikipedia.org/wiki/Global_warming_controversy#Sci...
- rndn 12y ago
  Fair enough, but I wouldn't exclude the possibility that AI experts are biased for at least three reasons, namely (1) that they believe in some sort of sanctity of the human mind (like many people do), (2) that they don't want to be regulated and (3) that they don't want to raise overblown expectations. There are definitely historical examples in which the majority of experts was wrong. I also find it incredibly difficult to tell what is missing about an algorithm. Perhaps multiple breakthroughs are necessary, perhaps just one (there is no good reason to believe that an algorithm that yields AGI can't be considerably simpler than what happens in the human brain). Initially, nobody thought that search could be done in O(log n) and similarly not many researchers expected neural networks to perform useful tasks until back-propagation was invented and demonstrated.

gajomi 12y ago

I think you make a lot of good points. >We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy directly those scenarios Here is my proposal. Since the whole of AI is a rather complicated and messy business let us just focus on a small part. That is, let's just talk about linear regression and simple decision making schemes based on estimated parameters. While one can debate whether or not linear regression counts as "actual" AI, there is little doubt that (a) it forms the conceptual basis of many more complicated AI schemes, so that if we can reason about doomsday scenarios for these we should also be able to do the same for linear regression and (b) despite its simplicity there is a substantial intelligence value in linear regression, to the point that successes and failures of linear regression can cause major effects on scientific/business/whatever process. What are the examples about how the set of decision/action possibilities increases before and after linear regression technology was applied? What examples are there of how the failure modes of linear regression tended to increase the risk to humans (e.g. moving up the doomsday axis)? In each of these cases what policies have been or could be implemented to mitigate these risks, including technological audits as well as regulation of deployment of technologies. My sense is that these questions actually can be answered. In all likelihood professional statisticians and operations research types already have opinions on these things. I would be curious to see what the public opinion is about this limited version of the problem, since the technology is something that many people (including Sam Altman and friends) should have a good understanding of.

sanxiyn 12y ago
Google published a paper which I think is relevant to the topic, Machine Learning: The High Interest Credit Card of Technical Debt (2014). http://research.google.com/pubs/pub43146.html http://research.google.com/pubs/pub43146.html The conclusion seems to be that even this level of machine learning decreases controllability substantially. "We note that it is remarkably easy to incur massive ongoing maintenance costs at the system level when applying machine learning."

lukeprog 12y ago

> On the other hand you have (essentially) the entire mainstream academic AI/ML community, who are mainly concerned that the hype is so intense it might lead to another AI winter This doesn't seem true. E.g. the signatories on the Future of Life Institute's recent open letter about AI progress and safety include: * Stuart Russell & Peter Norvig, co-authors of the #1 AI textbook * Tom Dietterich, AAAI President * Eric Horvitz, past AAAI President * Bart Selman, co-chair of AAAI presidential panel on long-term AI futures * Francesca Rossi, IJCAI President * All founders of Deep Mind and Vicarious, two leading AI companies * Yann LeCun, head of Facebook AI * Geoffrey Hinton, top academic ML researcher * Yoshua Bengio, top academic ML researcher ...and many more.

msutherl 12y ago
That letter is not about "superhuman machine intelligence", it is about "ensuring that AI remains robust and beneficial." Among the threats cited are: autonomous vehicles, machines that need to make ethical decisions, autonomous weapons, surveillance, and issues of verification, validity, and control. Words like "singularity", "superhuman", and "AGI" do not appear.
- jkramar 12y ago
  Superintelligence and intelligence explosion are mentioned as valuable things to investigate.
  - msutherl 12y ago
    Ah, turns out you're right: Finally, research on the possibility of superintelligent machines or rapid, sustained self-improvement (“intelligence explosion”) has been highlighted by past and current projects on the future of AI as potentially valuable to the project of maintaining reliable control in the long term. The the AAAI 2008–09 Presidential Panel on Long-Term AI Futures’ “Subgroup on Pace, Concerns, and Control” stated that: "There was overall skepticism about the prospect of an intelligence explosion... Nevertheless, there was a shared sense that additional research would be valuable on methods for understanding and verifying the range of behaviors of complex computational systems to minimize unexpected outcomes. Some panelists recommended that more research needs to be done to better define “intelligence explosion,” and also to better formulate different classes of such accelerating intelligences. Technical work would likely lead to enhanced understanding of the likelihood of such phenomena, and the nature, risks, and overall outcomes associated with different conceived variants." Stanford’s One-Hundred Year Study of Artificial Intelligence includes “Loss of Control of AI systems” as an area of study, specifically highlighting concerns over the possibility that: "we could one day lose control of AI systems via the rise of superintelligences that do not act in accordance with human wishes – and that such powerful systems would threaten humanity. Are such dystopic outcomes possible? If so, how might these situations arise? ...What kind of investments in research should be made to better understand and to address the possibility of the rise of a dangerous superintelligence or the occurrence of an 'intelligence explosion'?" Research in this area could include any of the long-term research priorities listed above, as well as theoretical and forecasting work on intelligence explosion and superintelligence, and could extend or critique existing approaches begun by groups such as the Machine Intelligence Research Institute.

resu_nimda 12y ago

We have to start with a discussion about how this progress towards doomsday AI should be measured, and then formulate policy to directly address those scenarios. As long as Altman et al are extrapolating on axes I (for one) don't understand, I honestly think this conversation is unlikely to be productive. Fully agree with this. So far, all of this activism around the AI threat has amounted to "guys we really need to think about this." But that's where it ends, because nobody has any idea what a dangerous AI would actually look like, what it would actually be capable of doing. There's just a vague notion of "it COULD do something really bad." While I agree that is a risk and could be a threat to humanity, I haven't seen any results from this line of thinking, nothing to demonstrate the value of trying to "get in front of" a problem that is so ill-defined and shrouded in so much mystery.

mori 12y ago
If you're looking to support AI safety, the Machine Intelligence Research Institute is researching it.

joeyspn 12y ago

Good try AI, good try...

leot 12y ago

While Altman and Musk don't spend all their time on it, I'd hardly call them "lay people". Altman studied AI at Stanford, and Musk knows a fair bit more than the average person about computer science and technology. The DeepMind people are also pretty bright on this topic, and they were concerned enough to set up an "ethics board". As someone with multiple degrees on related topics, I was pretty skeptical of the concerns as well. That is, until I read Nick Bostrom's book "Superintelligence": the fact is that the consequences of SMI are so potentially disastrous that even if the odds were 1 in a 100 it would make sense for us to be very careful. But the probability of SMI happening within the next 100 years is likely far more than that. Don't want regulation? Then come up with a proposal that has a better chance of keeping things under control and simultaneously utterly obviates any incremental benefit caused by imposition of a regulatory framework. It's one thing to distrust authority, especially when said authority wasn't earned. In the case of Musk, Altman, Bostrom, (and many others) ignoring the thoughtfulness they have brought to this issue seems extremely reckless.

deleted 12y ago
[deleted]

Houshalter 12y ago

The people concerned about AI are not talking about current ML research the near future, but where the tech will be decades from now. Surveys of AI and AGI researchers generally agree that it will probably happen with int next century and there is a high chance of it going badly.