6 ms·

That's… not at all true? There have been numerous thorough attempts at coming up with as reasonable an estimate as possible, involving in-depth conversations wi

That's… not at all true? There have been numerous thorough attempts at coming up with as reasonable an estimate as possible, involving in-depth conversations with most of the experts in the field. Maybe it's not strictly a "calculation", but it's a hell of a lot more than just an appeal to the idea of AGI.

joshuamorton 10y ago
Except that there's no such thing as the field of "AGI" (or if there is, its a subfield of philosophy). Asking modern ML and Deep Learning researchers about their thoughts on AGI is like asking the wright brothers about a mars mission. Sure, they're the closest thing we have to "experts", but there's not just one, but likely 5 or 10 more field, if not world-changing leaps we need to make before the technologies we have will resemble anything like AGI. And that's ignoring the people who ascribe diety-like powers to some potential AGI. Air gap the computer and control the inputs and outputs. We can formally prove what a system is capable of. That fixes the problem.
- nbouscal 10y ago
  Your opinion about how far off AGI is is a totally reasonable one, and some AI experts agree with it. Other AI experts disagree, and think that AGI is basically tractable with current research methods plus a breakthrough or two. Yes, there's some additional uncertainty about whether you're even asking the right people. But you can take account of that uncertainty, both by widening your error bars and by asking people from other fields as well (including philosophers, e.g. Bostrom). What you can't do is just throw up your hands and say "it's unknowable". This is all beside the original point, which is that these arguments are much more rigorously grounded than just a wave in the direction of AGI. Could they be better? Sure, and there are a lot of people who'd be interested in seeing some better estimates. But for now, they're the best we have, and they're a completely reasonable thing to base decisions on. > We can formally prove what a system is capable of. That fixes the problem. That's exactly what some of the people working on this problem are trying to do, but it's a hell of a lot harder than you make it sound. Formal methods have come a really long way, but they're not even remotely close to being able to prove that an AGI system is safe (yet).
  - joshuamorton 10y ago
    You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. I recognize that proving things about machine learning models is very hard, I tried to do some research in that area and got practically no where. And I think you and I have very different definitions of Rigorous. Like I said, unless you'd take Wilbur Wright's thoughts on a mars mission seriously, I don't think you should give much thought to Geoff Hinton's thoughts about when we'll get AGI, and I say that having enormous respect for him and his achievements. (and using him as a simple example) Its pseudoscience. We're notoriously bad about predicting the future. I don't see any reason to trust people going on about the dangers of AGI any more than the futurologists of my parents generation, who predicted flying cars and interstellar travel, but missed smartphones.
    - schoen 10y ago
      > You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. But then you can't benefit from it, or you can only benefit from it except in narrowly predefined ways.
      - joshuamorton 10y ago
        Perhaps, but if the alternative is either incredible fear of the technology, or the technology potentially killing humanity, then "making great strides in a select few fields" seems quite good.
    - nbouscal 10y ago
      I really, really don't think modern AI capabilities and AGI are as dissimilar as airplanes and rockets. Neither do many AI researchers. You're obviously welcome to disagree, but you don't just get to declare their opinions invalid because you think they're out of their depth.
      - rspeer 10y ago
        I don't think AI researchers are "out of their depth" (I am one myself). And yes, there is room for different opinions. However, I believe that the ones who say sensational things about AI doomsday are the ones who are disproportionately quoted by the media.
- schoen 10y ago
  > Air gap the computer and control the inputs and outputs. We can formally prove what a system is capable of. That fixes the problem. Debates about superhuman AI have focused quite a lot on what it would mean to "control the inputs and outputs" while still being able to get some kind of benefit from the AI. You can indeed formally prove that a computer will or won't do certain things, so you could use that for isolation purposes. But in order to be useful, the AI needs to interact with people and/or the world in some way. Otherwise it might as well be switched off or never have been built in the first place. If it's really a superhuman intelligence with superhuman knowledge about the world, then interacting with people is where the risk creeps back in, because the AI could make suggestions, recommendations, requests, offers, promises, or threats. Although there are plenty of ideas about limiting the nature of questions and answers, having some kind of separate person or machine judge whether information from the AI's communications should be used or how, or limiting what the AI is programmed to attempt to do or how, none of these measures are straightforward to formally prove correct in the way that simpler isolation properties are. If we made contact with intelligent aliens, would formal proofs of correctness of the computers through which we (say, exclusively) communicate with them guarantee that they couldn't massively disrupt our society by means of what they had to say?
- Smaug123 10y ago
  "We can formally prove what a system is capable of." This is not true. By Rice's theorem, either the system is too dumb to be useful, or we can't prove what it can do.
  - joshuamorton 10y ago
    An airgapped computer is not going to be capable of taking over the world. Further, Turing completeness is not required to be "useful". You can get to the moon without Turing completeness.
    - Smaug123 10y ago
      I didn't address airgappedness at all, but you may still be wrong: we use the superintelligence's output, or else why would we have created it, so we are the channel by which it affects the world. Anyway, who knows what can be done by a superintelligence which can control a stream of electrons emitted from its own CPU!
      - joshuamorton 10y ago
        Well, exactly as much as any other cpus can do by solely controlling it's output electrons: not much. Let's not ascribe deity-like powers to things that we can understand fairly well.
        Smaug123 10y ago
        Have you ever seen a CPU running a program which is trying to communicate with the outside world through its electron side-channels? As far as I can see, your argument is "no computer has ever done this, and also we understand computers well enough to know that no computer ever will". The first clause is obvious, since no computer has ever been made to do this. The second clause is assuming without proof that we will never make a superintelligent AI. Just because you don't see how to exploit the side-channels of your system, doesn't mean they're unexploitable. This is the lesson of all security research ever.
BaronSamedi 10y ago
Experts in the field of AI have a long history of being wrong. "AI is 20 years" away has become the standard joke in the field. There is no reason to grant expert opinion on this topic any weight.
- nbouscal 10y ago
  What's your suggested alternative? It's really easy to criticize, but a lot harder to do any better. If you aren't suggesting any other methods, you're implicitly saying we should just ignore the problem. I really don't think that will end well.
  - argonaut 10y ago
    You admit that you don't know what the actual probabilities are. Or you admit you have an extremely low confidence in your estimate of the probabilities.
    - nbouscal 10y ago
      Have you read any of the estimates? Basically every expert gives a huge confidence interval, as do most of the people actively working on AI risk.
      - argonaut 10y ago
        Saying you are 95% confident AGI will come between 50-100 years (a "huge" confidence interval), is still far too confident.
        nbouscal 10y ago
        I don't recall seeing 95% confidence intervals that narrow. Even the 90% confidence intervals I see are usually like 5-150 or something. (Also it would be super weird to me if the lower bound of someone's 95%CI was 50.)
  - rspeer 10y ago

joshuamorton 10y ago

Except that there's no such thing as the field of "AGI" (or if there is, its a subfield of philosophy). Asking modern ML and Deep Learning researchers about their thoughts on AGI is like asking the wright brothers about a mars mission. Sure, they're the closest thing we have to "experts", but there's not just one, but likely 5 or 10 more field, if not world-changing leaps we need to make before the technologies we have will resemble anything like AGI. And that's ignoring the people who ascribe diety-like powers to some potential AGI. Air gap the computer and control the inputs and outputs. We can formally prove what a system is capable of. That fixes the problem.

nbouscal 10y ago
Your opinion about how far off AGI is is a totally reasonable one, and some AI experts agree with it. Other AI experts disagree, and think that AGI is basically tractable with current research methods plus a breakthrough or two. Yes, there's some additional uncertainty about whether you're even asking the right people. But you can take account of that uncertainty, both by widening your error bars and by asking people from other fields as well (including philosophers, e.g. Bostrom). What you can't do is just throw up your hands and say "it's unknowable". This is all beside the original point, which is that these arguments are much more rigorously grounded than just a wave in the direction of AGI. Could they be better? Sure, and there are a lot of people who'd be interested in seeing some better estimates. But for now, they're the best we have, and they're a completely reasonable thing to base decisions on. > We can formally prove what a system is capable of. That fixes the problem. That's exactly what some of the people working on this problem are trying to do, but it's a hell of a lot harder than you make it sound. Formal methods have come a really long way, but they're not even remotely close to being able to prove that an AGI system is safe (yet).
- joshuamorton 10y ago
  You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. I recognize that proving things about machine learning models is very hard, I tried to do some research in that area and got practically no where. And I think you and I have very different definitions of Rigorous. Like I said, unless you'd take Wilbur Wright's thoughts on a mars mission seriously, I don't think you should give much thought to Geoff Hinton's thoughts about when we'll get AGI, and I say that having enormous respect for him and his achievements. (and using him as a simple example) Its pseudoscience. We're notoriously bad about predicting the future. I don't see any reason to trust people going on about the dangers of AGI any more than the futurologists of my parents generation, who predicted flying cars and interstellar travel, but missed smartphones.
  - schoen 10y ago
    > You don't need to prove anything about an AGI system if you can prove things about its I/O capabilities. But then you can't benefit from it, or you can only benefit from it except in narrowly predefined ways.
    - joshuamorton 10y ago
      Perhaps, but if the alternative is either incredible fear of the technology, or the technology potentially killing humanity, then "making great strides in a select few fields" seems quite good.
  - nbouscal 10y ago
    I really, really don't think modern AI capabilities and AGI are as dissimilar as airplanes and rockets. Neither do many AI researchers. You're obviously welcome to disagree, but you don't just get to declare their opinions invalid because you think they're out of their depth.
    - rspeer 10y ago
      I don't think AI researchers are "out of their depth" (I am one myself). And yes, there is room for different opinions. However, I believe that the ones who say sensational things about AI doomsday are the ones who are disproportionately quoted by the media.
schoen 10y ago
> Air gap the computer and control the inputs and outputs. We can formally prove what a system is capable of. That fixes the problem. Debates about superhuman AI have focused quite a lot on what it would mean to "control the inputs and outputs" while still being able to get some kind of benefit from the AI. You can indeed formally prove that a computer will or won't do certain things, so you could use that for isolation purposes. But in order to be useful, the AI needs to interact with people and/or the world in some way. Otherwise it might as well be switched off or never have been built in the first place. If it's really a superhuman intelligence with superhuman knowledge about the world, then interacting with people is where the risk creeps back in, because the AI could make suggestions, recommendations, requests, offers, promises, or threats. Although there are plenty of ideas about limiting the nature of questions and answers, having some kind of separate person or machine judge whether information from the AI's communications should be used or how, or limiting what the AI is programmed to attempt to do or how, none of these measures are straightforward to formally prove correct in the way that simpler isolation properties are. If we made contact with intelligent aliens, would formal proofs of correctness of the computers through which we (say, exclusively) communicate with them guarantee that they couldn't massively disrupt our society by means of what they had to say?
Smaug123 10y ago
"We can formally prove what a system is capable of." This is not true. By Rice's theorem, either the system is too dumb to be useful, or we can't prove what it can do.
- joshuamorton 10y ago
  An airgapped computer is not going to be capable of taking over the world. Further, Turing completeness is not required to be "useful". You can get to the moon without Turing completeness.
  - Smaug123 10y ago
    I didn't address airgappedness at all, but you may still be wrong: we use the superintelligence's output, or else why would we have created it, so we are the channel by which it affects the world. Anyway, who knows what can be done by a superintelligence which can control a stream of electrons emitted from its own CPU!
    - joshuamorton 10y ago
      Well, exactly as much as any other cpus can do by solely controlling it's output electrons: not much. Let's not ascribe deity-like powers to things that we can understand fairly well.
      - Smaug123 10y ago
        Have you ever seen a CPU running a program which is trying to communicate with the outside world through its electron side-channels? As far as I can see, your argument is "no computer has ever done this, and also we understand computers well enough to know that no computer ever will". The first clause is obvious, since no computer has ever been made to do this. The second clause is assuming without proof that we will never make a superintelligent AI. Just because you don't see how to exploit the side-channels of your system, doesn't mean they're unexploitable. This is the lesson of all security research ever.

BaronSamedi 10y ago

Experts in the field of AI have a long history of being wrong. "AI is 20 years" away has become the standard joke in the field. There is no reason to grant expert opinion on this topic any weight.

nbouscal 10y ago
What's your suggested alternative? It's really easy to criticize, but a lot harder to do any better. If you aren't suggesting any other methods, you're implicitly saying we should just ignore the problem. I really don't think that will end well.
- argonaut 10y ago
  You admit that you don't know what the actual probabilities are. Or you admit you have an extremely low confidence in your estimate of the probabilities.
  - nbouscal 10y ago
    Have you read any of the estimates? Basically every expert gives a huge confidence interval, as do most of the people actively working on AI risk.
    - argonaut 10y ago
      Saying you are 95% confident AGI will come between 50-100 years (a "huge" confidence interval), is still far too confident.
      - nbouscal 10y ago
        I don't recall seeing 95% confidence intervals that narrow. Even the 90% confidence intervals I see are usually like 5-150 or something. (Also it would be super weird to me if the lower bound of someone's 95%CI was 50.)
- rspeer 10y ago