5 ms·

I'm more in your camp than MIRI's but... Could you argue against the position directly? You kinda took a swerve in the middle there. Bostrom, Yudkowsky, and Ba

by maaku 12y ago

I'm more in your camp than MIRI's but... Could you argue against the position directly? You kinda took a swerve in the middle there. Bostrom, Yudkowsky, and Barrat's positions are basically "superintelligence is possible, it will eventually happen in some way, and when it does ..." It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence. So you would like to see unencumbered research in this area. They argue that the long term outcome of this research is existentially risky and therefore should be tightly regulated, and the sooner the better. Do you have any points regarding this particular difference of opinion?

csallen 12y ago
I'm not worried about the immediate future at all. We definitely haven't reached the point where hastily-implemented regulations will do more good than harm. That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again.
- darkmighty 12y ago
  Maybe it's precisely because the it can't be argued against that it's not much use? There's no way we could tell it will ever happen in the moment, no way to tell how intelligent we can make computers, and so on. You can't reliably argue for or against. So what's the point? I mean, you may want to form a group and discuss those scenarios, but it's not excusable to spread fear mongering in the media with no substance like that. It's unscientific in a way. We don't really need to start writing traffic rules for flying cars.
  - AndrewKemendo 12y ago
    We don't really need to start writing traffic rules for flying cars. This is a great metaphor and one I wish I would have thought up sooner.
    - maaku 12y ago
      It's a terrible metaphor, or at least it argues against your own position. If flying cars were in fact something on the horizon, we would need discussions about air spaces, commute lanes, and traffic rules. Otherwise lethal mid-air collisions would be much more probable, resulting in significant bystander and property damage as these wreckage fall out of the sky. And, no surprise, we're seeing exactly this discussion going on right now about drones.
  - csallen 12y ago
    I disagree that it can't be argued against. It's a set of logical steps, any one of which can be critiqued: 1 - There's no real dispute that computers will eventually be as "powerful" as brains. In fact, it's likely that they will one day be far more powerful. 2 - Assuming the brain evolved naturally, there's no reason to assume humans won't eventually duplicate the software once we have the hardware. It's really only a matter of when. 3 - An AGI with equal/more power than the human brain will, eventually, be able change its own code and improve upon itself. 4 - Given the characteristics of intelligence as it has been observed thus far in nature, even small increases in IQ lead to massive improvements in capability that are difficult for lesser intelligences to comprehend. A sufficiently advanced AGI would not only be highly capable, but would quite possibly be too smart for us to predict how it might behave. This is a dangerous combination. Further complicating things, we might hit step 2 on accident, without realizing that we hit it. Or some group might accomplish step 2 in secret. What I'd like to see someone do is argue that the chance of these things happening so small and/or the consequences so minuscule that it's not worth worrying about or planning for.
    - maaku 12y ago
      Step 4 is not at all obvious and would require some significant justification. General intelligence is, well, general. My primate brain may not be able at all to intuit what it is like to move on a 7-dimensional hyper-surface in an 11-dimensional space, but thanks to the wonders of formalized mathematics I can work out and tell you anything you want to know about higher-dimensional geometry. If the super-intelligence itself is computable, and we have the machinery to verify portions of the computation ourselves, it is in principle understandable. Of course there will be computational limits, but that's hardly anything new. There is no single person who can tell you absolutely everything about how a Boeing 787 works. Not even the engineers that built the thing. They work in groups as a collective intelligence, and use cognitive artifacts (CAD systems, simulators, automated design) to enhance their productive capacity. But still, we manage to build and fly them safely and routinely. There is no law of nature which says that human beings can't understand something which is smarter than them. Deep Blue is way smarter than me or any other human being at Chess. But its operation and move selection is no mystery to anyone who cares to investigate its inner workings.
- maaku 12y ago
  > That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again. Actually I am almost completely unpersuaded by the arguments of Bostrom, Yudkowsky, et al, at least in their presented strong form. Superintelligent AI is not a magical black box with infinite compute capacity, the two assumptions which really underlie the scary existential risk scenarios if you look closely. It is not at all clear to me that there will not exist mechanisms by which superintelligences can be contained or their proposed actions properly vetted. I have a number of ideas for this myself. In the weak form it is true that for most of the history of AI its proponents have more or less ignored the issue of machine morality. There were people in the 90's and earlier who basically advocated for building superintelligent AI and then sitting back and letting it do its thing to enact a positive singularity. That indeed would have been crazy stupid. Kudos to Yudkowsky and others for pointing out that intelligence and morality (mostly?) orthogonal. But it's a huge unjustified leap from "machines don't necessarily share our values" to "the mere existence of a superintelligent amoral machine will inevitably result in the destruction of humanity." (not an actual quote)
  - csallen 12y ago
    You're right: AGI might not be as powerful as some predict it can be. And there might be a way to contain it if it is. And we might be able to enforce said containment strategy on researches worldwide. And the AGI might be benevolent even if it is super powerful and uncontainable. The authors above are unpersuasive in arguing that these things are inevitable, but they are persuasive in arguing that these things are possible. And if they are possible, then what matters to me is their likelihood. Based on current estimates, are we talking about a 50-50 chance here? 80-20? 99-1? 99.999-0.001? If we're talking about the future of the human race, these chances matter. But instead it seems everyone is just waving their hands at the what-ifs simply because they aren't 100%.
  - Micaiah_Chang 12y ago
    I agree with a lesser form (timescale on which the AI self improves is probably going to be long enough for humans to deal with it) of your first argument, but I'm confused at how, given 'merely' lots of compute capacity can be countered by 'a number of ideas for this myself'. Security for human level threats is already very poor, and we already have a relatively good threat model. If you suppose that an AI could have radically different incentives and vectors than than a human, it seems implausible that you could be secure, even in practice. I suppose you could say that these would be implemented in time, but it's not at all clear to me that a humanity which has trouble coordinating to stop global warming or unilateral nuclear disarmament would recognize this in time. On the other hand, I'm slightly puzzled by why you think there's a huge unjustified leap between lack of value alignment and threat to the human race. Does most of your objection lie in 1) the lack of threat any given superintelligent AI would pose, because they're not going to be that much smarter than humans or 2) the lack of worry that they'll do anything too harmful to humans, because they'll do something relatively harmless, like trade with us or go into outer space? For 1, I buy that it'd be a lot smarter than humans, because even if it initially starts out as humanlike, it can copy itself and be productive for longer periods of time (imagine what you could do if you didn't have to sleep, or could eat more to avoid sleeping). And we know that any "superintelligent" machine can be at least as efficient as the smartest humans alive. I would still not want to be in a war against a nation full of von Neumanns on weapons development, Muhammads on foreign policy and Napoleons on Military strategy. For 2... I would need to hear specifics on how their morality would be close enough to ours to be harmless. But judging by your posts cross thread this doesn't seem to be the main point. By the way, I must thank you for your measured tone and willingness to engage on this issue. You seem to be familiar with some of the arguments, and perhaps just give different weights to them than I do. I've seen many gut level dismissals and I'm very happy to see that you're laying out your reasoning process.
AndrewKemendo 12y ago
Could you argue against the position directly? No I can't because there isn't a way to argue it. I can't argue that it would be safe because I have nothing to point to that says it will be or even could be. We simply do not know enough about how it would be built to reason on it. It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence. Correct, but they don't even reference a technical road map because there isn't one. It's too early to tell. Do you have any points regarding this particular difference of opinion? Not really, other than my own opinions which are as baseless as the others. I mean I have my own opinions and thoughts on things but they aren't empirically based at all, which is my main contention. We can't reason, let alone start making policy on stuff that we haven't the slightest idea how to build.
- maaku 12y ago
  Well, color me disappointed. I think there are many arguments against the viewpoints of MIRI and Bostrom, both discrediting the perceived risk of super-intelligences and the the idea that we could build a "provably friendly" alternative. But I don't think it's at all a fair criticism to say that you need a technical roadmap to engage in that debate. I would rather say that they need to be willing to discuss specific architectural details in order to even sit at the table, sure. The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates.
  - AndrewKemendo 12y ago
    I would rather say that they need to be willing to discuss specific architectural details in order to even sit at the table, sure. We are in violent agreement then. When I say roadmap I certainly don't intend to mean that each step needs to be perfectly detailed. The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates. Exactly exactly and well said! That is 100% the point. Mind meld complete.
    - 12y ago

by maaku 12y ago

csallen 12y ago

I'm not worried about the immediate future at all. We definitely haven't reached the point where hastily-implemented regulations will do more good than harm. That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again.

darkmighty 12y ago
Maybe it's precisely because the it can't be argued against that it's not much use? There's no way we could tell it will ever happen in the moment, no way to tell how intelligent we can make computers, and so on. You can't reliably argue for or against. So what's the point? I mean, you may want to form a group and discuss those scenarios, but it's not excusable to spread fear mongering in the media with no substance like that. It's unscientific in a way. We don't really need to start writing traffic rules for flying cars.
- AndrewKemendo 12y ago
  We don't really need to start writing traffic rules for flying cars. This is a great metaphor and one I wish I would have thought up sooner.
  - maaku 12y ago
    It's a terrible metaphor, or at least it argues against your own position. If flying cars were in fact something on the horizon, we would need discussions about air spaces, commute lanes, and traffic rules. Otherwise lethal mid-air collisions would be much more probable, resulting in significant bystander and property damage as these wreckage fall out of the sky. And, no surprise, we're seeing exactly this discussion going on right now about drones.
- csallen 12y ago
  I disagree that it can't be argued against. It's a set of logical steps, any one of which can be critiqued: 1 - There's no real dispute that computers will eventually be as "powerful" as brains. In fact, it's likely that they will one day be far more powerful. 2 - Assuming the brain evolved naturally, there's no reason to assume humans won't eventually duplicate the software once we have the hardware. It's really only a matter of when. 3 - An AGI with equal/more power than the human brain will, eventually, be able change its own code and improve upon itself. 4 - Given the characteristics of intelligence as it has been observed thus far in nature, even small increases in IQ lead to massive improvements in capability that are difficult for lesser intelligences to comprehend. A sufficiently advanced AGI would not only be highly capable, but would quite possibly be too smart for us to predict how it might behave. This is a dangerous combination. Further complicating things, we might hit step 2 on accident, without realizing that we hit it. Or some group might accomplish step 2 in secret. What I'd like to see someone do is argue that the chance of these things happening so small and/or the consequences so minuscule that it's not worth worrying about or planning for.
  - maaku 12y ago
    Step 4 is not at all obvious and would require some significant justification. General intelligence is, well, general. My primate brain may not be able at all to intuit what it is like to move on a 7-dimensional hyper-surface in an 11-dimensional space, but thanks to the wonders of formalized mathematics I can work out and tell you anything you want to know about higher-dimensional geometry. If the super-intelligence itself is computable, and we have the machinery to verify portions of the computation ourselves, it is in principle understandable. Of course there will be computational limits, but that's hardly anything new. There is no single person who can tell you absolutely everything about how a Boeing 787 works. Not even the engineers that built the thing. They work in groups as a collective intelligence, and use cognitive artifacts (CAD systems, simulators, automated design) to enhance their productive capacity. But still, we manage to build and fly them safely and routinely. There is no law of nature which says that human beings can't understand something which is smarter than them. Deep Blue is way smarter than me or any other human being at Chess. But its operation and move selection is no mystery to anyone who cares to investigate its inner workings.
maaku 12y ago
> That said, you put it exactly right. The arguments about the potential long-term risk are persuasive regardless of where we are today. It bothers me to see this argument ignored time and time again. Actually I am almost completely unpersuaded by the arguments of Bostrom, Yudkowsky, et al, at least in their presented strong form. Superintelligent AI is not a magical black box with infinite compute capacity, the two assumptions which really underlie the scary existential risk scenarios if you look closely. It is not at all clear to me that there will not exist mechanisms by which superintelligences can be contained or their proposed actions properly vetted. I have a number of ideas for this myself. In the weak form it is true that for most of the history of AI its proponents have more or less ignored the issue of machine morality. There were people in the 90's and earlier who basically advocated for building superintelligent AI and then sitting back and letting it do its thing to enact a positive singularity. That indeed would have been crazy stupid. Kudos to Yudkowsky and others for pointing out that intelligence and morality (mostly?) orthogonal. But it's a huge unjustified leap from "machines don't necessarily share our values" to "the mere existence of a superintelligent amoral machine will inevitably result in the destruction of humanity." (not an actual quote)
- csallen 12y ago
  You're right: AGI might not be as powerful as some predict it can be. And there might be a way to contain it if it is. And we might be able to enforce said containment strategy on researches worldwide. And the AGI might be benevolent even if it is super powerful and uncontainable. The authors above are unpersuasive in arguing that these things are inevitable, but they are persuasive in arguing that these things are possible. And if they are possible, then what matters to me is their likelihood. Based on current estimates, are we talking about a 50-50 chance here? 80-20? 99-1? 99.999-0.001? If we're talking about the future of the human race, these chances matter. But instead it seems everyone is just waving their hands at the what-ifs simply because they aren't 100%.
- Micaiah_Chang 12y ago
  I agree with a lesser form (timescale on which the AI self improves is probably going to be long enough for humans to deal with it) of your first argument, but I'm confused at how, given 'merely' lots of compute capacity can be countered by 'a number of ideas for this myself'. Security for human level threats is already very poor, and we already have a relatively good threat model. If you suppose that an AI could have radically different incentives and vectors than than a human, it seems implausible that you could be secure, even in practice. I suppose you could say that these would be implemented in time, but it's not at all clear to me that a humanity which has trouble coordinating to stop global warming or unilateral nuclear disarmament would recognize this in time. On the other hand, I'm slightly puzzled by why you think there's a huge unjustified leap between lack of value alignment and threat to the human race. Does most of your objection lie in 1) the lack of threat any given superintelligent AI would pose, because they're not going to be that much smarter than humans or 2) the lack of worry that they'll do anything too harmful to humans, because they'll do something relatively harmless, like trade with us or go into outer space? For 1, I buy that it'd be a lot smarter than humans, because even if it initially starts out as humanlike, it can copy itself and be productive for longer periods of time (imagine what you could do if you didn't have to sleep, or could eat more to avoid sleeping). And we know that any "superintelligent" machine can be at least as efficient as the smartest humans alive. I would still not want to be in a war against a nation full of von Neumanns on weapons development, Muhammads on foreign policy and Napoleons on Military strategy. For 2... I would need to hear specifics on how their morality would be close enough to ours to be harmless. But judging by your posts cross thread this doesn't seem to be the main point. By the way, I must thank you for your measured tone and willingness to engage on this issue. You seem to be familiar with some of the arguments, and perhaps just give different weights to them than I do. I've seen many gut level dismissals and I'm very happy to see that you're laying out your reasoning process.

AndrewKemendo 12y ago

Could you argue against the position directly? No I can't because there isn't a way to argue it. I can't argue that it would be safe because I have nothing to point to that says it will be or even could be. We simply do not know enough about how it would be built to reason on it. It's not within the scope of the quoted works to elaborate a technical roadmap or provide firm dates for the arrival of superhuman machine intelligence. Correct, but they don't even reference a technical road map because there isn't one. It's too early to tell. Do you have any points regarding this particular difference of opinion? Not really, other than my own opinions which are as baseless as the others. I mean I have my own opinions and thoughts on things but they aren't empirically based at all, which is my main contention. We can't reason, let alone start making policy on stuff that we haven't the slightest idea how to build.

maaku 12y ago
Well, color me disappointed. I think there are many arguments against the viewpoints of MIRI and Bostrom, both discrediting the perceived risk of super-intelligences and the the idea that we could build a "provably friendly" alternative. But I don't think it's at all a fair criticism to say that you need a technical roadmap to engage in that debate. I would rather say that they need to be willing to discuss specific architectural details in order to even sit at the table, sure. The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates.
- AndrewKemendo 12y ago
  I would rather say that they need to be willing to discuss specific architectural details in order to even sit at the table, sure. We are in violent agreement then. When I say roadmap I certainly don't intend to mean that each step needs to be perfectly detailed. The field of "AI safety" needs more engineering and less philosophy. But that doesn't require unity around a single roadmap, or any concept of deliverable dates. Exactly exactly and well said! That is 100% the point. Mind meld complete.
  - 12y ago