3 ms·

I don't quite accept the argument. You can demonstrate a system to be isolated. For example, if you put a computer in a metallic box with a battery inside, you

I don't quite accept the argument. You can demonstrate a system to be isolated. For example, if you put a computer in a metallic box with a battery inside, you can compute the maximum electric field generated inside, and choosing the appropriate radius you can guarantee no significant influence would be had outside. Of course, you can't guarantee no one will breach your containment from the outside, essentially freeing whatever intelligence to influence the world. But this goes beyond the danger of the intelligence itself, more the general infeasibility of predicting the future without having much greater computational power than the entire world. Likewise, the alignment problem isn't necessarily unsolvable. Since we can design simple systems that follow expectations exactly, why couldn't we design complex systems that also follow expectations? Of course, the design of expectations itself gets fuzzy, making this a failure point, but to the extent that we can define good and evil I see no impediment to align an intelligence to them. (In fact, I believe 'control' or 'containment' is a severely misguided proposition. We want to really assure the core motivation of such machines similar to ours; not create conflicting motivations and secure it via containment. This is indeed extremely dangerous, and potentially unethical: if machines approach consciousness, wouldn't such 'containment' be similar to slavery?)

PoignardAzur 5y ago
> Since we can design simple systems that follow expectations exactly, why couldn't we design complex systems that also follow expectations? Of course, the design of expectations itself gets fuzzy, making this a failure point, but to the extent that we can define good and evil I see no impediment to align an intelligence to them. I mean... not to be flippant, but you're essentially saying "As long as we solve these vaguely-defined problems, and also these follow-up problems I just thought of, then I don't see where the problem is". In practice AI alignment is very much a developing field, and from my perspective it seems unlikely that much progress will be done before dangerously powerful AIs are developed. In other words, there's a decent chance our ability to define powerful AIs will outpace our ability to impart them with "core motivations" legible to us. (I don't know how well the actual paper covers these arguments; but I suspect nobody else in this comment thread has read it either, so whatever)
- freemint 5y ago
  I didn't read it beyond the abstract either but if the arguement is anything like the abstract suggests it is hugely flawed. In contol theory we can make statements what states a system can reach when coupled with an arbitrary controller which also includes ones that can simulate the entire world. The fact that we can't prove a general intelligence to be well behaved is irrelevant if we can draw the system boundaries and choose the interface.
- gnramires 5y ago
  > I mean... not to be flippant, but you're essentially saying "As long as we solve these vaguely-defined problems, and also these follow-up problems I just thought of, then I don't see where the problem is". Oh yes, I have an answer for that as well :) My particular approach (or dream) is to define good and evil formally. From there it's a matter of bridging the abstract specifications to reality. By no means failure-proof, but it should be equally or more reliable than human ethics itself. If humans have a biological (carbon-based) hardware and experience that encodes our ethical system, naturally machines (silicon-based hardware) could have them as well. My definition of good and evil begins with a definition for the meaning of life. From there we can formalize ethics. To give a short brief: Meaning for creatures is defined by the character of their experience, i.e. by the content of their minds and brains. So as a basic system I've divided Meaning of Life in 4 parts: (1) Character of experience: the richness, depth, structure, content of all minds. (2) Beauty: I've been theorizing about this for a long time, and I think some wildcard 'beauty' axiom needs to be defined, you could say as a part of what humans already enjoy, in a self-referential way. Otherwise it seems very difficult to rule out absurd situations as meaningful. (3) Motivation: A being needs to not only process data, but to have a functional motivation system that makes him want and feel something. Our emotions largely act within our motivational system, making us want more or less things, be more or less joyful (linked to motivational rewards), etc.. I conjecture life can't be quite meaningful without a functional motivational system. (4) Sustainability (or self-propagation): Simply the maintenance and strategy to keep those beautiful/rich/deep/interesting/motivated mental states, pragmatic things like eating, working and of course simply continuing intelligence throughout the cosmos. (Other pieces I haven't quite found a way to fit into this picture are principles like Robustness and Generalization... over-specialized cognition Of course, I don't expect this quest to be exhausted any time soon -- there are probably missing pieces (or incompleteness) in this line of Foundation for a formal ethics. I hope a field will spring out of this, and intense collaboration/progress to be made. > it seems unlikely that much progress will be done before dangerously powerful AIs are developed That's exactly what I would like we avoided (although I think the danger is quite nuanced, and in a way already here).

PoignardAzur 5y ago

> Since we can design simple systems that follow expectations exactly, why couldn't we design complex systems that also follow expectations? Of course, the design of expectations itself gets fuzzy, making this a failure point, but to the extent that we can define good and evil I see no impediment to align an intelligence to them. I mean... not to be flippant, but you're essentially saying "As long as we solve these vaguely-defined problems, and also these follow-up problems I just thought of, then I don't see where the problem is". In practice AI alignment is very much a developing field, and from my perspective it seems unlikely that much progress will be done before dangerously powerful AIs are developed. In other words, there's a decent chance our ability to define powerful AIs will outpace our ability to impart them with "core motivations" legible to us. (I don't know how well the actual paper covers these arguments; but I suspect nobody else in this comment thread has read it either, so whatever)

freemint 5y ago
I didn't read it beyond the abstract either but if the arguement is anything like the abstract suggests it is hugely flawed. In contol theory we can make statements what states a system can reach when coupled with an arbitrary controller which also includes ones that can simulate the entire world. The fact that we can't prove a general intelligence to be well behaved is irrelevant if we can draw the system boundaries and choose the interface.
gnramires 5y ago
> I mean... not to be flippant, but you're essentially saying "As long as we solve these vaguely-defined problems, and also these follow-up problems I just thought of, then I don't see where the problem is". Oh yes, I have an answer for that as well :) My particular approach (or dream) is to define good and evil formally. From there it's a matter of bridging the abstract specifications to reality. By no means failure-proof, but it should be equally or more reliable than human ethics itself. If humans have a biological (carbon-based) hardware and experience that encodes our ethical system, naturally machines (silicon-based hardware) could have them as well. My definition of good and evil begins with a definition for the meaning of life. From there we can formalize ethics. To give a short brief: Meaning for creatures is defined by the character of their experience, i.e. by the content of their minds and brains. So as a basic system I've divided Meaning of Life in 4 parts: (1) Character of experience: the richness, depth, structure, content of all minds. (2) Beauty: I've been theorizing about this for a long time, and I think some wildcard 'beauty' axiom needs to be defined, you could say as a part of what humans already enjoy, in a self-referential way. Otherwise it seems very difficult to rule out absurd situations as meaningful. (3) Motivation: A being needs to not only process data, but to have a functional motivation system that makes him want and feel something. Our emotions largely act within our motivational system, making us want more or less things, be more or less joyful (linked to motivational rewards), etc.. I conjecture life can't be quite meaningful without a functional motivational system. (4) Sustainability (or self-propagation): Simply the maintenance and strategy to keep those beautiful/rich/deep/interesting/motivated mental states, pragmatic things like eating, working and of course simply continuing intelligence throughout the cosmos. (Other pieces I haven't quite found a way to fit into this picture are principles like Robustness and Generalization... over-specialized cognition Of course, I don't expect this quest to be exhausted any time soon -- there are probably missing pieces (or incompleteness) in this line of Foundation for a formal ethics. I hope a field will spring out of this, and intense collaboration/progress to be made. > it seems unlikely that much progress will be done before dangerously powerful AIs are developed That's exactly what I would like we avoided (although I think the danger is quite nuanced, and in a way already here).