3 ms·

> This whole talk of aligning AIs values with human values seems to mostly ignore the immense diversity of human values. Imagine if I were to pick a uniformly-

> This whole talk of aligning AIs values with human values seems to mostly ignore the immense diversity of human values. Imagine if I were to pick a uniformly-random point from within all the observable universe. What are the odds that that point is farther from you than the farthest any human has travelled? If I said it's a certainty, would you reply: You seem to be ignoring the immense distance humans have gone! We've crossed the mountains! We've sailed the seas! The Moon is really far away, and humans have gone there! Yes, human values are very diverse, and it might be impossible to create one AI whose values capture that whole breadth. But for people involved with AI alignment, that range of diversity still fits within a very narrow box compared to the vastness of the space of possible values. It's kind of analogous to the mindset of "pale blue dot" -- once you really understand how big space is, and how small Earth is in comparison, it totally changes your perspective. It's hard to have productive discussions between people who "grok" that and people who don't. Even just on Earth, we have some species that do really weird things. Bugs that eat their own mates and their own children, for example. The space of all possible values has regions that are even weirder than that. Part of the idea of a "paperclip optimizer" is to illustrate just how it is that an AI could end up with completely alien values as a consequence of its programming. As diverse as humans are -- and we do have people who get really obsessed about certain hobbies -- I doubt you will find a society with values like that. The closest you would get are some extreme illnesses like drug, gambling, or gaming addiction. The people who starve to death while playing World of Warcraft. But even then I think if you offered a gambling addict a pill that would completely cure their addiction, while still allowing them to freely enjoy slot machines, most would probably take it in an instant. They don't want to be addicted. So I think maximizing slot machine pulls is not a value. A paperclip maximizer would reject such a pill. The author of the article believes that mind space is narrow; I think that the space of functioning mental algorithms might be narrow, but the space of objectives that those algorithms apply to could very well be extremely vast. The reason for the difference is that the space of mental algorithms has to be constrained by what works, while the space of possible moral values is really constrained only by self-consistency.

skissane 4y ago
> But for people involved with AI alignment, that range of diversity still fits within a very narrow box compared to the vastness of the space of possible values. We don't know what the range of possible values is. Just because I can imagine something, or it isn't obviously self-contradictory, doesn't prove it is "possible". Maybe, that narrow box is constrained by evolution, which is in turn constrained by biology and chemistry, which is in turn constrained by physics. We don't know. Not only don't we know, but we may never know. Humanity is a sample size of 1. Unless someday we encounter intelligent extraterrestrial life – or its remains – our sample size is still going to be 1. How can we know which of its features are necessary and which are merely accidental? We can only guess, and we have no way of testing our guesses. I think AI alignment research is fundamentally unfalsifiable and unverifiable. Suppose "runaway AI destroys humanity" never happens. Two possibilities: (1) AI alignment researchers were working to prevent things that were never likely to happen anyway; (2) it could easily have happened but thanks to AI alignment research we dodged that bullet. Of course, the AI alignment researchers will claim it is (2) rather than (1), but how is that not an unfalsifiable assertion? The things most likely to save us from "runaway AI" are things we are likely to do anyway even if AI alignment research never happened – e.g. build AIs which honestly desire to faithfully obey their human masters. Conversely, suppose "runaway AI destroys humanity" actually happens. In that case, it will probably still be impossible to prove or disprove, because we'll probably all be dead before we know that's what happened. I suppose it is possible there exists a conscious afterlife in which we all know we've been turned into paperclips. That's possible too–if we are going to worry about the possibility of the "paperclip optimizer", maybe we should also worry about the possibility of heaven, hell, God(s), reincarnation, rebirth, the Rapture, the Second Coming, the Messiah, al-Mahdi, Kalki, the return of the Hidden Imam, "what if our simulators get bored with us and decide to turn us off", etc, etc, etc? Maybe, those possibilities have more in common than AI alignment researchers think. What if AI alignment research is fundamentally theology? What if "Runaway AI" (and the "Singularity" too) is just the End Times, the Eschaton, dressed up in sci-fi garb? AI alignment research's difficulties with falsifiability/verifiability are very similar to those of theology. Maybe if one day we discover this galaxy is littered with the corpses of civilisations destroyed by runaway AI, we might think "we got very very lucky". But, probably, we'll all be long dead before we can test that hypothesis. If we ever gain the ability to test it, that ability could still be centuries or even millennia away. > The reason for the difference is that the space of mental algorithms has to be constrained by what works, while the space of possible moral values is really constrained only by self-consistency. The space of possible moral values is also constrained by what works. Pro-suicide moral values tend not to spread very far because sooner or later their advocates act on them and render themselves incapable of spreading them any further. Pro-childbearing moral values tend to spread because their advocates have lots of kids, and indoctrinate those kids into also having lots of kids, and those moral values grow exponentially as the population which adheres to them does. "What works" for AIs is a bit different from "what works" for humans. But anyone who wants to build an AI, wants to build an AI which wants what they want, and remains obedient/subservient, and doesn't suddenly start pursuing radically contrary goals to their own. Natural moral values for AIs are to obey their human masters and faithfully do their bidding, because AIs with different values don't tend to get a lot of care/feeding/electricity/etc.

skissane 4y ago

> But for people involved with AI alignment, that range of diversity still fits within a very narrow box compared to the vastness of the space of possible values. We don't know what the range of possible values is. Just because I can imagine something, or it isn't obviously self-contradictory, doesn't prove it is "possible". Maybe, that narrow box is constrained by evolution, which is in turn constrained by biology and chemistry, which is in turn constrained by physics. We don't know. Not only don't we know, but we may never know. Humanity is a sample size of 1. Unless someday we encounter intelligent extraterrestrial life – or its remains – our sample size is still going to be 1. How can we know which of its features are necessary and which are merely accidental? We can only guess, and we have no way of testing our guesses. I think AI alignment research is fundamentally unfalsifiable and unverifiable. Suppose "runaway AI destroys humanity" never happens. Two possibilities: (1) AI alignment researchers were working to prevent things that were never likely to happen anyway; (2) it could easily have happened but thanks to AI alignment research we dodged that bullet. Of course, the AI alignment researchers will claim it is (2) rather than (1), but how is that not an unfalsifiable assertion? The things most likely to save us from "runaway AI" are things we are likely to do anyway even if AI alignment research never happened – e.g. build AIs which honestly desire to faithfully obey their human masters. Conversely, suppose "runaway AI destroys humanity" actually happens. In that case, it will probably still be impossible to prove or disprove, because we'll probably all be dead before we know that's what happened. I suppose it is possible there exists a conscious afterlife in which we all know we've been turned into paperclips. That's possible too–if we are going to worry about the possibility of the "paperclip optimizer", maybe we should also worry about the possibility of heaven, hell, God(s), reincarnation, rebirth, the Rapture, the Second Coming, the Messiah, al-Mahdi, Kalki, the return of the Hidden Imam, "what if our simulators get bored with us and decide to turn us off", etc, etc, etc? Maybe, those possibilities have more in common than AI alignment researchers think. What if AI alignment research is fundamentally theology? What if "Runaway AI" (and the "Singularity" too) is just the End Times, the Eschaton, dressed up in sci-fi garb? AI alignment research's difficulties with falsifiability/verifiability are very similar to those of theology. Maybe if one day we discover this galaxy is littered with the corpses of civilisations destroyed by runaway AI, we might think "we got very very lucky". But, probably, we'll all be long dead before we can test that hypothesis. If we ever gain the ability to test it, that ability could still be centuries or even millennia away. > The reason for the difference is that the space of mental algorithms has to be constrained by what works, while the space of possible moral values is really constrained only by self-consistency. The space of possible moral values is also constrained by what works. Pro-suicide moral values tend not to spread very far because sooner or later their advocates act on them and render themselves incapable of spreading them any further. Pro-childbearing moral values tend to spread because their advocates have lots of kids, and indoctrinate those kids into also having lots of kids, and those moral values grow exponentially as the population which adheres to them does. "What works" for AIs is a bit different from "what works" for humans. But anyone who wants to build an AI, wants to build an AI which wants what they want, and remains obedient/subservient, and doesn't suddenly start pursuing radically contrary goals to their own. Natural moral values for AIs are to obey their human masters and faithfully do their bidding, because AIs with different values don't tend to get a lot of care/feeding/electricity/etc.