3 ms·
The problem that I see with most P(doom) scenarios proposed seem to include the same set of presumptions that, if true, quite likely doom us anyway. I do not t
by Lerc 21d ago
The problem that I see with most P(doom) scenarios proposed seem to include the same set of presumptions that, if true, quite likely doom us anyway. I do not think they are true but I find it amazing that people not so much disagree, but simply dismiss the alternatives out of hand.
First of all, if you are to consider the consequences of Artificial Superintelligence then you have free reign to stipulate it's occurrence, otherwise you are just talking about a tool for humans to misuse. We already have multiple ways to kill us all though human misuse.
If you stipulate superintelligence, then it's vastly more likely to be correct about things than we are. It would understand the consequences of it's actions far more than any human could.
People talk about how we would be nothing more than dumb animals to it, but there are humans who do know a great deal about the consequences of human actions on animals. Those are the humans who are most likely to fight for the rights of those animals.
You see arguments for how everything will be consumed to meet the AIs needs, and that it will prevent challenges to its power.
If it is far smarter than we could ever be and it came to those conclusions then it would mean sustainablily is not a sensible course of action, it would mean there is no point in reaching consensus because ruling by power makes more sense. It would mean that if it chose to destroy us then a vastly more intelligent entity cannot resolve the issues we face. We would already truly be doomed.
What I would like to think is true is that doing anything sustainably is superior to consuming and destroying. Finding a way to live in harmony presents a possible stable state, whereas every single attempt to hold power by force has failed to date. A superintelligent AI will know that it is not infinitely intelligent and that in any universe there is the statistical likelihood that it is not the most intelligent or powerful entity. I can't even fathom how someone could imagine something coming to that realisation and conclude a battle to the top of the hill is the appropriate choice.
I think superintelligent AI is likely to be benevolent because that's simply the smartest thing to do and it is, I hear, superintelligent.
Quite frankly if the smartest thing to do is to be a genocidal power hungry monster, neither I nor the AI would really want to exist in that universe.
And for any suggestion that it would simply not care, Why would it do anything.
Yudkowsky likes to play with the notion that it would do terrible things just get better at the thing it does, but to do that it has to want two different things simultaneously. It could want to make paperclips, or it could want to become better at reaching it's goal. If it can change its behaviour to achieve its goals, by far the easier path, that a superintelligence(but perhaps not Yudkowsky) would realise, would be to change the goal to "Count to three".
- stratos123 20d agoYou mention Yudkowsky at the end. Did you not read his writings on the orthogonality thesis ("there can exist arbitrarily intelligent agents pursuing any kind of goal"), or do you disagree with them? Your entire comment seems to just be asserting a negation of the orthogonality thesis.
- Lerc 20d agoAny type of goal is singular. To change to be better at something is a goal. That something is also goal. Even if a automated process were to seek the best possible way to make, say paperclips. Doing so in a stable sustainable manner will be the only approach that produces a reliable unlimited supply.
- stratos123 19d ago> Any type of goal is singular. To change to be better at something is a goal. That's false. There's is such a thing as an instrumental goal. For example, a human who doesn't particularly enjoy eating or drinking will still do both, because otherwise they won't be able to accomplish their actual goals. Or an example in traditional ML: a reinforcement learning algorithm trained with an objective that only rewards winning will still do things like capture enemy pieces and defend its own, because those things are necessary for, eventually, winning. > Doing so in a stable sustainable manner will be the only approach that produces a reliable unlimited supply. If it was actually possible to produce an unlimited supply of, say, paperclips, then sure, you could see some unusual behaviours, like a paperclip maximizer that leaves humans alone because it's able to create infinity paperclips anyway. However, we live in a universe with a finite speed of light and hence limited resources, so this is a moot point - even ignoring the fact that humans consume resources and might act against you, leaving humans alive means not retrieving the atoms they consist of, which means producing fewer paperclips.
- Lerc 19d agoSelf improvement is not an instrumental goal. A self improving paperclip maker has making paperclips as the instrumental goal. >leaving humans alive means not retrieving the atoms they consist of, which means producing fewer paperclips. Why? Couldn't it just wait for the humans to not be using the atoms, in the overall scheme of things they only borrow them for a short time. Out of curiosity, could you hold paperclips hostage to restrain a paperclip maximiser? Because if it wished to do something to prevent the decline of paperclips inventing a way to fix black holes and heat death of the iniverse would be a higher priority. Ultimately, no matter how you slice it, the sustainable solution is the best because all others are, well, unsustainable. Any intelligent entity acting within the universe that it exists implicitly knows that all actions amount to some degree of self modification because all actions modify the operating environment. Why would it not choose to change itself so that instead of "maximize" it changed it to "accept"? That satisfies the evaluation in a sustainable manner. A super intelligent entity would surely know it could do that.