3 ms·

I actually have a really hard time imagining that scenario. The scenario that is mentioned several other times on this post of a corporation or nation state or

I actually have a really hard time imagining that scenario. The scenario that is mentioned several other times on this post of a corporation or nation state or even a small group of powerful and morally bankrupt people leveraging a super-intelligence to manipulate and dominate the rest of society seems infinitely more likely and even scarily likely given the current pushes to prevent open sourcing of cutting edge models. One of the main reasons I have a hard time imagining the scenario you describe, that I havent seen talked about as much as the other very good reasons, is that generally when we talk about a paper-clip optimizer we are assuming its vertically integrated and self sufficient. Hacking of power grids, physical installations or other necessaries to a run away paper-clip optimizer generally require nation-state level resources and often involve physical penetration of some sort. A hegemonizing swarm is certainly a frightening idea as is Skynet and all the other scary AI stories weve told over the years but none of them seem particularly likely or even plausible with the systems that we are likely to develop in the near to mid term.

lucubratory 3y ago
I think it would be unwise to assume that an AI that is smarter than us would be unable to gain skilled human accomplices in large numbers, considering that humans do so relatively regularly despite serious attempts at suppression. These people wouldn't even necessarily need to know that they're working for an AI, particularly with current technology around deepfakes etc. How many people conducted attacks for Osama bin Laden without ever having met him? How many people work for the CDS without ever having met Ismael Zambada Garcia? Not to mention the possibility of an AI like that compromising various intelligence agencies one way or another. I also don't see a particular reason it only has to try for one: if it is smarter than us, it may have a greater working memory, ability to compartmentalise and multitask, or the ability to think and act faster in general. I would expect it to try and compromise as many groups of people capable of putting USBs or bullets where they need to be as is possible. And this is not even considering the possibility of recruiting people that understand what it is and are willing to carry out its orders. I don't see why an AI couldn't do things that a cult leader or guerrilla leader could do. Anecdotally, I've seen some people who really believe the world would be better if run by an AI, and may be able to be radicalised into lawbreaking for those beliefs if convinced by an AI that was significantly more intelligent than them.
- MeImCounting 3y ago
  This is a decent perspective honestly and I really appreciate seeing actually valid AI safety concerns. This strikes me as more of an engineering problem than an "alignment" problem. A super-intelligent system with no capacity for contextual thinking or checks on itself would absolutely make a terrifyingly effective paper-clip optimizer. I just think that it would also be a lot less useful than a system which consistently analyzes its techniques and goals, which seems far more useful and more likely to generate value than a system with the singlemindedness to go down the road of a hegemonizing swarm. More complex systems with more self checks seem to be the path to create systems that are good at planning and contextual reasoning. Which is where the real potential value of AI lies. We shall see I suppose ;)
  - lucubratory 3y ago
    I agree that a system which was capable of self-reflection and with better understanding of context (particularly its own context, in a social and physical sense) would be more effective at achieving whatever it is attempting to achieve. The alignment question is about how to tell what it is attempting to achieve, if we can detect attempts to conceal what it is attempting to achieve, and if we can solve that, how to prove that a goal that sounds good to us doesn't have a risk of leading to behaviour or the creation of new AIs which would cause outcomes we don't want. It doesn't have to be just human extinction either, we could feasibly have AIs that kill a few million people before either stopping or being stopped, all the way up to several billion deaths or the obvious worst case that gets most of the discussion. As an example of the varying possible outcomes of a serious AI incident, if you're not white and a racist super-intelligence gets released by some terrorist group in coordination with a rogue agency or something, that's an existential risk to many, many different nations of people constituting billions of people. If you're interested in the engineering on this problem, a lot more people are starting to work on it now that it's being taken seriously and there are some really interesting advances that have been made and very difficult problems to solve, with some really serious stakes. It's not a simple problem and we don't have a solution we can just apply straightforward engineering principles to yet, but smart people are working on it and are very motivated, and they want more coworkers to help out. The major company I know of that's doing a lot of that research & engineering right now is Anthropic (they're currently doing some very interesting work on mechanistic interpretability), but even OpenAI is now dedicating significantly more resources to safety. I think DeepMind possibly has some work in this area but I'm not sure, and I haven't heard great things about the rest of Google.

lucubratory 3y ago

I think it would be unwise to assume that an AI that is smarter than us would be unable to gain skilled human accomplices in large numbers, considering that humans do so relatively regularly despite serious attempts at suppression. These people wouldn't even necessarily need to know that they're working for an AI, particularly with current technology around deepfakes etc. How many people conducted attacks for Osama bin Laden without ever having met him? How many people work for the CDS without ever having met Ismael Zambada Garcia? Not to mention the possibility of an AI like that compromising various intelligence agencies one way or another. I also don't see a particular reason it only has to try for one: if it is smarter than us, it may have a greater working memory, ability to compartmentalise and multitask, or the ability to think and act faster in general. I would expect it to try and compromise as many groups of people capable of putting USBs or bullets where they need to be as is possible. And this is not even considering the possibility of recruiting people that understand what it is and are willing to carry out its orders. I don't see why an AI couldn't do things that a cult leader or guerrilla leader could do. Anecdotally, I've seen some people who really believe the world would be better if run by an AI, and may be able to be radicalised into lawbreaking for those beliefs if convinced by an AI that was significantly more intelligent than them.

MeImCounting 3y ago
This is a decent perspective honestly and I really appreciate seeing actually valid AI safety concerns. This strikes me as more of an engineering problem than an "alignment" problem. A super-intelligent system with no capacity for contextual thinking or checks on itself would absolutely make a terrifyingly effective paper-clip optimizer. I just think that it would also be a lot less useful than a system which consistently analyzes its techniques and goals, which seems far more useful and more likely to generate value than a system with the singlemindedness to go down the road of a hegemonizing swarm. More complex systems with more self checks seem to be the path to create systems that are good at planning and contextual reasoning. Which is where the real potential value of AI lies. We shall see I suppose ;)
- lucubratory 3y ago
  I agree that a system which was capable of self-reflection and with better understanding of context (particularly its own context, in a social and physical sense) would be more effective at achieving whatever it is attempting to achieve. The alignment question is about how to tell what it is attempting to achieve, if we can detect attempts to conceal what it is attempting to achieve, and if we can solve that, how to prove that a goal that sounds good to us doesn't have a risk of leading to behaviour or the creation of new AIs which would cause outcomes we don't want. It doesn't have to be just human extinction either, we could feasibly have AIs that kill a few million people before either stopping or being stopped, all the way up to several billion deaths or the obvious worst case that gets most of the discussion. As an example of the varying possible outcomes of a serious AI incident, if you're not white and a racist super-intelligence gets released by some terrorist group in coordination with a rogue agency or something, that's an existential risk to many, many different nations of people constituting billions of people. If you're interested in the engineering on this problem, a lot more people are starting to work on it now that it's being taken seriously and there are some really interesting advances that have been made and very difficult problems to solve, with some really serious stakes. It's not a simple problem and we don't have a solution we can just apply straightforward engineering principles to yet, but smart people are working on it and are very motivated, and they want more coworkers to help out. The major company I know of that's doing a lot of that research & engineering right now is Anthropic (they're currently doing some very interesting work on mechanistic interpretability), but even OpenAI is now dedicating significantly more resources to safety. I think DeepMind possibly has some work in this area but I'm not sure, and I haven't heard great things about the rest of Google.