4 ms·
I agree that a system which was capable of self-reflection and with better understanding of context (particularly its own context, in a social and physical sens
by lucubratory 3y ago
I agree that a system which was capable of self-reflection and with better understanding of context (particularly its own context, in a social and physical sense) would be more effective at achieving whatever it is attempting to achieve.
The alignment question is about how to tell what it is attempting to achieve, if we can detect attempts to conceal what it is attempting to achieve, and if we can solve that, how to prove that a goal that sounds good to us doesn't have a risk of leading to behaviour or the creation of new AIs which would cause outcomes we don't want. It doesn't have to be just human extinction either, we could feasibly have AIs that kill a few million people before either stopping or being stopped, all the way up to several billion deaths or the obvious worst case that gets most of the discussion. As an example of the varying possible outcomes of a serious AI incident, if you're not white and a racist super-intelligence gets released by some terrorist group in coordination with a rogue agency or something, that's an existential risk to many, many different nations of people constituting billions of people.
If you're interested in the engineering on this problem, a lot more people are starting to work on it now that it's being taken seriously and there are some really interesting advances that have been made and very difficult problems to solve, with some really serious stakes. It's not a simple problem and we don't have a solution we can just apply straightforward engineering principles to yet, but smart people are working on it and are very motivated, and they want more coworkers to help out.
The major company I know of that's doing a lot of that research & engineering right now is Anthropic (they're currently doing some very interesting work on mechanistic interpretability), but even OpenAI is now dedicating significantly more resources to safety. I think DeepMind possibly has some work in this area but I'm not sure, and I haven't heard great things about the rest of Google.