4 ms·

The idea of "alignment" is pretty troubling to me. Do these people think that they, or those in power have achieved moral perfection, such that it would be good

The idea of "alignment" is pretty troubling to me. Do these people think that they, or those in power have achieved moral perfection, such that it would be good to have extremely powerful AI systems under their control, aligned with them? Imagine if the US or any other government of 1800s came gained so much power, 'locking-in' their repugnant values as the moral truth, backed by total control of the world.

cwillu 3y ago
Locking in values in that way would be considered a failure of alignment by anyone I've ever read talk about alignment. Not the worst possible failure of alignment (compared to locking in “the value of the entity legally known as OpenAI”, for example), but definitely a straightforward failure to achieve alignment.
- archon1410 3y ago
  I know it's a theme, MacAskill discusess it in his book. In practice, this is the direction all the "AI safety" departments and organisations seem to be going into. A world where everyone is paperclipped is probably better than one controlled by psychopathic totalitarian human overlords supported by AI, yet the direction of current research seems to leading us into the latter scenario.
  - oefnak 3y ago
    Are you serious? Paperclipped means the end of the human race. Why would you prefer that over anything?
space_fountain 3y ago
I think the worry with the vision of AI under the control of who ever happens to want to use them is that someday that might be the equivalent of giving everyone the keys to a nuclear silo. We know the universe makes it easier to destroy than to create and we know that AI may unleash tremendous power and there's nothing we've seen about the world that means it's guaranteed to stay nice and stable
- bobthepanda 3y ago
  If anything it’ll be a subtle bug that wipes us out. The 2003 Northeast blackout that affected 50 million people was partially caused by a race condition. https://www.theregister.com/2004/04/08/blackout_bug_report/ https://www.theregister.com/2004/04/08/blackout_bug_report/
- JohnFen 3y ago
  If they think what they're working on is as bad as nuclear weapons, then why are they even working on it?
ethanbond 3y ago
No, I don’t think they do, which is another point in the “alignment is a very big problem” column. The problem of defining “what’s a good outcome” is a sub problem of alignment.
m4rtink 3y ago
Don't nuclear weapons kinda cause ssomething like this ? At least the blocks that have the become effectively impossible to destroy & can better spread their ideology.

cwillu 3y ago

Locking in values in that way would be considered a failure of alignment by anyone I've ever read talk about alignment. Not the worst possible failure of alignment (compared to locking in “the value of the entity legally known as OpenAI”, for example), but definitely a straightforward failure to achieve alignment.

archon1410 3y ago
I know it's a theme, MacAskill discusess it in his book. In practice, this is the direction all the "AI safety" departments and organisations seem to be going into. A world where everyone is paperclipped is probably better than one controlled by psychopathic totalitarian human overlords supported by AI, yet the direction of current research seems to leading us into the latter scenario.
- oefnak 3y ago
  Are you serious? Paperclipped means the end of the human race. Why would you prefer that over anything?

space_fountain 3y ago

I think the worry with the vision of AI under the control of who ever happens to want to use them is that someday that might be the equivalent of giving everyone the keys to a nuclear silo. We know the universe makes it easier to destroy than to create and we know that AI may unleash tremendous power and there's nothing we've seen about the world that means it's guaranteed to stay nice and stable

bobthepanda 3y ago
If anything it’ll be a subtle bug that wipes us out. The 2003 Northeast blackout that affected 50 million people was partially caused by a race condition. https://www.theregister.com/2004/04/08/blackout_bug_report/ https://www.theregister.com/2004/04/08/blackout_bug_report/
JohnFen 3y ago
If they think what they're working on is as bad as nuclear weapons, then why are they even working on it?

ethanbond 3y ago

No, I don’t think they do, which is another point in the “alignment is a very big problem” column. The problem of defining “what’s a good outcome” is a sub problem of alignment.

m4rtink 3y ago

Don't nuclear weapons kinda cause ssomething like this ? At least the blocks that have the become effectively impossible to destroy & can better spread their ideology.