3 ms·

He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from

He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. His risk calculations derive primarily from the AI itself, not weaponization of AI by others. Rich Sutton seems to agree with this take and embraces our extinction: https://www.youtube.com/watch?v=NgHFMolXs3U https://www.youtube.com/watch?v=NgHFMolXs3U

jeffparsons 3y ago
You don't have to be sure that it's possible, because the alternative we're comparing it to is absolutely useless. You only need to not be sure that it's impossible. Which is where most of us, him included, are today.
hollerith 3y ago
>He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. That's a little bit inaccurate: he believes that it is humanly possible to acquire a body of knowledge sufficient to align an AI (i.e., to aim it at basically any goal the creators decide to aim it at), but that it is extremely unlikely that any group of humans will manage to do before unaligned AI kills us all. There is simply not enough time because (starting from the state of human knowledge we have now) it is much easier to create an unaligned AI capable enough that we would be defenseless against it than it is to create an aligned AI capable enough to prevent the creation of the former. Yudkowsky and his team have been working the alignment problem for 20 years (though 20 years ago they were calling it Friendliness, not alignment). Starting around 2003, his team's plan was to create an aligned AI to prevent the creation of dangerously-capable unaligned AIs. He is so pessimistic and so unimpressed with his team's current plan (ie., to lobby for governments to stop or at least slow down frontier AI research to give humanity more time for some unforeseen miracle to rescue us) that he only started executing on it about 2 years ago even though he had mostly given up on his previous plan by 2015.
- oceanplexian 3y ago
  I wonder if I, too, can simply rehash the plot of Terminator 2: Judgment Day and then publish research papers on how we will use AI’s to battle it out with other AI’s.
  - pixl97 3y ago
    Yes, and honestly it's a good damned idea to start now because we're already researching autonomous drones and counter autonomous drones that are autonomous themselves. While the general plot of T2 is bullshit, the idea of autonomous weapon systems at scale should be an 'oh shit' moment for everyone.

jeffparsons 3y ago

You don't have to be sure that it's possible, because the alternative we're comparing it to is absolutely useless. You only need to not be sure that it's impossible. Which is where most of us, him included, are today.

hollerith 3y ago

>He is not really convinced you can align an AI that is intelligent past some threshold with aims of a single entity. That's a little bit inaccurate: he believes that it is humanly possible to acquire a body of knowledge sufficient to align an AI (i.e., to aim it at basically any goal the creators decide to aim it at), but that it is extremely unlikely that any group of humans will manage to do before unaligned AI kills us all. There is simply not enough time because (starting from the state of human knowledge we have now) it is much easier to create an unaligned AI capable enough that we would be defenseless against it than it is to create an aligned AI capable enough to prevent the creation of the former. Yudkowsky and his team have been working the alignment problem for 20 years (though 20 years ago they were calling it Friendliness, not alignment). Starting around 2003, his team's plan was to create an aligned AI to prevent the creation of dangerously-capable unaligned AIs. He is so pessimistic and so unimpressed with his team's current plan (ie., to lobby for governments to stop or at least slow down frontier AI research to give humanity more time for some unforeseen miracle to rescue us) that he only started executing on it about 2 years ago even though he had mostly given up on his previous plan by 2015.

oceanplexian 3y ago
I wonder if I, too, can simply rehash the plot of Terminator 2: Judgment Day and then publish research papers on how we will use AI’s to battle it out with other AI’s.
- pixl97 3y ago
  Yes, and honestly it's a good damned idea to start now because we're already researching autonomous drones and counter autonomous drones that are autonomous themselves. While the general plot of T2 is bullshit, the idea of autonomous weapon systems at scale should be an 'oh shit' moment for everyone.