3 ms·

How are we doomed? Or what is the doomsday scenario you have in mind? From what I read we have: massive unemployment, AI making military decisions. What else?

hollerith 3y ago
Massive unemployment is not the main problem. No one has a good idea on how to control an AI that is more capable than people are over a broad range of pursuits. (Actually we don't need to be able to control such an AI, it suffices for it to be aligned with our long-term interests. No one knows how to do that either.) The Machine Intelligence Research Institute has been paying researchers to think up ideas for over 20 years. The executive director of that institute and its most senior researcher (Nate Soares and Eliezer Yudkowsky) have been publicly saying now for a few years that it is hopeless and that there is no way any of their institute's research directions will bear fruit before AI kills us all. They say that people just aren't smart enough to figure out how to control or align the powerful AIs that will be created soon if things keep on going the way they are going; consequently some sort of pause or halt on AI research -- almost certainly for at least a few decades -- is going to be necessary for us to survive. The people in charge of the AI labs care enough about their own survival and the survival of their families that they could probably be persuaded to adopt an effective alignment method if someone or some group can come up with one, but it is highly unlikely that anyone will do so in time. Note that whether the effective alignment method is used to align the AI with the broad interests of the entire human population or whether it is used to align the AI to the particular interests of only the leaders of the AI lab is not a particularly important consideration at the current time: it only becomes important in the very unlikely event that someone somewhere finds an effective alignment method. AI is essentially a subfield of the broader field of optimization; an AI applies an impressive amount of optimization pressure. (The human brain can, too.) When you (or a team of researchers) apply tons of optimization pressure, or create an AI that does so, you steer reality towards some "optimization point" or goal. Most optimization points are incompatible with continued human survival when enough optimization pressure is applied towards that point. And no one knows how to aim a powerful optimization process at the few points that are consistent with human survival and human flourishing. No one has even provided a mathematically precise specification of an optimization point consistent with human survival (and mathematical precision would be quite useful and I suspect also necessary for the specification to be useful to a team creating an AI). The closest anyone has gotten is a 2004 publication by Yudkowsky called "coherent extrapolated volition". "The hidden complexity of wishes" is one way in which applying tremendous optimization pressure can go wrong, like in the story of the powerful genie who grants 3 wishes to the person (robber?) who found a magic lamp, but the hidden complexity of wishes is not the main snag. If an AI is better at discovering scientific laws and inventing technologies than people are, it doesn't have to gain control over a nuclear arsenal: it can create its own arsenal that is better than nukes out of materials that people did not even suspect could be used as weapons (just as before about 1930 no one suspected that uranium could be turned into a bomb).
- barfbagginus 3y ago
  [flagged]

hollerith 3y ago

Massive unemployment is not the main problem. No one has a good idea on how to control an AI that is more capable than people are over a broad range of pursuits. (Actually we don't need to be able to control such an AI, it suffices for it to be aligned with our long-term interests. No one knows how to do that either.) The Machine Intelligence Research Institute has been paying researchers to think up ideas for over 20 years. The executive director of that institute and its most senior researcher (Nate Soares and Eliezer Yudkowsky) have been publicly saying now for a few years that it is hopeless and that there is no way any of their institute's research directions will bear fruit before AI kills us all. They say that people just aren't smart enough to figure out how to control or align the powerful AIs that will be created soon if things keep on going the way they are going; consequently some sort of pause or halt on AI research -- almost certainly for at least a few decades -- is going to be necessary for us to survive. The people in charge of the AI labs care enough about their own survival and the survival of their families that they could probably be persuaded to adopt an effective alignment method if someone or some group can come up with one, but it is highly unlikely that anyone will do so in time. Note that whether the effective alignment method is used to align the AI with the broad interests of the entire human population or whether it is used to align the AI to the particular interests of only the leaders of the AI lab is not a particularly important consideration at the current time: it only becomes important in the very unlikely event that someone somewhere finds an effective alignment method. AI is essentially a subfield of the broader field of optimization; an AI applies an impressive amount of optimization pressure. (The human brain can, too.) When you (or a team of researchers) apply tons of optimization pressure, or create an AI that does so, you steer reality towards some "optimization point" or goal. Most optimization points are incompatible with continued human survival when enough optimization pressure is applied towards that point. And no one knows how to aim a powerful optimization process at the few points that are consistent with human survival and human flourishing. No one has even provided a mathematically precise specification of an optimization point consistent with human survival (and mathematical precision would be quite useful and I suspect also necessary for the specification to be useful to a team creating an AI). The closest anyone has gotten is a 2004 publication by Yudkowsky called "coherent extrapolated volition". "The hidden complexity of wishes" is one way in which applying tremendous optimization pressure can go wrong, like in the story of the powerful genie who grants 3 wishes to the person (robber?) who found a magic lamp, but the hidden complexity of wishes is not the main snag. If an AI is better at discovering scientific laws and inventing technologies than people are, it doesn't have to gain control over a nuclear arsenal: it can create its own arsenal that is better than nukes out of materials that people did not even suspect could be used as weapons (just as before about 1930 no one suspected that uranium could be turned into a bomb).

barfbagginus 3y ago
[flagged]