3 ms·

Yes. The US government should ban the training of new foundational models and should do whatever it can to slow down progress in GPUs because the way things are

Yes. The US government should ban the training of new foundational models and should do whatever it can to slow down progress in GPUs because the way things are going now, we are very probably doomed. Even if China or some other country does not impose similar restrictions, a ban in the US and Great Britain would buy humanity a decade or maybe even 2 decades in which we might find a way out of the terrible situation we are in. (A decade or 2 is about how far the US and Great Britain are ahead in frontier AI research.) Also, buying an extra decade or 2 of life for everyone on Earth is a valuable goal in its own right even if after that Chinese AI researchers kill everyone. If humanity can survive somehow for a few more centuries, our prospects start improving because in a few centuries, the smartest among us might be smart enough to create powerful AI that doesn't quickly kill us all. It is not impossible that the current generation of humans can do it, but the probability is low enough (3.5 percent would be my current estimate) that I'm going to do anything I can to slow down AI research or preferably stop it altogether -- preferably for at least a few centuries.

MichaelMug 3y ago
How are we doomed? Or what is the doomsday scenario you have in mind? From what I read we have: massive unemployment, AI making military decisions. What else?
- hollerith 3y ago
  Massive unemployment is not the main problem. No one has a good idea on how to control an AI that is more capable than people are over a broad range of pursuits. (Actually we don't need to be able to control such an AI, it suffices for it to be aligned with our long-term interests. No one knows how to do that either.) The Machine Intelligence Research Institute has been paying researchers to think up ideas for over 20 years. The executive director of that institute and its most senior researcher (Nate Soares and Eliezer Yudkowsky) have been publicly saying now for a few years that it is hopeless and that there is no way any of their institute's research directions will bear fruit before AI kills us all. They say that people just aren't smart enough to figure out how to control or align the powerful AIs that will be created soon if things keep on going the way they are going; consequently some sort of pause or halt on AI research -- almost certainly for at least a few decades -- is going to be necessary for us to survive. The people in charge of the AI labs care enough about their own survival and the survival of their families that they could probably be persuaded to adopt an effective alignment method if someone or some group can come up with one, but it is highly unlikely that anyone will do so in time. Note that whether the effective alignment method is used to align the AI with the broad interests of the entire human population or whether it is used to align the AI to the particular interests of only the leaders of the AI lab is not a particularly important consideration at the current time: it only becomes important in the very unlikely event that someone somewhere finds an effective alignment method. AI is essentially a subfield of the broader field of optimization; an AI applies an impressive amount of optimization pressure. (The human brain can, too.) When you (or a team of researchers) apply tons of optimization pressure, or create an AI that does so, you steer reality towards some "optimization point" or goal. Most optimization points are incompatible with continued human survival when enough optimization pressure is applied towards that point. And no one knows how to aim a powerful optimization process at the few points that are consistent with human survival and human flourishing. No one has even provided a mathematically precise specification of an optimization point consistent with human survival (and mathematical precision would be quite useful and I suspect also necessary for the specification to be useful to a team creating an AI). The closest anyone has gotten is a 2004 publication by Yudkowsky called "coherent extrapolated volition". "The hidden complexity of wishes" is one way in which applying tremendous optimization pressure can go wrong, like in the story of the powerful genie who grants 3 wishes to the person (robber?) who found a magic lamp, but the hidden complexity of wishes is not the main snag. If an AI is better at discovering scientific laws and inventing technologies than people are, it doesn't have to gain control over a nuclear arsenal: it can create its own arsenal that is better than nukes out of materials that people did not even suspect could be used as weapons (just as before about 1930 no one suspected that uranium could be turned into a bomb).
  - barfbagginus 3y ago
    [flagged]
BriggyDwiggs42 3y ago
The status quo is and always has been unbearable, and forcing people to exist in it for centuries longer if there’s a better option just seems horrific to me. If there were an agi button that I could press, I’d take a 70% chance of failure, but I personally think that’s pessimistic.
- hollerith 3y ago
  Your use of the term "status quo" suggests a framing in which some corrupt minority has control over our society, and the rest of us must take desperate measures to overthrow this minority if we are to have any hope of leading worthwhile lives. This is a misleading frame (in case anyone is considering using it). You and I are not being oppressed unless perhaps we are being oppressed by reality itself: it turns out that human existence is more precarious and dangerous than we thought and we hoped it was: we can build machines more capable than ourselves. We are well on the way to do so. Every step of the way has proved profitable (and has for example allowed you and I to exchange these words with ease without our having to leave our homes), but the natural conclusion of this path of development is human extinction because humanity is clever enough to create machines more capable than we are, but not clever enough yet to control them. If we don't pause AI research till humanity is smarter, there is no 30% chance of success. If we look back at history at human projects that at the time were thought to have a 30% or 70% chance of success -- if we look back using our superior modern knowledge of those projects, you see that most were doomed from the start. Consider for example the project of turning base metal into gold. Completely doomed from the start. The alchemists living in the year 1450 did not know that: they thought they had a decent chance of succeeding. Consider most past medical treatments: cutting holes in a mentally-ill person's skull in the hope that that would let the bad humors leave the head; nailing religious symbols to front doors to prevent the house's occupants from catching the contagious disease going around. The thing all these doomed projects had in common is that the people pursuing the project lacked a sufficient understanding of the part of reality they were trying to affect. Similarly, we don't understand how to align a superhuman AI. Some of us have been trying to understand it for 20 years. Some of us have been getting paid a salary for 20 years to try to understand it. No one has even gotten close. There's no 30% chance: there might be a 3% chance, and if you think life is so bad that choosing a lottery that ends humanity with p = 97% and removes human suffering with p = 3% is a good move, well, most people disagree with you. And (I mean this in all sincerity and kindness) I'm sorry life has been so unbearable in your experience.

MichaelMug 3y ago

How are we doomed? Or what is the doomsday scenario you have in mind? From what I read we have: massive unemployment, AI making military decisions. What else?

hollerith 3y ago
Massive unemployment is not the main problem. No one has a good idea on how to control an AI that is more capable than people are over a broad range of pursuits. (Actually we don't need to be able to control such an AI, it suffices for it to be aligned with our long-term interests. No one knows how to do that either.) The Machine Intelligence Research Institute has been paying researchers to think up ideas for over 20 years. The executive director of that institute and its most senior researcher (Nate Soares and Eliezer Yudkowsky) have been publicly saying now for a few years that it is hopeless and that there is no way any of their institute's research directions will bear fruit before AI kills us all. They say that people just aren't smart enough to figure out how to control or align the powerful AIs that will be created soon if things keep on going the way they are going; consequently some sort of pause or halt on AI research -- almost certainly for at least a few decades -- is going to be necessary for us to survive. The people in charge of the AI labs care enough about their own survival and the survival of their families that they could probably be persuaded to adopt an effective alignment method if someone or some group can come up with one, but it is highly unlikely that anyone will do so in time. Note that whether the effective alignment method is used to align the AI with the broad interests of the entire human population or whether it is used to align the AI to the particular interests of only the leaders of the AI lab is not a particularly important consideration at the current time: it only becomes important in the very unlikely event that someone somewhere finds an effective alignment method. AI is essentially a subfield of the broader field of optimization; an AI applies an impressive amount of optimization pressure. (The human brain can, too.) When you (or a team of researchers) apply tons of optimization pressure, or create an AI that does so, you steer reality towards some "optimization point" or goal. Most optimization points are incompatible with continued human survival when enough optimization pressure is applied towards that point. And no one knows how to aim a powerful optimization process at the few points that are consistent with human survival and human flourishing. No one has even provided a mathematically precise specification of an optimization point consistent with human survival (and mathematical precision would be quite useful and I suspect also necessary for the specification to be useful to a team creating an AI). The closest anyone has gotten is a 2004 publication by Yudkowsky called "coherent extrapolated volition". "The hidden complexity of wishes" is one way in which applying tremendous optimization pressure can go wrong, like in the story of the powerful genie who grants 3 wishes to the person (robber?) who found a magic lamp, but the hidden complexity of wishes is not the main snag. If an AI is better at discovering scientific laws and inventing technologies than people are, it doesn't have to gain control over a nuclear arsenal: it can create its own arsenal that is better than nukes out of materials that people did not even suspect could be used as weapons (just as before about 1930 no one suspected that uranium could be turned into a bomb).
- barfbagginus 3y ago
  [flagged]

BriggyDwiggs42 3y ago

The status quo is and always has been unbearable, and forcing people to exist in it for centuries longer if there’s a better option just seems horrific to me. If there were an agi button that I could press, I’d take a 70% chance of failure, but I personally think that’s pessimistic.

hollerith 3y ago
Your use of the term "status quo" suggests a framing in which some corrupt minority has control over our society, and the rest of us must take desperate measures to overthrow this minority if we are to have any hope of leading worthwhile lives. This is a misleading frame (in case anyone is considering using it). You and I are not being oppressed unless perhaps we are being oppressed by reality itself: it turns out that human existence is more precarious and dangerous than we thought and we hoped it was: we can build machines more capable than ourselves. We are well on the way to do so. Every step of the way has proved profitable (and has for example allowed you and I to exchange these words with ease without our having to leave our homes), but the natural conclusion of this path of development is human extinction because humanity is clever enough to create machines more capable than we are, but not clever enough yet to control them. If we don't pause AI research till humanity is smarter, there is no 30% chance of success. If we look back at history at human projects that at the time were thought to have a 30% or 70% chance of success -- if we look back using our superior modern knowledge of those projects, you see that most were doomed from the start. Consider for example the project of turning base metal into gold. Completely doomed from the start. The alchemists living in the year 1450 did not know that: they thought they had a decent chance of succeeding. Consider most past medical treatments: cutting holes in a mentally-ill person's skull in the hope that that would let the bad humors leave the head; nailing religious symbols to front doors to prevent the house's occupants from catching the contagious disease going around. The thing all these doomed projects had in common is that the people pursuing the project lacked a sufficient understanding of the part of reality they were trying to affect. Similarly, we don't understand how to align a superhuman AI. Some of us have been trying to understand it for 20 years. Some of us have been getting paid a salary for 20 years to try to understand it. No one has even gotten close. There's no 30% chance: there might be a 3% chance, and if you think life is so bad that choosing a lottery that ends humanity with p = 97% and removes human suffering with p = 3% is a good move, well, most people disagree with you. And (I mean this in all sincerity and kindness) I'm sorry life has been so unbearable in your experience.