3 ms·

The status quo is and always has been unbearable, and forcing people to exist in it for centuries longer if there’s a better option just seems horrific to me. I

The status quo is and always has been unbearable, and forcing people to exist in it for centuries longer if there’s a better option just seems horrific to me. If there were an agi button that I could press, I’d take a 70% chance of failure, but I personally think that’s pessimistic.

hollerith 3y ago
Your use of the term "status quo" suggests a framing in which some corrupt minority has control over our society, and the rest of us must take desperate measures to overthrow this minority if we are to have any hope of leading worthwhile lives. This is a misleading frame (in case anyone is considering using it). You and I are not being oppressed unless perhaps we are being oppressed by reality itself: it turns out that human existence is more precarious and dangerous than we thought and we hoped it was: we can build machines more capable than ourselves. We are well on the way to do so. Every step of the way has proved profitable (and has for example allowed you and I to exchange these words with ease without our having to leave our homes), but the natural conclusion of this path of development is human extinction because humanity is clever enough to create machines more capable than we are, but not clever enough yet to control them. If we don't pause AI research till humanity is smarter, there is no 30% chance of success. If we look back at history at human projects that at the time were thought to have a 30% or 70% chance of success -- if we look back using our superior modern knowledge of those projects, you see that most were doomed from the start. Consider for example the project of turning base metal into gold. Completely doomed from the start. The alchemists living in the year 1450 did not know that: they thought they had a decent chance of succeeding. Consider most past medical treatments: cutting holes in a mentally-ill person's skull in the hope that that would let the bad humors leave the head; nailing religious symbols to front doors to prevent the house's occupants from catching the contagious disease going around. The thing all these doomed projects had in common is that the people pursuing the project lacked a sufficient understanding of the part of reality they were trying to affect. Similarly, we don't understand how to align a superhuman AI. Some of us have been trying to understand it for 20 years. Some of us have been getting paid a salary for 20 years to try to understand it. No one has even gotten close. There's no 30% chance: there might be a 3% chance, and if you think life is so bad that choosing a lottery that ends humanity with p = 97% and removes human suffering with p = 3% is a good move, well, most people disagree with you. And (I mean this in all sincerity and kindness) I'm sorry life has been so unbearable in your experience.

hollerith 3y ago

Your use of the term "status quo" suggests a framing in which some corrupt minority has control over our society, and the rest of us must take desperate measures to overthrow this minority if we are to have any hope of leading worthwhile lives. This is a misleading frame (in case anyone is considering using it). You and I are not being oppressed unless perhaps we are being oppressed by reality itself: it turns out that human existence is more precarious and dangerous than we thought and we hoped it was: we can build machines more capable than ourselves. We are well on the way to do so. Every step of the way has proved profitable (and has for example allowed you and I to exchange these words with ease without our having to leave our homes), but the natural conclusion of this path of development is human extinction because humanity is clever enough to create machines more capable than we are, but not clever enough yet to control them. If we don't pause AI research till humanity is smarter, there is no 30% chance of success. If we look back at history at human projects that at the time were thought to have a 30% or 70% chance of success -- if we look back using our superior modern knowledge of those projects, you see that most were doomed from the start. Consider for example the project of turning base metal into gold. Completely doomed from the start. The alchemists living in the year 1450 did not know that: they thought they had a decent chance of succeeding. Consider most past medical treatments: cutting holes in a mentally-ill person's skull in the hope that that would let the bad humors leave the head; nailing religious symbols to front doors to prevent the house's occupants from catching the contagious disease going around. The thing all these doomed projects had in common is that the people pursuing the project lacked a sufficient understanding of the part of reality they were trying to affect. Similarly, we don't understand how to align a superhuman AI. Some of us have been trying to understand it for 20 years. Some of us have been getting paid a salary for 20 years to try to understand it. No one has even gotten close. There's no 30% chance: there might be a 3% chance, and if you think life is so bad that choosing a lottery that ends humanity with p = 97% and removes human suffering with p = 3% is a good move, well, most people disagree with you. And (I mean this in all sincerity and kindness) I'm sorry life has been so unbearable in your experience.