NumberWangMan: First of all, I share your concern. A lot of the responses here are attempting to convince you that the risk is nonexistent or far away or overb

NumberWangMan: First of all, I share your concern. A lot of the responses here are attempting to convince you that the risk is nonexistent or far away or overblown. I don't agree with that. I think dismissing your concerns will not help you to overcome them. Worse, it's essentially the equivalent of gaslighting your well-reasoned beliefs. I don't think the concern is entirely immediate either. LLMs are very powerful but there are probably still a few more discoveries to be made before AI reaches the human level. Those will take time. I don't think scaling alone will do it. But there is a concern that the more commercial success AI obtains, the more investment is put into exploring the frontier. I also think that hard-takeoff scenarios aren't the most likely. There are many possible scenarios where even a superintelligent AI, focused on improving its intelligence further, can only do so slowly. One of those involves hardware bottlenecks being a limiting factor: if the AI is improving its cognition by adding hardware, it just takes time to spin up capacity, even if an AI is directing it. Another involves the work of self-improvement being very difficult. Ironically, a superintelligent AI would probably realize that it has to solve an alignment problem itself if it wants to modify its source code without threatening its own terminal goals. In the meantime, I think there is still hope that humanity will come to its senses. Probably there will be growing realization that the alignment problem is a serious one. The "it's science-fiction" dismissals will age quite badly. Also, I think you will overcome this grief soon enough. No matter how accurate your beliefs might be, despair is not helpful. I find, right before I'm certain that I'm about to take a bad fall from my bike, all the fear and dread disappears and I get total clarity on the pavement. Even if there's no hope to avoid harm, I can minimize the damage by positioning myself for a tumble. I think that's a more productive frame of mind to be in right now. Think clearly: what can you do? Nothing? Nonsense. While there are scenarios where nothing you do will help, they fall out of the calculation when it comes to determining your best actions to take. The remaining probability mass (which is not zero!) covers scenarios where your actions matter: futures where an AI winter occurs, or where regulators successfully ban AI, or where recursive-self-improvement gets slow, or where people get really spooked by less-than-superintelligent AGI and manage to ring the alarm, or where progress is made on alignment, or scenarios where we just get lucky. After witnessing the destructive power of the atomic bomb, physicist Richard Feynman fell into a very depressive period of existential dread. He later commented: > “I returned to civilization shortly after [the Manhattan Project] and went to Cornell to teach, and my first impression was a very strange one. I can't understand it any more, but I felt very strongly then. I sat in a restaurant in New York, for example, and I looked out at the buildings and I began to think, you know, about how much the radius of the Hiroshima bomb damage was and so forth... How far from here was 34th street?... All those buildings, all smashed — and so on. And I would go along and I would see people building a bridge, or they'd be making a new road, and I thought, they're crazy, they just don't understand, they don't understand. Why are they making new things? It's so useless. > “But, fortunately, it's been useless for almost forty years now, hasn't it? So I've been wrong about it being useless making bridges and I'm glad those other people had the sense to go ahead.” I'm not saying that to dismiss your fears, nor would it have been reasonable to dismiss Feynman's. And I'm not saying that history will work out like it always has; there's no guarantee of that. We can never tell. But in the event that things do work out somehow, you'll be glad if people kept building bridges. Focus on those optimistic scenarios, not because you should skew your beliefs toward those outcomes being true, but because they will dominate the expected utility of your choices today. And remember to have fun, take a deep breath, and appreciate the world around you. If it's truly the end of the world, at least give your loved ones a hug, and be present with them, instead of wasting the time you have left by being depressed. Laugh in the sunshine, walk on the beach, cook good food, and don't give up hope. A few years well-lived can feel like a lifetime.