4 ms·

As someone in the P(doom) > 90% category, I think in general making overly precise predictions are a really bad way to highlight AI risks (assuming that was the

by kypro 1y ago

As someone in the P(doom) > 90% category, I think in general making overly precise predictions are a really bad way to highlight AI risks (assuming that was the goal of AI 2027). Making predictions that are too specific just opens you up to pushback from people who are more interested in critiquing the exact details of your softer predictions (such as those around timelines) rather than your hard predictions about likely outcomes. And while I think articles like this are valuable to refine timeline predictions, I find a lot of people use them as evidence to dismiss the stronger predictions made about the risks of ASI. I think people like Nick Bostrom make much more convincing arguments about AI risk because they don't depend on overly detailed predictions which can be easily nit-picked at, but are instead much more general and focus more on the unique nature of the risks AI presents. For me the risk of timelines is that they're unknowable due to the unpredictable nature of ASI. The fact we are rapidly developing a technology which most people would accept comes with at least some existential risk, that we can't predict the progress curve of, and where solutions would come with significant coordination problems should concern people without having to say it will happen in x number of years. I think AI 2027 is interesting as a science fiction about potential futures we could be heading towards, but that's really it. The problem with being an AI doomer is that you can't say "I told you so" if you're right so any personal predictions you make have no close to no expected pay-out, either socially or economically. This is different to other risks which if you predict accurately when others don't you can still benefit from. I have no meaningful voice in this space so I'll just keep saying we're fucked because what does it matter what I think, but I wish there were more people with influence out there who were seriously thinking about how they can best influence rather than stroking their own own egos with future predictions, which even if I happen agree with do next to nothing to improve the distribution of outcomes.

Fraterkes 1y ago
I’m not trying to be disingenuous, but in what ways have you changed your life now that you belief theres >90% chance to an end to civilization/humanity? Are you living like a terminal cancer patient? (Im sorry, I know its a crass question)
- allturtles 1y ago
  The person you're replying to said "For me the risk of timelines is that they're unknowable due to the unpredictable nature of ASI." So they are predicting >90% chance of doom, but not when that will happen. Given that there is already a 100% chance of death at some unknown point in the future, why would this cause GP to start living like a terminal cancer patient (presumably defined as someone with a >99% chance of death in the next year)?
  - lava_pidgeon 1y ago
    I like to point out, that the existence of AGI in the future does change my potential future planning. So I am 35. Do I need save for pensions? Does it make to sense to start family? These aren't 1 year questions but 20 years ahead questions...
    - amarcheschi 1y ago
      If you're so terrorized of Ai to not start a family despite wanting and being able to, it must be miserable (if) to eventually live through the years as everyone that tried to predict the end of the world did (except for those who died of other causes before the predicted end)
  - kypro 1y ago
    To be clear, currently I think it's likely within the next 10 years we'll have ASI. But I think it's also quite possible that we won't have ASI for 20-30 years, and it's quite possible that moderate advancements in AI will be destabilising enough that we never reach ASI. I think a lot of people who talk about AI risk underweight the fairly likely scenario that highly capable narrow AIs are leveraged in ways that lead to civilisational collapse. Humans getting to ASI assumes that prior advancements are not destabilising or if they are destabilising the advancements happen quickly enough that it doesn't matter. That said, I think ASI is more likely that not. And I think ASI within 5-10 years is very likely.
- kypro 1y ago
  No – I like this. Most people who talk about AI risk, do not take it seriously in my opinion. I've been having nightmares about AIs since I was kid and these have only become more frequent and more realistic the older I've gotten. I studied AI at university because I was always scared and equally fascinated by AI. Even today most of my friends are friends because they're the only people I've been able to find who are willing to talk to me about AI risk for hours on end. Everyone else thinks I'm crazy, but I take this super seriously and I get frustrated by people who treat it purely as a critical thinking exercise, given in all likelihood their family and children will be dead or worse in a few years. To answer your question at very a high-level, I'm pretty depressed so I'm not bothered at all about dying which helps with the emotional side of this. I'm not bothered that I think I'll be dead soon – and a quick death is my highest probability positive outcome for what is coming for me personally. My primary concern, and the thing that keeps me up at night, is the risk of new forms of torture which are so inconceivably bad that any positive outcomes from ASI would never be worth the risk. In a few years if things go "well" and we get ASI it should be completely feasible to 3d print a torture helmet which will simulate the feeling of being burnt to death for 100,000+ years. Assuming this is an experience that's physically possible to experience then signalling the brain to experience it is just an engineering problem – and likely a fairly trivial one for an ASI. Again, death should be seen as a very positive outcome. ASI will create hell in the literal sense. The question here isn't whether ASI could create hells, or whether someone out there might be messed up enough to send someone to hell, but whether humanity will create ASI. If you think we will then we will create hell and we will banish people to it. So if there's even a 0.1% chance of something this bad happening to me I plan to check out early. That said, I see this as a low-probability risk on an individual basis – I think it's much much more likely I'll die of a designer virus or something similar. But should this not happen then we should assume many people alive today will be subjected to unimaginable horrors. Use cases of new forms of torture are very obvious and will be rapidly adopted in totalitarian states, but I'd also question whether democracies will end up as totalitarian states in a world where ASI exists – I think this depends on whether you believe ASI is likely to concentrate power (which I suspect it probably will). I appreciate what I'm saying sounds crazy today, but magical sticks which go "bang" and immediately kill the people they're pointed towards also sounds crazy to most humans who ever lived. We always underestimate how much technology can alter the boundaries of fiction, and ASI will likely change those boundaries faster and more significantly than we can possibly imagine today. Even I as a AI doomer am probably understating the extent of the horror which could be coming to you and your family. Your kids may suffer unimaginably for millions of years, trillions even. More likely though is that ASI introduces several new civilisational risks in rapid succession and one or two of them kill most humans alive today. I'd consider this a mostly neutral outcome. Other low probability, slightly bad outcomes include scenarios where AI advancements are fundamentally destabilising and where events like mass-job loss, rioting and war results in a halting of civilisational progress and mass death – largely from famine. I'm preparing for these outcomes as much as I possibly can since this these are the only outcomes I have any agency over. I grow a lot of food, I have chickens and keep years of food supplies (needed for nuclear winter scenarios). I'm in the process of fitting a water butt in my garden so I'm not dependant on the public water supply. I keep a large sash of firewood and heating fuel for cooking and heat if energy is cut off. I have been amassing tools for a few years now so I can repair things and can produce various items (including weapons). I will continue preparing for these scenarios for as long as I have left. Realistically though there's very little I can do in most scenarios and I'm no where near as prepared as I'd like to be. Really I just hope to avoid suffering and must accept most outcomes I have very little agency over. I just hope to die quickly or have the strength to check out while I have the chance. I guess the sad truth is I'd consider being diagnosed with terminal cancer right now as a positive improvement in my expected life outcome. It's quite hard to over state how concerned I am. The end of humanity isn't the bad outcome imo, it's neutral from the perspective of human suffering and dramatically understates the risks coming. But hey, hopefully I'm wrong =)
siddboots 1y ago
I think both approaches are useful. AI2027 presents a specific timeline in which a) the trajectory of tech is at least somewhat empirically grounded, and b) each step of the plot arc is plausible. There's a chance of it being convincing to a skeptic who had otherwise thought of the whole "rogue AI" scenario as a kind of magical thinking.
- kypro 1y ago
  I agree, but I think you're assuming a certain type of person who understands that a detailed prediction can be both wrong and right simultaneously. And that it's not so much about getting all the details right, but being in the right ballpark. Unfortunately there's a huge number of people who get obsessed about details and then nit pick. I see this with Eliezer Yudkowsky all the time where 90% of the criticism of his views are just nit picking the weaker predictions he makes while ignoring his stronger predictions regarding the core risks which could result in those bad things happening. I think Yudkowsky opens himself up to this though because he often makes very detailed predictions about how things might play out and this largely why he's so controversial, in my opinion. I really liked AI 2027 personally. I thought specifically the tabletop exercises were a nice heuristic for predicting how actors might behave in certain scenarios. I also agree that it presented a plausible narrative for how things could play out. I'm also glad they did wimp out with the bad ending. Another problem I have with people are concerned about AI risk is that they scare away from speaking plainly about the fact if things go poorly your love ones in a few years will probably either be either be dead, in suspended animation on a memory chip, or in a literal digital hell.
- eholk 1y ago
  For what it's worth, I've read both Bostrom's Superintelligence and AI 2027. Reading Superintelligence was interesting and for me really drove home how hard setting aligned goals for an AI is, but the timelines seemed far enough out that it wasn't likely to be something that would matter in my lifetime. AI 2027 was much more impactful on me. It probably helps that I read it the same week I started playing with agent mode on GitHub Copilot. Seeing what AI can already do, especially compared to six months ago, and then seeing their projections made AI seem like something much more worth paying attention to. Yeah, getting from here to being killed by rogue AI nanobots in less than five years still seems pretty far fetched to me. But, each of the steps in their scenario didn't seem completely outside the realm of possiblity. So for me personally, my 80% confidence interval includes both things stagnating pretty much where they are now, but also something more like AI 2027. I suspect we'll be fine, but AGI seems like a real enough possibility that it's worth working on a contingency plan.
avoutos 1y ago
As someone who does not buy into the ai doom, I'd like to understand your position better. Do you believe doom will come about from a malevolant asi or an asi that cannot understand human thinking and thus unintentionally causes disaster from operating on a different plane of intelligence?

by kypro 1y ago

Fraterkes 1y ago

I’m not trying to be disingenuous, but in what ways have you changed your life now that you belief theres >90% chance to an end to civilization/humanity? Are you living like a terminal cancer patient? (Im sorry, I know its a crass question)

allturtles 1y ago
The person you're replying to said "For me the risk of timelines is that they're unknowable due to the unpredictable nature of ASI." So they are predicting >90% chance of doom, but not when that will happen. Given that there is already a 100% chance of death at some unknown point in the future, why would this cause GP to start living like a terminal cancer patient (presumably defined as someone with a >99% chance of death in the next year)?
- lava_pidgeon 1y ago
  I like to point out, that the existence of AGI in the future does change my potential future planning. So I am 35. Do I need save for pensions? Does it make to sense to start family? These aren't 1 year questions but 20 years ahead questions...
  - amarcheschi 1y ago
    If you're so terrorized of Ai to not start a family despite wanting and being able to, it must be miserable (if) to eventually live through the years as everyone that tried to predict the end of the world did (except for those who died of other causes before the predicted end)
- kypro 1y ago
  To be clear, currently I think it's likely within the next 10 years we'll have ASI. But I think it's also quite possible that we won't have ASI for 20-30 years, and it's quite possible that moderate advancements in AI will be destabilising enough that we never reach ASI. I think a lot of people who talk about AI risk underweight the fairly likely scenario that highly capable narrow AIs are leveraged in ways that lead to civilisational collapse. Humans getting to ASI assumes that prior advancements are not destabilising or if they are destabilising the advancements happen quickly enough that it doesn't matter. That said, I think ASI is more likely that not. And I think ASI within 5-10 years is very likely.
kypro 1y ago
No – I like this. Most people who talk about AI risk, do not take it seriously in my opinion. I've been having nightmares about AIs since I was kid and these have only become more frequent and more realistic the older I've gotten. I studied AI at university because I was always scared and equally fascinated by AI. Even today most of my friends are friends because they're the only people I've been able to find who are willing to talk to me about AI risk for hours on end. Everyone else thinks I'm crazy, but I take this super seriously and I get frustrated by people who treat it purely as a critical thinking exercise, given in all likelihood their family and children will be dead or worse in a few years. To answer your question at very a high-level, I'm pretty depressed so I'm not bothered at all about dying which helps with the emotional side of this. I'm not bothered that I think I'll be dead soon – and a quick death is my highest probability positive outcome for what is coming for me personally. My primary concern, and the thing that keeps me up at night, is the risk of new forms of torture which are so inconceivably bad that any positive outcomes from ASI would never be worth the risk. In a few years if things go "well" and we get ASI it should be completely feasible to 3d print a torture helmet which will simulate the feeling of being burnt to death for 100,000+ years. Assuming this is an experience that's physically possible to experience then signalling the brain to experience it is just an engineering problem – and likely a fairly trivial one for an ASI. Again, death should be seen as a very positive outcome. ASI will create hell in the literal sense. The question here isn't whether ASI could create hells, or whether someone out there might be messed up enough to send someone to hell, but whether humanity will create ASI. If you think we will then we will create hell and we will banish people to it. So if there's even a 0.1% chance of something this bad happening to me I plan to check out early. That said, I see this as a low-probability risk on an individual basis – I think it's much much more likely I'll die of a designer virus or something similar. But should this not happen then we should assume many people alive today will be subjected to unimaginable horrors. Use cases of new forms of torture are very obvious and will be rapidly adopted in totalitarian states, but I'd also question whether democracies will end up as totalitarian states in a world where ASI exists – I think this depends on whether you believe ASI is likely to concentrate power (which I suspect it probably will). I appreciate what I'm saying sounds crazy today, but magical sticks which go "bang" and immediately kill the people they're pointed towards also sounds crazy to most humans who ever lived. We always underestimate how much technology can alter the boundaries of fiction, and ASI will likely change those boundaries faster and more significantly than we can possibly imagine today. Even I as a AI doomer am probably understating the extent of the horror which could be coming to you and your family. Your kids may suffer unimaginably for millions of years, trillions even. More likely though is that ASI introduces several new civilisational risks in rapid succession and one or two of them kill most humans alive today. I'd consider this a mostly neutral outcome. Other low probability, slightly bad outcomes include scenarios where AI advancements are fundamentally destabilising and where events like mass-job loss, rioting and war results in a halting of civilisational progress and mass death – largely from famine. I'm preparing for these outcomes as much as I possibly can since this these are the only outcomes I have any agency over. I grow a lot of food, I have chickens and keep years of food supplies (needed for nuclear winter scenarios). I'm in the process of fitting a water butt in my garden so I'm not dependant on the public water supply. I keep a large sash of firewood and heating fuel for cooking and heat if energy is cut off. I have been amassing tools for a few years now so I can repair things and can produce various items (including weapons). I will continue preparing for these scenarios for as long as I have left. Realistically though there's very little I can do in most scenarios and I'm no where near as prepared as I'd like to be. Really I just hope to avoid suffering and must accept most outcomes I have very little agency over. I just hope to die quickly or have the strength to check out while I have the chance. I guess the sad truth is I'd consider being diagnosed with terminal cancer right now as a positive improvement in my expected life outcome. It's quite hard to over state how concerned I am. The end of humanity isn't the bad outcome imo, it's neutral from the perspective of human suffering and dramatically understates the risks coming. But hey, hopefully I'm wrong =)

siddboots 1y ago

I think both approaches are useful. AI2027 presents a specific timeline in which a) the trajectory of tech is at least somewhat empirically grounded, and b) each step of the plot arc is plausible. There's a chance of it being convincing to a skeptic who had otherwise thought of the whole "rogue AI" scenario as a kind of magical thinking.

kypro 1y ago
I agree, but I think you're assuming a certain type of person who understands that a detailed prediction can be both wrong and right simultaneously. And that it's not so much about getting all the details right, but being in the right ballpark. Unfortunately there's a huge number of people who get obsessed about details and then nit pick. I see this with Eliezer Yudkowsky all the time where 90% of the criticism of his views are just nit picking the weaker predictions he makes while ignoring his stronger predictions regarding the core risks which could result in those bad things happening. I think Yudkowsky opens himself up to this though because he often makes very detailed predictions about how things might play out and this largely why he's so controversial, in my opinion. I really liked AI 2027 personally. I thought specifically the tabletop exercises were a nice heuristic for predicting how actors might behave in certain scenarios. I also agree that it presented a plausible narrative for how things could play out. I'm also glad they did wimp out with the bad ending. Another problem I have with people are concerned about AI risk is that they scare away from speaking plainly about the fact if things go poorly your love ones in a few years will probably either be either be dead, in suspended animation on a memory chip, or in a literal digital hell.
eholk 1y ago
For what it's worth, I've read both Bostrom's Superintelligence and AI 2027. Reading Superintelligence was interesting and for me really drove home how hard setting aligned goals for an AI is, but the timelines seemed far enough out that it wasn't likely to be something that would matter in my lifetime. AI 2027 was much more impactful on me. It probably helps that I read it the same week I started playing with agent mode on GitHub Copilot. Seeing what AI can already do, especially compared to six months ago, and then seeing their projections made AI seem like something much more worth paying attention to. Yeah, getting from here to being killed by rogue AI nanobots in less than five years still seems pretty far fetched to me. But, each of the steps in their scenario didn't seem completely outside the realm of possiblity. So for me personally, my 80% confidence interval includes both things stagnating pretty much where they are now, but also something more like AI 2027. I suspect we'll be fine, but AGI seems like a real enough possibility that it's worth working on a contingency plan.

avoutos 1y ago

As someone who does not buy into the ai doom, I'd like to understand your position better. Do you believe doom will come about from a malevolant asi or an asi that cannot understand human thinking and thus unintentionally causes disaster from operating on a different plane of intelligence?