7 ms·

> And why focus on extinction in particular? > runaway AI is a way off yet, and that it will take a significant scientific advance to get there—one that we can

• ImHereToVote 3y ago
  Intellectuals are gonna intellectualize. As soon as a large enough number of people are holding an opinion, the intellectual pops up his head. Ew, how gauche, only stupid people are concerned with what most people are concerned about.
• Capricorn2481 3y ago
  Global warming literally will kill everyone if it isn't stopped. The fact that you're more worried about AI than global warming is a real HN moment.
  • michaelmrose 3y ago
    There is no reason to believe this is so. A less advantageous climate will almost certainly in any reasonable projections kill/impoverish only some and in all probability a small minority of the human race unless we under stress decide to kill the rest. No reasonable scientists are projecting human extinction and by positing it as such you are erecting a trivially demolished straw man for the opposition to mock.
  • travisjungroth 3y ago
    Source? Every model I’ve seen from scientists is that climate change has a very high probability of killing a minority of people. I’m not acting like that’s a small amount. “Minority” would be tens of millions, hundreds of millions of people. I think it will be one of the greatest causes of human suffering. But it’s not an existential threat, it’s a different category.
    • mcguire 3y ago
      Look up "Runaway greenhouse effect". Think, "Venus". Is that likely? No. Is that more or less likely than a rogue superintelligent AI? Well, we have one example of the first and none of the second...
      • lamontcg 3y ago
        That's at the extreme of unlikely. The carbon that we're digging up was in the atmosphere before, it has just been sequestered, we're returning to a state that the Planet has seen before. Across the entire Earth's history we're still at a fairly cold point and a long way from "Greenhouse Earth" and the temperatures at the Eocene Optimum. And according to the IPCC: "a 'runaway greenhouse effect'—analogous to [that of] Venus—appears to have virtually no chance of being induced by anthropogenic activities"
        • Freebytes 3y ago
          If the Earth warms significantly, we could have a situation where it triggers an ice age. (There are mechanisms for this to happen.) If the surface temperature of the water surface rises to a very high level, we could also have hypercanes.
        • mcguire 3y ago
          More or less unlikely than a superintelligent AI converting all of us into paperclips?
  • digging 3y ago
    > The fact that you're more worried about AI than global warming is a real HN moment. This isn't an opinion the GP comment expressed, you assumed it, which is a real reddit moment. People can be equally worried about two existential threats. Being tied to the train tracks and hearing a whistle (this is climate change) is terrifying, but it doesn't mean you wouldn't care if somebody walked up and pointed a gun at you (this is AI, potentially). Either one's going to kill you.
  • waterheater 3y ago
    Global warming may kill everyone...eventually. I remember reading articles from the 90s discussing scientific research predicting that the East Coast will be underwater by 2020. My point in highlighting that misprediction is to demonstrate the difficulty in knowing the precise effects of higher temperatures on a planet. AI has the possibility but not guarantee to kill everyone. We could shift to a lifestyle using electricity but avoiding modern computing technology. AI can be unplugged given sufficient will, whereas a planetary system cannot.
• toolz 3y ago
  There are enough nuclear weapons lost and unaccounted for from the cold war to send humanity into extinction many times over. I think there are far more viable human extinction events that could occur that don't involve AI and further I don't exactly see how we halt the progress of AI. What would the language of such a law look like? Presuming it would have to be rather ambiguous, who in the government would be competent enough to enforce this well meaning law that wasn't just going to abuse their power to aide competing interests? AI is a tricky advancement that will be difficult to get right, but I think humanity has been so far successful at dealing with a much more dangerous technology (nuclear weaponry) - so that gives me hope.
  • kalkin 3y ago
    I have never seen a number for lost nukes higher than the dozens. Do you have a source for enough to "send humanity into extinction many times over"?
    • deleted 3y ago
      [deleted]
  • bee_rider 3y ago
    Is that true? I thought the number of lost nukes numbered in, like, the dozens at most. It would take a ton of nukes to wipe out humanity (although only one to really ruin somebody’s day). Unless you are counting strategies like: try to pretend you are one of the two (US, Russia) and try to bait the other into a “counterattack,” but hypothetically you could do that with 0 nukes (you “just” need a good enough fake missile I guess).
  • nradov 3y ago
    Nonsense. There are at most only a handful of nuclear weapons unaccounted for. And those that may have been lost are no longer going to be really operational. They aren't like rifle cartridges that you can stick in a box and store for decades. The physics package has to be periodically overhauled or else it just won't work.
• jazzyjackson 3y ago
  "cannot anticipate" == cannot know whether it will happen we also cannot anticipate an earth-killing asteroid appearing any day now, and no, i'm not bothered in the least by this possibility, any more than my usual existential angst as a mortal human. sometimes I think the AI safety people haven't come to terms with their own death and have a weird fixation on the world ending as a way to avoid their discomfort with the more routine varieties of the inevitable.
  • SpicyLemonZest 3y ago
    NASA does in fact run a planetary defense program (https://www.nasa.gov/planetarydefense https://www.nasa.gov/planetarydefense), investing $100m annually into the goal of anticipating earth-killing asteroids.
  • eterevsky 3y ago
    We can certainly calculate the probability of an Earth-killing asteroid and it's quite low. In case of AI, it's unclear whether we need any additional scientific advances at all beyond scaling existing methods. Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent.
    • AnimalMuppet 3y ago
      > Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent. You don't have any idea whether additional advances are needed. You don't have any idea what advances are needed. You don't have any idea whether those advances are even possible. But you're confident that they will happen with probability > 10%? You're making very confident assertions, but you have no factual basis for doing so. (Neither does anyone else. We don't know - we're all guessing.)
      • comp_throw7 3y ago
        People are actively trying to make it happen! This is literally the mission statement of DeepMind, OpenAI, etc. Obviously things that nobody is trying to make happen are not likely to happen, and sufficiently difficult things that people are trying to make happen might take a while (e.g. fusion). But AGI is a thing that people are trying to make happen, and are making progress on, and basically nobody predicted in advance the progress that has been made so far.
• majormajor 3y ago
  > So an AI that may cause our extinction may be a result of a scientific advance "we cannot anticipate"? Is that true? Are there unimaginably many ways in which some hypothetical AI or algorithm could cause extinction? I don't think so, I think the people who control [further research] are still the most important in that scenario. Maybe don't hook "it" up to the nuke switch. Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems (even in this sentence, the amount of "not currently technically possible" is extremely high). Let's consider the war in Ukraine, on the other hand? How might it cause extinction? That's MUCH easier to imagine. So why would it be less of an concern?
  • lumenwrites 3y ago
    > Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems If we knew how to make sure that this does not happen, the problem would be solved and there would be nothing to worry about. The problem is that we have no idea how to prevent that from happening, and if you look at the trajectory of where things are going, we're clearly moving in the direction where this occurs. "just not doing it" would have to involve everyone in the world agreeing to not do it, and banning all large AI training runs in all countries, which is what many people are hoping will happen.
    • majormajor 3y ago
      If you have a function that you call, that returns a value, do you not think we know enough to understand that it's not "conscious" when it's not called, and not executing, and the hardware is sitting idle? EDIT: we understand enough to know that today, "runaway GPT" is not a major concern compared to, say, a war between nuclear-armed world powers.
      • SpicyLemonZest 3y ago
        I don't think we know enough to understand if "consciousness" can be sliced and diced in the way you're describing. Are you conscious while your neurotransmitters cross their respective synaptic gaps, or only when they arrive at the receptors? I don't know how we'd begin to evaluate the question.
• AnimalMuppet 3y ago
  Sure, that would be a more pressing concern.... if it were to happen. What's the probability of it happening? What's the probability that an AI powerful enough to do that is even possible? Meanwhile, we've got a war in Ukraine with probability 1. So AI risk has to get in line with global nuclear war, and giant meteor strikes, and supervolcanoes - risks that are serious concerns, and could cause massive damage if they happened, but are not drop-everything-now-and-focus-your-entire-existence-on-this-one-threat levels of probability.
• a_shovel 3y ago
  Runaway AI could cause the extinction of humanity, but The Big Red Button That Turns The Universe Into Pudding would cause the extinction of all life everywhere, including extraterrestrials, so it's obviously the more pressing concern. Why are you wasting time on AI when the Button is so much more important? No, The Button doesn't currently exist, and all available science says it cannot ever exist. But the chance that all available science is wrong is technically not zero, because quantum, so that means The Button is possible, so unless you want everything to be turned into pudding, you need to start panicking about The Button right now.
  • mcguire 3y ago
    I know you're getting downvoted, but that's a legitimate comment. The rogue, super-intelligent AI singularity involves creating an actual God after all.
  • digging 3y ago
    > No, The Button doesn't currently exist, and all available science says it cannot ever exist. In what way is this an analogy for misaligned superhuman AGI? I've never heard an assertion that it can't exist based on available knowledge. This seems a very flimsy argument. Anyway, the button not only can exist, some would say it probably does exist. Some would say it's likely to have been pressed already, somewhere in the universe. It's called false vacuum decay, and it moves at the speed of light, so as long as it never gets pressed inside the galaxy it may never reach us.
• mcguire 3y ago
  There are an unbounded number of concerns that could result in the COMPLETE EXTINCTION OF ALL HUMANS AND ALL LIFE that cannot be anticipated. Why are you fixated on this particular one?
  • digging 3y ago
    Because some of the smartest and most well funded humans all over the planet are spending their careers making it more and more likely by the day. Nobody is aiming asteroids at Earth or trying to rotate nearby massive stars to point their poles at Earth.
    • mcguire 3y ago
      One notes that historically (and probably currently), some of the smartest and most well funded humans all over the planet have spent their careers preventing one or more deities from killing us all. And yet I find myself an atheist.
      • digging 3y ago
        That pithy analogy makes no sense. Nobody in history has ever been actively working on creating a deity with measurable progress. Say what you mean - do you believe it is physically impossible to create a superhuman AGI? If so, how do you argue that our physical brains can't be replicated or surpassed by metal without asserting the existence of some metaphysical soul that makes us possible?
• visarga 3y ago
  Being scared of existential AI risks does not mean we should take a knee jerk reaction. By over-regulating or restricting access to AI early on we might sabotage our chances of successful alignment. People are catching issues every day, exposure is the best way to find out what are the risks. Let's do it now before everything runs on it. Even malicious use for spam or manipulation should be treated as an ongoing war, a continual escalation. We should focus on keeping up. No way to avoid it.