7 ms·

Can someone ELI5 the motivation to worry about "runaway A.I."? To me, it just seems to be a slippery slope argument. Genuinely curious.

• finexplained 4y ago
  You decide you want to optimize for number of lives saved. You decide that future lives, those of people yet to be born, are worth as much as those currently alive. You place small, but importantly not zero, probability on existential risks to humanity, so that when you do the expected value calculation, even an infinitesimally small risk of humanity's complete extinction results in negative infinite utility. You're also very smart and realize that smart people can do damage if their objectives are misaligned, and you start to worry about something much much smarter than you with objectives misaligned to humanity's more broadly. In their defense reward specification is indeed a hard problem, RL agents find unexpected policies that maximize reward in even toy settings. At this point you're down the rabbit hole and no other problem seems to compare. Climate change will leave some people alive, pandemics leave some people alive, AIs have no such kindness.
  • deleted 4y ago
    [deleted]
• frgtpsswrdlame 4y ago
  Basically they're doing a utilitarian calculus. But since our calculus is basically all pretend, we can assign arbitrary importance to anything we want just by daydreaming the costs and percentages of these super unlikely, super catastrophic events.
  • NateEag 4y ago
    Thank you for so clearly and simply articulating the fundamental problem of utilitarianism.
    • Planktonne 4y ago
      I don't think this is a problem with utilitarianism itself; the problem is utilitarianism as practiced by people who are unable to estimate risk practically. This is not unusual - it's a core symptom of various mental health problems. There's even a fairy tale about this - a family of foolish people who catastrophise so much that they are unable to accomplish anything [1]. People assigning bizarre weights to unlikely probabilities is very human, and problematic regardless of ethics. [1] https://en.wikipedia.org/wiki/Clever_Elsie https://en.wikipedia.org/wiki/Clever_Elsie
      • NateEag 4y ago
        I personally think an honest look at human ability to estimate risk shows that as a species we're bad enough at it that utilitarianism is intractable. I may, of course, suffer from the very problem you're describing.
• Imnimo 4y ago
  It's basically Pascal's Wager but for tech billionaires. Once you imagine of a suitably large magnitude of potential harm, it doesn't really matter how likely it is, you can convince yourself that you should worry about that rather than more realistic problems. Maybe in the future, humanity will spread throughout the galaxy, number in the quintillions, and then be tortured for all eternity by an evil AI. That's a lot of QALYs!
  • giraffe_lady 4y ago
    pascal's mugging
    • teddyh 4y ago
      https://en.wikipedia.org/wiki/Pascal%27s_mugging https://en.wikipedia.org/wiki/Pascal%27s_mugging
  • strken 4y ago
    This isn't how they think about it. Their estimates of the likelihood of AGI are often single or double digits. You don't need a huge number of theoretical humans to worry about if you think there's a 10% chance of skynet happening before 2050.
    • yxwvut 4y ago
      The point is that it only takes an arbitrarily small probability for an infinite cost event to have an infinite Expected Value if you prevent it (and an infinite ROI if you even marginally improve the probability of prevention), so all of these outlandish far future hypotheses with catastrophic costs get priority from EA proponents over saving real lives today with certainty.
      • kalkin 4y ago
        That's not actually the argument that the AI Risk people use. It is a possible fallacy for sure but what people actually argue if you read Eliezer Yudkowsky, Nick Bostrom etc is that there is a high probability of building superhuman AI this century, and that by default the outcome is we all die. You can debate these premises but they aren't Pascal's Wager.
• jbay808 4y ago
  If you believe that the timeline for runaway AI is comparable to the timeline for runaway climate change, and if you believe that AI is unlikely to be aligned with furthering human values, then it's a concern that you may view as having significance comparable to runaway climate change.
  • jazzyjackson 4y ago
    can I get a tl;dr on why I would worry about AI alignment? When is an AI running without someone paying the electricity bill, should we not be more concerned with "venture capitalist alignment" ?
    • 6177c40f 4y ago
      They're worried about the case where the AI figures out how to get others to (unknowingly) pay its energy bill for it, in a manner of speaking.
      • paulproteus 4y ago
        I think I figured it out. An AI will make a cryptocurrency and people will mine it, and mining will directly advance the AI. It will be negative utility to humanity, and it will doom us.
    • srinivgp 4y ago
      'cause humans beat apes and if you plunked down 20,000 humans onto a world with 10 billion apes, humans would beat apes call it a slippery slope if you want, the fallacy is when you take it too far, not inherently any time you extrapolate anything whatsoever
      • nonameiguess 4y ago
        Except all of these arguments rely upon AI causing human extinction. Humans haven't driven other apes to extinction. They occupy the same ecological niches they did before, just usually in slightly smaller jungles as our cities and farms have encroached on some of it.
• psi75 4y ago
  My personal belief is that we will see catastrophic results from the use of AI by authoritarian governments or corporations long before we invent AIs capable of killing us; however, it is correct to view it as an X-risk. In a war against robots, we will lose. Why? Machines are stronger and faster than we are. Our only advantages are intellectual; but if we reach a technical level at which what we consider true intelligence is reduced to mere computation, then we have none. The distance between the first integrated circuit and an artificial human is massive (and we may never cross the chasm) but the distance between an artificial human and an artificial superhero/villain is not. If we ever create a machine that's "as smart as" an average person, we're probably not very far from having one that's smarter than all of us put together. (Let's be honest; except in highly technical fields, the differences between a 150 IQ human's abilities and an average person's are not that great.) Should this happen, we'll soon enough be outmatched; we'll be just another animal in a world whose rules (ever changing) we cannot possibly understand... and just as humans have destroyed countless species often without conscious ill intent, it does not seem unreasonable that a robot successor would do the same to us. Here's a thought experiment. Let's say we want to build not just a great Go player, but the best Go player that is possible in any world. Go is harmless, right? Sure, in isolation, but now let's imagine an AGI smarter than any of us which only "cares" about maximizing its Go proficiency. It's going to need a lot of energy. It'll hack or create robots and drill new oil wells (bypassing any constraints we put on it, since it in essence is spending thousands of human years figuring out how to escape). It'll devise nuclear technologies that may be unsafe from our perspective. Consider all the horrible things humans do in order to procure excess money (a commodity loosely correlated to energy) they don't actually need; there's no reason to think a badly aligned AI won't be just as inventive and horrible in its quest for more exajoules. If capitalism and war are still the way of the world in 100 years, we are guaranteed to see some idiot inventing robot soldiers, hundreds of times faster and stronger than us, and with at least as much operational intelligence. He might never deploy the technology--a lab leak is in my opinion more likely--but at some point, it will get out. It won't be one robot; it won't have a body we can destroy. It'll be able to replicate itself and beam copies to satellites. It'll have control of our power infrastructure (which it will divert to its own "wants") within hours. We'll be screwed, not because it will want to harm us, because our lives require the use of resources that it will divert toward some objective function we technically invented but do not understand.
• bnralt 4y ago
  Whats interesting to me is how which fringe disaster scenarios people decide are extremely important and which they decide are ridiculous seems pretty closely tied to their cultural circles. So, for instance, it's extremely unlikely than anyone here will get hurt by nuclear power, GMOs, mass surveillance, or artificial general intelligence anytime soon. We could take any one of those technologies and imagine extreme SciFi levels of future tech and incompetent or hostile governments, and then postulate the possibility of a large threat, based upon our imagination. At least in the online groups I frequent, it seems that the latter two are seen as self-evidently dangerous, not because of any particular harm they have done, but because of imagined future harm. The former two are considered ridiculous things to worry about, because it's considered silly to imagine future potential harm. Perhaps because the former seem tied to uncool hippy types, and the latter seems more tied to cutting edge tech types? I personally don't worry about any of these techs, since all of the danger I've seen appears greatly exaggerated. But it's curious to see how important trendiness is to fringe doomsday scenarios.
  • quickthrower2 4y ago
    The meta is how to make society ready for new big threats. How can counties communicate and collaborate better. Using climate change and covid as learnings. If we can do that then we have a better chance of surviving $BIGPROBLEM when we don’t know in advance what that problem is.
  • concordDance 4y ago
    Nuclear power is one we actually know the harm for and what we can do about it.
• kalkin 4y ago
  I'm uncertain where I stand on this but I'm going to give a shot at a less dismissive answer than most of the replies so far. There's a very abstract argument that goes like this: * "Orthogonality thesis": an arbitrarily smart agent could in theory value ~anything, e.g. maximize Facebook's valuation (say as measured by some specific dataset published by S&P). * "Instrumental convergence thesis": most things that you might conceivably value are easier to optimize if you have a lot of power and resources (e.g. if you just run S&P, and hey maybe also control the worlds militaries so you can ensure nobody challenges that). * Belief in the power of intelligence: it's theoretically possible to be much more capable at ~anything than the most capable human just by thinking better and faster, including building even better AIs. Eventually some human will build an agent that does that, and plausibly this will happen in the current century. * Alignment problem: just because we make something doesn't automatically mean it wants exactly what we want, and in fact getting it to share our values is hard. Children disagree with parents, human values aren't those of natural selection, and ML language models repeat white supremacist ideas about Black people from their Internet training data. Extrapolate out these points - which FWIW I find individually plausible - and you get doom: sooner or later humanity loses control of its destiny to whatever superintelligence we stumble on first, which in the worst case re-uses our atoms for something else (digits in some high-density storage representation that meets the loss function's technical criterion for Facebook's S&P valuation) about even in the best case seems fairly dystopian (whatever human Zuckerberg would actually want). Now this is all pretty scifi and that's about all there was to it a few decades ago, but more recently we've started hitting ~parity milestones (chess, Go, protein folding, translation, arguably even fiction and visual art) that many experts once thought would require fully general human-level intelligence. It's clear to me that we haven't built that yet, but (as I said in another comment upthread) it also seems to me that our models are advancing in capability much more rapidly than we are gaining the ability to understand their internals or make guarantees of any kind about their worst-case or out-of-training-distribution behavior. Seems entirely sane to be worried by that trajectory and to start thinking seriously about how it might connect to the scary abstract speculations. (edited for typos)
• concordDance 4y ago
  > Can someone ELI5 the motivation to worry about "runaway A.I."? No and anyone who tells you otherwise is wrong. A proper understanding of AI risk is a very thick book. There are a ton of normally useful intuitions, heuristics and habits of thought that are implicitly applied when understanding the world that DON'T apply in the AGI case. It cannot be explained to a five year old.
  • Noosphere89 4y ago
    This. Trying to explain why mesa-optimizers are a problem, why deception is common, and instrumental convergence harms humanity is quite complicated, and any proper explanation can't be told to a 5 year old.