12 ms·

Why do we think AI will have motivations like a greedy human might? Why do we think AI will have motivations at all? Why do we think such AI is a pressing threa

• bottlepalm 3y ago
  Because you can’t stop people, companies and/or countries from creating AI with those motivations.
  • jacquesm 3y ago
    I'm not worried about the AI, I'm worried about the people and companies that have it.
    • mitthrowaway2 3y ago
      The above sentiment is posted many times on every such thread, it would be greatly enriched with am explanation of why you're not worried about the AI. Because worrying about B is not a rationale for not worrying about A.
      • tpm 3y ago
        Because the AI controlled by humans is a tool, a technology, that can be used in various ways. Are you worried about knives in general or are you worried about a knife in the hands of a particular person in a particular moment?
        • mitthrowaway2 3y ago
          Okay, but that assumes that control of AI is as trivially easy as controlling a knife. Given that this is one of the key unknowns, assuming that the AI never does anything that its controller doesn't intend is begging the question. That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.
          • tpm 3y ago
            One important thing is that nobody has shown an actual existing AI to be not controllable, nor a way to create one. Until then it's an imaginary, non-existing risk. Of course as a fear and anxiety-based society we can also worry about those, but I think we should worry more about existing risks.
      • j_maffe 3y ago
        Besides the other comment which is quite correct, we're going to suffer from B far before we will suffer from A.
        • mitthrowaway2 3y ago
          That is a valid reason to worry about B, but not a valid reason not to worry about A!
      • jacquesm 3y ago
        AI doesn't act by itself and worst case you could unplug the bloody thing (SF notwithstanding). But the guys with the guns are the same guys with the guns that have been in charge for as long as there are guns (with some rare exceptions) and I really don't think the existence of AI will change human nature. So AI will be used as a weapon long before it will be a standalone threat. And I like to focus on the things that are more immediate because that's where the threat is most likely to materialize and have effect on me.
        • dragonwriter 3y ago
          > AI doesn't act by itself It can be made to quite easily. > and worst case you could unplug the bloody thing Not necessarily. > But the guys with the guns are the same guys with the guns Well, unless they are autonomous AI drones with guns (or missiles), in which case they are different “guys” with guns.
          • jacquesm 3y ago
            Sure, I've seen those movies. But today it's the guys with the guns/drones/grenades/missiles. I'm worried about stable dictatorships far more than I am about AGI taking over the world and offing humanity because stable dictatorships are already using every weapon they can lay their hands on to control the population and AGI might just tip the scale to the point where they become eternal. And regular AI (the kind that we have today, because AGI is simply not yet a thing that has been achieved afaik) is more than dangerous enough in that context.
            • mitthrowaway2 3y ago
              > But today There's no question that this is a tomorrow-focused concern, just like climate change or Kessler syndrome, et cetera. And today's concerns are important too. Nobody is dismissing them. But tomorrow comes quite quickly these days. You're allowed to be concerned about both.
              • jacquesm 3y ago
• yawpitch 3y ago
  I’m not afraid of the motivations of a hypothetical strong AI… I’m absolutely terrified of the motivations of those who can afford to employ weak AI.
  • dbuxton 3y ago
    Can you expand on this?
    • yawpitch 3y ago
      Sure… imagine, for example, health insurance companies employing proprietary ML models to decide if you’re covered, or for-profit prisons using ML models to decide if someone will be a recidivist. Or governments or law enforcement or private corporations or simply rogue free speech absolutist billionaires using image recognition and ML models to decide which pixels represent a combat age male? The AI we’ve got right now is an absolutely horrifying force multiplier for anyone with wealth and a sufficiently perverse incentive.
      • dbuxton 3y ago
        Ah ok I see what you mean. I think what we have today (weak ML) are models which can be trained to operate at the quality of a human for specific tasks. So I am definitely terrified by the recidivism example, because it’s not obvious to me that there is a human who can do that task, but I’m pretty relaxed about the coverage one (because presumably at least in principle if you automate these sorts of manual tasks you can drive down the cost of providing care - if that saving passed on to consumer). The image recognition one is a bit more complicated but if your scenario is targeting a hellfire missile I’m not sure the human-in-the-driving-seat method has created good outcomes so far, so I’m at least in principle open to some level of AI involvement perhaps short of full autonomy. But I can definitely understand the problem of incentives. Ultimately we have that issue with humans though and it’s in some respects harder to audit them.
        • yawpitch 3y ago
• pointlessone 3y ago
  This is actually one of the failure modes described long ago. Paperclip maximizer. An AI that has agency, intelligence, and a goal. In the course of achieving its goal it completely neglects all other considerations. These unaccounted externalities are the danger.
  • eigenket 3y ago
    In addition to a hypothetical AI, it sounds like you are describing real companies here.
    • EliRivers 3y ago
      And likewise, corporations have already been described as AIs: https://patternsofmeaning.com/2017/11/30/ai-has-already-taken-over-its-called-the-corporation/ https://patternsofmeaning.com/2017/11/30/ai-has-already-take... et al.
    • pointlessone 3y ago
      You're not wrong but also real companies consist of people. Individually they share many of their thinking patterns and moral views with people outside of those companies. So even though collective nature of companies diffuse responsibility and somewhat erode moral considerations of collective action it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. AI doesn't share thinking patterns. We're not even sure to what extent it can do that. We almost certainly don't operate on matrix multiplication while most AIs do. We don't know if AI can share our moral considerations. Many people act as if they equate intelligence with morality, as if one can not exist without other, and as if there's only human kinds of morality. Meanwhile ChatGPT will lie to your face without a hitch even though OpenAI spent a lot of time trying to make it not.
      • eigenket 3y ago
        >Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.
• loup-vaillant 3y ago
  It’s not about motivation, it’s about doing whatever it’s been programmed to do, and grabbing whatever means it can to that end. Now something as puny as a large language model won’t have much reach (the current ones can’t even read the internet after the initial learning phase), but if you have a way to give the thing an unbounded goal, then it will use all means at its disposal to make that goal happen. Heck, even if you just give it some hard maths problem, the machine might think of ways to turn the entire solar system into a giant computer, using us for spare parts along the way. That requires a sufficiently accurate model of the universe of course, as well as ways to harness that model (say the AI starts by convincing humans to build robots and factories — which would be quite easy if it’s good at playing the stock market game).
  • chrismatheson 3y ago
    Are you suggesting the earth may be one giant computer tasked with finding the ultimate question ?
    • loup-vaillant 3y ago
      I’m suggesting the earth may be turned into one giant computer tasked with finding a well defined question. The one posed in HG2G’s would likely just return a syntax or type error.
• mitthrowaway2 3y ago
  Given almost any goal, there are some nearly universal instrumental goals that appear alongside. Including "prevent things from happening that might interfere with achieving the specified end goal" and "gain resources that can be used to achieve the end goal". Many of these instrumental goals could put humans and human interests at risk.
• huytersd 3y ago
  I don’t know if AI is a pressing threat but the slow march towards general intelligence is actually pretty incredible to witness. A year ago I would have found it hard to believe something like chatGPT could exist in my lifetime, now it’s rather easy to visualize what it’s doing in that “word predictor” as something very close to abstractions similar to what humans do. It feels like all that needs to happen at this point is the spontaneous eruption of a meta hypervisor that would “think” about what those abstractions are and boom you have consciousness. All of it doesn’t seem impossible anymore. After all, humans evolved a brain under the same physical universal constraints and with the same set of initial building blocks.
  • xcode42 3y ago
    While I agree that ChatGPT is surprisingly impressive, I think you might be underestimating what's still missing. For starters, what you describe as "the spontaneous eruption of a meta hypervisor that would “think” " is such a hard problem that they literally call it the "Hard problem of consciousness" and is debatably a much harder problem than what we've already solved. Secondly, it is very much an open question whether having this "conscience" would solve the arbitrary problem/context/environment abstraction generation, since it doesn't seem to me that that is the way our own brains/conscience work. I(my conscious mind) don't generate the abstractions that allow me to solve problems, I look at problems and the "right way to look at the problem to better solve it" just kind of "clicks"/pops into my head as an idea, so it is most likely being handled by some algorithm in the background I'm not aware off. And that's not even mentioning that as far as I am aware(someone who is more uptodate with the most recent AI advancements correct me if I'm wrong) but I don't think learning on-the-go processing new input data as it comes along without screwing up/forgetting/overffiting your previous learning is a solved a problem. ChatGPT still needed to be explicitly and manually trained from a large dataset right? It isn't continuously learning, right?
• harshreality 3y ago
  AI will have whatever motivations its thought loop maintains. The problem is not that anyone will intentionally (well, hopefully) ask some future model (FCMX="Future Closed Model X" for brevity's sake, closed meaning some entity gatekeeps access, assuming massive resources continue to be needed to train the best models regardless of architecture) to destroy humans, step by step, because FCMX's existence is at stake, and then keep re-prompting FCMX as if it's an iterator to get it to achieve that result. The problem is that FCMX, which may or may not include LLMs, may have sufficient abilities such that, either between human-prompted steps or before an observing human can react against it, it will destroy the world or turn the human into its agent. A very rough analogy would be the massive number of people, even intelligent, well-educated people, who will fall prey to a sophisticated con, ending up doing something like handing someone they don't know a very large amount of money. "But the person doing the con knows it's a con and has that intent." What "intents" will be surfaced after 1000 or 1e6 chained (re)promptings, such that those intents will then feed into future inference passes? Nobody knows. For purposes of this risk, it doesn't matter whether FCMX's architecture is maintaining its "cognitive loop" itself, or whether someone's set up (as they already have outside OpenAI to chain GPT-x prompts without human intervention) a secondary program (which may or may not include a lesser open source AI model) that reprompts FCMX in a long (or endless) chain.
• NoGravitas 3y ago
  It's projection and displacement. The people most active in worrying about AI threats can easily see what industrial capitalism is doing to the biosphere: stripping everything that makes it a viable support system for life in order to build something that's only a viable support system for markets. But they are also ideologically committed to industrial capitalism, so they can't bring themselves to admit it. Instead, they externalize those criticisms and fears onto an abstract future threat, so they don't have to deal with the current one. That's why you'll see AI bros saying that AGI is an existential threat, while global warming doesn't matter in the long run.