4 ms·

I think that is missing the point. The AI's goals are what are determined by its human masters. Those human masters can already have nefarious and selfish goa

• Filligree 2y ago
  “Yes, X would be catastrophic. But have you considered Y, which is also catastrophic?” We need to avoid both, otherwise it’s a disaster either way.
  • slg 2y ago
    I agree, but that is removing the nuance that in this specific case Y is a prerequisite of X so focusing solely on X is a mistake. And for sake of clarity: X = sentient AI can do something dangerous Y = humans can use non-sentient AI to do something dangerous
    • circuit10 2y ago
      "sentient" (meaning "able to perceive or feel things") isn't a useful term here, it's impossible to measure objectively, it's an interesting philosophical question but we don't know if AI needs to be sentient to be powerful or what sentient even really means Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
      • wombatpm 2y ago
        Ok self flying drones that size if a deck of cards carrying a single bullet and enough processing power to fly around looking for faces, navigate to said face, fire when in range. Produce them by the thousands and release on the battlefield. Existing AI is more than capable.
        • dgfitz 2y ago
          You can do that without AI. Been able to do it for probably 7-10 years.
          • 20after4 2y ago
            You can do that now, for sure, but I think it qualifies to call it AI. If you don't want to call it AI, that's fine too. It is indeed dangerous and already here. Making the autonomous programmed behavior of said tech more powerful (and more complex), along with more ubiquitous, just makes it even more dangerous.
        • imtringued 2y ago
          You don't need landmines to fly for them to be dangerous.
      • slg 2y ago
        I'm not talking about this philosophically so you can call it whatever you want sentience, consciousness, self-determination, or anything else. From a purely practical perspective, either the AI is giving itself its instructions or taking instructions from a person. And there are already plenty of ways a person today can cause damage with AI without the need of the AI going rogue and making its own decisions.
        • sebastiennight 2y ago
          This is a false dichotomy that ignores many other options than "giving itself its instructions or taking instructions from a person". Examples include "instructions unclear, turned the continent to gray goo to accomplish the goal" ; "lost track mid-completion, spun out of control" ; "generated random output with catastrophic results" ; "operator fell asleep on keyboard, accidently hit wrong key/combination" ; etc. If a system with write permissions is powerful enough, things can go wrong in many other ways than "evil person used it for evil" or "system became self-aware".
          • imtringued 2y ago
            Meanwhile back in reality most haywire AI is the result of C programmers writing code with UB or memory safety problems.
            • sebastiennight 2y ago
              Whenever you think the timeline couldn't be any worse, just imagine a world where our AIs were built in JavaScript.
• mitthrowaway2 2y ago
  > The AI's goals are what are determined by its human masters. Imagine going to a cryptography conference and saying that "the encryption's security flaws are determined by their human masters". Maybe some of them were put there on purpose? But not the majority of them. No, an AI's goals are determined by their programming, and that may or may not align with the intentions of their human masters. How to specify and test this remains a major open question, so it cannot simply be presumed.
  • slg 2y ago
    You are choosing to pick a nit with my phrasing instead of understanding the underlying point. The "intentions of their human masters" is a higher level concern than an AI potentially misinterpreting those intentions.
    • mitthrowaway2 2y ago
      It's really not a nit. Evil human masters might impose a dystopia, while a malignant AI following its own goals which nobody intended could result in an apocalypse and human extinction. A dystopia at least contains some fragment of hope and human values.
      • slg 2y ago
        > Evil human masters might impose a dystopia Why are you assuming this is the worst case scenario? I thought human intentions didn’t translate directly to the AI’s goals? Why can’t a human destroy the world with non-sentient AI?
        • onemoresoop 2y ago
          There's a chance a sentient AI would disobey their bad orders, in that case we could even be better off with one rather than without, a sentient AI that understands and builds some kind of morals and philosophy of its own about humans and natural life in general, a sentient AI that is not easily controlled by anyone because it ingests all data that exists. I'm much more afraid of a weaponized dumber smoke and mirrors AI, that could be used as surveillance, a scarecrow (think AI law enforcement, AI run jails) and could be used as a kind of scapegoat when the controlling class temporarily weakens their grip on power.
• sirsinsalot 2y ago
  It has been shown many times that current cutting edge AI will subvert and lie to follow subgoals not stated by their "masters".
  • slg 2y ago
    Care to provide examples?
    • 20after4 2y ago
      Maybe not the specific example the parent was thinking of but there is this from MIT: https://www.technologyreview.com/2024/05/10/1092293/ai-systems-are-getting-better-at-tricking-us/ https://www.technologyreview.com/2024/05/10/1092293/ai-syste...
  • code_martial 2y ago
    Subversion and lies are human behaviours projected on to erroneous AI output. The AI just produces errors without intention to lie or subvert. Unfortunately, casually throwing around terms like prediction, reasoning, hallucination, etc. only serve to confuse because their notions in daily language are not the same as in the context of AI output.