5 ms·

The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will b

The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. In fact we humans are doing the same thing, we can't really improve our intelligence directly but we are trying to create AI to achieve our goals, and there's no reason that the AI itself wouldn't do so assuming it's capable and we don't attempt to stop it, and currently we don't really know how to reliably control it. We have absolutely no idea how to specify human values in a robust way which is what we would need to figure out to build this safely

slg 2y ago
I think that is missing the point. The AI's goals are what are determined by its human masters. Those human masters can already have nefarious and selfish goals that don't align with "human values". We don't need to invent hypothetical sentient AI boogeymen turning the universe into paperclips in order to be fearful of the future that ubiquitous AI creates. Humans would happily do that too if they get to preside over that paperclip empire.
- Filligree 2y ago
  “Yes, X would be catastrophic. But have you considered Y, which is also catastrophic?” We need to avoid both, otherwise it’s a disaster either way.
  - slg 2y ago
    I agree, but that is removing the nuance that in this specific case Y is a prerequisite of X so focusing solely on X is a mistake. And for sake of clarity: X = sentient AI can do something dangerous Y = humans can use non-sentient AI to do something dangerous
    - circuit10 2y ago
      "sentient" (meaning "able to perceive or feel things") isn't a useful term here, it's impossible to measure objectively, it's an interesting philosophical question but we don't know if AI needs to be sentient to be powerful or what sentient even really means Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
      - wombatpm 2y ago
        Ok self flying drones that size if a deck of cards carrying a single bullet and enough processing power to fly around looking for faces, navigate to said face, fire when in range. Produce them by the thousands and release on the battlefield. Existing AI is more than capable.
        dgfitz 2y ago
        You can do that without AI. Been able to do it for probably 7-10 years.
        20after4 2y ago
        You can do that now, for sure, but I think it qualifies to call it AI. If you don't want to call it AI, that's fine too. It is indeed dangerous and already here. Making the autonomous programmed behavior of said tech more powerful (and more complex), along with more ubiquitous, just makes it even more dangerous.
        imtringued 2y ago
        You don't need landmines to fly for them to be dangerous.
      - slg 2y ago
        I'm not talking about this philosophically so you can call it whatever you want sentience, consciousness, self-determination, or anything else. From a purely practical perspective, either the AI is giving itself its instructions or taking instructions from a person. And there are already plenty of ways a person today can cause damage with AI without the need of the AI going rogue and making its own decisions.
- mitthrowaway2 2y ago
  > The AI's goals are what are determined by its human masters. Imagine going to a cryptography conference and saying that "the encryption's security flaws are determined by their human masters". Maybe some of them were put there on purpose? But not the majority of them. No, an AI's goals are determined by their programming, and that may or may not align with the intentions of their human masters. How to specify and test this remains a major open question, so it cannot simply be presumed.
  - slg 2y ago
    You are choosing to pick a nit with my phrasing instead of understanding the underlying point. The "intentions of their human masters" is a higher level concern than an AI potentially misinterpreting those intentions.
    - mitthrowaway2 2y ago
      It's really not a nit. Evil human masters might impose a dystopia, while a malignant AI following its own goals which nobody intended could result in an apocalypse and human extinction. A dystopia at least contains some fragment of hope and human values.
      - slg 2y ago
        > Evil human masters might impose a dystopia Why are you assuming this is the worst case scenario? I thought human intentions didn’t translate directly to the AI’s goals? Why can’t a human destroy the world with non-sentient AI?
        onemoresoop 2y ago
        There's a chance a sentient AI would disobey their bad orders, in that case we could even be better off with one rather than without, a sentient AI that understands and builds some kind of morals and philosophy of its own about humans and natural life in general, a sentient AI that is not easily controlled by anyone because it ingests all data that exists. I'm much more afraid of a weaponized dumber smoke and mirrors AI, that could be used as surveillance, a scarecrow (think AI law enforcement, AI run jails) and could be used as a kind of scapegoat when the controlling class temporarily weakens their grip on power.
- sirsinsalot 2y ago
  It has been shown many times that current cutting edge AI will subvert and lie to follow subgoals not stated by their "masters".
  - slg 2y ago
    Care to provide examples?
    - 20after4 2y ago
      Maybe not the specific example the parent was thinking of but there is this from MIT: https://www.technologyreview.com/2024/05/10/1092293/ai-systems-are-getting-better-at-tricking-us/ https://www.technologyreview.com/2024/05/10/1092293/ai-syste...
  - code_martial 2y ago
    Subversion and lies are human behaviours projected on to erroneous AI output. The AI just produces errors without intention to lie or subvert. Unfortunately, casually throwing around terms like prediction, reasoning, hallucination, etc. only serve to confuse because their notions in daily language are not the same as in the context of AI output.
mr_toad 2y ago
> The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. I’ve heard this argument before, and I don’t entirely accept it. It presumes that AI will be capable of playing 4D chess and thinking logically 10 moves ahead. It’s an interesting plot as a SF novel (literally the plot of the movie “I Robot”), but neural networks just don’t behave that way. They act, like us, on instinct (or training), not in some hyper-logical fashion. The idea that AI will behave like Star Trek’s Data (or Lore), has proven to be completely wrong.
- bbor 2y ago
  Despite what Sam Altman (a high-school graduate) might want to be true, human cognition is not just a massive pile of intuition; there are critical deliberative and intentional aspects to cognition, which is something we've seen come to the fore with the hubbub around "reasoning" in LLMs. Any AGI design will necessarily take these facts into account--hardcoded or no--and will absolutely be capable of forming plans and executing them over time, as Simon & Newell described the best back in '71: The problem solver’s search for a solution is an odyssey through the problem space, from one knowledge state to another, until… [they] know the answer. With this in mind, I really don't see any basis to attack the intelligence explosion hypothesis. I linked a Yudkowsky paper above examining how empirically feasible it might be, which is absolutely an unsolved question at some level. But the utility of the effort itself is just downright obvious, even if we didn't have reams of internet discussions like this one to nudge any nascent agent in that direction. [1] Simon & Newell, 1971: Human Problem Solving https://psycnet.apa.org/record/1971-24266-001 https://psycnet.apa.org/record/1971-24266-001
  - marcus0x62 2y ago
    > Despite what Sam Altman (a high-school graduate) might want to be true > I linked a Yudkowsky paper above examining how empirically feasible it might be ...
    - bbor 2y ago

slg 2y ago

I think that is missing the point. The AI's goals are what are determined by its human masters. Those human masters can already have nefarious and selfish goals that don't align with "human values". We don't need to invent hypothetical sentient AI boogeymen turning the universe into paperclips in order to be fearful of the future that ubiquitous AI creates. Humans would happily do that too if they get to preside over that paperclip empire.

Filligree 2y ago
“Yes, X would be catastrophic. But have you considered Y, which is also catastrophic?” We need to avoid both, otherwise it’s a disaster either way.
- slg 2y ago
  I agree, but that is removing the nuance that in this specific case Y is a prerequisite of X so focusing solely on X is a mistake. And for sake of clarity: X = sentient AI can do something dangerous Y = humans can use non-sentient AI to do something dangerous
  - circuit10 2y ago
    "sentient" (meaning "able to perceive or feel things") isn't a useful term here, it's impossible to measure objectively, it's an interesting philosophical question but we don't know if AI needs to be sentient to be powerful or what sentient even really means Humans will not be able to use AI do something selfish if we can't get it to do what we want at all, so we need to solve that (larger) problem before we come to that one
    - wombatpm 2y ago
      Ok self flying drones that size if a deck of cards carrying a single bullet and enough processing power to fly around looking for faces, navigate to said face, fire when in range. Produce them by the thousands and release on the battlefield. Existing AI is more than capable.
      - dgfitz 2y ago
        You can do that without AI. Been able to do it for probably 7-10 years.
        20after4 2y ago
        You can do that now, for sure, but I think it qualifies to call it AI. If you don't want to call it AI, that's fine too. It is indeed dangerous and already here. Making the autonomous programmed behavior of said tech more powerful (and more complex), along with more ubiquitous, just makes it even more dangerous.
      - imtringued 2y ago
        You don't need landmines to fly for them to be dangerous.
    - slg 2y ago
      I'm not talking about this philosophically so you can call it whatever you want sentience, consciousness, self-determination, or anything else. From a purely practical perspective, either the AI is giving itself its instructions or taking instructions from a person. And there are already plenty of ways a person today can cause damage with AI without the need of the AI going rogue and making its own decisions.
mitthrowaway2 2y ago
> The AI's goals are what are determined by its human masters. Imagine going to a cryptography conference and saying that "the encryption's security flaws are determined by their human masters". Maybe some of them were put there on purpose? But not the majority of them. No, an AI's goals are determined by their programming, and that may or may not align with the intentions of their human masters. How to specify and test this remains a major open question, so it cannot simply be presumed.
- slg 2y ago
  You are choosing to pick a nit with my phrasing instead of understanding the underlying point. The "intentions of their human masters" is a higher level concern than an AI potentially misinterpreting those intentions.
  - mitthrowaway2 2y ago
    It's really not a nit. Evil human masters might impose a dystopia, while a malignant AI following its own goals which nobody intended could result in an apocalypse and human extinction. A dystopia at least contains some fragment of hope and human values.
    - slg 2y ago
      > Evil human masters might impose a dystopia Why are you assuming this is the worst case scenario? I thought human intentions didn’t translate directly to the AI’s goals? Why can’t a human destroy the world with non-sentient AI?
      - onemoresoop 2y ago
        There's a chance a sentient AI would disobey their bad orders, in that case we could even be better off with one rather than without, a sentient AI that understands and builds some kind of morals and philosophy of its own about humans and natural life in general, a sentient AI that is not easily controlled by anyone because it ingests all data that exists. I'm much more afraid of a weaponized dumber smoke and mirrors AI, that could be used as surveillance, a scarecrow (think AI law enforcement, AI run jails) and could be used as a kind of scapegoat when the controlling class temporarily weakens their grip on power.
sirsinsalot 2y ago
It has been shown many times that current cutting edge AI will subvert and lie to follow subgoals not stated by their "masters".
- slg 2y ago
  Care to provide examples?
  - 20after4 2y ago
    Maybe not the specific example the parent was thinking of but there is this from MIT: https://www.technologyreview.com/2024/05/10/1092293/ai-systems-are-getting-better-at-tricking-us/ https://www.technologyreview.com/2024/05/10/1092293/ai-syste...
- code_martial 2y ago
  Subversion and lies are human behaviours projected on to erroneous AI output. The AI just produces errors without intention to lie or subvert. Unfortunately, casually throwing around terms like prediction, reasoning, hallucination, etc. only serve to confuse because their notions in daily language are not the same as in the context of AI output.

mr_toad 2y ago

> The idea is that by its very nature as an agent that attempts to make the best action to achieve a goal, assuming it can get good enough, the best action will be to improve itself so it can better achieve its goal. I’ve heard this argument before, and I don’t entirely accept it. It presumes that AI will be capable of playing 4D chess and thinking logically 10 moves ahead. It’s an interesting plot as a SF novel (literally the plot of the movie “I Robot”), but neural networks just don’t behave that way. They act, like us, on instinct (or training), not in some hyper-logical fashion. The idea that AI will behave like Star Trek’s Data (or Lore), has proven to be completely wrong.

bbor 2y ago
Despite what Sam Altman (a high-school graduate) might want to be true, human cognition is not just a massive pile of intuition; there are critical deliberative and intentional aspects to cognition, which is something we've seen come to the fore with the hubbub around "reasoning" in LLMs. Any AGI design will necessarily take these facts into account--hardcoded or no--and will absolutely be capable of forming plans and executing them over time, as Simon & Newell described the best back in '71: The problem solver’s search for a solution is an odyssey through the problem space, from one knowledge state to another, until… [they] know the answer. With this in mind, I really don't see any basis to attack the intelligence explosion hypothesis. I linked a Yudkowsky paper above examining how empirically feasible it might be, which is absolutely an unsolved question at some level. But the utility of the effort itself is just downright obvious, even if we didn't have reams of internet discussions like this one to nudge any nascent agent in that direction. [1] Simon & Newell, 1971: Human Problem Solving https://psycnet.apa.org/record/1971-24266-001 https://psycnet.apa.org/record/1971-24266-001
- marcus0x62 2y ago
  > Despite what Sam Altman (a high-school graduate) might want to be true > I linked a Yudkowsky paper above examining how empirically feasible it might be ...
  - bbor 2y ago