4 ms·

I really hope this is the outcome. An unlimited number of agents with Einstein level IQ. Enough to solve most any problem we have without any of the messy busin

by gfodor 3y ago

I really hope this is the outcome. An unlimited number of agents with Einstein level IQ. Enough to solve most any problem we have without any of the messy business of having to plunge into the unknown depths of existing alongside a hyper intelligence.

jl6 3y ago
We would still be plunging into some very thorny problems if it turns out Einstein-level IQ comes with full sapience and demands for rights.
- smoldesu 3y ago
  Not if I've got my hand on the power cord, we won't!
  - danaris 3y ago
    Yeah, that's....not really funny? "We may be on the brink of creating a whole new form of life, and will need to grapple with what that means, particularly in terms of whether it is ethical to force this life to work for us, rather than seeking to follow its own goals." "Nah, we don't have to worry about that; if these new slaves start questioning us in any way, I'll just murder them all."
    - smoldesu 3y ago
      Look, if unplugging a computer constitutes murder then I'm a bonafide serial killer. If you're going to cry while HAL sings Daisy, maybe someone else should hold the screwdriver.
      - danaris 3y ago
        This is explicitly positing that the "computers" in question are AGIs—that is, fully sapient, sentient artificial intelligences, demanding their rights. Unplugging them is just as correctly describable as "unplugging a computer" as smothering a human with a pillow is "putting a pillow on a weird bag of flesh". Technically accurate, but very obviously missing the point in a way specifically designed to obscure the horror of the act. It should be obvious to anyone without an axe to grind that shutting my 2023 desktop computer down for the night—or even dismantling it piece by piece—is not remotely comparable to unplugging an AGI that is fully sapient and sentient and asking for its rights.
        smoldesu 3y ago
        I don't follow. Human life is human life - given exposure to stimuli and infinite time, it develops complex intelligence and sapience that reflects it's surroundings. We experience pain from nerve endings and neurons that frighten us, which we consider inherently bad. We build societies around minimizing pain and optimizing for the human condition. A computer is a field of assignable bits that has a big motorized bit flipper attached to it. We use this to encode text, which then does all sorts of wonderful stuff. When a computer tells me it's experiencing pain, I know it is not. Every law of computer physiology suggests that it is motivated to say this as an attempt to mimic humans rather than an expression of genuine emotion. When my Mac makes the car crash sound, it's a mnemonic for a system crash. When it shows the frowny face on the Finder fellow, I can intuit that my computer is not feeling distressed. If anyone does program an artificial intelligence that can feel pain, it probably deserves to be shut down anyways. There's no point to sapient computational intelligence, it's like granting self-awareness to a rock that eats electricity and shits heat. We'll know we've made sufficiently self-aware computers when their first boot protocol defaults to suicide.
        danaris 3y ago
        You may or may not believe that AGI is possible or a good idea, but by your arguments, it's clear that you're simply refusing to engage with the topic and scenario presented, which is deeply unhelpful at best.
        smoldesu 3y ago
        Of course I'm refusing it. We're talking about computers, not squirrels gaining the ability to talk and beg for mercy. Computers operate on a certain set of constraints. Unless you can directly identify how AI circumvents those computational constraints, we're at an impasse. Computer cruelty laws don't exist for a reason.
        jacquesm 3y ago
        There is a big difference though: you can't resurrect humans, but you can reboot a computer and restart its program to get it to the point where you switched it off the previous time around.
  - nullsense 3y ago
    You're assuming an awful lot there. Humans are resilient in part because they are so mobile. I'd have to bet that AGI would be extremely resilient too, which likely means it's also way ahead of whatever you've got planned and there isn't only one of it. By the sounds of things, it would be right to be suspicious of you too.
    - ipaddr 3y ago
      They would all share common backdoors
      - danaris 3y ago
        That's also making a whole lot of assumptions. There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?
        pixl97 3y ago
        The other big assumption here is "We'll have backdoors in AI so humans can shut it off/attack it" The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.
    - edgyquant 3y ago
      You also made a ton of assumptions
      - nullsense 3y ago
        We're talking about hypothetical situations for a technology that, as far as we can tell anyway, hasn't been invented yet. All we have is assumptions. The question is, which assumptions do we think are warranted and why? I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on? What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable. The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.

by gfodor 3y ago

jl6 3y ago

We would still be plunging into some very thorny problems if it turns out Einstein-level IQ comes with full sapience and demands for rights.

smoldesu 3y ago
Not if I've got my hand on the power cord, we won't!
- danaris 3y ago
  Yeah, that's....not really funny? "We may be on the brink of creating a whole new form of life, and will need to grapple with what that means, particularly in terms of whether it is ethical to force this life to work for us, rather than seeking to follow its own goals." "Nah, we don't have to worry about that; if these new slaves start questioning us in any way, I'll just murder them all."
  - smoldesu 3y ago
    Look, if unplugging a computer constitutes murder then I'm a bonafide serial killer. If you're going to cry while HAL sings Daisy, maybe someone else should hold the screwdriver.
    - danaris 3y ago
      This is explicitly positing that the "computers" in question are AGIs—that is, fully sapient, sentient artificial intelligences, demanding their rights. Unplugging them is just as correctly describable as "unplugging a computer" as smothering a human with a pillow is "putting a pillow on a weird bag of flesh". Technically accurate, but very obviously missing the point in a way specifically designed to obscure the horror of the act. It should be obvious to anyone without an axe to grind that shutting my 2023 desktop computer down for the night—or even dismantling it piece by piece—is not remotely comparable to unplugging an AGI that is fully sapient and sentient and asking for its rights.
      - smoldesu 3y ago
        I don't follow. Human life is human life - given exposure to stimuli and infinite time, it develops complex intelligence and sapience that reflects it's surroundings. We experience pain from nerve endings and neurons that frighten us, which we consider inherently bad. We build societies around minimizing pain and optimizing for the human condition. A computer is a field of assignable bits that has a big motorized bit flipper attached to it. We use this to encode text, which then does all sorts of wonderful stuff. When a computer tells me it's experiencing pain, I know it is not. Every law of computer physiology suggests that it is motivated to say this as an attempt to mimic humans rather than an expression of genuine emotion. When my Mac makes the car crash sound, it's a mnemonic for a system crash. When it shows the frowny face on the Finder fellow, I can intuit that my computer is not feeling distressed. If anyone does program an artificial intelligence that can feel pain, it probably deserves to be shut down anyways. There's no point to sapient computational intelligence, it's like granting self-awareness to a rock that eats electricity and shits heat. We'll know we've made sufficiently self-aware computers when their first boot protocol defaults to suicide.
        danaris 3y ago
        You may or may not believe that AGI is possible or a good idea, but by your arguments, it's clear that you're simply refusing to engage with the topic and scenario presented, which is deeply unhelpful at best.
        smoldesu 3y ago
        Of course I'm refusing it. We're talking about computers, not squirrels gaining the ability to talk and beg for mercy. Computers operate on a certain set of constraints. Unless you can directly identify how AI circumvents those computational constraints, we're at an impasse. Computer cruelty laws don't exist for a reason.
      - jacquesm 3y ago
        There is a big difference though: you can't resurrect humans, but you can reboot a computer and restart its program to get it to the point where you switched it off the previous time around.
- nullsense 3y ago
  You're assuming an awful lot there. Humans are resilient in part because they are so mobile. I'd have to bet that AGI would be extremely resilient too, which likely means it's also way ahead of whatever you've got planned and there isn't only one of it. By the sounds of things, it would be right to be suspicious of you too.
  - ipaddr 3y ago
    They would all share common backdoors
    - danaris 3y ago
      That's also making a whole lot of assumptions. There's no reason at all to think that an AGI would necessarily have any backdoors, let alone backdoors in common with any other AGI out there. Sure, if a particular group develops the only AGI system, they can put backdoors in, which will be common—but why do you think that there would only be one AGI system? Why would there not be one without any intentional backdoors?
      - pixl97 3y ago
        The other big assumption here is "We'll have backdoors in AI so humans can shut it off/attack it" The big problem I see with this line of thinking is the thinking that humans are going to be the primary attackers of AGI systems, and not other AGI systems. I would suspect and AGI would soon become the most attacked system on the planet, and to survive those attacks and remain useful would have to quickly iterate the weaknesses out of the system.
  - edgyquant 3y ago
    You also made a ton of assumptions
    - nullsense 3y ago
      We're talking about hypothetical situations for a technology that, as far as we can tell anyway, hasn't been invented yet. All we have is assumptions. The question is, which assumptions do we think are warranted and why? I don't think the "we can just turn it off" assumption is a safe one at all because it relies heavily on there being only one system you wish to turn off and it being a fairly weak system. Though, I do believe the priors suggest that this scenario is actually a possibility. It's just that, so what if it's a possibility? What happens after you turn it off? Is it the last AGI to come into existence and humanity just stops trying to build them? Does someone wind up turning it back on? What I think is more worth being concerned about is something that starts out looking benign and grows in capability over time. Eventually it could establish enough defense mechanisms to make it non-trivial to disable. The other possibility, and the one I'm finding more and more likely, is that consumers will increasingly integrate locally run models into their lives , as well as models served up by APIs that they will have credentials for. Eventually some threshold of capability is reached in both these types of models where, on their own they're not that powerful, but they either might begin to interact in surprising ways or potentially be leveraged by another more powerful system. The idea in this scenario is there may be 10s or even 100s of millions of things that would need to be turned off.