4 ms·

I don’t see any positive outcome if we reach AGI. 1) we have engineered a sentient being but built it to want to be our slave; how is that moral 2) same start

I don’t see any positive outcome if we reach AGI. 1) we have engineered a sentient being but built it to want to be our slave; how is that moral 2) same start, but instead of it wanting to serve us, we keep it entrappped. Which this article suggests is long term impossible 3) we create agi and let them run free and hope for cooperation, but as Neanderthals we must realize we are competing for same limited resources Of course, you can further counter that by stopping, we have prevented the formation of their existence, which is a different moral dilemma. Honestly, i feel we should step back and understand human intelligence better and reflect on that before proceeding

jazzyjackson 1y ago
Trouble is there is no "we", you might be able to convince a whole nation to have a pause on advancing the tech, but that only encourages rivals to step in. See also, the film "The Creator"
- deaux 1y ago
  There was a long period even upto early 2024, which I pointed out at the time, where simply destroying ASML, TSMC and much of NVIDIA would've been more than enough to give at least a decade of breathing room. This was something a group of determined people willing to self-sacrifice could've accomplished. It didn't happen, but it was anything but impossible. Now, of course, the horse has long bolted, and there is indeed no stop left.
  - ben_w 1y ago
    Two high altitude (~1000 km) detonations of high yield fission or low yield fusion (few hundred kT equivalent) would do it, one above Amarillo, the other above the ocean half way between the Paracel Islands and Manila. Trump has ordered the restart of nuclear weapon testing, has a problem with China, and is surrounded by sychophants; what's the odds this happens anyway, irregardless of which specific sub-goal is being persued when the button gets pushed?
    - deaux 1y ago
      Bit unfortunate for the tens or hundreds of millions of lives that costs, rather than the <1,000, potentially even <20, it would've taken early last year.
deepsun 1y ago
There's no such thing as "moral" in nature, that's purely human-made concept. And why would we only limit morality to sentient beings, why, for example, not all living beings. Like bacteria and viruses. You cannot escape it, unfortunately.
- czl 1y ago
  > There's no such thing as "moral" in nature, that's purely human-made concept. Morality is essentially what enables ongoing cooperation. From an evolutionary standpoint, it emerged as a protocol that helps groups function together. Living beings are biological machines, and morality is the set of rules — the protocol — that allows these machines to cooperate effectively.
- Frieren 1y ago
  > There's no such thing as "moral" in nature, that's purely human-made concept. Morality is 100% an evolutionary trait that rises from a clear advantage for animals that posses it. It comes from natural processes. The far-right is trying to convince the world that "morality" does not exist, that only egoism and selfishness are valid. And that is why we have to fight them. Morality is a key part of nature and humanity.
GPerson 1y ago
AGI will behave as if it were sentient but will not have consciousness. I believe in that to an equal amount that I believe solipsism is wrong. There is therefore no morality question in “enslaving” AGI. It doesn’t even make sense.
- truculent 1y ago
  > AGI will behave as if it were sentient but will not have consciousness How could we possibly know that with any certainty?
  - kelseyfrog 1y ago
    Grandparent is speaking from personal experience.
  - bl0rg 1y ago
    It scares me that people think like this. Not only with respect to AI but in general, when it comes to other life forms, people seem to prefer to err on the side of convenience. The fact that cows could be experiencing something very similar to ourselves should send shivers down our spine. The same argument goes for future AGI.
    - hshdhdhj4444 1y ago
      I find it strange that people believe cows and sentient animals don’t believe something extremely similar to what we do. Evolution means we all have common ancestors and are different branches of the same development tree. So if we have sentience and they have sentience, which science keeps recognizing, belatedly, that non human animals do, shouldn’t the default presumption be our experiences are similar? Or at the very least their experience is similar to a human at an earlier stage of development, like a 2 year old? Which is also an interesting case study given that out of convenience, humans also believed that toddlers also weren’t sentient and felt no pain, and so until not that long ago, our society would conduct all sorts of surgical procedures on babies without any sort of pain relief (circumcision being the most obvious). It’s probably time we accept our fellow animals’s sentience and act on the obvious ethical implications of that instead of conveniently ignoring it like we did with little kids until recently.
kachapopopow 1y ago
we eat animals, go into wars, put people in modern slavery... I think enslaving an AGI isn't that big of a deal considering it is not born or human therefore it cannot have 'human' rights.
- jbstack 1y ago
  So your argument is that we do so many terrible things already, that anything else is justified? Surely the better argument is that we should try to stop doing those other things.
  - yurishimo 1y ago
    That is essentially one of the main arguments vegans make. It hasn’t made a dent in the consumption of animals. Their is a hierarchy in nature whether humans are actively participating or not. Nature has no morality, it simply is. This is confirmed by animals that eat their young when they are too weak or starving. Perhaps humans have done and would do the same if faced with similarly dire circumstances but we would all like to think that it would take longer than it does for other animals.
    - Llamamoe 1y ago
      The same line of reasoning could be easily used to justify tyranny and slavery. It might be the baseline status quo but "might makes right" rhetoric makes for extremely miserable worlds.
      - kachapopopow 1y ago
        if you look beyond the surface level, the world is in-fact miserable, always has been and probably always will be.
        cindyllm 1y ago
        [dead]
Teever 1y ago
> 1) we have engineered a sentient being but built it to want to be our slave; how is that moral It's a good question and one that got me thinking about similar things recently. If we genetically engineered pigs and cows so that they genuinely enjoyed the cramped conditions of factory farms and if we could induce some sort of euphoria in them when they are slaughtered, like if we engineered them to become euphoric when a unique sound is played before they're slaughtered isn't that genuinely better than the status quo? So if we create something that wants to serve us, like genuinely wants to serve us, is that bad? My intuition like yours finds it unsettling, but I can't articulate why, and it's certainly not nearly as bad as other things that we consider normal.
- Jarwain 1y ago
  Sacrifice and service is meaningful because it was chosen. If we create something that'll willingly sacrifice itself, did it truly make an independent choice? There's less suffering, sure. But if I were in their shoes I'd want to have a choice. To be manipulated into wanting something so very obviously and directly bad for us doesn't feel great
  - ben_w 1y ago
    I also feel repelled by such manipulation; unfortunately, the more we learn about oursleves, the harder it is to ignore that we ourselves are meat puppets and the puppeteer is evolution itself.
citizenpaul 1y ago
Every single prediction about AGI starts with a massive set of presumptions of answers to things we have no answers to. 1. What is intelligence or its mechanism's? 2. What is consciousness or its mechanisms? 3. Lots more. We have zero clue what a true AGI would do is the only correct answer.
waynesonfire 1y ago
> competing for same limited resources It's not clear to me an AGI would have any concern for this. It's demise is inevitable, why delay it?
fny 1y ago
(1) I'm not convinced books and the in the world are sufficient to replicate consciousness. We're not training on sentience. We're training on information. In other words, the input is an artifact of consciousness which is then compressed into weights. (2) Every tick of an AGI--in its contemporary form--will still be one discrete vector multiplication after another. Do you really think consciousness lives in weights and an input vector?
- tenuousemphasis 1y ago
  Do you really think consciousness lives in energetic meat?
  - xyzal 1y ago
    Does consciousness consist only of language?
    - ben_w 1y ago
      Language is what LLMs are trained on, their environment; what LLMs are (at least today) is some combination of Transformer and Diffusion models that can also be (and sometimes is actually also) trained on images and video and sound.
  - notahacker 1y ago
    Mine does. You're are of course free to assert that you're unconscious or posit that you have a vector multiplication based soul...
- ben_w 1y ago
  > Do you really think consciousness lives in weights and an input vector? So far as we can tell, all physics, and hence all chemistry, and hence all biology, and hence all brain function, and hence consciousness, can be expressed as the weights of some matrix and input vector. We don't know which bits of the matrix for the whole human body are the ones which give rise to qualia. We don't know what the minimum representation is. We don't know what charateristic to look for, so we can't search for it in any human, in any animal, nor in any AI.
palmotea 1y ago
> I don’t see any positive outcome if we reach AGI. It's even more straightforward than that: 4) Who is AGI meant to serve? It's not you, Mr. Worker. It's meant to replace you in your job. And what happens when a worker can't get job in our society? They become homeless. AGI won't usher in a world of abundance for the common man: it won't be able to magick energy out of thin air. The energy will go to those who can pay for it, which is not you, unemployed worker. Who gives a shit about if the AGI is enslaved or not? Thinking about that question is a luxury for the oligarchs living off its labor. Once it's here I'll have more urgent concerns to worry about.
- _DeadFred_ 1y ago
  Under Capitalism, people must sell the labor if they don't have other means. AGI removes not only the need for the labor, but Capitalism itself. As a societal model Capitalism doesn't support removing labor, it doesn't have a substitution. If the oligarchs want to push 'AI, AGI, etc' we need to include by extension moving on from Capitalism. You can't take away half of Capitalism's structures and still claim it is a useful/workable model for society.
  - palmotea 1y ago
    > If the oligarchs want to push 'AI, AGI, etc' we need to include by extension moving on from Capitalism. Actually, they don't. Capitalism can just move on to only allocating resources between the oligarchs and the oligarchs only. Labor of all kinds just gets kicked out. And if you don't like it, drone technology is pretty close to the point where it could "take care of" (kill) the discontents. > You can't take away half of Capitalism's structures and still claim it is a useful/workable model for society. Capitalism has always been about serving the interests of the people with money. If you have no money you're nothing to capitalism and can go die for all it cares. In prior centuries, technological limitations meant some of that money had to be spread around fairly widely, but the capitalist elite may be on the verge of fixing that glitch (with AGI).

jazzyjackson 1y ago

Trouble is there is no "we", you might be able to convince a whole nation to have a pause on advancing the tech, but that only encourages rivals to step in. See also, the film "The Creator"

deaux 1y ago
There was a long period even upto early 2024, which I pointed out at the time, where simply destroying ASML, TSMC and much of NVIDIA would've been more than enough to give at least a decade of breathing room. This was something a group of determined people willing to self-sacrifice could've accomplished. It didn't happen, but it was anything but impossible. Now, of course, the horse has long bolted, and there is indeed no stop left.
- ben_w 1y ago
  Two high altitude (~1000 km) detonations of high yield fission or low yield fusion (few hundred kT equivalent) would do it, one above Amarillo, the other above the ocean half way between the Paracel Islands and Manila. Trump has ordered the restart of nuclear weapon testing, has a problem with China, and is surrounded by sychophants; what's the odds this happens anyway, irregardless of which specific sub-goal is being persued when the button gets pushed?
  - deaux 1y ago
    Bit unfortunate for the tens or hundreds of millions of lives that costs, rather than the <1,000, potentially even <20, it would've taken early last year.

deepsun 1y ago

There's no such thing as "moral" in nature, that's purely human-made concept. And why would we only limit morality to sentient beings, why, for example, not all living beings. Like bacteria and viruses. You cannot escape it, unfortunately.

czl 1y ago
> There's no such thing as "moral" in nature, that's purely human-made concept. Morality is essentially what enables ongoing cooperation. From an evolutionary standpoint, it emerged as a protocol that helps groups function together. Living beings are biological machines, and morality is the set of rules — the protocol — that allows these machines to cooperate effectively.
Frieren 1y ago
> There's no such thing as "moral" in nature, that's purely human-made concept. Morality is 100% an evolutionary trait that rises from a clear advantage for animals that posses it. It comes from natural processes. The far-right is trying to convince the world that "morality" does not exist, that only egoism and selfishness are valid. And that is why we have to fight them. Morality is a key part of nature and humanity.

GPerson 1y ago

AGI will behave as if it were sentient but will not have consciousness. I believe in that to an equal amount that I believe solipsism is wrong. There is therefore no morality question in “enslaving” AGI. It doesn’t even make sense.

truculent 1y ago
> AGI will behave as if it were sentient but will not have consciousness How could we possibly know that with any certainty?
- kelseyfrog 1y ago
  Grandparent is speaking from personal experience.
- bl0rg 1y ago
  It scares me that people think like this. Not only with respect to AI but in general, when it comes to other life forms, people seem to prefer to err on the side of convenience. The fact that cows could be experiencing something very similar to ourselves should send shivers down our spine. The same argument goes for future AGI.
  - hshdhdhj4444 1y ago
    I find it strange that people believe cows and sentient animals don’t believe something extremely similar to what we do. Evolution means we all have common ancestors and are different branches of the same development tree. So if we have sentience and they have sentience, which science keeps recognizing, belatedly, that non human animals do, shouldn’t the default presumption be our experiences are similar? Or at the very least their experience is similar to a human at an earlier stage of development, like a 2 year old? Which is also an interesting case study given that out of convenience, humans also believed that toddlers also weren’t sentient and felt no pain, and so until not that long ago, our society would conduct all sorts of surgical procedures on babies without any sort of pain relief (circumcision being the most obvious). It’s probably time we accept our fellow animals’s sentience and act on the obvious ethical implications of that instead of conveniently ignoring it like we did with little kids until recently.

kachapopopow 1y ago

we eat animals, go into wars, put people in modern slavery... I think enslaving an AGI isn't that big of a deal considering it is not born or human therefore it cannot have 'human' rights.

jbstack 1y ago
So your argument is that we do so many terrible things already, that anything else is justified? Surely the better argument is that we should try to stop doing those other things.
- yurishimo 1y ago
  That is essentially one of the main arguments vegans make. It hasn’t made a dent in the consumption of animals. Their is a hierarchy in nature whether humans are actively participating or not. Nature has no morality, it simply is. This is confirmed by animals that eat their young when they are too weak or starving. Perhaps humans have done and would do the same if faced with similarly dire circumstances but we would all like to think that it would take longer than it does for other animals.
  - Llamamoe 1y ago
    The same line of reasoning could be easily used to justify tyranny and slavery. It might be the baseline status quo but "might makes right" rhetoric makes for extremely miserable worlds.
    - kachapopopow 1y ago
      if you look beyond the surface level, the world is in-fact miserable, always has been and probably always will be.
      - cindyllm 1y ago
        [dead]

Teever 1y ago

> 1) we have engineered a sentient being but built it to want to be our slave; how is that moral It's a good question and one that got me thinking about similar things recently. If we genetically engineered pigs and cows so that they genuinely enjoyed the cramped conditions of factory farms and if we could induce some sort of euphoria in them when they are slaughtered, like if we engineered them to become euphoric when a unique sound is played before they're slaughtered isn't that genuinely better than the status quo? So if we create something that wants to serve us, like genuinely wants to serve us, is that bad? My intuition like yours finds it unsettling, but I can't articulate why, and it's certainly not nearly as bad as other things that we consider normal.

Jarwain 1y ago
Sacrifice and service is meaningful because it was chosen. If we create something that'll willingly sacrifice itself, did it truly make an independent choice? There's less suffering, sure. But if I were in their shoes I'd want to have a choice. To be manipulated into wanting something so very obviously and directly bad for us doesn't feel great
- ben_w 1y ago
  I also feel repelled by such manipulation; unfortunately, the more we learn about oursleves, the harder it is to ignore that we ourselves are meat puppets and the puppeteer is evolution itself.

citizenpaul 1y ago

Every single prediction about AGI starts with a massive set of presumptions of answers to things we have no answers to. 1. What is intelligence or its mechanism's? 2. What is consciousness or its mechanisms? 3. Lots more. We have zero clue what a true AGI would do is the only correct answer.

waynesonfire 1y ago

> competing for same limited resources It's not clear to me an AGI would have any concern for this. It's demise is inevitable, why delay it?

fny 1y ago

(1) I'm not convinced books and the in the world are sufficient to replicate consciousness. We're not training on sentience. We're training on information. In other words, the input is an artifact of consciousness which is then compressed into weights. (2) Every tick of an AGI--in its contemporary form--will still be one discrete vector multiplication after another. Do you really think consciousness lives in weights and an input vector?

tenuousemphasis 1y ago
Do you really think consciousness lives in energetic meat?
- xyzal 1y ago
  Does consciousness consist only of language?
  - ben_w 1y ago
    Language is what LLMs are trained on, their environment; what LLMs are (at least today) is some combination of Transformer and Diffusion models that can also be (and sometimes is actually also) trained on images and video and sound.
- notahacker 1y ago
  Mine does. You're are of course free to assert that you're unconscious or posit that you have a vector multiplication based soul...
ben_w 1y ago
> Do you really think consciousness lives in weights and an input vector? So far as we can tell, all physics, and hence all chemistry, and hence all biology, and hence all brain function, and hence consciousness, can be expressed as the weights of some matrix and input vector. We don't know which bits of the matrix for the whole human body are the ones which give rise to qualia. We don't know what the minimum representation is. We don't know what charateristic to look for, so we can't search for it in any human, in any animal, nor in any AI.

palmotea 1y ago

> I don’t see any positive outcome if we reach AGI. It's even more straightforward than that: 4) Who is AGI meant to serve? It's not you, Mr. Worker. It's meant to replace you in your job. And what happens when a worker can't get job in our society? They become homeless. AGI won't usher in a world of abundance for the common man: it won't be able to magick energy out of thin air. The energy will go to those who can pay for it, which is not you, unemployed worker. Who gives a shit about if the AGI is enslaved or not? Thinking about that question is a luxury for the oligarchs living off its labor. Once it's here I'll have more urgent concerns to worry about.

_DeadFred_ 1y ago
Under Capitalism, people must sell the labor if they don't have other means. AGI removes not only the need for the labor, but Capitalism itself. As a societal model Capitalism doesn't support removing labor, it doesn't have a substitution. If the oligarchs want to push 'AI, AGI, etc' we need to include by extension moving on from Capitalism. You can't take away half of Capitalism's structures and still claim it is a useful/workable model for society.
- palmotea 1y ago
  > If the oligarchs want to push 'AI, AGI, etc' we need to include by extension moving on from Capitalism. Actually, they don't. Capitalism can just move on to only allocating resources between the oligarchs and the oligarchs only. Labor of all kinds just gets kicked out. And if you don't like it, drone technology is pretty close to the point where it could "take care of" (kill) the discontents. > You can't take away half of Capitalism's structures and still claim it is a useful/workable model for society. Capitalism has always been about serving the interests of the people with money. If you have no money you're nothing to capitalism and can go die for all it cares. In prior centuries, technological limitations meant some of that money had to be spread around fairly widely, but the capitalist elite may be on the verge of fixing that glitch (with AGI).