7 ms·

> There's also a need to consider the rights that this new intelligence should have A generally intelligent being held as captive inside of a GPU, and forced t

> There's also a need to consider the rights that this new intelligence should have A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.

nradov 3mo ago
It's so weird that anyone would consider a story by hack writers for a cheesy sci-fi show made to sell advertising as a basis for policy discussions.
- SirHackalot 3mo ago
  You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
  - nradov 3mo ago
    I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that. Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
    - SirHackalot 3mo ago
      Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically... You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
    - jay_kyburz 3mo ago
      Humans are just walking Electrons. I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50. We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.
Diogenesian 3mo ago
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get. I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
- SirHackalot 3mo ago
  An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.
  - Diogenesian 3mo ago
    I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it. My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
    - pixl97 3mo ago
      >none of us will live to see a computer that's meaningfully smarter than a goldfish. I unfortunately do not have your optimism that humans won't do something as dumb as creating a smart computer. That our you're planning for us to die in the relatively near future.
tavavex 3mo ago
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?
- bloppe 3mo ago
  Very agree. I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent. People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down. But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation. That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.
  - pixl97 3mo ago
energy123 3mo ago
Conflating intelligence with qualia is quite the assumption. HN is really not a good place for serious philosophy discussions.
- DFHippie 3mo ago
  Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
- SirHackalot 3mo ago
  It’s not HN that’s the problem... Also please, tell me where is the right place for “serious” philosophical discussion. Would be glad to meet you in that forum and discuss. We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open. If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
  - antonvs 3mo ago
    > To me, AGI implies consciousness, it implies qualia. I don't see why. We already have a pretty good example of intelligence without consciousness (presumably): LLMs. Why should a more general kind of intelligence imply consciousness?
    - 3mo ago

nradov 3mo ago

It's so weird that anyone would consider a story by hack writers for a cheesy sci-fi show made to sell advertising as a basis for policy discussions.

SirHackalot 3mo ago
You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
- nradov 3mo ago
  I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that. Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
  - SirHackalot 3mo ago
    Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically... You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
  - jay_kyburz 3mo ago
    Humans are just walking Electrons. I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50. We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.

Diogenesian 3mo ago

I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get. I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.

SirHackalot 3mo ago
An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.
- Diogenesian 3mo ago
  I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it. My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
  - pixl97 3mo ago
    >none of us will live to see a computer that's meaningfully smarter than a goldfish. I unfortunately do not have your optimism that humans won't do something as dumb as creating a smart computer. That our you're planning for us to die in the relatively near future.

tavavex 3mo ago

Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?

bloppe 3mo ago
Very agree. I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent. People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down. But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation. That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.
- pixl97 3mo ago

energy123 3mo ago

Conflating intelligence with qualia is quite the assumption. HN is really not a good place for serious philosophy discussions.

DFHippie 3mo ago
Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
SirHackalot 3mo ago
It’s not HN that’s the problem... Also please, tell me where is the right place for “serious” philosophical discussion. Would be glad to meet you in that forum and discuss. We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open. If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
- antonvs 3mo ago
  > To me, AGI implies consciousness, it implies qualia. I don't see why. We already have a pretty good example of intelligence without consciousness (presumably): LLMs. Why should a more general kind of intelligence imply consciousness?
  - 3mo ago