4 ms·

I understood from the comment, given the remark of "over the long term", the meaning over decades/centuries/millennia. Which isn't that nonsensical, if an AGI

by piva00 3y ago

I understood from the comment, given the remark of "over the long term", the meaning over decades/centuries/millennia. Which isn't that nonsensical, if an AGI is created and follows similar processes as humans to acquire knowledge, skills, etc. it is a quite logical conclusion that it will be able to render humans useless: a very advanced AGI can imagine and develop technologies just like we do, it can direct resources and machinery to build whatever it creates, including self-replication, given access to resources it can potentially render humans useless. That's at least what I interpreted from the comment.

strogonoff 3y ago
No technology can render humans useless, since technology is the “how” whereas humans are the “why”. Technology can be seen as good/bad/neutral to humans in some ways, but rationally speaking—religious views aside—humans cannot logically be qualified in those terms with respect to technology. One could say some future technology might be so good that humans need to do almost nothing in order to survive. Perhaps that’d pose some challenges in terms of mental health, but that is a very different issue, assuming it would be a issue at all (and assuming such technology is even possible in theory). One could say some advanced technology could go out of control, or magically acquire self-awareness and agency (same thing in a philosophically dubious wrapper), or (more prosaically) be (intentionally or not) misused by humans so that it harms humans, but that just puts it into the bad category.
- piva00 3y ago
  I think where we diverge is that you are assuming we will be in control of such technology. I'm in the camp which doesn't think we would be really in control of a superhuman intelligence.
  - strogonoff 3y ago
    Well, in that scenario I stand by my comment: if there is some technology that humans are in control of at all (as in, it literally performs in ways we are unable to predict), then that technology is harmful (because inability to predict significantly complicates things like survival). However, crucially, in no scenario it can make humans useless. That said, I do doubt it is likely. What is much more likely is that, more prosaically, humans can control it, but some humans misuse it and, intentionally or not, harm other humans; a story as old as civilisation itself.

by piva00 3y ago

strogonoff 3y ago

No technology can render humans useless, since technology is the “how” whereas humans are the “why”. Technology can be seen as good/bad/neutral to humans in some ways, but rationally speaking—religious views aside—humans cannot logically be qualified in those terms with respect to technology. One could say some future technology might be so good that humans need to do almost nothing in order to survive. Perhaps that’d pose some challenges in terms of mental health, but that is a very different issue, assuming it would be a issue at all (and assuming such technology is even possible in theory). One could say some advanced technology could go out of control, or magically acquire self-awareness and agency (same thing in a philosophically dubious wrapper), or (more prosaically) be (intentionally or not) misused by humans so that it harms humans, but that just puts it into the bad category.

piva00 3y ago
I think where we diverge is that you are assuming we will be in control of such technology. I'm in the camp which doesn't think we would be really in control of a superhuman intelligence.
- strogonoff 3y ago
  Well, in that scenario I stand by my comment: if there is some technology that humans are in control of at all (as in, it literally performs in ways we are unable to predict), then that technology is harmful (because inability to predict significantly complicates things like survival). However, crucially, in no scenario it can make humans useless. That said, I do doubt it is likely. What is much more likely is that, more prosaically, humans can control it, but some humans misuse it and, intentionally or not, harm other humans; a story as old as civilisation itself.