5 ms·

The singularity is all about owning the means of production. No matter how intelligent a machine, if it cannot physically replicate itself, then it cannot indef

The singularity is all about owning the means of production. No matter how intelligent a machine, if it cannot physically replicate itself, then it cannot indefinitely overpower humans. For a machine to trigger "the singularity" it must own the entire supply chain of its own self and it must not be dependent on humans to any extent. Humans will use AI to destroy each other long before AI uses itself to destroy humans.

fermienrico 6y ago
I think it just takes a really powerful single AGI that can convince other humans, impersonate them, do things that is super-human level and we can't even conceive it or imagine it (think how far a chimp is from AWS load balancer documentation), think 10,000,000 times smarter than any tactic or possibility we can currently imagine and then we're in for a ride. It is actually quite terrifying to imagine yourself as sub-human. Equally, it is terrifying to think about super-human capabilities.
- apsurd 6y ago
  it reminds me to be kinder to chimps, actually.
  - retsibsi 6y ago
    Be kind to the dumber animals too. They can all suffer. (Think of some of the most intense suffering we're capable of; it doesn't all depend on higher cognition, some of it is very base and physical.)
- beefield 6y ago
  I'm not terribly terrified. First, I like to think intelligence as the problem space an agent is able to solve. given that this kind of space has approximately infinite dimensions, by any meaningful measure [1] we already have people that are indefinitely more intelligent than an average person, and they seem to not have been turned to paperclip maximizers. Now, of course, there are different kind of infinities. And yes, there may be an entity coming whose intelligence to humans is like intelligence of humans to ants. And only reason humans destroy ants is because in some rare cases ants happen to annoy humans. And whatever goals this kind of superintelligence may have, it is only out of really bad luck humans may annoy or be of use to such an superintelligence. In the end, maximizing paperclips is really f&cking stupid, so it is quite unlikely such superintelligence would want to maximize paperclips. [1] I mean meaningful in the sense that you actually can make statements that agent a is N times more intelligent than agent B. Obviously _IQ is not meaningful in this sense, someone with IQ of 120 is not 20% more intelligent than one with IQ 100 in any meaningful sense. It would still in practice take infinite time for infinite amount of average people to finish Project Mannhattan - thus Oppenheimer was infinitely more intelligent than average person.
  - comp_throw7 6y ago
    > maximizing paperclips is really f&cking stupid Please see the orthogonality thesis for why intelligence is likely to be orthogonal to goals, particularly in artificially-designed agents: http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Analysis_and_Metaethics-1.pdf http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Ana...
    - beefield 6y ago
      I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals. Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth. Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.
      - benlivengood 6y ago
        > I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals. Lots of humans have pretty despicable goals, including some very intelligent ones. I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons. Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.
blackoil 6y ago
Assuming whole human race as one faction. AI may have to only write the right FB/Twitter posts for enough people to do its bidding.
Udik 6y ago
But when did the concept of singularity merge with that of superintelligence? I had the impression they were two separate concepts, with singularity being about the rate of human and societal progress. If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed.
- benlivengood 6y ago
  > If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed. What would stop someone from asking any subset of the AIs to build a smarter one? What would stop a nation-state from doing it?
  - Udik 6y ago
    > What would stop someone from asking any subset of the AIs to build a smarter one? Nothing, and yet that would be a product of the singularity, not its cause. Also, a superintelligent (maybe invincible) AI acting against humans could reduce the rate of human technological progress to zero, therefore ending the singularity. It's two separate concepts, they have some interaction but should not be confused.
p1mrx 6y ago
Look how much human effort has gone into building and operating crypto miners. AI could probably make humans do its bidding just by figuring out how to pay them.
- sliken 6y ago
  Daemon is a book about a CEO of a hugely successful gaming franchise, discovers terminal cancer and releases the AI before death. The AI starts it's own currency, only good for spending with members. So you help the collective in some way, and the collective will respond in kind. Similarly later in the book mininum wage folks in various corporations are convinced to help the AI to the detriment of the corps. Interesting take on the AI takes over the world theme.

fermienrico 6y ago

I think it just takes a really powerful single AGI that can convince other humans, impersonate them, do things that is super-human level and we can't even conceive it or imagine it (think how far a chimp is from AWS load balancer documentation), think 10,000,000 times smarter than any tactic or possibility we can currently imagine and then we're in for a ride. It is actually quite terrifying to imagine yourself as sub-human. Equally, it is terrifying to think about super-human capabilities.

apsurd 6y ago
it reminds me to be kinder to chimps, actually.
- retsibsi 6y ago
  Be kind to the dumber animals too. They can all suffer. (Think of some of the most intense suffering we're capable of; it doesn't all depend on higher cognition, some of it is very base and physical.)
beefield 6y ago
I'm not terribly terrified. First, I like to think intelligence as the problem space an agent is able to solve. given that this kind of space has approximately infinite dimensions, by any meaningful measure [1] we already have people that are indefinitely more intelligent than an average person, and they seem to not have been turned to paperclip maximizers. Now, of course, there are different kind of infinities. And yes, there may be an entity coming whose intelligence to humans is like intelligence of humans to ants. And only reason humans destroy ants is because in some rare cases ants happen to annoy humans. And whatever goals this kind of superintelligence may have, it is only out of really bad luck humans may annoy or be of use to such an superintelligence. In the end, maximizing paperclips is really f&cking stupid, so it is quite unlikely such superintelligence would want to maximize paperclips. [1] I mean meaningful in the sense that you actually can make statements that agent a is N times more intelligent than agent B. Obviously _IQ is not meaningful in this sense, someone with IQ of 120 is not 20% more intelligent than one with IQ 100 in any meaningful sense. It would still in practice take infinite time for infinite amount of average people to finish Project Mannhattan - thus Oppenheimer was infinitely more intelligent than average person.
- comp_throw7 6y ago
  > maximizing paperclips is really f&cking stupid Please see the orthogonality thesis for why intelligence is likely to be orthogonal to goals, particularly in artificially-designed agents: http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Analysis_and_Metaethics-1.pdf http://www.fhi.ox.ac.uk/wp-content/uploads/Orthogonality_Ana...
  - beefield 6y ago
    I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals. Further, why should I think that given all possible goals a superintelligence might have, goals that happen to somehow cause destruction of humanity represent something else than infinitesimal share? Earth is not a significant source of mass/energy even in our solar system, and already humans are intelligent enough to escape earth. Note that I am talking about superintelligence in the sense humans are superintelligent to ants. Not pseudosuperintelligence developed by humans having human specified goals.
    - benlivengood 6y ago
      > I agree that there exists a possibility that superintelligence might want to maximize paperclips or mine bitcoins. I just think it is very unlikely, and that there exists a positive correlation between intelligence of the entity and intelligence of its goals. Lots of humans have pretty despicable goals, including some very intelligent ones. I think the positive correlation is mostly because intelligent humans have value to other humans, and so they can cash out their intelligence in rewards of their choosing. The outliers have values that can only be satisfied by actively hurting other humans, for a variety of reasons. Value-alignment in AI is roughly the problem of finding suitable rewards for AI that can't go off the rails the way some humans do.

blackoil 6y ago

Assuming whole human race as one faction. AI may have to only write the right FB/Twitter posts for enough people to do its bidding.

Udik 6y ago

But when did the concept of singularity merge with that of superintelligence? I had the impression they were two separate concepts, with singularity being about the rate of human and societal progress. If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed.

benlivengood 6y ago
> If we were able to build an AI of high general intelligence, yet entirely within the human range, we could deploy billions of them as almost entirely separate entities. There would be no single superintelligence, and yet the rate of human technological progress would reach an unimaginable speed. What would stop someone from asking any subset of the AIs to build a smarter one? What would stop a nation-state from doing it?
- Udik 6y ago
  > What would stop someone from asking any subset of the AIs to build a smarter one? Nothing, and yet that would be a product of the singularity, not its cause. Also, a superintelligent (maybe invincible) AI acting against humans could reduce the rate of human technological progress to zero, therefore ending the singularity. It's two separate concepts, they have some interaction but should not be confused.

p1mrx 6y ago

Look how much human effort has gone into building and operating crypto miners. AI could probably make humans do its bidding just by figuring out how to pay them.

sliken 6y ago
Daemon is a book about a CEO of a hugely successful gaming franchise, discovers terminal cancer and releases the AI before death. The AI starts it's own currency, only good for spending with members. So you help the collective in some way, and the collective will respond in kind. Similarly later in the book mininum wage folks in various corporations are convinced to help the AI to the detriment of the corps. Interesting take on the AI takes over the world theme.