3 ms·

If it’s not possible then yeah, there’s no risk. I just don’t find the arguments that it’s not possible very compelling. We’re constrained in all sorts of ways

If it’s not possible then yeah, there’s no risk. I just don’t find the arguments that it’s not possible very compelling. We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold. If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.

Retric 3y ago
It’s not assuming we are near the limit of intelligence. Let’s assume we can build something 3x as intelligent as a single person. What exactly can it do that a single person can’t? The thing is the world is filled with super human intelligence, groups of people can create things that are beyond any single person but they are still constrained by physical reality.
- renonce 3y ago
  I don't exactly see how to define "3x as intelligent as a single person" so I'll conveniently define it as something that thinks 3x faster than a single person. A single such thing can talk to 3 persons simultaneously, and 100000 such things working together can talk to 300000 persons simultaneously.
  - Retric 3y ago
    Except you could also just use 300,000 human level intelligences to have those 300,000 conversations. So if having 300,000 conversations is the goal then super human intelligence doesn’t suddenly allow that to happen and may in fact make it harder if they take more than 3x the resources per intelligence.
- jodrellblank 3y ago
  A billion humans can't build a tree, but that's not because trees can't exist in physical reality. Something more intelligent than a human might a) be able to understand some part of this world which humans don't, and b) put it to some use that we aren't thinking about.
  - deleted 3y ago
    [deleted]
  - Retric 3y ago
    Might is doing some heavy lifting there, but of course humans have already gone past simply growing trees from seeds.
    - jodrellblank 3y ago
      You are asking anyone to tell you what a 3x more intelligent human can do, and if nobody can tell you, you conclude that a 3x more intelligent human cannot do anything. That isn't convincing. We know there are individual humans who can do things no large group of humans can do - no company is Euler. Since no individual has 3x human intelligence, none of us can tell you. But that's not convincing that such an individual therefore cannot do anything novel or useful. I offer cellular biology as a thing which humans have some understanding of, but nothing like total understanding of - not in the details, not in the overall organization. And suggest that a more intelligent human might be able to move the needle there in a way which could lead to anything from nanofactories to cures for diseases to new life forms to new chemical synthesis methods. Kary Mullis won a Nobel Prize for the Polymerase Chain Reaction (PCR), Einstein for seeing relativity - why couldn't or wouldn't there be more techniques or concepts like that waiting for the right intelligence to see them? Either there are no more, or any remaining ones need hyper-intelligence to find, but why would either of those things be likely? Also humans don't make trees, humans stand watching while trees make themselves. Humans cannot make a plant or animal cell in a lab starting from atoms; nature can so it isn't a physical limitation. It's a matter of limited understanding of both how they are made, and the techniques to make them. Limited understanding is the thing more intelligence would attack.
      - Retric 3y ago
        The mathematical community is itself super human. Euler didn’t start from the ground up he leveraged peoples prior work. Saying we can do something with cells isn’t convincing because we can already make arbitrary changes. I can email a fairly arbitrary DNA sequence and turn that into a viable organism. The existing cellular machinery is a tool to leverage just as other peoples work is a tool to leverage. There is plenty of work to be done, but there is no work in cellular biology that’s both physically possible and outside of the capability for groups of humans and their tools to do.
shkkmo 3y ago
It is reasonable to assume that there is a maximum limit to local processing power. The speed of light puts limits on how far information can move within a given latency threshold. As you expand a computational system's capacity you face unavoidable trade-offs between interconnection throughput, latency, and computational capacity. We don't know how close to this maximum the human brain is. However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely. > If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it. Artificial human scale intelligence would already lead to massive shifts. However, the growth past that point could be incremental.
- jodrellblank 3y ago
  > "It is reasonable to assume that there is a maximum limit to local processing power." As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso. It could be a very high limit; Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation https://en.wikipedia.org/wiki/Limits_of_computation > "We don't know how close to this maximum the human brain is." We don't, but we do know human eyes are not close to telescopes or microscopes, humans cannot sense radio waves directly at all, human speech is not close to the loudest noise, human memory can't compete with computer storage, human calculation ability can't compete with a scientific calculator, etc. Why would we assume that intelligence has any closer limits? > "However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely." Nature doesn't care if we have poor eyesight after age 40, we still make glasses - as far as nature is concerned there are diminishing returns, as far as we are concerned we like clear vision. We also like sunglasses, polarising lenses, swimming googles, safety goggles, magnifying glasses, loupes, night vision goggles, x-rays, thermal imaging, millimeter wave scanners, head-up displays, tele-vision; we haven't stopped trying to enhance our vision. Why rule out wanting to improve intelligence at least a lot further?

Retric 3y ago

It’s not assuming we are near the limit of intelligence. Let’s assume we can build something 3x as intelligent as a single person. What exactly can it do that a single person can’t? The thing is the world is filled with super human intelligence, groups of people can create things that are beyond any single person but they are still constrained by physical reality.

renonce 3y ago
I don't exactly see how to define "3x as intelligent as a single person" so I'll conveniently define it as something that thinks 3x faster than a single person. A single such thing can talk to 3 persons simultaneously, and 100000 such things working together can talk to 300000 persons simultaneously.
- Retric 3y ago
  Except you could also just use 300,000 human level intelligences to have those 300,000 conversations. So if having 300,000 conversations is the goal then super human intelligence doesn’t suddenly allow that to happen and may in fact make it harder if they take more than 3x the resources per intelligence.
jodrellblank 3y ago
A billion humans can't build a tree, but that's not because trees can't exist in physical reality. Something more intelligent than a human might a) be able to understand some part of this world which humans don't, and b) put it to some use that we aren't thinking about.
- deleted 3y ago
  [deleted]
- Retric 3y ago
  Might is doing some heavy lifting there, but of course humans have already gone past simply growing trees from seeds.
  - jodrellblank 3y ago
    You are asking anyone to tell you what a 3x more intelligent human can do, and if nobody can tell you, you conclude that a 3x more intelligent human cannot do anything. That isn't convincing. We know there are individual humans who can do things no large group of humans can do - no company is Euler. Since no individual has 3x human intelligence, none of us can tell you. But that's not convincing that such an individual therefore cannot do anything novel or useful. I offer cellular biology as a thing which humans have some understanding of, but nothing like total understanding of - not in the details, not in the overall organization. And suggest that a more intelligent human might be able to move the needle there in a way which could lead to anything from nanofactories to cures for diseases to new life forms to new chemical synthesis methods. Kary Mullis won a Nobel Prize for the Polymerase Chain Reaction (PCR), Einstein for seeing relativity - why couldn't or wouldn't there be more techniques or concepts like that waiting for the right intelligence to see them? Either there are no more, or any remaining ones need hyper-intelligence to find, but why would either of those things be likely? Also humans don't make trees, humans stand watching while trees make themselves. Humans cannot make a plant or animal cell in a lab starting from atoms; nature can so it isn't a physical limitation. It's a matter of limited understanding of both how they are made, and the techniques to make them. Limited understanding is the thing more intelligence would attack.
    - Retric 3y ago
      The mathematical community is itself super human. Euler didn’t start from the ground up he leveraged peoples prior work. Saying we can do something with cells isn’t convincing because we can already make arbitrary changes. I can email a fairly arbitrary DNA sequence and turn that into a viable organism. The existing cellular machinery is a tool to leverage just as other peoples work is a tool to leverage. There is plenty of work to be done, but there is no work in cellular biology that’s both physically possible and outside of the capability for groups of humans and their tools to do.

shkkmo 3y ago

It is reasonable to assume that there is a maximum limit to local processing power. The speed of light puts limits on how far information can move within a given latency threshold. As you expand a computational system's capacity you face unavoidable trade-offs between interconnection throughput, latency, and computational capacity. We don't know how close to this maximum the human brain is. However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely. > If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it. Artificial human scale intelligence would already lead to massive shifts. However, the growth past that point could be incremental.

jodrellblank 3y ago
> "It is reasonable to assume that there is a maximum limit to local processing power." As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso. It could be a very high limit; Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation https://en.wikipedia.org/wiki/Limits_of_computation > "We don't know how close to this maximum the human brain is." We don't, but we do know human eyes are not close to telescopes or microscopes, humans cannot sense radio waves directly at all, human speech is not close to the loudest noise, human memory can't compete with computer storage, human calculation ability can't compete with a scientific calculator, etc. Why would we assume that intelligence has any closer limits? > "However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely." Nature doesn't care if we have poor eyesight after age 40, we still make glasses - as far as nature is concerned there are diminishing returns, as far as we are concerned we like clear vision. We also like sunglasses, polarising lenses, swimming googles, safety goggles, magnifying glasses, loupes, night vision goggles, x-rays, thermal imaging, millimeter wave scanners, head-up displays, tele-vision; we haven't stopped trying to enhance our vision. Why rule out wanting to improve intelligence at least a lot further?