6 ms·

It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architecture

by valine 3y ago

It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architectures and end up with helpful assistants similar in capability to George Lucas's C3PO. The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.

pixl97 3y ago
>It's not clear at all that we have an avenue to super intelligence. All problems in reality are probability problems. If we don't have a path to superintelligence, then the worst problems just don't manifest themselves. If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty. It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario". >We simply don't know how to build a system like that. You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
- RandomLensman 3y ago
  Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.
  - pixl97 3y ago
    >What's the model to get to that Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now. I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it. There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to. Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
    - RandomLensman 3y ago
      Do you think we could wipe out all furry mammals, for example? Could another intelligence have the same level of difference to us as in your story we to furry mammals? We don't even know if the mythical superintelligence could manifest the way you assume. It assumes that intelligence basically can overcome any obstacles - I'd say we actually see that seems not to be the case currently and claims that that is just a function of sufficient intelligence are unproven (setting aside physical limits to certain actions and results).
      - pixl97 3y ago
        >Do you think we could wipe out all furry mammals, for example? lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point. > It assumes that intelligence basically can overcome any obstacles In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.
        RandomLensman 3y ago
        So that is a "no" on all furry mammals. Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.
patrec 3y ago
> It's not clear at all that we have an avenue to super intelligence AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again. But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about. Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me. > The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that. Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.
- mrtranscendence 3y ago
  > we are still seeing exponential progress I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware. We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.
- AnimalMuppet 3y ago
  Right, and if someone can join the two, that could be something genuinely formidable. But does anyone have a credible path to joining the different flavors to produce a unity that actually works?
mitthrowaway2 3y ago
> We simply don't know how to build a system like that. Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems. "We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search. [1] https://www.wired.com/2014/05/the-world-of-computer-go/ https://www.wired.com/2014/05/the-world-of-computer-go/ [2] https://kotaku.com/humans-triumph-over-machines-in-protein-folding-game-fo-5605576 https://kotaku.com/humans-triumph-over-machines-in-protein-f... [3] https://www.jstor.org/stable/24354221 https://www.jstor.org/stable/24354221 [4] https://davidol.medium.com/will-ai-ever-be-able-to-make-a-joke-808a656b53a6 https://davidol.medium.com/will-ai-ever-be-able-to-make-a-jo...
TheOtherHobbes 3y ago
We don't need an avenue to super-intelligence. We just need a system that is better at manipulating human beliefs and behaviour than our existing media, PR, and ad industries. The problem is not science fiction god-mode digital quetta-smart hypercomputing. This is about political, social, and economic influence, and who controls it.
- AnimalMuppet 3y ago
  That risk isn't about AI-as-AI. That risk is about AI-as-better-persuasive-nonsense-generator. But the same risk is there for any better-persuasive-nonsense-generator, completely independent from whether it's an AI. It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.
- babyshake 3y ago
  Indeed, an epistemological crisis seems to be the most realistic problem in the next few years.
- patrec 3y ago
  Effective dystopian mass-manipulation and monitoring are a real concern and we're closer to it[1] than to super intelligence. But super-intelligence going wrong is almost incomparably worse. So we should very much worry about it as well. [1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.
HDThoreaun 3y ago
A super intelligent AI is not necessary for AI to be an threat. Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage. We are not far from such an AI being accessible to the masses. You can try to frame this like the gun debate "it's not the AI it's the people using it" but the AI would be acting autonomously here. I have no faith that people won't do extremely risky things if given the opportunity.
- tome 3y ago
  > Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage North Korea and Iran are (essentially) already trying to do that, so I think that particular risk is well understood.

by valine 3y ago

pixl97 3y ago

>It's not clear at all that we have an avenue to super intelligence. All problems in reality are probability problems. If we don't have a path to superintelligence, then the worst problems just don't manifest themselves. If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty. It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario". >We simply don't know how to build a system like that. You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.

RandomLensman 3y ago
Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.
- pixl97 3y ago
  >What's the model to get to that Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now. I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it. There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to. Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
  - RandomLensman 3y ago
    Do you think we could wipe out all furry mammals, for example? Could another intelligence have the same level of difference to us as in your story we to furry mammals? We don't even know if the mythical superintelligence could manifest the way you assume. It assumes that intelligence basically can overcome any obstacles - I'd say we actually see that seems not to be the case currently and claims that that is just a function of sufficient intelligence are unproven (setting aside physical limits to certain actions and results).
    - pixl97 3y ago
      >Do you think we could wipe out all furry mammals, for example? lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point. > It assumes that intelligence basically can overcome any obstacles In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.
      - RandomLensman 3y ago
        So that is a "no" on all furry mammals. Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.

patrec 3y ago

> It's not clear at all that we have an avenue to super intelligence AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again. But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about. Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me. > The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that. Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.

mrtranscendence 3y ago
> we are still seeing exponential progress I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware. We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.
AnimalMuppet 3y ago
Right, and if someone can join the two, that could be something genuinely formidable. But does anyone have a credible path to joining the different flavors to produce a unity that actually works?

mitthrowaway2 3y ago

> We simply don't know how to build a system like that. Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems. "We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search. [1] https://www.wired.com/2014/05/the-world-of-computer-go/ https://www.wired.com/2014/05/the-world-of-computer-go/ [2] https://kotaku.com/humans-triumph-over-machines-in-protein-folding-game-fo-5605576 https://kotaku.com/humans-triumph-over-machines-in-protein-f... [3] https://www.jstor.org/stable/24354221 https://www.jstor.org/stable/24354221 [4] https://davidol.medium.com/will-ai-ever-be-able-to-make-a-joke-808a656b53a6 https://davidol.medium.com/will-ai-ever-be-able-to-make-a-jo...

TheOtherHobbes 3y ago

We don't need an avenue to super-intelligence. We just need a system that is better at manipulating human beliefs and behaviour than our existing media, PR, and ad industries. The problem is not science fiction god-mode digital quetta-smart hypercomputing. This is about political, social, and economic influence, and who controls it.

AnimalMuppet 3y ago
That risk isn't about AI-as-AI. That risk is about AI-as-better-persuasive-nonsense-generator. But the same risk is there for any better-persuasive-nonsense-generator, completely independent from whether it's an AI. It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.
babyshake 3y ago
Indeed, an epistemological crisis seems to be the most realistic problem in the next few years.
patrec 3y ago
Effective dystopian mass-manipulation and monitoring are a real concern and we're closer to it[1] than to super intelligence. But super-intelligence going wrong is almost incomparably worse. So we should very much worry about it as well. [1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.

HDThoreaun 3y ago

A super intelligent AI is not necessary for AI to be an threat. Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage. We are not far from such an AI being accessible to the masses. You can try to frame this like the gun debate "it's not the AI it's the people using it" but the AI would be acting autonomously here. I have no faith that people won't do extremely risky things if given the opportunity.

tome 3y ago
> Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage North Korea and Iran are (essentially) already trying to do that, so I think that particular risk is well understood.