6 ms·

you are pretty optimistic about your AI's ability to deal with novel situations.

by lsc 8y ago

nopinsight 8y ago
To be unconcerned, you need to be very optimistic and have a very strong believe (99% or higher probability) that no AI to be invented in the next four or five decades can deal with novel situations (which we didn’t anticipate beforehand).
- lsc 8y ago
  Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances. Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future. But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button. The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.
  - nopinsight 8y ago
    Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years. Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war. A general AI would have certain immediate advantages over humans: - Immensely larger communication bandwidth - Broader knowledge than any single group of humans - Faster thinking speed (at least by enlisting other processors) Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?
    - lsc 8y ago
      >Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years. My understanding is that there are physical limits, and we will asymtodically approach those physical limits. So while progress will never stop, each year, progress will be less. (you are right that we don't know where it will stop. we could have more breakthroughs in us. even without compute density breakthroughs, there a breakthroughs in compute type, like the TPU you mention. But there are physical limits, and it seems that we are approaching them.) >Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you? I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst. Really, I'm far more concerned with the renewed stirrings of nationalism than anything else; We've got the technology to end humanity already; we've had that for like half a century now. The thing was? last time Nationalism was a real force, we didn't have that technology. That is what scares me. If we combine early 20th century politics with late 20th century weapons, civilization will end. humanity will end.
      - vidarh 8y ago
        > But there are physical limits, and it seems that we are approaching them.) Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain. It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints. We know the processes involved are tremendously inefficient and full of redundancies, because the cost of failure is immensely high, and because the manufacturing environment is incredibly messy and error prone. It may be reasonable to think we won't get there in the next couple of decades, but unless it's fundamentally impossible (because our brains are mere projections of something more complex "outside" our universe), it's a question of when, not if. > I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst. That misses the point of the question. It's not "how horrible can an AI get?" It's "at what point is it too late to stop them if they're going the wrong direction?" The problem is that by the time you realise you were dealing with a human-level AGI, it may already be far too late - it may have cloned itself many times over, and found multiple optimisations, each making the next step up easier. As for something worse, even the worst human regimes have fundamentally depended on humans for survival. Their own survival have been a strong motivation for trying to avoid the most extreme reactions. An AGI has potentially entirely different parameters - it doesn't have the same needs in order to survive. Put another way: Imagine an AGI caring as little about our survival as we have for other species. Now imagine if humanity didn't itself need to breathe the air, didn't clean water, and so on.
  - TeMPOraL 8y ago
    > Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances. Why do you think this is relevant? Consider that a) human brain does intelligence within a small package made out of structures that are large compared to what silicon fabs deal with, and b) we've already beaten the brain in straight compute density by many orders of magnitude. The answer doesn't seem to be "more compute"; it seems to lie in what we're doing with that computing power. > the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future. Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.
    - lsc 8y ago
      >b) we've already beaten the brain in straight compute density by many orders of magnitude. citation needed. computers are better at arithmetic, sure, but computers can only do things that you can map to arithmetic. I'm given to understand that we don't understand how the brain works well enough to simulate even a very small one. Not because we lack the compute power, but because we don't know what to simulate. I think there's a lot of evidence that brains are doing something powerful (that we don't yet fully understand) that computers are unable to do. It's quite possible that brains are doing something that can't be done in a practical way with transistors. In short, I don't know of any solid evidence that the brain is only a turing machine. I mean, brains can be used as a turing machine, but a brain is terrible at that, and can't compete with even really primitive purpose built turing machines. I mean, I'm not saying that it is theoretically impossible to map what a brain does on to sufficiently powerful silicon, or that we wont figure out, at some point in the future, whatever non-arithmatic primitives the brain uses... just as far as I can tell, we haven't yet, and that means it's likely that a brain is still more powerful in some ways than even our largest computers. (obviously, even a small computer is more powerful when the problem maps cleanly to mathematical primitives. ) >Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter. What if we just got lucky?
harshreality 8y ago
You're essentially arguing that AGI can't exist, and you're assuming when people say "AI will turn the world to paperclips" they mean a really smart go-playing neural net could do that. Of course it can't. Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI. It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.
- lsc 8y ago
  Eh, if you are talking about something truly human-like rather than what people usually mean when they say AI these days, I think there are other limits. We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before. And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.
  - deleted 8y ago
    [deleted]
  - exoesquitur 8y ago
    What you say is true for silicon, but Turing complete molecular machinery is already out there, we just have yet to master its intricacies. The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode. This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?
vidarh 8y ago
I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence)
- lsc 8y ago
  >I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence) right, but all our computers right now are based on lumping transistors together so you can do math. It's likely that our brains work on a different (but still physical) principle. It's possible our brains can't reasonably be reduced to arithmatic. It's possible (I think likely) that we need different primitives to reasonably simulate a brain. I mean, brains make absolutely terrible Turing machines. And Turing machines make terrible brains. I suspect they just are very different sorts of things, and running something meant for one on the other produces poor results. Sure, it's possible we can discover some new compute primitive that is more powerful (or powerful in a different way) that can do the things the brain can do. It's possible it could do those things better than the brain can. My point is just that for a long time it looked like our current transistor based computers were getting more and more powerful, and that eventually, even if the emulation was terribly inefficient, we'd eventually have enough power to make it work. As we approach the limits of our abilities to make tiny transistors, this is looking less likely.
  - vidarh 8y ago
    What computers are like right now is irrelevant to the issue being discussed. If/when we develop AGI we will necessarily have figured out what the computational model needed is. And the odds that we can not then improve on it seems extremely unlikely to me. > As we approach the limits of our abilities to make tiny transistors, this is looking less likely. Is it? Computational power is still growing like before. What is getting harder is speeding up individual threads of computation but that has never been a likely efficient model for AGI. Things like Google's TPUs etc. represents just the very beginning in optimising hardware specifically for AI. There are many fields where single-thread performance will be a limiting factor, but AI is not it.

you are pretty optimistic about your AI's ability to deal with novel situations.

by lsc 8y ago

you are pretty optimistic about your AI's ability to deal with novel situations.

To be unconcerned, you need to be very optimistic and have a very strong believe (99% or higher probability) that no AI to be invented in the next four or five decades can deal with novel situations (which we didn’t anticipate beforehand).

lsc 8y ago
Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances. Personally, I find it amazing that humanity survived the cold war. the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future. But, the fission bombs were expensive, centralized weapons. there was a small number of men who could have pushed the button. The problems I see now aren't general AI; the problems I think you should fear is that we have very powerful tools that are very cheap. I mean, like the deepfakes thing. If instead of making creepy porn, what if people used that technology to cause chaos? Make targeted ads/videos of leaders saying nutty things? I mean, I think we could adapt to deepfakes style attacks fairly quickly; someone would have to use it decisively while it's new. But things like that are being invented all the time.
- nopinsight 8y ago
  Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years. Malicious humans are in general easier to deal with since we know most of them are subject to ingrained psychological tendencies we have learned much about over the course of history and they depend on other humans to pull off something as catastrophic as a world war. A general AI would have certain immediate advantages over humans: - Immensely larger communication bandwidth - Broader knowledge than any single group of humans - Faster thinking speed (at least by enlisting other processors) Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you?
  - lsc 8y ago
    >Slower progress in microprocessors is a good point but specialized hardware like TPUs could overcome some limitations. Also, slower does not mean stop. When we talk about decades into the future, even a 20% average annual improvement amounts to 3 orders of magnitude in 40 years. My understanding is that there are physical limits, and we will asymtodically approach those physical limits. So while progress will never stop, each year, progress will be less. (you are right that we don't know where it will stop. we could have more breakthroughs in us. even without compute density breakthroughs, there a breakthroughs in compute type, like the TPU you mention. But there are physical limits, and it seems that we are approaching them.) >Let me ask you (or anyone else) a question: What would be a minimum demonstrated capability of an AI that starts to worry you? I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst. Really, I'm far more concerned with the renewed stirrings of nationalism than anything else; We've got the technology to end humanity already; we've had that for like half a century now. The thing was? last time Nationalism was a real force, we didn't have that technology. That is what scares me. If we combine early 20th century politics with late 20th century weapons, civilization will end. humanity will end.
    - vidarh 8y ago
      > But there are physical limits, and it seems that we are approaching them.) Unless you believe in something supernatural providing extra computational capacity, we know that the physical limits allows the manufacture of a sub 20W computational device in about 1.5kg of matter with the capacity of a human brain. It may be impossible to reach the same using silicon, but we know it is possible to get there somehow within those constraints. We know the processes involved are tremendously inefficient and full of redundancies, because the cost of failure is immensely high, and because the manufacturing environment is incredibly messy and error prone. It may be reasonable to think we won't get there in the next couple of decades, but unless it's fundamentally impossible (because our brains are mere projections of something more complex "outside" our universe), it's a question of when, not if. > I lived through the end of the 20th century, and have studied a lot of 20th century history. I'm having a really difficult time imagining anything that could be worse than humanity at it's worst. That misses the point of the question. It's not "how horrible can an AI get?" It's "at what point is it too late to stop them if they're going the wrong direction?" The problem is that by the time you realise you were dealing with a human-level AGI, it may already be far too late - it may have cloned itself many times over, and found multiple optimisations, each making the next step up easier. As for something worse, even the worst human regimes have fundamentally depended on humans for survival. Their own survival have been a strong motivation for trying to avoid the most extreme reactions. An AGI has potentially entirely different parameters - it doesn't have the same needs in order to survive. Put another way: Imagine an AGI caring as little about our survival as we have for other species. Now imagine if humanity didn't itself need to breathe the air, didn't clean water, and so on.
- TeMPOraL 8y ago
  > Yeah, deal with novel situations better than humans. In a world where we are approaching the physical limits of integrated circuit feature size? I like my chances. Why do you think this is relevant? Consider that a) human brain does intelligence within a small package made out of structures that are large compared to what silicon fabs deal with, and b) we've already beaten the brain in straight compute density by many orders of magnitude. The answer doesn't seem to be "more compute"; it seems to lie in what we're doing with that computing power. > the fact we did not self-immolate goes against everything I believe about humanity and human nature. The fact that we made it through that gives me a lot of hope for the future. Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter.
  - lsc 8y ago
    >b) we've already beaten the brain in straight compute density by many orders of magnitude. citation needed. computers are better at arithmetic, sure, but computers can only do things that you can map to arithmetic. I'm given to understand that we don't understand how the brain works well enough to simulate even a very small one. Not because we lack the compute power, but because we don't know what to simulate. I think there's a lot of evidence that brains are doing something powerful (that we don't yet fully understand) that computers are unable to do. It's quite possible that brains are doing something that can't be done in a practical way with transistors. In short, I don't know of any solid evidence that the brain is only a turing machine. I mean, brains can be used as a turing machine, but a brain is terrible at that, and can't compete with even really primitive purpose built turing machines. I mean, I'm not saying that it is theoretically impossible to map what a brain does on to sufficiently powerful silicon, or that we wont figure out, at some point in the future, whatever non-arithmatic primitives the brain uses... just as far as I can tell, we haven't yet, and that means it's likely that a brain is still more powerful in some ways than even our largest computers. (obviously, even a small computer is more powerful when the problem maps cleanly to mathematical primitives. ) >Back then, we've learned something new about nature of human societies. I don't believe the concept of MAD was known before nuclear weapons. The individual human nature didn't change, nor did the social one; it's just that we already know a lot about the former, but we're still discovering the latter. What if we just got lucky?

harshreality 8y ago

You're essentially arguing that AGI can't exist, and you're assuming when people say "AI will turn the world to paperclips" they mean a really smart go-playing neural net could do that. Of course it can't. Everyone agrees there is some magic smoke necessary for AGI that we haven't figured out yet. The arguments that AI will eat the world are about AGI, not current instances of narrow AI. It may be true that an AGI will never be created, but I don't see a reason to believe that just because nobody has figured it out yet. That philosophy is just like the physicist who, shortly before fission was discovered, didn't think splitting the atom was possible. Bohr, maybe? I don't remember.

lsc 8y ago
Eh, if you are talking about something truly human-like rather than what people usually mean when they say AI these days, I think there are other limits. We're running up against limits on feature-size in our integrated circuits. If you invent an AGI that can make smarter copies of itself... it's pretty likely those same physical limits will continue to apply, meaning that each iteration will be better than the last, but like this year's latest CPU, it will be less better than the previous generation than that generation was better than the one that came before. And yeah, I am a whole lot less optimistic about AGI in general than when I was as a younger man, simply because when I was younger, there was a lot more headroom in how much better computer hardware could get.
- deleted 8y ago
  [deleted]
- exoesquitur 8y ago
  What you say is true for silicon, but Turing complete molecular machinery is already out there, we just have yet to master its intricacies. The nightmare AGI scenario likely involves molecular computing using a chemically optimal code-base and engineered lifeforms a version or two up from current DNA microcode. This is most likely to be created by a human-AI collaboration, which will be innocuous enough in itself.... it's the creations at this level that have the potential to replace us. Would that nesecarily be bad? Or just an enhanced form of human evolution?

vidarh 8y ago

I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence)

lsc 8y ago
>I've seen humans deal with novel situations. The only assumption needed is that humans and AI are subject to the same laws of nature (that is, that there is no supernatural / outside element to human intelligence) right, but all our computers right now are based on lumping transistors together so you can do math. It's likely that our brains work on a different (but still physical) principle. It's possible our brains can't reasonably be reduced to arithmatic. It's possible (I think likely) that we need different primitives to reasonably simulate a brain. I mean, brains make absolutely terrible Turing machines. And Turing machines make terrible brains. I suspect they just are very different sorts of things, and running something meant for one on the other produces poor results. Sure, it's possible we can discover some new compute primitive that is more powerful (or powerful in a different way) that can do the things the brain can do. It's possible it could do those things better than the brain can. My point is just that for a long time it looked like our current transistor based computers were getting more and more powerful, and that eventually, even if the emulation was terribly inefficient, we'd eventually have enough power to make it work. As we approach the limits of our abilities to make tiny transistors, this is looking less likely.
- vidarh 8y ago
  What computers are like right now is irrelevant to the issue being discussed. If/when we develop AGI we will necessarily have figured out what the computational model needed is. And the odds that we can not then improve on it seems extremely unlikely to me. > As we approach the limits of our abilities to make tiny transistors, this is looking less likely. Is it? Computational power is still growing like before. What is getting harder is speeding up individual threads of computation but that has never been a likely efficient model for AGI. Things like Google's TPUs etc. represents just the very beginning in optimising hardware specifically for AI. There are many fields where single-thread performance will be a limiting factor, but AI is not it.