5 ms·

> 2. The claims by the Bostrom/Yudkowsky/etc. crowd that an AI intelligence explosion will cause the extinction of humanity Maybe someone can correct me if I'm

by ffwd 9y ago

> 2. The claims by the Bostrom/Yudkowsky/etc. crowd that an AI intelligence explosion will cause the extinction of humanity Maybe someone can correct me if I'm wrong here, but I have a hard time understanding what /any/ "utility function" would be, that the super ai people talk about. It can't be a passive deep learning network that parses information and gives an output, it has to be some kind of complex perception / action loop of many neural nets and actuators in the real world that somehow leads to an intelligent self improving behavior? I guess you could make a deep learning controller for self driving cars say, and if an input to many cars is wrong, all the cars crash and create a big cascading mess of wrong input values, but that kind of accident is a far cry from an intelligent chain of events where every link in the chain is an intelligent decision but the ultimate goal is bad. And, do we even know any way to chain many deep learning networks together that accurately give correct output values, that we then can hook up to a controller to give a utility function, which can then lead to a cascade of intelligent decisions across domains?

jononor 9y ago
Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
- goatlover 9y ago
  The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips". My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.
  - Smaug123 9y ago
    Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else. For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.
    - goatlover 9y ago
      If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.
      - Smaug123 9y ago
        I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it. What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…
        goatlover 9y ago
        I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips. It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me. What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate. After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world. Alternatively, it could just ask me how many "some" means.
        Hannan 9y ago
        - Hill Climbing - Local Maximum - Gradient Descent I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.
  - lazaroclapp 9y ago
    No. It means you could program an AI to a) make paperclips and b) increase its own intelligence. Assume you program (a) incorrectly (in the sense of lacking the appropriate safeguards about when not to make more paperclips) but (b) correctly. Then, it doesn't matter that the AI will eventually understand what you want perfectly, because what it will want to do is to keep making paperclips. The whole argument is that if you have a self-improving agent with a goal and the ability to improve in ways you can't predict at all, then you better make sure its goal matches yours. Of course, the obvious solution is: "make the objective function equivalent to 'do what I want you to do'", but the problem is we might not know how to encode that without help from a super-AI.
    - Smaug123 9y ago
      Even "do what I mean" is not obviously correct. Yudkowsky used to advocate the idea of "coherent extrapolated volition": the idea of what I would want the AI to do, were I much more intelligent, knew more, could think faster, were more how I wished I were, and so on.
      - lazaroclapp 9y ago
        "Do what I mean" is correct in that it reduces super-human AI to the same category of other human technologies: as good or as bad as whatever we choose to do with them. "Coherent extrapolated volition" might be preferable, but if we could get to the first, we could then ask for the second.
  - deleted 9y ago
    [deleted]
  - CM30 9y ago
    I don't think the intelligence aspect is the biggest problem with that paperclip maximiser concept. It's the physical ability to actually turn stuff into paperclips. Or whatever else the nearest analogy would be. Doesn't matter how determined the machine is to make them, the rest of the world won't exactly lay down and let it. Nor would its goals often be possible with the amount of resources available in its surroundings. It's basically the cat problems from this tongue in cheek article: http://idlewords.com/talks/superintelligence.htm http://idlewords.com/talks/superintelligence.htm
- ffwd 9y ago
  Yeah the problem is where would you put in the command to maximize paperclips? Even though deep learning is very powerful, it doesn't contain a utility function. The utility function to maximize paperclips would be some kind of module separate from a set of deep learning nets that would coordinate and read inputs from all the nets and then somehow give commands to some actuators somewhere (which it would also need intelligence about). All those deep learning networks would have to already be tuned to be very accurate in their own right before they are hooked up to any kind of controller. That was my point that, AI has been confused with the deep learning gains in the past years, even though, the hard part about how to coordinate all those inputs and also give the right commands to actuators is still not near to solved, much less a completely virtual superintelligence that has its own "virtual" goals and virtual simulations of both its inputs and potential actuators in the real world. This would be akin to a kind of intelligence algorithm, and not deep learning vector algorithms that reveal structures in data.
Smaug123 9y ago
We don't know how to do this yet, but humans have things which are a bit like utility functions, "built in" in a way we don't understand. However it's done, it leads to self-improving behaviour in humans (many humans value learning and becoming better at things). Just because we don't know how to do it with deep learning and/or neural networks doesn't mean that it's a) not possible with those techniques, and b) that it's not possible with some other technique. (When a neural network becomes sophisticated enough to model its own workings and its relation to the environment, I don't see a knockdown argument for why it couldn't in principle optimise itself, given help from the outside.)

by ffwd 9y ago

jononor 9y ago

Check for instance the 'Paperclip maximizer' thought experiment, https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer

goatlover 9y ago
The Paperclip maximizer assumes that you could have an AI smart enough to turn the world into paperclips, but not smart enough to understand what the human meant by "make me some paperclips". My guess is that paperclipping the world takes magnitudes more intelligence than understanding what a human means when saying something ambiguous.
- Smaug123 9y ago
  Not necessarily. It could be smart enough to know what you meant (and indeed to deduce the entire history of your thoughts!) and simply not care, because you told it to care about something else. For example, maybe you inadvertently programmed it to "be totally literally 100% certain that you've completed the task I told you to do". Then from its perspective, there's always a tiny, tiny chance that its sensors are being fooled or malfunctioning, so it can't be literally 100% certain that it's ever made a paperclip successfully, so by your programming, it should keep making more paperclips. This is independent of whether or not you wanted it to do that: it's a result of what you programmed it to do and what you told it to do.
  - goatlover 9y ago
    If a human being is smart enough to realize that you don't want to paperclip the entire world, then surely a super AI would be smart enough, by definition. Part of intelligence is knowing when to ignore all those tiny chances. Human intelligence is less brittle than artificial because we can handle ambiguity and know when it's ridiculous to continue a task, just because there might be some small chance that it's not finished. We also know that paperclipping the world will get in the way of other goals, like going to the show or making money.
    - Smaug123 9y ago
      I did indeed say that a superintelligent AI could realise that we don't want to paperclip the world. But it need not care what we want, unless we're really, really careful about how we program it. What is "ridiculous" about continuing a task because we're not certain that it's done yet? It's only your human moral system saying that. Just because humans usually don't value things enough to pursue them to the exclusion of all else…
      - goatlover 9y ago
        I tell my super intelligent assistant, who's much better at understanding human language than Siri, to make me some paperclips. It understands some to mean more than one and less than infinity. In fact, some means less than "a lot". The meaning of a lot depends on the context, which happens to be paperclips for me. What is "some" paperclips for me? It depends on how many papers I might need to clip (or whatever use I might have for paperclips). My super intelligent assistant would be able to work out a good estimate. After having an estimate, it can go make me "some" paperclips, and then stop somewhere short of paper clipping the entire world. Alternatively, it could just ask me how many "some" means.
        Hannan 9y ago
        - Hill Climbing - Local Maximum - Gradient Descent I don't pretend to have any sort of expertise in these sorts of discussions, so I thought I would throw out some easily wikipedia'd terms that seem to back your thoughts regarding convergence vs. exponentiation.
- lazaroclapp 9y ago
  No. It means you could program an AI to a) make paperclips and b) increase its own intelligence. Assume you program (a) incorrectly (in the sense of lacking the appropriate safeguards about when not to make more paperclips) but (b) correctly. Then, it doesn't matter that the AI will eventually understand what you want perfectly, because what it will want to do is to keep making paperclips. The whole argument is that if you have a self-improving agent with a goal and the ability to improve in ways you can't predict at all, then you better make sure its goal matches yours. Of course, the obvious solution is: "make the objective function equivalent to 'do what I want you to do'", but the problem is we might not know how to encode that without help from a super-AI.
  - Smaug123 9y ago
    Even "do what I mean" is not obviously correct. Yudkowsky used to advocate the idea of "coherent extrapolated volition": the idea of what I would want the AI to do, were I much more intelligent, knew more, could think faster, were more how I wished I were, and so on.
    - lazaroclapp 9y ago
      "Do what I mean" is correct in that it reduces super-human AI to the same category of other human technologies: as good or as bad as whatever we choose to do with them. "Coherent extrapolated volition" might be preferable, but if we could get to the first, we could then ask for the second.
- deleted 9y ago
  [deleted]
- CM30 9y ago
  I don't think the intelligence aspect is the biggest problem with that paperclip maximiser concept. It's the physical ability to actually turn stuff into paperclips. Or whatever else the nearest analogy would be. Doesn't matter how determined the machine is to make them, the rest of the world won't exactly lay down and let it. Nor would its goals often be possible with the amount of resources available in its surroundings. It's basically the cat problems from this tongue in cheek article: http://idlewords.com/talks/superintelligence.htm http://idlewords.com/talks/superintelligence.htm
ffwd 9y ago
Yeah the problem is where would you put in the command to maximize paperclips? Even though deep learning is very powerful, it doesn't contain a utility function. The utility function to maximize paperclips would be some kind of module separate from a set of deep learning nets that would coordinate and read inputs from all the nets and then somehow give commands to some actuators somewhere (which it would also need intelligence about). All those deep learning networks would have to already be tuned to be very accurate in their own right before they are hooked up to any kind of controller. That was my point that, AI has been confused with the deep learning gains in the past years, even though, the hard part about how to coordinate all those inputs and also give the right commands to actuators is still not near to solved, much less a completely virtual superintelligence that has its own "virtual" goals and virtual simulations of both its inputs and potential actuators in the real world. This would be akin to a kind of intelligence algorithm, and not deep learning vector algorithms that reveal structures in data.

Smaug123 9y ago

We don't know how to do this yet, but humans have things which are a bit like utility functions, "built in" in a way we don't understand. However it's done, it leads to self-improving behaviour in humans (many humans value learning and becoming better at things). Just because we don't know how to do it with deep learning and/or neural networks doesn't mean that it's a) not possible with those techniques, and b) that it's not possible with some other technique. (When a neural network becomes sophisticated enough to model its own workings and its relation to the environment, I don't see a knockdown argument for why it couldn't in principle optimise itself, given help from the outside.)